Two Phase Simplex Method

Two Phase Simplex Algorithm

Given a linear program, how do we check that it is feasible?

Two Phase Simplex Algorithm

Starting with an LP in SEF with $b \ge 0$, we can write down the auxilliary LP

$m$ is the number of equations (the number of rows of $A$).

The auxilliary LP is always feasible and its optimal value is 0 if and only if the original LP was feasible.

Two Phase Simplex Algorithm

Mechanically, the new LP is defined from the old components via the new matrices/vectors

\[ \color{#bd17fe}{A'} = \begin{pmatrix} A & I \end{pmatrix}\qquad \color{#bd17fe}{b'} = b\qquad \color{#bd17fe}{c'}^{\intercal} = (\vec{0}_n, -\vec{1}_m) \]

where $\vec{1}_m$ is the all one's vector of length $m$, and $\vec{0}_n$ is the all zeros vector of length $n$.

Note that $\color{#bd17fe}{A'}^{\intercal} \vec{1} = (A^{\intercal}\vec{1}_m, \vec{1}_m)$, and the following LP is in canonical form whose basis is the auxilliary variables $\{x_{n+1},\dots,x_{n+m}\}$.

Example

Canonical Form

Phase 1 — Iteration 1

We start in canonical form with respect to the basis $\color{blue}{\{4,5\}}$:

The positive objective coefficients are $\{x_1, x_2, x_3\}$; Bland's rule enters the smallest index, $x_1$.

Ratio test on $A_1 = \begin{pmatrix}1\\2\end{pmatrix}$: $\tfrac{4}{1} = 4$ (row $\color{blue}{x_4}$), $\tfrac{5}{2}$ (row $\color{blue}{x_5}$). The minimum is $\tfrac{5}{2}$, so $x_5$ leaves.

Phase 1 — Iteration 1

\[ \begin{pmatrix} \color{red}2 & 1 & 1 & \color{blue}0 & \color{green}1 \\ \color{red}1 & 1 & 2 & \color{blue}1 & \color{green}0 \end{pmatrix} x' = \begin{pmatrix} 5 \\ 4 \end{pmatrix} \] \[ \downarrow_{\ \rho_1 \,=\, \frac{1}{2}\rho_1} \] \[ \begin{pmatrix} \color{red}1 & \tfrac{1}{2} & \tfrac{1}{2} & \color{blue}0 & \color{green}{\tfrac{1}{2}} \\ \color{red}1 & 1 & 2 & \color{blue}1 & \color{green}0 \end{pmatrix} x' = \begin{pmatrix} \tfrac{5}{2} \\ 4 \end{pmatrix} \] \[ \downarrow_{\ \rho_2 \,=\, \rho_2 - \rho_1} \] \[ \begin{pmatrix} \color{red}1 & \tfrac{1}{2} & \tfrac{1}{2} & \color{blue}0 & \color{green}{\tfrac{1}{2}} \\ \color{red}0 & \tfrac{1}{2} & \tfrac{3}{2} & \color{blue}1 & \color{green}{-\tfrac{1}{2}} \end{pmatrix} x' = \begin{pmatrix} \tfrac{5}{2} \\ \tfrac{3}{2} \end{pmatrix} \]

Phase 1 — Iteration 1

First equation says that $\begin{pmatrix} \color{red}1 & \tfrac{1}{2} & \tfrac{1}{2} & \color{blue}0 & \color{green}{\tfrac{1}{2}}\end{pmatrix}x' - \frac{5}2 = 0$, so we can subtract this from the objective:

\[ \begin{aligned} &\left(\begin{pmatrix} \color{red}3 & 2 & 3 & \color{blue}0 & \color{green}0 \end{pmatrix} x' - 9\right)\\ - &3\left(\begin{pmatrix} \color{red}1 & \tfrac{1}{2} & \tfrac{1}{2} & \color{blue}0 & \color{green}{\tfrac{1}{2}}\end{pmatrix}x' - \frac{5}2\right)\\ =& \begin{pmatrix} \color{blue}0 & \tfrac{1}{2} & \tfrac{3}{2} & \color{blue}0 & -\tfrac{3}{2} \end{pmatrix} x' - \tfrac{3}{2} \end{aligned} \]

Phase 1 — Iteration 1

Phase 1 — Iteration 2

Canonical form with respect to $\color{blue}{\{1,4\}}$, basic feasible solution $x' = \left(\tfrac{5}{2},0,0,\tfrac{3}{2},0\right)$, objective value $-\tfrac{3}{2}$:

The positive objective coefficients are $\{x_2, x_3\}$; Bland's rule enters $x_2$.

Ratio test on $A_2 = \begin{pmatrix}\tfrac{1}{2}\\\tfrac{1}{2}\end{pmatrix}$: $\tfrac{5/2}{1/2} = 5$ (row $\color{blue}{x_1}$), $\tfrac{3/2}{1/2} = 3$ (row $\color{blue}{x_4}$). The minimum is $3$, so $x_4$ leaves.

Phase 1 — Iteration 2

\[ \begin{pmatrix} \color{blue}1 & \color{red}{\tfrac{1}{2}} & \tfrac{1}{2} & \color{green}0 & \tfrac{1}{2} \\ \color{blue}0 & \color{red}{\tfrac{1}{2}} & \tfrac{3}{2} & \color{green}1 & -\tfrac{1}{2} \end{pmatrix} x' = \begin{pmatrix} \tfrac{5}{2} \\ \tfrac{3}{2} \end{pmatrix} \] \[ \downarrow_{\ \rho_2 \,=\, 2\rho_2} \] \[ \begin{pmatrix} \color{blue}1 & \color{red}{\tfrac{1}{2}} & \tfrac{1}{2} & \color{green}0 & \tfrac{1}{2} \\ \color{blue}0 & \color{red}1 & 3 & \color{green}2 & -1 \end{pmatrix} x' = \begin{pmatrix} \tfrac{5}{2} \\ 3 \end{pmatrix} \] \[ \downarrow_{\ \rho_1 \,=\, \rho_1 - \frac{1}{2}\rho_2} \] \[ \begin{pmatrix} \color{blue}1 & \color{red}0 & -1 & \color{green}{-1} & 1 \\ \color{blue}0 & \color{red}1 & 3 & \color{green}2 & -1 \end{pmatrix} x' = \begin{pmatrix} 1 \\ 3 \end{pmatrix} \]

Phase 1 — Iteration 2

Canonical form with respect to $\color{blue}{\{1,4\}}$, basic feasible solution $x' = \left(\tfrac{5}{2},0,0,\tfrac{3}{2},0\right)$, objective value $-\tfrac{3}{2}$:

Second equation says that $\begin{pmatrix} \color{blue}0 & \color{red}1 & 3 & \color{green}2 & -1 \end{pmatrix}x' - 3 = 0$, so we can subtract this from the objective:

\[ \begin{aligned} &\left(\begin{pmatrix} \color{blue}0 & \color{red}{\tfrac{1}{2}} & \tfrac{3}{2} & \color{blue}0 & -\tfrac{3}{2} \end{pmatrix} x' - \tfrac{3}{2}\right)\\ - &\tfrac{1}{2}\left(\begin{pmatrix} \color{blue}0 & \color{red}1 & 3 & \color{green}2 & -1 \end{pmatrix}x' - 3\right)\\ =& \begin{pmatrix} \color{blue}0 & \color{blue}0 & 0 & -1 & -1 \end{pmatrix} x' - 0 \end{aligned} \]

Phase 1 — Iteration 3

This gives the canonical form with respect to $\color{blue}{\{1,2\}}$ (sorted), with basic feasible solution $x' = (1,3,0,0,0)$ and objective value $0$:

The objective has no positive coefficient, so phase 1 is optimal with value $0$. The artificial variables $x_4, x_5$ are both $0$, so the original LP is feasible, and $\{x_1, x_2\}$ is a starting basis for phase 2.

Phase 2 — Setup

Drop the artificial variables $x_4, x_5$ and restore the original objective $\max\ 3x_1 + 2x_2 + x_3$. Phase 1 ended at the basis $\color{blue}{\{1,2\}}$, which we use to start phase 2.

Taking the phase-1 constraint rows (restricted to $x_1, x_2, x_3$) already in canonical form with respect to $\color{blue}{\{1,2\}}$:

The objective is not yet reduced: the basic variables $x_1, x_2$ still have nonzero coefficients.

Phase 2 — Optimal

Reduce the objective so the basic variables $x_1, x_2$ have zero coefficients. This gives the canonical form with respect to $\color{blue}{\{1,2\}}$, with basic feasible solution $x = (1,3,0)$ and objective value $9$:

The objective vector has no positive coefficient, so this is already optimal. The optimal solution is $x_1 = 1,\ x_2 = 3,\ x_3 = 0$ with value $3(1) + 2(3) + 1(0) = 9$.

Another Example

Phase 1

Phase 1 — Iteration 1

Canonical form with respect to $\color{blue}{\{4,5\}}$, basic feasible solution $x' = (0,0,0,1,5)$, with $x_4 + x_5 = 6$:

The positive objective coefficients are $\{x_1, x_2, x_3\}$; Bland's rule enters $x_1$.

Ratio test on $A_1 = \begin{pmatrix}1\\2\end{pmatrix}$: $\tfrac{1}{1} = 1$ (row $\color{blue}{x_4}$), $\tfrac{5}{2}$ (row $\color{blue}{x_5}$). The minimum is $1$, so $x_4$ leaves.

Phase 1 — Iteration 2

After the pivot, the basis is $\color{blue}{\{1,5\}}$ (already sorted). Canonical form with basic feasible solution $x' = (1,0,0,0,3)$, with $x_4 + x_5 = 3$:

The only positive objective coefficient is $x_3$, so $x_3$ enters. Ratio test on $A_3 = \begin{pmatrix}1\\1\end{pmatrix}$: $\tfrac{1}{1} = 1$ (row $\color{blue}{x_1}$), $\tfrac{3}{1} = 3$ (row $\color{blue}{x_5}$). The minimum is $1$, so $x_1$ leaves.

IPhase 1 — Iteration 3

After the pivot, the basis is $\color{blue}{\{3,5\}}$ (already sorted). Canonical form with basic feasible solution $x' = (0,0,1,0,2)$, with $x_4 + x_5 = 2$:

The objective has no positive coefficient, so phase 1 is optimal with value $\max\big(-(x_4+x_5)\big) = -2$, i.e. $\min(x_4 + x_5) = 2 > 0$.

The artificials cannot all be driven to $0$ (here $x_5 = 2$), so the original LP is infeasible.

Geometry of Polytopes

Half-Spaces

Every polytope is built out of inequalities of the form $a^{\intercal}x \le b$, which defines a half-space

Half-Spaces

A halfspace defined by

\[a^{\intercal}x \le b\] has the important property of convexity. An average of two points satisfying this condition still satisfies this condition. For $0 \le t \le 1$,

\[\begin{aligned}a^{\intercal}(tx+(1-t)y) &= ta^{\intercal}x + (1-t)a^{\intercal}y\\ &\le tb + (1-t)b\\ &= b.\end{aligned}\]

Half-Spaces

The feasible region for an LP is an intersection of halfspaces.

Intersections of convex sets are still convex, so every feasible region of an LP is convex.

Standard Equality Form

We have seen that a linear program with feasible set $Ax \le b$ can also be described in standard equality form by adding some extra variables.

This corresponds to `lifting' the polytope into a higher dimensional space, or in other words, drawing the polytope as a projection of a higher dimensional polytope.

Dimensions of Polytopes

What are the dimensions of the following polytopes?

Faces of Polytopes

The faces consist of the 3 red edges, the 3 green vertices and the whole triangle (blue).

Let $P$ be a polytope $\{x \in \R^n : Ax \le b\}$. For a set $S \subseteq [m]$, the associated face of $P$ is the set of points where the associated inequalities are met with equality. \[ F = \{x \in \R^n : Ax \le b, \text{ and }\forall i \in S, a_i^{\intercal}x = b_i \}. \]

Each face is a polytope with its own dimension.

The vertices of $P$ are the 0 dimensional faces. These are points where as many of the inequalities are equations as possible (the number of equations should equal the dimension of the affine hull).

An edge in a polytope is a 1 dimensional face.

Faces of Polytopes

What do vertices correspond to for a polytope of the form $P = \{x \in \R^n : Ax = b, x \ge 0\}$?

Let's assume that the affine hull is $\{x \in \R^n : Ax = b\}$ and $A$ is full rank. The dimension of this affine hull is $n-m$.

Thus, the vertices should correspond to points in $P$ where $n-m$ of the inequalities are tight. Each inequality is of the form $x_i \ge 0$, so these are points where $n-m$ entries are 0.

These are exactly the basic feasible solutions!