PDF version of these slides (demo: this PDF version is experimental and may contain errors or differ from the slides)
Primal
\[\max\{c^{\intercal}x:Ax=b,\ x\ge0\}.\]
Dual
\[\min\{b^{\intercal}y:A^{\intercal}y\ge c\}.\]
Theorem (Strong duality). If a linear program is feasible and bounded, then its dual also has an optimal solution, and their optimal objective values are equal.
That is, if both primal/dual LPs are feasible, then there are optimal solutions $x^*,y^*$ with
\[\boxed{c^{\intercal}x^*=b^{\intercal}y^*.}\]
Primal
\[\max\{c^{\intercal}x:Ax=b,\ x\ge0\}.\]
Dual
\[\min\{b^{\intercal}y:A^{\intercal}y\ge c\}.\]
Proof of strong duality
Primal
\[\max\{c^{\intercal}x:Ax=b,\ x\ge0\}.\]
Dual
\[\min\{b^{\intercal}y:A^{\intercal}y\ge c\}.\]
We've seen that if we have an optimal solution to the primal, then there is an optimal solution to the dual (and one construction for such a dual solution).
Can we recover the dual solution in an easier way?
Let $x^*$ and $y^*$ be optimal solutions for the primal/dual LPs, where the primal is in SEF.
Recall that each primal variable $x_i \ge 0$ is associated to a dual constraint that \[(A^{\intercal}y^*)_i \ge c_i.\]
Observation: For each primal variable, if $x^*_i$ is not 0 (i.e. $x^*_i > 0$), then the dual constraint is tight: \[ (A^{\intercal}y^*)_i = c_i. \]
We will first prove this, then we'll use this to find the value of $y^*$ given $x^*.$
Let's recall the proof of weak duality, we looked at \[ y^{\intercal}Ax, \] and we showed that \[ c^{\intercal} x \le y^{\intercal}Ax = b^{\intercal}y. \]
If $x^*$ and $y^*$ are primal/dual optimal, then the inequality on the left must be an equality (so that the whole inequality is an equality).
So we have that \[ c^{\intercal} x = (A^{\intercal}y)^{\intercal}x. \]
Rewriting the previous equation gives \[ (A^{\intercal}y - c)^{\intercal}x = 0 \]
Let's expand the inner product: \[ \sum_{i=1}^n (A^{\intercal}y - c)_i x_i = 0. \] What do we notice about the terms in this sum?
Both $(A^{\intercal}y - c)_i \ge 0$ and $x_i \ge 0$ (by primal/dual feasibility). The only way a sum of nonnegative terms can be 0 is if every term is 0.
Therefore, $(A^{\intercal}y - c)_i x_i = 0$ for each $i$. If $x_i > 0$, then this implies that $(A^{\intercal}y - c)_i = 0$.
Theorem (Complementary Slackness). Let $x^*$ be an optimal solution for a primal linear program in SEF and let $y^*$ be the optimal solution to its dual linear program.
For each $i$, either $x_i^* = 0$, or $(A^{\intercal} y)_i = c_i$.
Conversely, if $x^*$ is an optimal solution for the primal linear program, and $y^*$ is feasible for the dual program, and it satisfies the property that $(A^{\intercal} y^*)_i = c_i$ whenever $x_i^* > 0$, then $y^*$ is optimal for the dual program.
Primal
\[\begin{aligned}\max\quad &2x_1+3x_2\\ &x_1+x_2+x_3=4\\ &x_1+3x_2+x_4=6\\ &x_1,x_2,x_3,x_4\ge0\end{aligned}\]
Dual
\[\begin{aligned}\min\quad &4y_1+6y_2\\ &y_1+y_2\ge2\\ &y_1+3y_2\ge3\\ &y_1\ge0,\ y_2\ge0\end{aligned}\]
The optimal solutions are \[ x^*=(3,1,0,0), \qquad y^* = \left(\frac{3}{2}, \frac{1}{2}\right),\] with objective values both equal to 9.
Data: $c = (2,3,0,0)$, $b = (4,6)$, $A = \begin{pmatrix}1 & 1 & 1 & 0 \\ 1 & 3 & 0 & 1\end{pmatrix}$, $x^*=(3,1,0,0)$ and $y^*=(\tfrac32,\tfrac12)$:
\[ c^{\intercal}x^* \;\le\; (A^{\intercal}y^*)^{\intercal}x^* \;=\; b^{\intercal}y^*. \]
Here $c^{\intercal}x^*=9=b^{\intercal}y^*$, so the inequality on the left must be an equality: $(A^{\intercal}y^*)^{\intercal}x^*=c^{\intercal}x^*$.
We get from this that $(A^{\intercal}y^* - c)^{\intercal}x^*=0$.
Data: $c = (2,3,0,0)$, $b = (4,6)$, $A = \begin{pmatrix}1 & 1 & 1 & 0 \\ 1 & 3 & 0 & 1\end{pmatrix}$, $x^*=(3,1,0,0)$ and $y^*=(\tfrac32,\tfrac12)$:
Let's look at the terms in $(A^{\intercal}y^* - c)^{\intercal}x^*=0$.
\[ \begin{aligned} A^{\intercal}y^* - c = \begin{pmatrix}\color{red}{0}\\\color{red}{0}\\\tfrac32\\\tfrac12\end{pmatrix}. \end{aligned} \qquad \begin{aligned} x^* = \begin{pmatrix} 3 \\ 1 \\ \color{red}{0} \\ \color{red}{0}\end{pmatrix}. \end{aligned} \]
This is what complementary slackness means: if an entry of $x^*$ is nonzero, then the corresponding entry of $A^{\intercal}y - c$ must be 0.
| Chair | Table | Available | |
|---|---|---|---|
| Wood (metres) | 2 | 4 | 80 |
| Labor (hours) | 1 | 3 | 50 |
| Profit (dollars) | 40 | 90 |
Primal
\[\begin{aligned} \max\quad &40c+90t\\ \text{s.t.}\quad &2c+4t+s_1=80,\\ &c+3t+s_2=50,\\ &c,t,s_1,s_2\ge0. \end{aligned}\]
Dual
\[\begin{aligned} \min\quad &80u+50v\\ \text{s.t.}\quad &2u+v\ge40,\\ &4u+3v\ge90,\\ &u,v\ge0. \end{aligned}\]
Optimal primal: $(c,t,s_1,s_2)=(20,10,0,0)$, Optimal dual : $(u,v)=(15,10)$.
Let's say we have an LP in SEF, where $A \in \R^{m \times n}$. \[\max\{c^{\intercal}x:Ax=b,\ x\ge0\}.\]
We saw from our simplex method discussion that an optimal solution to this LP is a basic feasible solution with respect to some basis. How many nonzero entries do we expect that such a solution should have?
Every basis in of $A$ has $m$ entries, so every BFS has at most $m$ nonzero entries. A nondegenerate BFS has exactly $m$ nonzero entries (we'll talk about degenerate solutions in a second.)
What does complementary slackness imply in this case?
Primal (SEF)
\[\begin{aligned}\max\quad &7x_1+4x_2\\ &x_1+2x_2+x_3=8\\ &3x_1+x_2+x_4=9\\ &x_1,x_2,x_3,x_4\ge0\end{aligned}\]
Dual
\[\begin{aligned}\min\quad &8y_1+9y_2\\ &y_1+3y_2\ge7\\ &2y_1+y_2\ge4\\ &y_1\ge0,\ y_2\ge0\end{aligned}\]
Let's take an optimal solution to the primal LP: \[ x^*=(2,3,0,0), \qquad c^{\intercal}x^*=26. \]
It is nondegenerate because it has exactly $m=2$ nonzero entries.
Primal (SEF)
\[\begin{aligned}\max\quad &7x_1+4x_2\\ &x_1+2x_2+x_3=8\\ &3x_1+x_2+x_4=9\\ &x_1,x_2,x_3,x_4\ge0\end{aligned}\]
Dual
\[\begin{aligned}\min\quad &8y_1+9y_2\\ &y_1+3y_2\ge7\\ &2y_1+y_2\ge4\\ &y_1\ge0,\ y_2\ge0\end{aligned}\]
Since $x_1^*=2>0$ and $x_2^*=3>0$, complementary slackness implies that the dual constraints are tight: \[ y_1+3y_2=7, \qquad 2y_1+y_2=4. \]
This has a unique solution $y = (1,2)$. Complementary slackness implies that any optimal dual solution must satisfy these constraints, so $(1,2)$ must be an optimal dual solution!
Primal (SEF)
\[\begin{aligned}\max\quad &7x_1+4x_2\\ &x_1+2x_2+x_3=8\\ &3x_1+x_2+x_4=9\\ &x_1,x_2,x_3,x_4\ge0\end{aligned}\]
Dual
\[\begin{aligned}\min\quad &8y_1+9y_2\\ &y_1+3y_2\ge7\\ &2y_1+y_2\ge4\\ &y_1\ge0,\ y_2\ge0\end{aligned}\]
Let's check for $y = (1,2)$: $ \begin{aligned}y_1+3y_2 = 7 \ge7\\ 2y_1+y_2 = 4\ge4\\ y_1 = 1\ge0\\ y_2 = 2\ge0 \\ 8y_1 + 9 y_2 = 26 = c^{\intercal}x^*\end{aligned}. $
Primal (SEF)
\[\begin{aligned}\max\quad &3x_1+3x_2+2x_3+5x_4\\ &x_1+x_2+2x_4=3\\ &x_2+x_3+x_4=4\\ &x_1+x_3+x_4=5\\ &x_1,x_2,x_3,x_4\ge0\end{aligned}\]
Dual
\[\begin{aligned}\min\quad &3y_1+4y_2+5y_3\\ &y_1+y_3\ge3\\ &y_1+y_2\ge3\\ &y_2+y_3\ge2\\ &2y_1+y_2+y_3\ge5\end{aligned}\]
Let's take an optimal solution to the primal LP: \[ x^*=(2,1,3,0), \qquad c^{\intercal}x^*=15. \]
It is nondegenerate because it has exactly $m=3$ nonzero entries.
Primal (SEF)
\[\begin{aligned}\max\quad &3x_1+3x_2+2x_3+5x_4\\ &x_1+x_2+2x_4=3\\ &x_2+x_3+x_4=4\\ &x_1+x_3+x_4=5\\ &x_1,x_2,x_3,x_4\ge0\end{aligned}\]
Dual
\[\begin{aligned}\min\quad &3y_1+4y_2+5y_3\\ &y_1+y_3\ge3\\ &y_1+y_2\ge3\\ &y_2+y_3\ge2\\ &2y_1+y_2+y_3\ge5\end{aligned}\]
Since $x_1^*,x_2^*,x_3^*>0$, complementary slackness implies that the first three dual constraints are tight: \[ y_1+y_3=3, \qquad y_1+y_2=3, \qquad y_2+y_3=2. \]
This has a unique solution $y = (2,1,1)$. Complementary slackness implies that any optimal dual solution must satisfy these constraints, so $(2,1,1)$ must be an optimal dual solution!
Primal (SEF)
\[\begin{aligned}\max\quad &3x_1+3x_2+2x_3+5x_4\\ &x_1+x_2+2x_4=3\\ &x_2+x_3+x_4=4\\ &x_1+x_3+x_4=5\\ &x_1,x_2,x_3,x_4\ge0\end{aligned}\]
Dual
\[\begin{aligned}\min\quad &3y_1+4y_2+5y_3\\ &y_1+y_3\ge3\\ &y_1+y_2\ge3\\ &y_2+y_3\ge2\\ &2y_1+y_2+y_3\ge5\end{aligned}\]
Let's check for $y = (2,1,1)$: $ \begin{aligned}y_1+y_3 = 3 \ge3\\ y_1+y_2 = 3\ge3\\ y_2+y_3 = 2\ge2\\ 2y_1+y_2+y_3 = 6\ge5 \\ 3y_1 + 4 y_2 + 5y_3 = 15 = c^{\intercal}x^*\end{aligned}. $
Theorem. Let $x^*$ be an optimal solution of $\max\{c^{\intercal}x:Ax=b,\ x\ge0\}$ that is a nondegenerate (i.e. having $m$ nonzero entries) basic feasible solution with basis $B$. Then the dual $\min\{b^{\intercal}y:A^{\intercal}y\ge c\}$ has a unique optimal solution $y^*$, which is the solution of \[ A_B^{\intercal}y=c_B, \qquad\text{i.e.}\qquad y^*=(A_B^{\intercal})^{-1}c_B. \]
The proof is just that complementary slackness implies that $ A_B^{\intercal}y=c_B$ and the fact that $x^*$ is nondegenerate implies that this matrix is in fact full rank.
What happens when the optimal solution to an LP is degenerate?
One possibility is that there are multiple optimal dual solutions; this implies that there won't be a simple formula for the value of $y^*$ just from complementary slackness considerations.
This is not the only possibility; there may be just one optimal dual solution and the primal might just have a degenerate optimum.
Primal (SEF)
\[\begin{aligned}\max\quad &2x_1+x_2\\ &x_1+x_2+x_3=4\\ &x_1+3x_2+x_4=4\\ &x_1,x_2,x_3,x_4\ge0\end{aligned}\]
Dual
\[\begin{aligned}\min\quad &4y_1+4y_2\\ &y_1+y_2\ge2\\ &y_1+3y_2\ge1\\ &y_1\ge0,\ y_2\ge0\end{aligned}\]
Let's take an optimal solution to the primal LP: \[ x^*=(4,0,0,0), \qquad c^{\intercal}x^*=8. \]
It is degenerate because it has only $1$ nonzero entry, which is fewer than $m=2$.
Primal (SEF)
\[\begin{aligned}\max\quad &2x_1+x_2\\ &x_1+x_2+x_3=4\\ &x_1+3x_2+x_4=4\\ &x_1,x_2,x_3,x_4\ge0\end{aligned}\]
Dual
\[\begin{aligned}\min\quad &4y_1+4y_2\\ &y_1+y_2\ge2\\ &y_1+3y_2\ge1\\ &y_1\ge0,\ y_2\ge0\end{aligned}\]
Since $x_1^*>0$, complementary slackness says that the first dual constraint is tight: $y_1+y_2=2$.
Letting $y=(2-t,\,t)$, the remaining dual constraints hold as long as $0 \le t \le 2$, so there are infinitely many $y$ optimizing the dual program.
Unfortunately, for a degenerate primal solution, there is no easy way to extract an optimal dual solution.
We can run the simplex method on a smaller linear programming feasibility problem.
\[ \begin{aligned} A_T^{\intercal} y = c_T\\ A_S^{\intercal}y \ge c_S, \end{aligned} \] where $A_T$ is the submatrix of $A$ containing columns associated to variables $x_i > 0$, and $A_S$ is the submatrix containing columns associated to the variablaes $x_i = 0$. This encodes complementary slackness. Also note that we can use linear algebra to reduce the number of variables using $A_T^{\intercal} y = c_T$.
Unfortunately, for a degenerate primal solution, there is no easy way to extract an optimal dual solution.
We can run the simplex method on a smaller linear program feasibility problem.
If $x^*$ was obtained by finding a basis $B$ for which the simplex method terminates, then it turns out that the columns of $B$ can be used to solve for $y$, i.e. \[ y = (A_B^{\intercal})^{-1} c_B, \] where $A_B$ denotes the submatrix of $A$ consisting of the columns in $B$.
We can also use an optimal dual solution to recover an optimal primal solution using complementary slackness.
Primal (SEF)
\[\begin{aligned}\max\quad &4x_1+x_2\\ &2x_1+x_2+x_3=8\\ &x_1+3x_2+x_4=9\\ &x_1,x_2,x_3,x_4\ge0\end{aligned}\]
Dual
\[\begin{aligned}\min\quad &8y_1+9y_2\\ &2y_1+y_2\ge4\\ &y_1+3y_2\ge1\\ &y_1\ge0,\ y_2\ge0\end{aligned}\]
Let's take an optimal solution to the dual LP: \[ y^*=(2,0), \qquad b^{\intercal}y^*=16. \]
Primal (SEF)
\[\begin{aligned}\max\quad &4x_1+x_2\\ &2x_1+x_2+x_3=8\\ &x_1+3x_2+x_4=9\\ &x_1,x_2,x_3,x_4\ge0\end{aligned}\]
Dual
\[\begin{aligned}\min\quad &8y_1+9y_2\\ &2y_1+y_2\ge4\\ &y_1+3y_2\ge1\\ &y_1\ge0,\ y_2\ge0\end{aligned}\]
The dual constraints for $x_2$ and $x_3$ are not tight: $y_1+3y_2=2>1$ and $y_1=2>0$. Complementary slackness implies that \[ x_2^*=0, \qquad x_3^*=0. \]
Substituting into $Ax=b$ leaves $2x_1=8$ and $x_1+x_4=9$. This has a unique solution $x=(4,0,0,5)$, so $(4,0,0,5)$ must be an optimal primal solution!
Primal (SEF)
\[\begin{aligned}\max\quad &4x_1+x_2\\ &2x_1+x_2+x_3=8\\ &x_1+3x_2+x_4=9\\ &x_1,x_2,x_3,x_4\ge0\end{aligned}\]
Dual
\[\begin{aligned}\min\quad &8y_1+9y_2\\ &2y_1+y_2\ge4\\ &y_1+3y_2\ge1\\ &y_1\ge0,\ y_2\ge0\end{aligned}\]
Let's check for $x = (4,0,0,5)$: $ \begin{aligned}2x_1+x_2+x_3 = 8 \\ x_1+3x_2+x_4 = 9\\ x_1 = 4\ge0, \ x_2 = 0\ge0\\ x_3 = 0\ge0,\ x_4 = 5\ge0 \\ 4x_1 + x_2 = 16 = b^{\intercal}y^*\end{aligned}. $
Given a maximization LP that has a mixture of free variables and nonfree variables (and equality and inequality constraints), recall that the dual problem has
Complementary slackness in this case says that each dual constraint associated to a nonnegative variable is tight if that variable is positive.
Similarly, each dual variable associated to a $\le$ constraint must be $0$ if that constraint is not tight.
Primal
\[\begin{aligned}\max\quad &3x_1+2x_2\\ &x_1+x_2=4\\ &2x_1+x_2\le6\\ &x_2\le3\\ &x_1,x_2\ge0\end{aligned}\]
Dual
\[\begin{aligned}\min\quad &4y_1+6y_2+3y_3\\ &y_1+2y_2\ge3\\ &y_1+y_2+y_3\ge2\\ &y_1\text{ free},\ y_2,y_3\ge0\end{aligned}\]
Let's take an optimal solution to the primal LP: \[ x^*=(2,2), \qquad c^{\intercal}x^*=10. \]
The first constraint is an equality, the second is tight, and the third has slack: $x_2^*=2<3$.
Primal
\[\begin{aligned}\max\quad &3x_1+2x_2\\ &x_1+x_2=4\\ &2x_1+x_2\le6\\ &x_2\le3\\ &x_1,x_2\ge0\end{aligned}\]
Dual
\[\begin{aligned}\min\quad &4y_1+6y_2+3y_3\\ &y_1+2y_2\ge3\\ &y_1+y_2+y_3\ge2\\ &y_1\text{ free},\ y_2,y_3\ge0\end{aligned}\]
At $x^* = (2,2)$, the third primal constraint is not tight, so complementary slackness implies $y_3=0$. Since $x_1^*>0$ and $x_2^*>0$, the first two dual constraints are tight: \[ y_1+2y_2=3, \qquad y_1+y_2+y_3=2. \]
With $y_3=0$, this has a unique solution $y = (1,1,0)$. Complementary slackness implies that any optimal dual solution must satisfy these conditions, so $(1,1,0)$ must be an optimal dual solution!
Primal
\[\begin{aligned}\max\quad &3x_1+2x_2\\ &x_1+x_2=4\\ &2x_1+x_2\le6\\ &x_2\le3\\ &x_1,x_2\ge0\end{aligned}\]
Dual
\[\begin{aligned}\min\quad &4y_1+6y_2+3y_3\\ &y_1+2y_2\ge3\\ &y_1+y_2+y_3\ge2\\ &y_1\text{ free},\ y_2,y_3\ge0\end{aligned}\]
Let's check for $y = (1,1,0)$: $ \begin{aligned}y_1+2y_2 = 3 \ge3\\ y_1+y_2+y_3 = 2\ge2\\ y_1 = 1\text{ (free)},\ y_2 = 1\ge0,\ y_3=0\ge0 \\ 4y_1 + 6 y_2 + 3y_3 = 10 = c^{\intercal}x^*\end{aligned}. $
In summary, nonnegative primal variables are associated to inequality dual constraints, and those dual constraints are tight if that primal variable is positive.
Primal inequality constraints are associated to nonnegative dual variables, and those dual variables are 0 if the associated constraint is not held at equality for any optimal solution.