Introduction to Optimization

CO 250


Lecture 15


Scroll view

PDF version of these slides (demo: this PDF version is experimental and may contain errors or differ from the slides)

Lecture Outline

  • Review of strong duality
  • Recovering a dual solution from a primal optimum
  • Complementary slackness
  • Complementary slackness for LPs not in SEF

Review: Strong Duality

Primal

\[\max\{c^{\intercal}x:Ax=b,\ x\ge0\}.\]

Dual

\[\min\{b^{\intercal}y:A^{\intercal}y\ge c\}.\]

Theorem (Strong duality). If a linear program is feasible and bounded, then its dual also has an optimal solution, and their optimal objective values are equal.

That is, if both primal/dual LPs are feasible, then there are optimal solutions $x^*,y^*$ with

\[\boxed{c^{\intercal}x^*=b^{\intercal}y^*.}\]

Review: Strong Duality

Primal

\[\max\{c^{\intercal}x:Ax=b,\ x\ge0\}.\]

Dual

\[\min\{b^{\intercal}y:A^{\intercal}y\ge c\}.\]

Proof of strong duality

  • If the primal LP is bounded, then the simplex method terminates.
  • At this point, the LP will be in canonical form with $c \le 0$.
  • For such an LP, $y = \vec{0}$ is an optimal solution to its dual.
  • We get from this that strong duality holds for the canonical form LP.
  • Because the canonical form of the primal LP is obtained from row operations, we can recover strong duality for the original LP.
  • If $B$ is the basis that the simplex algorithm terminates at, then we can express $y^*$ as $(A_B^{\intercal})^{-1}c_B$, where $A_B$ is the submatrix of columns in $B$ and $c_B$ is the vector obtained from the entries of $c$ in $B$.

Review: Strong Duality

Primal

\[\max\{c^{\intercal}x:Ax=b,\ x\ge0\}.\]

Dual

\[\min\{b^{\intercal}y:A^{\intercal}y\ge c\}.\]

We've seen that if we have an optimal solution to the primal, then there is an optimal solution to the dual (and one construction for such a dual solution).

Can we recover the dual solution in an easier way?

Open the demo in its own tab

Complementary Slackness

Let $x^*$ and $y^*$ be optimal solutions for the primal/dual LPs, where the primal is in SEF.

Recall that each primal variable $x_i \ge 0$ is associated to a dual constraint that \[(A^{\intercal}y^*)_i \ge c_i.\]

Observation: For each primal variable, if $x^*_i$ is not 0 (i.e. $x^*_i > 0$), then the dual constraint is tight: \[ (A^{\intercal}y^*)_i = c_i. \]

We will first prove this, then we'll use this to find the value of $y^*$ given $x^*.$

Proof of Complementary Slackness

Let's recall the proof of weak duality, we looked at \[ y^{\intercal}Ax, \] and we showed that \[ c^{\intercal} x \le y^{\intercal}Ax = b^{\intercal}y. \]

If $x^*$ and $y^*$ are primal/dual optimal, then the inequality on the left must be an equality (so that the whole inequality is an equality).

So we have that \[ c^{\intercal} x = (A^{\intercal}y)^{\intercal}x. \]

Proof of Complementary Slackness

Rewriting the previous equation gives \[ (A^{\intercal}y - c)^{\intercal}x = 0 \]

Let's expand the inner product: \[ \sum_{i=1}^n (A^{\intercal}y - c)_i x_i = 0. \] What do we notice about the terms in this sum?

Both $(A^{\intercal}y - c)_i \ge 0$ and $x_i \ge 0$ (by primal/dual feasibility). The only way a sum of nonnegative terms can be 0 is if every term is 0.

Therefore, $(A^{\intercal}y - c)_i x_i = 0$ for each $i$. If $x_i > 0$, then this implies that $(A^{\intercal}y - c)_i = 0$.

Complementary Slackness

Theorem (Complementary Slackness). Let $x^*$ be an optimal solution for a primal linear program in SEF and let $y^*$ be the optimal solution to its dual linear program.

For each $i$, either $x_i^* = 0$, or $(A^{\intercal} y)_i = c_i$.

Conversely, if $x^*$ is an optimal solution for the primal linear program, and $y^*$ is feasible for the dual program, and it satisfies the property that $(A^{\intercal} y^*)_i = c_i$ whenever $x_i^* > 0$, then $y^*$ is optimal for the dual program.

Complementary Slackness Example

Primal

\[\begin{aligned}\max\quad &2x_1+3x_2\\ &x_1+x_2+x_3=4\\ &x_1+3x_2+x_4=6\\ &x_1,x_2,x_3,x_4\ge0\end{aligned}\]

Dual

\[\begin{aligned}\min\quad &4y_1+6y_2\\ &y_1+y_2\ge2\\ &y_1+3y_2\ge3\\ &y_1\ge0,\ y_2\ge0\end{aligned}\]

The optimal solutions are \[ x^*=(3,1,0,0), \qquad y^* = \left(\frac{3}{2}, \frac{1}{2}\right),\] with objective values both equal to 9.

Complementary Slackness Example

Data: $c = (2,3,0,0)$, $b = (4,6)$, $A = \begin{pmatrix}1 & 1 & 1 & 0 \\ 1 & 3 & 0 & 1\end{pmatrix}$, $x^*=(3,1,0,0)$ and $y^*=(\tfrac32,\tfrac12)$:

\[ c^{\intercal}x^* \;\le\; (A^{\intercal}y^*)^{\intercal}x^* \;=\; b^{\intercal}y^*. \]

Here $c^{\intercal}x^*=9=b^{\intercal}y^*$, so the inequality on the left must be an equality: $(A^{\intercal}y^*)^{\intercal}x^*=c^{\intercal}x^*$.

We get from this that $(A^{\intercal}y^* - c)^{\intercal}x^*=0$.

Complementary Slackness Example

Data: $c = (2,3,0,0)$, $b = (4,6)$, $A = \begin{pmatrix}1 & 1 & 1 & 0 \\ 1 & 3 & 0 & 1\end{pmatrix}$, $x^*=(3,1,0,0)$ and $y^*=(\tfrac32,\tfrac12)$:

Let's look at the terms in $(A^{\intercal}y^* - c)^{\intercal}x^*=0$.

\[ \begin{aligned} A^{\intercal}y^* - c = \begin{pmatrix}\color{red}{0}\\\color{red}{0}\\\tfrac32\\\tfrac12\end{pmatrix}. \end{aligned} \qquad \begin{aligned} x^* = \begin{pmatrix} 3 \\ 1 \\ \color{red}{0} \\ \color{red}{0}\end{pmatrix}. \end{aligned} \]

This is what complementary slackness means: if an entry of $x^*$ is nonzero, then the corresponding entry of $A^{\intercal}y - c$ must be 0.

Furniture Workshop

ChairTableAvailable
Wood (metres)2480
Labor (hours)1350
Profit (dollars)4090

Primal

\[\begin{aligned} \max\quad &40c+90t\\ \text{s.t.}\quad &2c+4t+s_1=80,\\ &c+3t+s_2=50,\\ &c,t,s_1,s_2\ge0. \end{aligned}\]

Dual

\[\begin{aligned} \min\quad &80u+50v\\ \text{s.t.}\quad &2u+v\ge40,\\ &4u+3v\ge90,\\ &u,v\ge0. \end{aligned}\]

Optimal primal: $(c,t,s_1,s_2)=(20,10,0,0)$, Optimal dual : $(u,v)=(15,10)$.

Why is complementary slackness useful?

Let's say we have an LP in SEF, where $A \in \R^{m \times n}$. \[\max\{c^{\intercal}x:Ax=b,\ x\ge0\}.\]

We saw from our simplex method discussion that an optimal solution to this LP is a basic feasible solution with respect to some basis. How many nonzero entries do we expect that such a solution should have?

Every basis in of $A$ has $m$ entries, so every BFS has at most $m$ nonzero entries. A nondegenerate BFS has exactly $m$ nonzero entries (we'll talk about degenerate solutions in a second.)

What does complementary slackness imply in this case?

Recovering a Dual Solution

Primal (SEF)

\[\begin{aligned}\max\quad &7x_1+4x_2\\ &x_1+2x_2+x_3=8\\ &3x_1+x_2+x_4=9\\ &x_1,x_2,x_3,x_4\ge0\end{aligned}\]

Dual

\[\begin{aligned}\min\quad &8y_1+9y_2\\ &y_1+3y_2\ge7\\ &2y_1+y_2\ge4\\ &y_1\ge0,\ y_2\ge0\end{aligned}\]

Let's take an optimal solution to the primal LP: \[ x^*=(2,3,0,0), \qquad c^{\intercal}x^*=26. \]

It is nondegenerate because it has exactly $m=2$ nonzero entries.

Recovering a Dual Solution

Primal (SEF)

\[\begin{aligned}\max\quad &7x_1+4x_2\\ &x_1+2x_2+x_3=8\\ &3x_1+x_2+x_4=9\\ &x_1,x_2,x_3,x_4\ge0\end{aligned}\]

Dual

\[\begin{aligned}\min\quad &8y_1+9y_2\\ &y_1+3y_2\ge7\\ &2y_1+y_2\ge4\\ &y_1\ge0,\ y_2\ge0\end{aligned}\]

Since $x_1^*=2>0$ and $x_2^*=3>0$, complementary slackness implies that the dual constraints are tight: \[ y_1+3y_2=7, \qquad 2y_1+y_2=4. \]

This has a unique solution $y = (1,2)$. Complementary slackness implies that any optimal dual solution must satisfy these constraints, so $(1,2)$ must be an optimal dual solution!

Recovering a Dual Solution

Primal (SEF)

\[\begin{aligned}\max\quad &7x_1+4x_2\\ &x_1+2x_2+x_3=8\\ &3x_1+x_2+x_4=9\\ &x_1,x_2,x_3,x_4\ge0\end{aligned}\]

Dual

\[\begin{aligned}\min\quad &8y_1+9y_2\\ &y_1+3y_2\ge7\\ &2y_1+y_2\ge4\\ &y_1\ge0,\ y_2\ge0\end{aligned}\]

Let's check for $y = (1,2)$: $ \begin{aligned}y_1+3y_2 = 7 \ge7\\ 2y_1+y_2 = 4\ge4\\ y_1 = 1\ge0\\ y_2 = 2\ge0 \\ 8y_1 + 9 y_2 = 26 = c^{\intercal}x^*\end{aligned}. $

Recovering a Dual Solution II

Primal (SEF)

\[\begin{aligned}\max\quad &3x_1+3x_2+2x_3+5x_4\\ &x_1+x_2+2x_4=3\\ &x_2+x_3+x_4=4\\ &x_1+x_3+x_4=5\\ &x_1,x_2,x_3,x_4\ge0\end{aligned}\]

Dual

\[\begin{aligned}\min\quad &3y_1+4y_2+5y_3\\ &y_1+y_3\ge3\\ &y_1+y_2\ge3\\ &y_2+y_3\ge2\\ &2y_1+y_2+y_3\ge5\end{aligned}\]

Let's take an optimal solution to the primal LP: \[ x^*=(2,1,3,0), \qquad c^{\intercal}x^*=15. \]

It is nondegenerate because it has exactly $m=3$ nonzero entries.

Recovering a Dual Solution II

Primal (SEF)

\[\begin{aligned}\max\quad &3x_1+3x_2+2x_3+5x_4\\ &x_1+x_2+2x_4=3\\ &x_2+x_3+x_4=4\\ &x_1+x_3+x_4=5\\ &x_1,x_2,x_3,x_4\ge0\end{aligned}\]

Dual

\[\begin{aligned}\min\quad &3y_1+4y_2+5y_3\\ &y_1+y_3\ge3\\ &y_1+y_2\ge3\\ &y_2+y_3\ge2\\ &2y_1+y_2+y_3\ge5\end{aligned}\]

Since $x_1^*,x_2^*,x_3^*>0$, complementary slackness implies that the first three dual constraints are tight: \[ y_1+y_3=3, \qquad y_1+y_2=3, \qquad y_2+y_3=2. \]

This has a unique solution $y = (2,1,1)$. Complementary slackness implies that any optimal dual solution must satisfy these constraints, so $(2,1,1)$ must be an optimal dual solution!

Recovering a Dual Solution II

Primal (SEF)

\[\begin{aligned}\max\quad &3x_1+3x_2+2x_3+5x_4\\ &x_1+x_2+2x_4=3\\ &x_2+x_3+x_4=4\\ &x_1+x_3+x_4=5\\ &x_1,x_2,x_3,x_4\ge0\end{aligned}\]

Dual

\[\begin{aligned}\min\quad &3y_1+4y_2+5y_3\\ &y_1+y_3\ge3\\ &y_1+y_2\ge3\\ &y_2+y_3\ge2\\ &2y_1+y_2+y_3\ge5\end{aligned}\]

Let's check for $y = (2,1,1)$: $ \begin{aligned}y_1+y_3 = 3 \ge3\\ y_1+y_2 = 3\ge3\\ y_2+y_3 = 2\ge2\\ 2y_1+y_2+y_3 = 6\ge5 \\ 3y_1 + 4 y_2 + 5y_3 = 15 = c^{\intercal}x^*\end{aligned}. $

Finding the Dual Solution

Theorem. Let $x^*$ be an optimal solution of $\max\{c^{\intercal}x:Ax=b,\ x\ge0\}$ that is a nondegenerate (i.e. having $m$ nonzero entries) basic feasible solution with basis $B$. Then the dual $\min\{b^{\intercal}y:A^{\intercal}y\ge c\}$ has a unique optimal solution $y^*$, which is the solution of \[ A_B^{\intercal}y=c_B, \qquad\text{i.e.}\qquad y^*=(A_B^{\intercal})^{-1}c_B. \]

The proof is just that complementary slackness implies that $ A_B^{\intercal}y=c_B$ and the fact that $x^*$ is nondegenerate implies that this matrix is in fact full rank.

Degenerate Optimal Solutions

What happens when the optimal solution to an LP is degenerate?

One possibility is that there are multiple optimal dual solutions; this implies that there won't be a simple formula for the value of $y^*$ just from complementary slackness considerations.

This is not the only possibility; there may be just one optimal dual solution and the primal might just have a degenerate optimum.

Degenerate Example: Many Dual Solutions

Primal (SEF)

\[\begin{aligned}\max\quad &2x_1+x_2\\ &x_1+x_2+x_3=4\\ &x_1+3x_2+x_4=4\\ &x_1,x_2,x_3,x_4\ge0\end{aligned}\]

Dual

\[\begin{aligned}\min\quad &4y_1+4y_2\\ &y_1+y_2\ge2\\ &y_1+3y_2\ge1\\ &y_1\ge0,\ y_2\ge0\end{aligned}\]

Let's take an optimal solution to the primal LP: \[ x^*=(4,0,0,0), \qquad c^{\intercal}x^*=8. \]

It is degenerate because it has only $1$ nonzero entry, which is fewer than $m=2$.

Degenerate Example: Many Dual Solutions

Primal (SEF)

\[\begin{aligned}\max\quad &2x_1+x_2\\ &x_1+x_2+x_3=4\\ &x_1+3x_2+x_4=4\\ &x_1,x_2,x_3,x_4\ge0\end{aligned}\]

Dual

\[\begin{aligned}\min\quad &4y_1+4y_2\\ &y_1+y_2\ge2\\ &y_1+3y_2\ge1\\ &y_1\ge0,\ y_2\ge0\end{aligned}\]

Since $x_1^*>0$, complementary slackness says that the first dual constraint is tight: $y_1+y_2=2$.

Letting $y=(2-t,\,t)$, the remaining dual constraints hold as long as $0 \le t \le 2$, so there are infinitely many $y$ optimizing the dual program.

What to do with a degenerate primal solution

Unfortunately, for a degenerate primal solution, there is no easy way to extract an optimal dual solution.

  1. We can run the simplex method on a smaller linear programming feasibility problem.

    \[ \begin{aligned} A_T^{\intercal} y = c_T\\ A_S^{\intercal}y \ge c_S, \end{aligned} \] where $A_T$ is the submatrix of $A$ containing columns associated to variables $x_i > 0$, and $A_S$ is the submatrix containing columns associated to the variablaes $x_i = 0$. This encodes complementary slackness. Also note that we can use linear algebra to reduce the number of variables using $A_T^{\intercal} y = c_T$.

What to do with a degenerate primal solution

Unfortunately, for a degenerate primal solution, there is no easy way to extract an optimal dual solution.

  1. We can run the simplex method on a smaller linear program feasibility problem.

  2. If $x^*$ was obtained by finding a basis $B$ for which the simplex method terminates, then it turns out that the columns of $B$ can be used to solve for $y$, i.e. \[ y = (A_B^{\intercal})^{-1} c_B, \] where $A_B$ denotes the submatrix of $A$ consisting of the columns in $B$.

Recovering a Primal Solution

We can also use an optimal dual solution to recover an optimal primal solution using complementary slackness.

Primal (SEF)

\[\begin{aligned}\max\quad &4x_1+x_2\\ &2x_1+x_2+x_3=8\\ &x_1+3x_2+x_4=9\\ &x_1,x_2,x_3,x_4\ge0\end{aligned}\]

Dual

\[\begin{aligned}\min\quad &8y_1+9y_2\\ &2y_1+y_2\ge4\\ &y_1+3y_2\ge1\\ &y_1\ge0,\ y_2\ge0\end{aligned}\]

Let's take an optimal solution to the dual LP: \[ y^*=(2,0), \qquad b^{\intercal}y^*=16. \]

Recovering a Primal Solution

Primal (SEF)

\[\begin{aligned}\max\quad &4x_1+x_2\\ &2x_1+x_2+x_3=8\\ &x_1+3x_2+x_4=9\\ &x_1,x_2,x_3,x_4\ge0\end{aligned}\]

Dual

\[\begin{aligned}\min\quad &8y_1+9y_2\\ &2y_1+y_2\ge4\\ &y_1+3y_2\ge1\\ &y_1\ge0,\ y_2\ge0\end{aligned}\]

The dual constraints for $x_2$ and $x_3$ are not tight: $y_1+3y_2=2>1$ and $y_1=2>0$. Complementary slackness implies that \[ x_2^*=0, \qquad x_3^*=0. \]

Substituting into $Ax=b$ leaves $2x_1=8$ and $x_1+x_4=9$. This has a unique solution $x=(4,0,0,5)$, so $(4,0,0,5)$ must be an optimal primal solution!

Recovering a Primal Solution

Primal (SEF)

\[\begin{aligned}\max\quad &4x_1+x_2\\ &2x_1+x_2+x_3=8\\ &x_1+3x_2+x_4=9\\ &x_1,x_2,x_3,x_4\ge0\end{aligned}\]

Dual

\[\begin{aligned}\min\quad &8y_1+9y_2\\ &2y_1+y_2\ge4\\ &y_1+3y_2\ge1\\ &y_1\ge0,\ y_2\ge0\end{aligned}\]

Let's check for $x = (4,0,0,5)$: $ \begin{aligned}2x_1+x_2+x_3 = 8 \\ x_1+3x_2+x_4 = 9\\ x_1 = 4\ge0, \ x_2 = 0\ge0\\ x_3 = 0\ge0,\ x_4 = 5\ge0 \\ 4x_1 + x_2 = 16 = b^{\intercal}y^*\end{aligned}. $

Complementary Slackness for LPs Not In SEF

LPs Not In SEF

Given a maximization LP that has a mixture of free variables and nonfree variables (and equality and inequality constraints), recall that the dual problem has

  1. An equality constraint for each free variable.
  2. A $\ge$ constraint for each nonnegative variable.
  3. A free variable for each equality constraint.
  4. A nonnegative variable for each $\le$ constraint.

Complementary slackness in this case says that each dual constraint associated to a nonnegative variable is tight if that variable is positive.

Similarly, each dual variable associated to a $\le$ constraint must be $0$ if that constraint is not tight.

Recovering a Dual Solution III

Primal

\[\begin{aligned}\max\quad &3x_1+2x_2\\ &x_1+x_2=4\\ &2x_1+x_2\le6\\ &x_2\le3\\ &x_1,x_2\ge0\end{aligned}\]

Dual

\[\begin{aligned}\min\quad &4y_1+6y_2+3y_3\\ &y_1+2y_2\ge3\\ &y_1+y_2+y_3\ge2\\ &y_1\text{ free},\ y_2,y_3\ge0\end{aligned}\]

Let's take an optimal solution to the primal LP: \[ x^*=(2,2), \qquad c^{\intercal}x^*=10. \]

The first constraint is an equality, the second is tight, and the third has slack: $x_2^*=2<3$.

Recovering a Dual Solution III

Primal

\[\begin{aligned}\max\quad &3x_1+2x_2\\ &x_1+x_2=4\\ &2x_1+x_2\le6\\ &x_2\le3\\ &x_1,x_2\ge0\end{aligned}\]

Dual

\[\begin{aligned}\min\quad &4y_1+6y_2+3y_3\\ &y_1+2y_2\ge3\\ &y_1+y_2+y_3\ge2\\ &y_1\text{ free},\ y_2,y_3\ge0\end{aligned}\]

At $x^* = (2,2)$, the third primal constraint is not tight, so complementary slackness implies $y_3=0$. Since $x_1^*>0$ and $x_2^*>0$, the first two dual constraints are tight: \[ y_1+2y_2=3, \qquad y_1+y_2+y_3=2. \]

With $y_3=0$, this has a unique solution $y = (1,1,0)$. Complementary slackness implies that any optimal dual solution must satisfy these conditions, so $(1,1,0)$ must be an optimal dual solution!

Recovering a Dual Solution III

Primal

\[\begin{aligned}\max\quad &3x_1+2x_2\\ &x_1+x_2=4\\ &2x_1+x_2\le6\\ &x_2\le3\\ &x_1,x_2\ge0\end{aligned}\]

Dual

\[\begin{aligned}\min\quad &4y_1+6y_2+3y_3\\ &y_1+2y_2\ge3\\ &y_1+y_2+y_3\ge2\\ &y_1\text{ free},\ y_2,y_3\ge0\end{aligned}\]

Let's check for $y = (1,1,0)$: $ \begin{aligned}y_1+2y_2 = 3 \ge3\\ y_1+y_2+y_3 = 2\ge2\\ y_1 = 1\text{ (free)},\ y_2 = 1\ge0,\ y_3=0\ge0 \\ 4y_1 + 6 y_2 + 3y_3 = 10 = c^{\intercal}x^*\end{aligned}. $

Complementary Slackness for More General LPs

In summary, nonnegative primal variables are associated to inequality dual constraints, and those dual constraints are tight if that primal variable is positive.

Primal inequality constraints are associated to nonnegative dual variables, and those dual variables are 0 if the associated constraint is not held at equality for any optimal solution.