Introduction to Optimization

CO 250


Lecture 16


Scroll view

PDF version of these slides (demo: this PDF version is experimental and may contain errors or differ from the slides)

Lecture Outline

  • Review of Complementary Slackness
  • Strong Duality and Farkas' Lemma
  • Normal Cones

Review: Complementary Slackness

Review: Duality

  • Every linear program has a dual linear program.
  • Dual constraints correspond to primal variables.
  • Dual variables correspond to primal constraints.
  • Variable sign conditions should be thought of as being part of the variable definition and not a separate constraint for these purposes.
  • Inequality constraints correspond to dual variables with sign constraints, and sign-constrained variables correspond to dual inequality constraints.

Review: Complementary Slackness

  • For each primal inequality constraint, the dual variable will either be 0 or nonzero.
  • For an optimal solution $x^*$ to the LP, a primal inequality is tight if the left and right sides of the inequality are equal at $x^*$. It is slack otherwise.
  • Theorem: If $x^*$ is an optimal solution to an LP, and a given primal inequality is slack, then the dual variable is equal to 0.
  • Each LP is the dual of its dual LP, so the dual concept holds.
  • Theorem: If $x^*$ is an optimal solution to an LP, and $x_i \ge 0$ is a nonnegativity constraint, and $x^*_i > 0$, then for any optimal solution $y^*$ to the dual LP, the corresponding inequality in the dual LP is tight.

Complementary Slackness Example

Primal

\[\begin{aligned} \max\quad &3x_1+4x_2-x_3\\ \text{s.t.}\quad &x_1+2x_2+x_3\le-1,\\ &2x_1+x_2-x_3=1,\\ &x_1-x_2+2x_3\ge-3,\\ &x_1\ge0,\ x_2\text{ free},\ x_3\le0. \end{aligned}\]

Dual

\[\begin{aligned} \min\quad &-y_1+y_2-3y_3\\ \text{s.t.}\quad &y_1+2y_2+y_3\ge3,\\ &2y_1+y_2-y_3=4,\\ &y_1-y_2+2y_3\le-1,\\ &y_1\ge0,\ y_2\text{ free},\ y_3\le0. \end{aligned}\]

The optimal primal solution: $x^*=(0,\,0,\,-1)$, with optimal value $1$.

Complementary Slackness Example

PrimalDual
Constraints $\leftrightarrow$ dual variables
$x_1+2x_2+x_3\le-1$$\leftrightarrow$$y_1\ge0$
$2x_1+x_2-x_3=1$$\leftrightarrow$$y_2$ free
$x_1-x_2+2x_3\ge-3$$\leftrightarrow$$y_3\le0$
Variables $\leftrightarrow$ dual constraints
$x_1\ge0$$\leftrightarrow$$y_1+2y_2+y_3\ge3$
$x_2$ free$\leftrightarrow$$2y_1+y_2-y_3=4$
$x_3\le0$$\leftrightarrow$$y_1-y_2+2y_3\le-1$

Complementary Slackness Example

Primal at $x^*$Dual
Constraints $\leftrightarrow$ dual variables
$x^*_1+2x^*_2+x^*_3=-1\le-1$$\leftrightarrow$$y_1\ge0$
$2x^*_1+x^*_2-x^*_3=1=1$$\leftrightarrow$$y_2$ free
$x^*_1-x^*_2+2x^*_3=-2\ge-3$$\leftrightarrow$$y_3\le0$ (tight)
Variables $\leftrightarrow$ dual constraints
$x^*_1=0\ge0$ $\leftrightarrow$ $y_1+2y_2+y_3\ge3$
$x^*_2=0$ (free) $\leftrightarrow$ $2y_1+y_2-y_3=4$ (tight)
$x^*_3=-1\le0$ $\leftrightarrow$ $y_1-y_2+2y_3\le-1$ (tight)

Red: inequalities with slack at $x^* = (0,0,-1)$. Green: rows with no slack (tight).

Complementary Slackness Example

We find from the previous considerations that for any dual optimal solution, $y_3=0$ (its primal constraint has slack), and the dual constraints for $x_2$ (free) and $x_3<0$ (sign constraint with slack) are tight:

\(\displaystyle y_3=0\)

\(\displaystyle 2y_1+y_2-y_3=4\)

\(\displaystyle y_1-y_2+2y_3=-1\)

There is a unique solution to these 3 equations in 3 unknowns.

\[y^*=(y_1,y_2,y_3)=(1,\,2,\,0).\]

Complementary Slackness Example II

Complementary Slackness Example II

Primal

\[\begin{aligned} \max\quad &4x_1+4x_2-3x_3\\ \text{s.t.}\quad &x_1+x_2+x_3=2,\\ &x_1-x_2+2x_3=5,\\ &2x_1+x_2-x_3=2,\\ &x_1\ge0,\ x_2\text{ free},\ x_3\ge0. \end{aligned}\]

Dual

\[\begin{aligned} \min\quad &2y_1+5y_2+2y_3\\ \text{s.t.}\quad &y_1+y_2+2y_3\ge4,\\ &y_1-y_2+y_3=4,\\ &y_1+2y_2-y_3\ge-3,\\ &y_1,y_2,y_3\text{ free}. \end{aligned}\]

The optimal primal solution: $x^*=(2,\,-1,\,1)$, with optimal value $1$.

Complementary Slackness Example II

PrimalDual
Constraints $\leftrightarrow$ dual variables
$x_1+x_2+x_3=2$$\leftrightarrow$$y_1$ free
$x_1-x_2+2x_3=5$$\leftrightarrow$$y_2$ free
$2x_1+x_2-x_3=2$$\leftrightarrow$$y_3$ free
Variables $\leftrightarrow$ dual constraints
$x_1\ge0$$\leftrightarrow$$y_1+y_2+2y_3\ge4$
$x_2$ free$\leftrightarrow$$y_1-y_2+y_3=4$
$x_3\ge0$$\leftrightarrow$$y_1+2y_2-y_3\ge-3$

Complementary Slackness Example II

Primal at $x^*$Dual
Constraints $\leftrightarrow$ dual variables
$x^*_1+x^*_2+x^*_3=2=2$$\leftrightarrow$$y_1$ free
$x^*_1-x^*_2+2x^*_3=5=5$$\leftrightarrow$$y_2$ free
$2x^*_1+x^*_2-x^*_3=2=2$$\leftrightarrow$$y_3$ free
Variables $\leftrightarrow$ dual constraints
$x^*_1=2\ge0$ $\leftrightarrow$ $y_1+y_2+2y_3\ge4$ (tight)
$x^*_2=-1$ (free) $\leftrightarrow$ $y_1-y_2+y_3=4$ (tight)
$x^*_3=1\ge0$ $\leftrightarrow$ $y_1+2y_2-y_3\ge-3$ (tight)

Red: inequalities with slack at $x^* = (2,-1,1)$. Green: rows with no slack (tight).

Complementary Slackness Example II

The dual constraints associated with $x_1^*,x_3^*>0$ and with the free variable $x_2$ are tight for any dual optimal solution:

\(\displaystyle y_1+y_2+2y_3=4\)

\(\displaystyle y_1-y_2+y_3=4\)

\(\displaystyle y_1+2y_2-y_3=-3\)

There is a unique solution to these 3 equations in 3 unknowns.

\[y^*=(y_1,y_2,y_3)=(1,\,-1,\,2).\]

Farkas' Lemma and Strong Duality

Farkas' Lemma

Recall Farkas' lemma.

Farkas' Lemma

For $A \in \R^{m \times n}$ and $b \in \R^m$, exactly one of the following two things holds:

  • There is some $x \in \R^n$ so that $Ax = b$, and $x \ge 0$.
  • There is some $y \in \R^m$ so that $A^{\intercal}y \ge 0$ and $b^{\intercal}y < 0$.

We discussed a geometric proof of this using the separating hyperplane theorem and now we will prove it using strong duality.

Farkas' Lemma Proof

We have already seen that if there is $y$ so that $A^{\intercal}y \ge 0$ and $b^{\intercal}y < 0$, then there cannot be an element of the set $\{x\in \R^n : Ax = b, x \ge 0\}$. If there were, then \[ \underbrace{y^{\intercal}Ax}_{\qquad \ge 0\\A^{\intercal}y \ge 0, x \ge 0} = \underbrace{y^{\intercal}b}_{<0}. \]

So to prove the other direction, we will need strong duality. What LP are we going to apply the strong duality theorem to?

Farkas' Lemma Proof

Consider the linear program \[ \begin{aligned} \max\qquad & \vec{0}^{\intercal}x \\ \text{s.t.}\qquad & Ax = b \\ & x \ge 0 \\ \end{aligned} \]

This is infeasible precisely when the set $\{x\in \R^n : Ax = b, x \ge 0\}$ is empty.

Strong duality implies that the dual LP does not have an optimal solution (if it did, then the primal LP would have an optimal solution by strong duality).

So the dual LP is either unbounded or infeasible.

Farkas' Lemma Proof

The dual LP is \[ \begin{aligned} \min\qquad & b^{\intercal}y \\ \text{s.t.}\qquad & A^{\intercal}y \ge 0 \\ \end{aligned} \]

Why is this feasible?

It's feasible since $y = 0$ is feasible.

Since the dual LP is unbounded, we have seen that there must in particular be a feasible $y$ so that $b^{\intercal} y < 0$.

We conclude that there is some $y$ so that $A^{\intercal}y \ge 0$ and $b^{\intercal}y< 0$, which is what we wanted.

Farkas' Lemma Variants

We can use more general linear programming duality results to get other versions of Farkas' lemma:

Farkas' Lemma Version II

For $A \in \R^{m \times n}$ and $b \in \R^m$, exactly one of the following two things holds:

  • There is some $x \in \R^n$ so that $Ax \le b$.
  • There is some $y \in \R^m$ so that $y \ge 0$, $A^{\intercal}y = 0$ and $b^{\intercal}y < 0$.

We can prove this by noting that the dual of the linear program \[ \begin{aligned} \max\qquad & \vec{0}^{\intercal}x \\ \text{s.t.}\qquad & Ax \le b \\ \end{aligned} \qquad \text{ is } \qquad \begin{aligned} \min\qquad & b^{\intercal}y \\ \text{s.t.}\qquad & A^{\intercal}y = 0 \\ & y \ge 0 \\ \end{aligned} \]

Normal Cones

Normal Cones

The feasible region of a linear program is a polytope, and we have seen that (typically) the unique maximizer of a linear function over a polytope is a vertex of that polytope.

A quadrilateral in the plane with an objective direction arrow and its maximizing vertex highlighted. $x^*$

\[\begin{aligned} \max\quad &x_1+2x_2\\ \text{s.t.}\quad &x_1\ge0,\\ &x_2\ge0,\\ &3x_1+x_2\le12,\\ &-x_1+3x_2\le6. \end{aligned}\]

Normal Cones

So while there are infinitely many possible objective directions $c$, there are only finitely many possible outcomes of the optimal solution of the LP.

The same quadrilateral with all four vertices highlighted. $x^*$

\[\begin{aligned} \max\quad &c^{\intercal}x\\ \text{s.t.}\quad &x_1\ge0,\\ &x_2\ge0,\\ &3x_1+x_2\le12,\\ &-x_1+3x_2\le6. \end{aligned}\]

Open the demo in its own tab

Normal Cones

For each vertex $v$ in a polytope $P$, we define the normal cone $N_v$ to be the set of $c$ so that the maximizer of the linear program $\max \{c^{\intercal}x : x \in P\}$ is $v$: \[ N_v = \{c \in \R^n : \text{For any }x \in P,\; c^{\intercal}v \ge c^{\intercal}x \}. \]

The plane of objective vectors divided into four colored normal cones, one for each vertex of the quadrilateral.

Normally, with linear programming, we are given the objective direction $c$ and want to find a vertex $v \in P$ maximizing $c^{\intercal}v$; the normal cone flips this idea on its head.

Can we get a more explicit description of the normal cone of a polytope?

Normal Cones

Normal Cone Characterization (SEF Polytopes)

Suppose that $P = \{x \in \R^n : Ax = b, x \ge 0\}$, and let $v \in P$ be a vertex (i.e. a basic feasible solution). Then $c \in N_v$ if and only if there exists some $y \in \R^m$ so that \[ c \le A^{\intercal}y, \] and $c_i = (A^{\intercal}y)_i$ whenever $v_i > 0$.

This is really a rephrasing of complementary slackness.

Normal Cones

Example of Theorem

Consider a polytope defined by \[ P=\{x\in\R^4:Ax=b,\ x\ge0\} \text{ with }\] \[ A=\begin{pmatrix}1&1&1&0\\1&3&0&1\end{pmatrix},\quad b=\begin{pmatrix}4\\6\end{pmatrix}. \]

Here is a vertex of this polytope: $v=(3,1,0,0)$.

Let's use the theorem to check that $c=(3,5,1,0) \in N_v$ (i.e. $v$ is a maximizer of $c^{\intercal}x$ in $P$).

Let $y = (2,1)$, and note that $A^{\intercal}y = (3,5,2,1)$. We see that \[ c = A^{\intercal}y - (0,0,1,1). \]

Normal Cones

Proof of Theorem

$c$ is in $N_v$ if and only if $v$ is an optimal solution of the following linear program: \[ \max \{c^{\intercal}x : Ax = b, x \ge 0\} \]

The dual linear program is \[ \min \{b^{\intercal}y : A^{\intercal}y \ge c\}. \]

Strong duality implies that the dual linear program is feasible, so there is some $y$ so that $A^{\intercal}y \ge c$, and by complementary slackness, we know that $(A^{\intercal}y)_i = c_i$ whenever $v_i > 0$.

Normal Cones

The above theorem shows one characterization of the normal cone, but it does not make it easy to check if a given $c$ is in the normal cone without solving a linear program. We can give a simpler characterization (at least when the vertex is nondegenerate).

Normal Cone Characterization (SEF Polytopes)

Let $P=\{x:Ax=b,\ x\ge0\}$, and let $v$ be a vertex of $P$ with basis $B$ with $m$ positive entries (so $v$ is nondegenerate). For each $c$, we can break up its coordinates into those coordinates in the basis $B$ and those outside it: $c = \begin{pmatrix}c_B & c_N\end{pmatrix}$. \[ N_v=\{c\in\R^n : c - A^{\intercal}A_B^{-\intercal}c_B\le0\}. \]

Normal Cones

Proof of Theorem This can be seen from the terminal condition of the simplex method.

We can also see it from complementary slackness; using these dual LPs:

$\max \{c^{\intercal}x : Ax = b, x \ge 0\}$

$\min \{b^{\intercal}y : A^{\intercal}y \ge c\}.$

Complementary slackness implies that $v$ is an optimal solution to the primal LP if and only if there is a feasible $y^*$ for the dual LP that satisfies $(A^{\intercal}y)_i = c_i$ whenever $v_i > 0$.

This implies that the unique such $y^*$ is precisely $y^* = A_B^{-\intercal}c_B$.

Plugging in, we get that $v$ is an optimal solution to the primal LP if and only if $A^{\intercal}A_B^{-\intercal}c_B \ge c$, which is what we wanted to show.

Normal Cones

We can also characterize normal cones for polytopes described by inequalities.

Normal Cone Characterization (Polytopes $Ax \le b$)

Suppose that $P = \{x \in \R^n : Ax \le b\}$, and let $v \in P$ be a vertex. Then $c \in N_v$ if and only if there exists some $y \in \R^m$ so that \[ c = A^{\intercal}y, \qquad y \ge 0, \] and $y_i = 0$ whenever $(Av)_i < b_i$.

One way to think of this is as follows: for each $v$ in $P$, there is a collection of tight constraints, $a_1^{\intercal}x \le b_1, \dots, a_k^{\intercal}x \le b_k$ where $k \ge n$, and where $a_i^{\intercal}v = b_i$ for each $i$. We then have that the normal cone is \[ N_v = \{\sum_{i=1}^k y_i a_i : y \ge 0\}. \]

Normal Cone Example

Let's recall that the $n$ dimensional hypercube is the polytope defined by \[ P = \{x \in \R^n : 0 \le x_i \le 1 \text{ for }i=1,\dots,n\}. \]

So $A = \begin{pmatrix} I \\ -I\end{pmatrix},$ and $b = \begin{pmatrix} \vec{1}\\\vec{0}\end{pmatrix}.$

A vertex $v$ of $P$ is a point where $n$ of the constraints are tight, i.e. a point where every entry is either 0 or 1.

The tight constraints at $v$ have normals $e_i$ (when $v_i=1$) and $-e_i$ (when $v_i=0$), so \[ N_v=\{c\in\R^n : c_i\ge0 \text{ if } v_i=1,\ c_i\le0 \text{ if } v_i=0\}. \]

A square with four colored vertices, and its normal fan: the four quadrants of the plane, each colored by its vertex.