Introduction to Optimization

CO 250


Lecture 14


Scroll view

PDF version of these slides (demo: this PDF version is experimental and may contain errors or differ from the slides)

  • Review of Dual Linear Programs
  • Strong Duality

The Dual of an LP in SEF

Primal

\[\begin{aligned} \max\quad &c^{\intercal}x\\ \text{s.t.}\quad &Ax=b,\\ &x\ge0. \end{aligned}\]

Dual

\[\begin{aligned} \min\quad &b^{\intercal}y\\ \text{s.t.}\quad &A^{\intercal}y\ge c,\\ &y\text{ free}. \end{aligned}\]

If $A$ has $m$ rows and $n$ columns, the dual has $m$ variables and $n$ constraints.

Weak Duality

If $x$ is primal feasible and $y$ is dual feasible, then

\[\boxed{c^{\intercal}x\le b^{\intercal}y.}\]

We can see this by looking at $y^{\intercal}Ax$, and expanding this expression in two different ways using the constraints.

$(y^{\intercal}A)x =$ $(A^{\intercal}y)^{\intercal}x =$ $\underbrace{c^{\intercal}x}_{\color{#1f6fd6}{A^{\intercal}y\ge c},\;x\ge0}\le$ $y^{\intercal}Ax =$ $y^{\intercal}(Ax)$ $\underbrace{b^{\intercal}}_{Ax=b}y$ .

Consequences of Weak Duality

  • If $c^{\intercal}x^*=b^{\intercal}y^*,$ for dual feasible $y^*$ and primal feasible $x^*$, then both are optimal.
  • In this case, $y^*$ is a certificate of optimality for the primal LP.
  • If the primal is unbounded above, the dual is infeasible.
  • If the dual is unbounded below, the primal is infeasible.
  • Note: both LPs can be infeasible; see the assignment for examples.

Duals of LPs Not in SEF

If an LP is not in SEF, we can take its dual by first giving an equivalent LP in SEF, and then taking the dual of the SEF LP.

\[\begin{aligned} \max\quad &x_1+2x_2\\ \text{s.t.}\quad &x_1 - x_2 = 1,\\ &x_1 + x_2 \ge 3,\\ &x_1 \ge 0,\\ &x_2\text{ free}. \end{aligned}\]

Duals of LPs Not in SEF

If an LP is not in SEF, we can take its dual by first giving an equivalent LP in SEF, and then taking the dual of the SEF LP.

\[\begin{aligned} \max\quad &x_1+2x_2^+ - 2 x_2^-\\ \text{s.t.}\quad &x_1 - x_2^+ + x_2^- = 1,\\ &x_1 + x_2^+-x_2^- - s_1 = 3,\\ &x_1, x_2^+, x_2^-, s_1 \ge 0 \end{aligned}\]

Duals of LPs Not in SEF

If an LP is not in SEF, we can take its dual by first giving an equivalent LP in SEF, and then taking the dual of the SEF LP.

\[\begin{aligned} \max\quad &\begin{pmatrix}1\\2\\-2\\0\end{pmatrix}^{\intercal}\begin{pmatrix}x_1\\x_2^+\\x_2^-\\s_1\end{pmatrix}\\ \text{s.t.}\quad & \begin{pmatrix} 1 & -1 & 1 & 0\\ 1 & 1 & -1 & -1\end{pmatrix}\begin{pmatrix}x_1\\x_2^+\\x_2^-\\s_1\end{pmatrix} = \begin{pmatrix}1\\3\end{pmatrix}\\ &x_1, x_2^+, x_2^-, s_1 \ge 0 \end{aligned}\]

How many dual variables will there be? How many dual constraints?

Duals of LPs Not in SEF

If an LP is not in SEF, we can take its dual by first giving an equivalent LP in SEF, and then taking the dual of the SEF LP.

\[\begin{aligned} \max\quad &\begin{pmatrix}1\\2\\-2\\0\end{pmatrix}^{\intercal}\begin{pmatrix}x_1\\x_2^+\\x_2^-\\s_1\end{pmatrix}\\ \text{s.t.}\quad & \begin{pmatrix} 1 & -1 & 1 & 0\\ 1 & 1 & -1 & -1\end{pmatrix}\begin{pmatrix}x_1\\x_2^+\\x_2^-\\s_1\end{pmatrix} = \begin{pmatrix}1\\3\end{pmatrix}\\ &x_1, x_2^+, x_2^-, s_1 \ge 0 \end{aligned}\]

\[\begin{aligned} \min\quad &\begin{pmatrix}1\\3\end{pmatrix}^{\intercal}\begin{pmatrix}y_1\\y_2\end{pmatrix}\\ \text{s.t.}\quad & \begin{pmatrix} 1 & 1 \\ -1 & 1 \\ 1 & -1 \\ 0 & -1 \end{pmatrix} \begin{pmatrix}y_1\\y_2\end{pmatrix} \ge \begin{pmatrix}1\\2\\-2\\0\end{pmatrix} \end{aligned}\]

Duals of LPs Not in SEF

If an LP is not in SEF, we can take its dual by first giving an equivalent LP in SEF, and then taking the dual of the SEF LP.

\[\begin{aligned} \min\quad &\begin{pmatrix}1\\3\end{pmatrix}^{\intercal}\begin{pmatrix}y_1\\y_2\end{pmatrix}\\ \text{s.t.}\quad & \begin{pmatrix} 1 & 1 \\ -1 & 1 \\ 1 & -1 \\ 0 & -1 \end{pmatrix} \begin{pmatrix}y_1\\y_2\end{pmatrix} \ge \begin{pmatrix}1\\2\\-2\\0\end{pmatrix} \end{aligned}\]

We can simplify the dual constraints though; note that the last row amounts to saying that $y_2 \le 0$ and the second and third inequalities amount to $y_1 - y_2 = -2$.

Duals of LPs Not in SEF

If an LP is not in SEF, we can take its dual by first giving an equivalent LP in SEF, and then taking the dual of the SEF LP.

\[\begin{aligned} \max\quad &x_1+2x_2\\ \text{s.t.}\quad &x_1 - x_2 = 1,\\ &x_1 + x_2 \ge 3,\\ &x_1 \ge 0,\\ &x_2\text{ free}. \end{aligned}\]

\[\begin{aligned} \min\quad &y_1 + 3y_2\\ \text{s.t.}\quad & y_1 + y_2 \ge 1\\ & y_1 - y_2 = -2 \\ & y_1\text{ free}\\ & y_2 \le 0. \end{aligned}\]

Note that the dual is infeasible; what does this imply about the primal?

Duals of LPs Not in SEF

Note that we have the following correspondence between variables/constraints in the primal/dual.

\[\begin{aligned} \max\quad &x_1+2x_2\\ \text{s.t.}\quad &{\color{#1f6fd6}{x_1-x_2=1}},\\ &{\color{#cb4b16}{x_1+x_2\ge3}},\\ &{\color{#708400}{x_1\ge0}},\\ &{\color{#8b4bad}{x_2\text{ free}}}. \end{aligned}\]

\[\begin{aligned} \min\quad &y_1+3y_2\\ \text{s.t.}\quad &{\color{#708400}{y_1+y_2\ge1}}\\ &{\color{#8b4bad}{y_1-y_2=-2}}\\ &{\color{#1f6fd6}{y_1\text{ free}}}\\ &{\color{#cb4b16}{y_2\le0}}. \end{aligned}\]

General Rule: For each variable in the primal, there should be a constraint in the dual. For each constraint in the primal other than a variable sign restriction, there should be a variable in the dual. Nonnegative variables in the primal correspond to inequality constraints. Free variables correspond to equality constraints, and vice versa.

Mixed Constraints and Variable Types

For a maximization primal and a minimization dual:

PrimalDual
$\le$ constraint$\leftrightarrow$Nonnegative variable
Equality constraint$\leftrightarrow$Free variable
$\ge$ constraint$\leftrightarrow$Nonpositive variable
Nonnegative variable$\leftrightarrow$$\ge$ constraint
Free variable$\leftrightarrow$Equality constraint
Nonpositive variable$\leftrightarrow$$\le$ constraint

Steps to Taking a Dual

  • Identify the number of dual variables and constraints.
  • For each nonnegative primal variable, the dual constraint is an inequality.
  • For each free primal variable, the dual constraint is an equality.
  • For each primal equality constraint, the dual variable is free.
  • For each primal $\le$ constraint, the dual variable is nonnegative.

Step 1: Count the Dual Variables

\[\begin{aligned} \max\quad &4x_1+2x_2-2x_3\\ \text{s.t.}\quad &2x_1+x_2+x_3\le4,\\ &x_1+3x_2-x_3=7,\\ &-x_1+2x_2+x_3\ge3,\\ &x_1\text{ free},\quad x_2,x_3\ge0. \end{aligned}\]

There are three constraints (apart from variable sign restrictions). So the dual has three variables: $y_1,y_2,y_3$.

For now, don't count $x_2,x_3\ge0$ as additional constraints: these will determine the directions of the dual constraints.

Step 2: Determine Their Signs

The primal is a maximization problem. Each constraint row determines the sign of its dual variable.

Primal rowDual variable
$2x_1+x_2+x_3\le4$$y_1\ge0$
$x_1+3x_2-x_3=7$$y_2$ is free
$-x_1+2x_2+x_3\ge3$$y_3\le0$

So $y_1$ is nonnegative, $y_2$ has no sign restriction, and $y_3$ is nonpositive.

Step 3: Build the Dual Constraints

For each primal variable, find the dual constraint. The coefficients in that constraint are the coefficients of that primal variable. The sense of the constraint is determined by the sign of the variable.

\[A=\begin{pmatrix}2&1&1\\1&3&-1\\-1&2&1\end{pmatrix}.\]

Primal variable / columnDual left side / relation
$x_1$ free: $(2,1,-1)$$2y_1+y_2-y_3$; equality
$x_2\ge0$: $(1,3,2)$$y_1+3y_2+2y_3$; $\ge$
$x_3\ge0$: $(1,-1,1)$$y_1-y_2+y_3$; $\ge$

Step 4: Objective and Right-Hand Sides

The primal maximizes, so the dual minimizes.

The primal right-hand sides $(4,7,3)$ become the dual objective coefficients:

\[\min\quad 4y_1+7y_2+3y_3.\]

The primal objective coefficients $(4,2,-2)$ become the dual constraint right-hand sides, in variable order:

\[\begin{aligned} 2y_1+y_2-y_3&=4,\\ y_1+3y_2+2y_3&\ge2,\\ y_1-y_2+y_3&\ge-2. \end{aligned}\]

Step 5: Assemble the Dual

Primal

\[\begin{aligned} \max\quad &4x_1+2x_2-2x_3\\ \text{s.t.}\quad &2x_1+x_2+x_3\le4,\\ &x_1+3x_2-x_3=7,\\ &-x_1+2x_2+x_3\ge3,\\ &x_1\text{ free},\quad x_2,x_3\ge0. \end{aligned}\]

Dual

\[\begin{aligned} \min\quad &4y_1+7y_2+3y_3\\ \text{s.t.}\quad &2y_1+y_2-y_3=4,\\ &y_1+3y_2+2y_3\ge2,\\ &y_1-y_2+y_3\ge-2,\\ &y_1\ge0,\quad y_2\text{ free},\quad y_3\le0. \end{aligned}\]

Check: $x=(1,2,0)$ and $y=(1,1,-1)$ are feasible and both have objective value $8$. Thus both are optimal.

Another Primal–Dual Pair

Primal

\[\begin{aligned} \max\quad &4x_1+5x_2\\ \text{s.t.}\quad &2x_1+x_2+x_3=8,\\ &x_1+2x_2+x_4=10,\\ &x_1,x_2,x_3,x_4\ge0. \end{aligned}\]

Dual

\[\begin{aligned} \min\quad &8y_1+10y_2\\ \text{s.t.}\quad &2y_1+y_2\ge4,\\ &y_1+2y_2\ge5,\\ &y_1,y_2\ge0. \end{aligned}\]

Optimal Solutions

Primal

\[\begin{aligned} \max\quad &4x_1+5x_2\\ \text{s.t.}\quad &2x_1+x_2+x_3=8,\\ &x_1+2x_2+x_4=10,\\ &x_1,x_2,x_3,x_4\ge0. \end{aligned}\]

Dual

\[\begin{aligned} \min\quad &8y_1+10y_2\\ \text{s.t.}\quad &2y_1+y_2\ge4,\\ &y_1+2y_2\ge5,\\ &y_1,y_2\ge0. \end{aligned}\]

Optimal solutions are $x^*=(2,4,0,0)$ and $y^*=(1,2)$.

Both are feasible, and their objective values agree:

\[4(2)+5(4)=28=8(1)+10(2).\]

By weak duality, both solutions are optimal.

A Diet Problem

Choose the cheapest mixture of three foods meeting three nutrient minimums. We also want exactly 600 g of food.

Use these illustrative values per 100 g portion, with prices in whole cents:

OatsLentilsAlmondsMinimum
Protein (g)1392174
Fibre (g)1081255
Iron (mg)43421
Price (cents)6030100

Let $x_1,x_2,x_3$ be the numbers of 100 g portions of oats, lentils, and almonds. Portions can be fractional.

The Diet LP

\[\begin{aligned} \min\quad &60x_1+30x_2+100x_3\\ \text{s.t.}\quad &13x_1+9x_2+21x_3\ge74 &\text{ (Protein)}\\ &10x_1+8x_2+12x_3\ge55 &\text{ (Fibre)}\\ &4x_1+3x_2+4x_3\ge21 &\text{ (Iron)}\\ &x_1+x_2+x_3=6 &\text{ (Total)}\\ &x_1,x_2,x_3\ge0. \end{aligned}\]

The objective is the cost in cents. The first three constraints are nutrient minimums. The equality fixes the total amount of food.

Step 1: Dual Variables and Signs

There are four dual variables: one per constraint row, excluding variable nonnegativity.

Primal constraintDual variable
Protein $\ge74$$u\ge0$
Fibre $\ge55$$v\ge0$
Iron $\ge21$$w\ge0$
Total portions $=6$$z$ free

This primal minimizes: $\ge$ rows give nonnegative dual variables, while an equality gives a free variable.

Step 2: Dual Constraints

Three primal variables give three dual constraints, one per food. All three variables are nonnegative, so the dual constraints are $\le$.

Read the food's nutrient column, then include its coefficient $1$ from the total-food equality:

FoodDual constraint
Oats$13u+10v+4w+z\le60$
Lentils$9u+8v+3w+z\le30$
Almonds$21u+12v+4w+z\le100$

Step 3: Objective and Right-Hand Sides

The primal minimizes, so the dual maximizes.

The nutrient minimums and total portions become the dual objective coefficients:

\[\max\quad74u+55v+21w+6z.\]

The three food prices become the right-hand sides of the dual constraints: $60$, $30$, and $100$ cents.

The Diet Primal and Dual

Primal

\[\begin{aligned} \min\quad &60x_1+30x_2+100x_3\\ \text{s.t.}\quad &13x_1+9x_2+21x_3\ge74,\\ &10x_1+8x_2+12x_3\ge55,\\ &4x_1+3x_2+4x_3\ge21,\\ &x_1+x_2+x_3=6,\\ &x_1,x_2,x_3\ge0. \end{aligned}\]

Dual

\[\begin{aligned} \max\quad &74u+55v+21w+6z\\ \text{s.t.}\quad &13u+10v+4w+z\le60,\\ &9u+8v+3w+z\le30,\\ &21u+12v+4w+z\le100,\\ &u,v,w\ge0,\quad z\text{ free}. \end{aligned}\]

Interpreting the Diet Dual: Pricing Nutrients

The dual variables associated to nutrients can be interpreted as the price of that nutrient, i.e. how much it costs to feed someone one gram of protein, fibre, etc.

Imagine that instead of buying food to give people nutrients, we could just buy nutrients directly; the constraints say that the value of each food is at least the total value of its component nutrients.

The value of $z$ represents the marginal cost of weight; this value can be positive or negative, depending on whether taking up more weight helps meet the requirement or makes the requirement harder to meet.

Interpreting the Diet Dual: Pricing Nutrients

The optimal solutions to this primal/dual pair are

\[x^*=(2,3,1),\]

\[(u^*,v^*,w^*,z^*)=(5,0,10,-45).\]

Both have objective value 310 cents:\[60(2)+30(3)+100(1)=310={\color{#1f6fd6}{74(5)+55(0)+21(10)+6(-45)}}.\]

Protein is worth 5 cents per gram and iron 10 cents per mg. Fibre is worth nothing, because the diet has 56 g, more than the 55 g required. Weight costs 45 cents per 100 g; if we had more room for food, we could use more of the cheaper food items.

Open the demo in its own tab

Strong Duality

Theorem (Strong duality). If a linear program is feasible and its objective is bounded in the direction of optimization, then its dual also has an optimal solution, and their optimal objective values are equal.

For a primal and its dual, there are optimal solutions $x^*,y^*$ with

\[\boxed{c^{\intercal}x^*=b^{\intercal}y^*.}\]

Weak duality says every primal value is at most every dual value. Strong duality says that when a finite optimum exists, the best values meet.

Strong Duality: Start with Simplex

Assume the primal LP has an optimal solution. Work in SEF, with redundant equations removed:

\[\max\{c^{\intercal}x:Ax=b,\ x\ge0\}.\]

Run the simplex method with a terminating pivot rule, such as Bland's rule. Since an optimum exists, simplex terminates at an optimal basic feasible solution $x^*$, with some basis $B$.

Put the LP into canonical form for this final basis $B$. Write its parameters as $c',A',b'$ and let $v=c^{\intercal}x^*$.

The Final Canonical Form

Original primal

\[\begin{aligned} \max\quad &c^{\intercal}x\\ \text{s.t.}\quad &Ax=b,\\ &x\ge0. \end{aligned}\]

Canonical primal

\[\begin{aligned} \max\quad &(c')^{\intercal}x+v\\ \text{s.t.}\quad &A'x=b',\\ &x\ge0. \end{aligned}\]

At the final simplex basis, $A'_B=I$, $b'\ge0$, $c'_B=0$, and $c'\le0$: there is no entering variable that can improve the objective.

The basic solution has $x_B^*=b'$ and $x_N^*=0$, so $(c')^{\intercal}x^*=0$. Its objective value is $v$.

Strong Duality in Canonical Form

Canonical primal

\[\begin{aligned} \max\quad &(c')^{\intercal}x+v\\ \text{s.t.}\quad &A'x=b',\\ &x\ge0. \end{aligned}\]

Its dual

\[\begin{aligned} \min\quad &(b')^{\intercal}y+v\\ \text{s.t.}\quad &(A')^{\intercal}y\ge c',\\ &y\text{ free}. \end{aligned}\]

Because $c'\le0$, $y=0$ is dual feasible. Its objective value is $v$, exactly the value of $x^*$.

By weak duality, both are optimal. We have proved strong duality for the canonical LP and its dual.

The constant $v$ is carried into the dual objective; it does not change the dual constraints.

Four LPs: What Have We Shown?

Original primal

\[\begin{aligned}\max\quad &c^{\intercal}x\\&Ax=b,\quad x\ge0\end{aligned}\]

$\longleftrightarrow$
dual

Original dual

\[\begin{aligned}\min\quad &b^{\intercal}y\\&A^{\intercal}y\ge c,\quad y\text{ free}\end{aligned}\]

$\Updownarrow$ equivalent

$\Updownarrow$ equivalent?

Canonical primal

\[\begin{aligned}\max\quad &(c')^{\intercal}x+v\\&A'x=b',\quad x\ge0\end{aligned}\]

$\longleftrightarrow$
strong duality

Canonical dual

\[\begin{aligned}\min\quad &(b')^{\intercal}y+v\\&(A')^{\intercal}y\ge c',\quad y\text{ free}\end{aligned}\]

While the original LP is ‘equivalent’ to its canonical form, it is not clear that the canonical dual is equivalent in the same way to the original dual.

Note that while $y = 0$ is an optimal solution to the canonical dual, it is not usually feasible for the original dual.

Completing the Proof of Strong Duality

How does the original LP differ from the canonical form LP?

Original primal

\[\begin{aligned}\max\quad &c^{\intercal}x\\&Ax=b,\quad x\ge0\end{aligned}\]

Canonical primal

\[\begin{aligned}\max\quad &(c')^{\intercal}x+v\\&A'x=b',\quad x\ge0\end{aligned}\]

They differ by row operations on $A$ and changing $c$ by an element of the row space of $A$.

That is, we have that $A' = EA$ and $b' = Eb$ for some invertible matrix $E$, and $c' = c - A^{\intercal}w$ and $v = b^{\intercal}w$ for some vector $w$.

Transformations of LPs and Duality

How does rewriting the equations in the primal affect the dual?

Original primal

\[\begin{aligned}\max\quad &c^{\intercal}x\\&Ax=b,\quad x\ge0\end{aligned}\]

Original dual

\[\begin{aligned}\min\quad &b^{\intercal}y\\&A^{\intercal}y\ge c,\quad y\text{ free}\end{aligned}\]

Transformed primal

\[\begin{aligned}\max\quad &c^{\intercal}x\\&EAx=Eb,\quad x\ge0\end{aligned}\]

Transformed dual

\[\begin{aligned}\min\quad &(Eb)^{\intercal}y\\&(EA)^{\intercal}y\ge c,\quad y\text{ free}\end{aligned}\]

The primals are ‘obviously equivalent’ because the feasible sets for both are identical, but this is less clear for the dual.

Transformations of LPs and Duality

How does rewriting the equations in the primal affect the dual?

Original primal

\[\begin{aligned}\max\quad &c^{\intercal}x\\&Ax=b,\quad x\ge0\end{aligned}\]

Original dual

\[\begin{aligned}\min\quad &b^{\intercal}y\\&A^{\intercal}y\ge c,\quad y\text{ free}\end{aligned}\]

Transformed primal

\[\begin{aligned}\max\quad &c^{\intercal}x\\&EAx=Eb,\quad x\ge0\end{aligned}\]

Transformed dual

\[\begin{aligned}\min\quad &b^{\intercal}(E^{\intercal}y)\\&A^{\intercal}(E^{\intercal}y)\ge c,\quad y\text{ free}\end{aligned}\]

This says that if $y$ is a feasible point for the transformed dual, then $E^{\intercal}y$ is feasible for the original dual with the same objective!

Transformations of LPs and Duality

We can also look at what happens when we modify the objective.

Original primal

\[\begin{aligned}\max\quad &c^{\intercal}x\\&Ax=b,\quad x\ge0\end{aligned}\]

Original dual

\[\begin{aligned}\min\quad &b^{\intercal}y\\&A^{\intercal}y\ge c,\quad y\text{ free}\end{aligned}\]

Transformed primal

\[\begin{aligned}\max\quad &c^{\intercal}x - w^{\intercal}(Ax - b)\\&Ax=b,\quad x\ge0\end{aligned}\]

Transformed dual

\[\begin{aligned}\min\quad &b^{\intercal}y + w^{\intercal}b\\&A^{\intercal}y\ge (c - A^{\intercal}w),\quad y\text{ free}\end{aligned}\]

Now, what do we need to do to $y$ to transform an original dual solution to a transformed dual solution?

Transformations of LPs and Duality

We can also look at what happens when we modify the objective.

Original primal

\[\begin{aligned}\max\quad &c^{\intercal}x\\&Ax=b,\quad x\ge0\end{aligned}\]

Original dual

\[\begin{aligned}\min\quad &b^{\intercal}y\\&A^{\intercal}y\ge c,\quad y\text{ free}\end{aligned}\]

Transformed primal

\[\begin{aligned}\max\quad &c^{\intercal}x - w^{\intercal}(Ax - b)\\&Ax=b,\quad x\ge0\end{aligned}\]

Transformed dual

\[\begin{aligned}\min\quad &b^{\intercal}(y + w)\\&A^{\intercal}(y + w)\ge c,\quad y\text{ free}\end{aligned}\]

So, if $y$ is an optimal solution to the transformed dual, then $y + w$ will be an optimal solution to the original dual.

Putting It Together

Original primal

\[\begin{aligned}\max\quad &c^{\intercal}x\\&Ax=b,\quad x\ge0\end{aligned}\]

$\longleftrightarrow$
dual

Original dual

\[\begin{aligned}\min\quad &b^{\intercal}y\\&A^{\intercal}y\ge c,\quad y\text{ free}\end{aligned}\]

$\Updownarrow$ equivalent

$\Updownarrow$ equivalent

Canonical primal

\[\begin{aligned}\max\quad &(c')^{\intercal}x+v\\&A'x=b',\quad x\ge0\end{aligned}\]

$\longleftrightarrow$
strong duality

Canonical dual

\[\begin{aligned}\min\quad &(b')^{\intercal}y+v\\&(A')^{\intercal}y\ge c',\quad y\text{ free}\end{aligned}\]

We have seen that the transformations taking the primal to its canonical form also can be applied to the dual.

The homework will go into more details about the exact calculations.