PDF version of these slides (demo: this PDF version is experimental and may contain errors or differ from the slides)
Primal
\[\begin{aligned} \max\quad &c^{\intercal}x\\ \text{s.t.}\quad &Ax=b,\\ &x\ge0. \end{aligned}\]
Dual
\[\begin{aligned} \min\quad &b^{\intercal}y\\ \text{s.t.}\quad &A^{\intercal}y\ge c,\\ &y\text{ free}. \end{aligned}\]
If $A$ has $m$ rows and $n$ columns, the dual has $m$ variables and $n$ constraints.
If $x$ is primal feasible and $y$ is dual feasible, then
\[\boxed{c^{\intercal}x\le b^{\intercal}y.}\]
We can see this by looking at $y^{\intercal}Ax$, and expanding this expression in two different ways using the constraints.
$(y^{\intercal}A)x =$ $(A^{\intercal}y)^{\intercal}x =$ $\underbrace{c^{\intercal}x}_{\color{#1f6fd6}{A^{\intercal}y\ge c},\;x\ge0}\le$ $y^{\intercal}Ax =$ $y^{\intercal}(Ax)$ $\underbrace{b^{\intercal}}_{Ax=b}y$ .
If an LP is not in SEF, we can take its dual by first giving an equivalent LP in SEF, and then taking the dual of the SEF LP.
\[\begin{aligned} \max\quad &x_1+2x_2\\ \text{s.t.}\quad &x_1 - x_2 = 1,\\ &x_1 + x_2 \ge 3,\\ &x_1 \ge 0,\\ &x_2\text{ free}. \end{aligned}\]
If an LP is not in SEF, we can take its dual by first giving an equivalent LP in SEF, and then taking the dual of the SEF LP.
\[\begin{aligned} \max\quad &x_1+2x_2^+ - 2 x_2^-\\ \text{s.t.}\quad &x_1 - x_2^+ + x_2^- = 1,\\ &x_1 + x_2^+-x_2^- - s_1 = 3,\\ &x_1, x_2^+, x_2^-, s_1 \ge 0 \end{aligned}\]
If an LP is not in SEF, we can take its dual by first giving an equivalent LP in SEF, and then taking the dual of the SEF LP.
\[\begin{aligned} \max\quad &\begin{pmatrix}1\\2\\-2\\0\end{pmatrix}^{\intercal}\begin{pmatrix}x_1\\x_2^+\\x_2^-\\s_1\end{pmatrix}\\ \text{s.t.}\quad & \begin{pmatrix} 1 & -1 & 1 & 0\\ 1 & 1 & -1 & -1\end{pmatrix}\begin{pmatrix}x_1\\x_2^+\\x_2^-\\s_1\end{pmatrix} = \begin{pmatrix}1\\3\end{pmatrix}\\ &x_1, x_2^+, x_2^-, s_1 \ge 0 \end{aligned}\]
How many dual variables will there be? How many dual constraints?
If an LP is not in SEF, we can take its dual by first giving an equivalent LP in SEF, and then taking the dual of the SEF LP.
\[\begin{aligned} \max\quad &\begin{pmatrix}1\\2\\-2\\0\end{pmatrix}^{\intercal}\begin{pmatrix}x_1\\x_2^+\\x_2^-\\s_1\end{pmatrix}\\ \text{s.t.}\quad & \begin{pmatrix} 1 & -1 & 1 & 0\\ 1 & 1 & -1 & -1\end{pmatrix}\begin{pmatrix}x_1\\x_2^+\\x_2^-\\s_1\end{pmatrix} = \begin{pmatrix}1\\3\end{pmatrix}\\ &x_1, x_2^+, x_2^-, s_1 \ge 0 \end{aligned}\]
\[\begin{aligned} \min\quad &\begin{pmatrix}1\\3\end{pmatrix}^{\intercal}\begin{pmatrix}y_1\\y_2\end{pmatrix}\\ \text{s.t.}\quad & \begin{pmatrix} 1 & 1 \\ -1 & 1 \\ 1 & -1 \\ 0 & -1 \end{pmatrix} \begin{pmatrix}y_1\\y_2\end{pmatrix} \ge \begin{pmatrix}1\\2\\-2\\0\end{pmatrix} \end{aligned}\]
If an LP is not in SEF, we can take its dual by first giving an equivalent LP in SEF, and then taking the dual of the SEF LP.
\[\begin{aligned} \min\quad &\begin{pmatrix}1\\3\end{pmatrix}^{\intercal}\begin{pmatrix}y_1\\y_2\end{pmatrix}\\ \text{s.t.}\quad & \begin{pmatrix} 1 & 1 \\ -1 & 1 \\ 1 & -1 \\ 0 & -1 \end{pmatrix} \begin{pmatrix}y_1\\y_2\end{pmatrix} \ge \begin{pmatrix}1\\2\\-2\\0\end{pmatrix} \end{aligned}\]
We can simplify the dual constraints though; note that the last row amounts to saying that $y_2 \le 0$ and the second and third inequalities amount to $y_1 - y_2 = -2$.
If an LP is not in SEF, we can take its dual by first giving an equivalent LP in SEF, and then taking the dual of the SEF LP.
\[\begin{aligned} \max\quad &x_1+2x_2\\ \text{s.t.}\quad &x_1 - x_2 = 1,\\ &x_1 + x_2 \ge 3,\\ &x_1 \ge 0,\\ &x_2\text{ free}. \end{aligned}\]
\[\begin{aligned} \min\quad &y_1 + 3y_2\\ \text{s.t.}\quad & y_1 + y_2 \ge 1\\ & y_1 - y_2 = -2 \\ & y_1\text{ free}\\ & y_2 \le 0. \end{aligned}\]
Note that the dual is infeasible; what does this imply about the primal?
Note that we have the following correspondence between variables/constraints in the primal/dual.
\[\begin{aligned} \max\quad &x_1+2x_2\\ \text{s.t.}\quad &{\color{#1f6fd6}{x_1-x_2=1}},\\ &{\color{#cb4b16}{x_1+x_2\ge3}},\\ &{\color{#708400}{x_1\ge0}},\\ &{\color{#8b4bad}{x_2\text{ free}}}. \end{aligned}\]
\[\begin{aligned} \min\quad &y_1+3y_2\\ \text{s.t.}\quad &{\color{#708400}{y_1+y_2\ge1}}\\ &{\color{#8b4bad}{y_1-y_2=-2}}\\ &{\color{#1f6fd6}{y_1\text{ free}}}\\ &{\color{#cb4b16}{y_2\le0}}. \end{aligned}\]
General Rule: For each variable in the primal, there should be a constraint in the dual. For each constraint in the primal other than a variable sign restriction, there should be a variable in the dual. Nonnegative variables in the primal correspond to inequality constraints. Free variables correspond to equality constraints, and vice versa.
For a maximization primal and a minimization dual:
| Primal | Dual | |
|---|---|---|
| $\le$ constraint | $\leftrightarrow$ | Nonnegative variable |
| Equality constraint | $\leftrightarrow$ | Free variable |
| $\ge$ constraint | $\leftrightarrow$ | Nonpositive variable |
| Nonnegative variable | $\leftrightarrow$ | $\ge$ constraint |
| Free variable | $\leftrightarrow$ | Equality constraint |
| Nonpositive variable | $\leftrightarrow$ | $\le$ constraint |
\[\begin{aligned} \max\quad &4x_1+2x_2-2x_3\\ \text{s.t.}\quad &2x_1+x_2+x_3\le4,\\ &x_1+3x_2-x_3=7,\\ &-x_1+2x_2+x_3\ge3,\\ &x_1\text{ free},\quad x_2,x_3\ge0. \end{aligned}\]
There are three constraints (apart from variable sign restrictions). So the dual has three variables: $y_1,y_2,y_3$.
For now, don't count $x_2,x_3\ge0$ as additional constraints: these will determine the directions of the dual constraints.
The primal is a maximization problem. Each constraint row determines the sign of its dual variable.
| Primal row | Dual variable |
|---|---|
| $2x_1+x_2+x_3\le4$ | $y_1\ge0$ |
| $x_1+3x_2-x_3=7$ | $y_2$ is free |
| $-x_1+2x_2+x_3\ge3$ | $y_3\le0$ |
So $y_1$ is nonnegative, $y_2$ has no sign restriction, and $y_3$ is nonpositive.
For each primal variable, find the dual constraint. The coefficients in that constraint are the coefficients of that primal variable. The sense of the constraint is determined by the sign of the variable.
\[A=\begin{pmatrix}2&1&1\\1&3&-1\\-1&2&1\end{pmatrix}.\]
| Primal variable / column | Dual left side / relation |
|---|---|
| $x_1$ free: $(2,1,-1)$ | $2y_1+y_2-y_3$; equality |
| $x_2\ge0$: $(1,3,2)$ | $y_1+3y_2+2y_3$; $\ge$ |
| $x_3\ge0$: $(1,-1,1)$ | $y_1-y_2+y_3$; $\ge$ |
The primal maximizes, so the dual minimizes.
The primal right-hand sides $(4,7,3)$ become the dual objective coefficients:
\[\min\quad 4y_1+7y_2+3y_3.\]
The primal objective coefficients $(4,2,-2)$ become the dual constraint right-hand sides, in variable order:
\[\begin{aligned} 2y_1+y_2-y_3&=4,\\ y_1+3y_2+2y_3&\ge2,\\ y_1-y_2+y_3&\ge-2. \end{aligned}\]
Primal
\[\begin{aligned} \max\quad &4x_1+2x_2-2x_3\\ \text{s.t.}\quad &2x_1+x_2+x_3\le4,\\ &x_1+3x_2-x_3=7,\\ &-x_1+2x_2+x_3\ge3,\\ &x_1\text{ free},\quad x_2,x_3\ge0. \end{aligned}\]
Dual
\[\begin{aligned} \min\quad &4y_1+7y_2+3y_3\\ \text{s.t.}\quad &2y_1+y_2-y_3=4,\\ &y_1+3y_2+2y_3\ge2,\\ &y_1-y_2+y_3\ge-2,\\ &y_1\ge0,\quad y_2\text{ free},\quad y_3\le0. \end{aligned}\]
Check: $x=(1,2,0)$ and $y=(1,1,-1)$ are feasible and both have objective value $8$. Thus both are optimal.
Primal
\[\begin{aligned} \max\quad &4x_1+5x_2\\ \text{s.t.}\quad &2x_1+x_2+x_3=8,\\ &x_1+2x_2+x_4=10,\\ &x_1,x_2,x_3,x_4\ge0. \end{aligned}\]
Dual
\[\begin{aligned} \min\quad &8y_1+10y_2\\ \text{s.t.}\quad &2y_1+y_2\ge4,\\ &y_1+2y_2\ge5,\\ &y_1,y_2\ge0. \end{aligned}\]
Primal
\[\begin{aligned} \max\quad &4x_1+5x_2\\ \text{s.t.}\quad &2x_1+x_2+x_3=8,\\ &x_1+2x_2+x_4=10,\\ &x_1,x_2,x_3,x_4\ge0. \end{aligned}\]
Dual
\[\begin{aligned} \min\quad &8y_1+10y_2\\ \text{s.t.}\quad &2y_1+y_2\ge4,\\ &y_1+2y_2\ge5,\\ &y_1,y_2\ge0. \end{aligned}\]
Optimal solutions are $x^*=(2,4,0,0)$ and $y^*=(1,2)$.
Both are feasible, and their objective values agree:
\[4(2)+5(4)=28=8(1)+10(2).\]
By weak duality, both solutions are optimal.
Choose the cheapest mixture of three foods meeting three nutrient minimums. We also want exactly 600 g of food.
Use these illustrative values per 100 g portion, with prices in whole cents:
| Oats | Lentils | Almonds | Minimum | |
|---|---|---|---|---|
| Protein (g) | 13 | 9 | 21 | 74 |
| Fibre (g) | 10 | 8 | 12 | 55 |
| Iron (mg) | 4 | 3 | 4 | 21 |
| Price (cents) | 60 | 30 | 100 |
Let $x_1,x_2,x_3$ be the numbers of 100 g portions of oats, lentils, and almonds. Portions can be fractional.
\[\begin{aligned} \min\quad &60x_1+30x_2+100x_3\\ \text{s.t.}\quad &13x_1+9x_2+21x_3\ge74 &\text{ (Protein)}\\ &10x_1+8x_2+12x_3\ge55 &\text{ (Fibre)}\\ &4x_1+3x_2+4x_3\ge21 &\text{ (Iron)}\\ &x_1+x_2+x_3=6 &\text{ (Total)}\\ &x_1,x_2,x_3\ge0. \end{aligned}\]
The objective is the cost in cents. The first three constraints are nutrient minimums. The equality fixes the total amount of food.
There are four dual variables: one per constraint row, excluding variable nonnegativity.
| Primal constraint | Dual variable |
|---|---|
| Protein $\ge74$ | $u\ge0$ |
| Fibre $\ge55$ | $v\ge0$ |
| Iron $\ge21$ | $w\ge0$ |
| Total portions $=6$ | $z$ free |
This primal minimizes: $\ge$ rows give nonnegative dual variables, while an equality gives a free variable.
Three primal variables give three dual constraints, one per food. All three variables are nonnegative, so the dual constraints are $\le$.
Read the food's nutrient column, then include its coefficient $1$ from the total-food equality:
| Food | Dual constraint |
|---|---|
| Oats | $13u+10v+4w+z\le60$ |
| Lentils | $9u+8v+3w+z\le30$ |
| Almonds | $21u+12v+4w+z\le100$ |
The primal minimizes, so the dual maximizes.
The nutrient minimums and total portions become the dual objective coefficients:
\[\max\quad74u+55v+21w+6z.\]
The three food prices become the right-hand sides of the dual constraints: $60$, $30$, and $100$ cents.
Primal
\[\begin{aligned} \min\quad &60x_1+30x_2+100x_3\\ \text{s.t.}\quad &13x_1+9x_2+21x_3\ge74,\\ &10x_1+8x_2+12x_3\ge55,\\ &4x_1+3x_2+4x_3\ge21,\\ &x_1+x_2+x_3=6,\\ &x_1,x_2,x_3\ge0. \end{aligned}\]
Dual
\[\begin{aligned} \max\quad &74u+55v+21w+6z\\ \text{s.t.}\quad &13u+10v+4w+z\le60,\\ &9u+8v+3w+z\le30,\\ &21u+12v+4w+z\le100,\\ &u,v,w\ge0,\quad z\text{ free}. \end{aligned}\]
The dual variables associated to nutrients can be interpreted as the price of that nutrient, i.e. how much it costs to feed someone one gram of protein, fibre, etc.
Imagine that instead of buying food to give people nutrients, we could just buy nutrients directly; the constraints say that the value of each food is at least the total value of its component nutrients.
The value of $z$ represents the marginal cost of weight; this value can be positive or negative, depending on whether taking up more weight helps meet the requirement or makes the requirement harder to meet.
The optimal solutions to this primal/dual pair are
\[x^*=(2,3,1),\]
\[(u^*,v^*,w^*,z^*)=(5,0,10,-45).\]
Both have objective value 310 cents:\[60(2)+30(3)+100(1)=310={\color{#1f6fd6}{74(5)+55(0)+21(10)+6(-45)}}.\]
Protein is worth 5 cents per gram and iron 10 cents per mg. Fibre is worth nothing, because the diet has 56 g, more than the 55 g required. Weight costs 45 cents per 100 g; if we had more room for food, we could use more of the cheaper food items.
Theorem (Strong duality). If a linear program is feasible and its objective is bounded in the direction of optimization, then its dual also has an optimal solution, and their optimal objective values are equal.
For a primal and its dual, there are optimal solutions $x^*,y^*$ with
\[\boxed{c^{\intercal}x^*=b^{\intercal}y^*.}\]
Weak duality says every primal value is at most every dual value. Strong duality says that when a finite optimum exists, the best values meet.
Assume the primal LP has an optimal solution. Work in SEF, with redundant equations removed:
\[\max\{c^{\intercal}x:Ax=b,\ x\ge0\}.\]
Run the simplex method with a terminating pivot rule, such as Bland's rule. Since an optimum exists, simplex terminates at an optimal basic feasible solution $x^*$, with some basis $B$.
Put the LP into canonical form for this final basis $B$. Write its parameters as $c',A',b'$ and let $v=c^{\intercal}x^*$.
Original primal
\[\begin{aligned} \max\quad &c^{\intercal}x\\ \text{s.t.}\quad &Ax=b,\\ &x\ge0. \end{aligned}\]
Canonical primal
\[\begin{aligned} \max\quad &(c')^{\intercal}x+v\\ \text{s.t.}\quad &A'x=b',\\ &x\ge0. \end{aligned}\]
At the final simplex basis, $A'_B=I$, $b'\ge0$, $c'_B=0$, and $c'\le0$: there is no entering variable that can improve the objective.
The basic solution has $x_B^*=b'$ and $x_N^*=0$, so $(c')^{\intercal}x^*=0$. Its objective value is $v$.
Canonical primal
\[\begin{aligned} \max\quad &(c')^{\intercal}x+v\\ \text{s.t.}\quad &A'x=b',\\ &x\ge0. \end{aligned}\]
Its dual
\[\begin{aligned} \min\quad &(b')^{\intercal}y+v\\ \text{s.t.}\quad &(A')^{\intercal}y\ge c',\\ &y\text{ free}. \end{aligned}\]
Because $c'\le0$, $y=0$ is dual feasible. Its objective value is $v$, exactly the value of $x^*$.
By weak duality, both are optimal. We have proved strong duality for the canonical LP and its dual.
The constant $v$ is carried into the dual objective; it does not change the dual constraints.
Original primal
\[\begin{aligned}\max\quad &c^{\intercal}x\\&Ax=b,\quad x\ge0\end{aligned}\]
$\longleftrightarrow$
dual
Original dual
\[\begin{aligned}\min\quad &b^{\intercal}y\\&A^{\intercal}y\ge c,\quad y\text{ free}\end{aligned}\]
$\Updownarrow$ equivalent
$\Updownarrow$ equivalent?
Canonical primal
\[\begin{aligned}\max\quad &(c')^{\intercal}x+v\\&A'x=b',\quad x\ge0\end{aligned}\]
$\longleftrightarrow$
strong duality
Canonical dual
\[\begin{aligned}\min\quad &(b')^{\intercal}y+v\\&(A')^{\intercal}y\ge c',\quad y\text{ free}\end{aligned}\]
While the original LP is ‘equivalent’ to its canonical form, it is not clear that the canonical dual is equivalent in the same way to the original dual.
Note that while $y = 0$ is an optimal solution to the canonical dual, it is not usually feasible for the original dual.
How does the original LP differ from the canonical form LP?
Original primal
\[\begin{aligned}\max\quad &c^{\intercal}x\\&Ax=b,\quad x\ge0\end{aligned}\]
Canonical primal
\[\begin{aligned}\max\quad &(c')^{\intercal}x+v\\&A'x=b',\quad x\ge0\end{aligned}\]
They differ by row operations on $A$ and changing $c$ by an element of the row space of $A$.
That is, we have that $A' = EA$ and $b' = Eb$ for some invertible matrix $E$, and $c' = c - A^{\intercal}w$ and $v = b^{\intercal}w$ for some vector $w$.
How does rewriting the equations in the primal affect the dual?
Original primal
\[\begin{aligned}\max\quad &c^{\intercal}x\\&Ax=b,\quad x\ge0\end{aligned}\]
Original dual
\[\begin{aligned}\min\quad &b^{\intercal}y\\&A^{\intercal}y\ge c,\quad y\text{ free}\end{aligned}\]
Transformed primal
\[\begin{aligned}\max\quad &c^{\intercal}x\\&EAx=Eb,\quad x\ge0\end{aligned}\]
Transformed dual
\[\begin{aligned}\min\quad &(Eb)^{\intercal}y\\&(EA)^{\intercal}y\ge c,\quad y\text{ free}\end{aligned}\]
The primals are ‘obviously equivalent’ because the feasible sets for both are identical, but this is less clear for the dual.
How does rewriting the equations in the primal affect the dual?
Original primal
\[\begin{aligned}\max\quad &c^{\intercal}x\\&Ax=b,\quad x\ge0\end{aligned}\]
Original dual
\[\begin{aligned}\min\quad &b^{\intercal}y\\&A^{\intercal}y\ge c,\quad y\text{ free}\end{aligned}\]
Transformed primal
\[\begin{aligned}\max\quad &c^{\intercal}x\\&EAx=Eb,\quad x\ge0\end{aligned}\]
Transformed dual
\[\begin{aligned}\min\quad &b^{\intercal}(E^{\intercal}y)\\&A^{\intercal}(E^{\intercal}y)\ge c,\quad y\text{ free}\end{aligned}\]
This says that if $y$ is a feasible point for the transformed dual, then $E^{\intercal}y$ is feasible for the original dual with the same objective!
We can also look at what happens when we modify the objective.
Original primal
\[\begin{aligned}\max\quad &c^{\intercal}x\\&Ax=b,\quad x\ge0\end{aligned}\]
Original dual
\[\begin{aligned}\min\quad &b^{\intercal}y\\&A^{\intercal}y\ge c,\quad y\text{ free}\end{aligned}\]
Transformed primal
\[\begin{aligned}\max\quad &c^{\intercal}x - w^{\intercal}(Ax - b)\\&Ax=b,\quad x\ge0\end{aligned}\]
Transformed dual
\[\begin{aligned}\min\quad &b^{\intercal}y + w^{\intercal}b\\&A^{\intercal}y\ge (c - A^{\intercal}w),\quad y\text{ free}\end{aligned}\]
Now, what do we need to do to $y$ to transform an original dual solution to a transformed dual solution?
We can also look at what happens when we modify the objective.
Original primal
\[\begin{aligned}\max\quad &c^{\intercal}x\\&Ax=b,\quad x\ge0\end{aligned}\]
Original dual
\[\begin{aligned}\min\quad &b^{\intercal}y\\&A^{\intercal}y\ge c,\quad y\text{ free}\end{aligned}\]
Transformed primal
\[\begin{aligned}\max\quad &c^{\intercal}x - w^{\intercal}(Ax - b)\\&Ax=b,\quad x\ge0\end{aligned}\]
Transformed dual
\[\begin{aligned}\min\quad &b^{\intercal}(y + w)\\&A^{\intercal}(y + w)\ge c,\quad y\text{ free}\end{aligned}\]
So, if $y$ is an optimal solution to the transformed dual, then $y + w$ will be an optimal solution to the original dual.
Original primal
\[\begin{aligned}\max\quad &c^{\intercal}x\\&Ax=b,\quad x\ge0\end{aligned}\]
$\longleftrightarrow$
dual
Original dual
\[\begin{aligned}\min\quad &b^{\intercal}y\\&A^{\intercal}y\ge c,\quad y\text{ free}\end{aligned}\]
$\Updownarrow$ equivalent
$\Updownarrow$ equivalent
Canonical primal
\[\begin{aligned}\max\quad &(c')^{\intercal}x+v\\&A'x=b',\quad x\ge0\end{aligned}\]
$\longleftrightarrow$
strong duality
Canonical dual
\[\begin{aligned}\min\quad &(b')^{\intercal}y+v\\&(A')^{\intercal}y\ge c',\quad y\text{ free}\end{aligned}\]
We have seen that the transformations taking the primal to its canonical form also can be applied to the dual.
The homework will go into more details about the exact calculations.