Duality for sets and functions

Chapter 3. Duality for sets and functions Chapter 3. Duality for sets and functions tvnguyen (University of Science) Convex Optimization 46 / 108 Chapter 3. Duality for sets and functions Dual representation of convex sets Several basic geometrical objects in IR n can be described by using linear forms. For example A closed hyperplane H can be written H = {x ∈ IR n | p, x = α} for some p ∈ IR n , p = 0, and α ∈ IR. Similarly, a closed half-space H can be written H = {x ∈ IR n | p, x ≤ α} We will show that arbitrary closed convex sets in IR n can be described by using only linear forms. This is what we call a dual representation. This theory is based on the Hahn-Banach theorem. tvnguyen (University of Science) Convex Optimization 47 / 108 Chapter 3. Duality for sets and functions Closest point theorem A well-known geometric fact is that, given a closed convex set A and a point x ∈ A, there exists a unique point y ∈ A¸ with minimum distance from x. Theorem. (Closest Point Theorem) Let A be a nonempty, closed convex set in IR n and x ∈ A. Then, there exists a unique point y ∈ A with minimum distance from x. Furthermore, y is the minimizing point, or closest point to x, if and only if x − y, z − y ≤ 0 for all z ∈ A. A z yx tvnguyen (University of Science) Convex Optimization 48 / 108 Chapter 3. Duality for sets and functions Separation of convex sets Almost all optimality conditions and duality relationships use some sort of separation or support of convex sets. Definition. (Separation of Sets) Let S1 and S2 be nonempty sets in IR n . A hyperplane H = {x|p, x = α} separates S1 and S2 if p, x ≥ α for each x ∈ S1 and p, x ≤ α for each x ∈ S2. If, in addition, p, x ≥ α + ε for each x ∈ S1 and p, x ≤ α for each x ∈ S2, where ε is a positive scalar, then the hyperplane H is said to strongly separate the sets S1 and S2. Notice that strong separation implies separation of sets. xxxxxxxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxxxxxx Separation Strong separation S 1 S 1 S 2 S 2 H H tvnguyen (University of Science) Convex Optimization 49 / 108 Chapter 3. Duality for sets and functions The following is the most fundamental separation theorem. Theorem. (Separation Theorem) Let A be a nonempty closed convex set in IR n and x ∈ A. Then, there exists a nonzero vector p and a scalar α such that p, x > α and p, x ≤ α for each z ∈ A. A z yx Here p = x − y. tvnguyen (University of Science) Convex Optimization 50 / 108 Chapter 3. Duality for sets and functions A direct consequence Proposition. Let A be a nonempty closed convex set in IR n . Then A is equal to the intersection of all closed half-spaces that contain it : A = ∩ { H p≤α | A ⊂ H p≤α } where H p≤α = {x ∈ IR n | p, x ≤ α} In such a representation of a closed convex set, it is natural to look for the simplest representation. Observe : (a) α  ≥ α and A ⊂ H p≤α ⇒ A ⊂ H p≤α  (b) fixing p = 0 and making α vary gives rise to parallel hyperplanes tvnguyen (University of Science) Convex Optimization 51 / 108 Chapter 3. Duality for sets and functions Support function Question. For a given p ∈ IR n ,p = 0, such that A ⊂ H p≤α for some α ∈ IR, what is the intersection of all the parallel half-spaces containing A. Proposition. For a given p ∈ IR n ,p = 0, such that A ⊂ H p≤α for some α ∈ IR, the intersection of all the parallel half-spaces containing A is the closed half-space H p≤σ A (p) , where σ A (p) = sup {p, x | x ∈ A}. Definition. Let A be a nonempty subset of IR n . The support function of A is defined by σ A : IR n → IR ∪ {+∞} σ A (p) = sup x∈A < p, x > tvnguyen (University of Science) Convex Optimization 52 / 108 Chapter 3. Duality for sets and functions Support function As a direct consequence of the definition we obtain the dual representation of a closed convex set. Proposition. Let A be a closed convex set in IR n . Then A is completely determined by its support function, i.e., A = {x| p, x ≤ σ A (p), ∀p ∈ IR n } Here are some properties of the support function Proposition. The support function σ A : IR n → IR ∪ {+∞} of a closed convex nonempty subset A is a function which is proper, closed, convex, and positively homogeneous of degree 1. Its epigraph is a closed convex cone in IR n × IR. Furthermore, dom σ A = IR n when A is bounded. tvnguyen (University of Science) Convex Optimization 53 / 108 Chapter 3. Duality for sets and functions Illustration A p p tvnguyen (University of Science) Convex Optimization 54 / 108 Chapter 3. Duality for sets and functions Supporting hyperplane. To have a sharper view of the dual generation of closed convex sets, it is interesting to introduce the notion of supporting hyperplane. This is related to the question : Question : In the definition of σ A (p) = sup x∈A < p, x >, is the supremum attained ? Definition. Let p ∈ IR, p = 0. The hyperplane H = {x ∈ IR n | p, x = σ A (p)} is a supporting hyperplane of A at x ∗ ∈ A if the closed half-space H p≤σ A (p) contains A and H = {x ∈ IR n | p, x = σ A (p)} intersects A at x ∗ Note that the intersection of H with A may contain some other points tvnguyen (University of Science) Convex Optimization 55 / 108 [...]... in IRn and x ∈ intS Then ¯ there is a nonzero vector p such that p, (x − x ) ≤ 0 for each x ∈ clS ¯ tvnguyen (University of Science) Convex Optimization 58 / 108 Chapter 3 Duality for sets and functions Separation of Disjoint Convex Sets If two convex sets are disjoint, then they can be separated by a hyperplane Theorem (Separation of Two Disjoint Convex Sets) Let S1 and S2 be nonempty convex sets in... 3 Duality for sets and functions Separation of Nondisjoint Convex Sets The previous result (Separation of Two Disjoint Convex Sets) holds true even if the two sets have some points in common, as long as their interiors are disjoint Corollary Let S1 and S2 be nonempty convex sets in IRn Suppose that int S2 is not empty and that S1 ∩ int S2 is empty Then, there exists a hyperplane that separates S1 and. ..Chapter 3 Duality for sets and functions Illustration A A (a) (b) (a) Several supporting hyperplanes to A ⊂ IR2 (b) Two supporting hyperplanes to A ⊂ IR3 with int A = ∅ tvnguyen (University of Science) Convex Optimization 56 / 108 Chapter 3 Duality for sets and functions Supporting hyperplane An equivalent definition Definition (Supporting Hyperplane at a Boundary Point) Let S be a nonempty set in IRn , and. .. is well defined and is proper, closed and convex Proposition The following properties hold : (i) If f is proper and convex then f ∗∗ ≤ f (ii) If f is proper, closed and convex then f ∗∗ = f tvnguyen (University of Science) Convex Optimization 62 / 108 Chapter 3 Duality for sets and functions Infimal convolution, support function and conjugacy Proposition Let f1 , , fm be proper convex functions Then... tvnguyen (University of Science) Convex Optimization 64 / 108 Chapter 3 Duality for sets and functions Illustration and main property Proposition (J.J Moreau) Let K be a closed convex cone For the three elements x, x1 , x2 in IRn , the properties below are equivalent : (i) x = x1 + x2 with x1 ∈ K , x2 ∈ K 0 and x1 , x2 = 0 ; (ii) x1 = PK (x) and x2 = PK (x) tvnguyen (University of Science) Convex Optimization... (f1∗ ⊕ · · · ⊕ fm ) Furthermore, if the sets ri dom fi , i = 1, , m, have a point in common, the closure operation can be omitted from the second formula Proposition The indicator function and the support function of a closed convex set are conjugate to each other tvnguyen (University of Science) Convex Optimization 63 / 108 Chapter 3 Duality for sets and functions Polar of a convex set It is well-known... Convex Optimization x 57 / 108 Chapter 3 Duality for sets and functions Supporting hyperplane A convex set has a supporting hyperplane at each boundary point Theorem (Supporting Hyperplane) Let S be a nonempty convex set in IRn , and let x ∈ ∂S Then there exists a hyperplane that supports S ¯ at x ; that is, there exists a nonzero vector p such that p, (x − x ) ≤ 0 ¯ ¯ for each x ∈ clS As a corollary, we... xxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxx S S tvnguyen (University of Science) Convex Optimization H 60 / 108 Chapter 3 Duality for sets and functions Conjugate of a convex function Motivation Let f be proper, closed and convex We know that f is the pointwise supremum of the collection of all affine functions h such that h ≤ f Consider F ∗ = {(x ∗ , µ∗ ) ∈ IRn × IR | h(x) = x, x ∗ − µ∗ ≤ f (x) Then h(x) ≤ f... Science) Convex Optimization 61 / 108 Chapter 3 Duality for sets and functions Conjugate of a convex function Definition Let f : IRn → IR ∪ {+∞} be proper and lower bounded by an affine function Then the function f ∗ : IRn → IR ∪ {+∞} defined by f ∗ (x) = sup {< x, y > −f (y )} y ∈R n is called the conjugate function of f Proposition Let f : IRn → IR ∪ {+∞} be proper and lower bounded by an affine function Then... decomposed in a unique way as x1 + x2 with x1 ∈ V and x2 ∈ V ⊥ More precisely x = PV (x) + PV ⊥ (x) Our aim is to replace V by a closed convex cone So what is V ⊥ ? Definition Let K be a nonempty convex set The polar of K is K 0 = {x ∗ ∈ IRn | x ∗ , x ≤ 1 for all x ∈ K } Proposition If K is a convex cone, then the polar of K is the cone K 0 = {x ∗ | x ∗ , x ≤ 0 for all x ∈ K } Furthermore, K 00 is the closure . 3. Duality for sets and functions Chapter 3. Duality for sets and functions tvnguyen (University of Science) Convex Optimization 46 / 108 Chapter 3. Duality. 3. Duality for sets and functions Illustration A p p tvnguyen (University of Science) Convex Optimization 54 / 108 Chapter 3. Duality for sets and functions