Công Nghệ Thông Tin, it, phầm mềm, website, web, mobile app, trí tuệ nhân tạo, blockchain, AI, machine learning - Khoa học tự nhiên - Công nghệ thông tin Projection onto nonnegative orthant, rectangular box and polyhedron Andersen Ang ECS, Uni. Southampton, UK andersen.angsoton.ac.uk Homepage angms.science Version: April 2, 2023 First draft: March 19, 2020 Content proj onto nonnegative orthant proj onto box alternating projection algorithm active set method interior point method dual proximal gradient method Projection onto nonnegative orthant ▶ The Euclidean projection of a given point y ∈ Rn onto a (non-empty compact) set S ⊆ Rn, denoted as projS (y), is a function Rn → Rn that output a point ˆx by solving ˆx = projS (y) = argmin x∈S ∥x − y∥2. Such optimization always has a unique solution, details here. ▶ Question: What if S is the nonnegative orthant? projS (y) = argmin x≥0 ∥x − y∥2, where x ≥ 0 means x is inside the nonnegative orthant S = x xi ≥ 0 ∀i . ▶ This problem has a trivial solution ˆx = y+, where ˆxi = yi+ with · + = max{ · , 0 }. 2 13 y+ = argmin x≥0 ∥x − y∥2 Among all the feasible value in the nonnegative orthant, 0 is the closest to negative value. Example ▶ The orange point y = 2 −2 is projected to the blue point 2 0 as the blue point is where the blue circle just touch the nonnegative orthant (violet region). ▶ Here the radius of the circle (=2) is the optimal cost value. 3 13 Projection onto a box A generalization of the nonnegativity is the box constraint PS (y) = argmin l≤x≤u ∥x − y∥2, with the close form solution PS (y)i = li yi ≤ li yi li < yi < ui ui yi ≥ ui An example li = 1, ui = 2 and a point at (5, 5). 4 13 Projection onto a polyhedron ▶ projnonnegative orthant and projbox are special cases of projpolyhedron Polyhedron S = n x Ax ≤ b o . where A ∈ Rm×n is full rank, and b ∈ Rm. ▶ Note that it does not matter Ax ≥ b or Ax ≤ b: we can absorb the negative sign into A or b. 5 13 Example 1 : m = n = 2 orange point y = 2, −2 blue point ˆx = 125, −65 Red region x + 2y ≥ 0 Blue region 2x + y ≥ 0 a1 2, 1 a2 1, 2 A = 2 1 1 2 b = 0 0 ▶ y fulfill Ax ≥ b for the first row of A, but not the second row. ▶ The optimal cost value = 4 5 , which is the distance between ˆx and y , and it is also the radius of the blue circle. ▶ We can see in this example that, proj to nonnegative orthant is simply the vector a1, a2 are rotated to the direction of the standard basis. 6 13 Example 2 : m = 3, n = 2 orange point y = 2, −2 blue point ˆx = 2, −1 Red region x + 2y ≥ 0 Blue region 2x + y ≥ 0 Green region x − y ≤ 3 A = 2 1 1 2 −1 1 b = 0 0 −3 ▶ The boundary of the feasible region is highlighted in yellow. 7 13 Solving projection onto polyhedron ▶ Unlike the projection onto nonnegative orthant or projection onto a box, the problem argmin Ax≤b ∥x − y∥2, has no simple close form solution in general. ▶ To solve it, we use an iterative optimization algorithms, such as 1. Alternating projection algorithm 2. Active set methods 3. Interior point methods 4. (Dual) Proximal gra...
Projection onto nonnegative orthant, rectangular box and polyhedron Andersen Ang Content proj onto nonnegative orthant ECS, Uni Southampton, UK proj onto box andersen.ang@soton.ac.uk alternating projection algorithm Homepage angms.science active set method interior point method Version: April 2, 2023 dual proximal gradient method First draft: March 19, 2020 Projection onto nonnegative orthant ▶ The Euclidean projection of a given point y ∈ Rn onto a (non-empty & compact) set S ⊆ Rn, denoted as projS(y), is a function Rn → Rn that output a point xˆ by solving xˆ = projS(y) = argmin ∥x − y∥2 x∈S Such optimization always has a unique solution, details here ▶ Question: What if S is the nonnegative orthant? projS(y) = argmin ∥x − y∥2, x≥0 where x ≥ 0 means x is inside the nonnegative orthant S = x | xi ≥ 0 ∀i ▶ This problem has a trivial solution xˆ = [y]+, where xˆi = [yi]+ with [ · ]+ = max{ · , 0 } 2 / 13 [y]+ = argmin ∥x − y∥2 x≥0 Among all the feasible value in the nonnegative orthant, 0 is the closest to negative value Example ▶ The orange point y = 2 is projected to the −2 blue point 2 as the blue point is where the 0 blue circle just touch the nonnegative orthant (violet region) ▶ Here the radius of the circle (=2) is the optimal cost value 3 / 13 Projection onto a box A generalization of the nonnegativity is the box constraint PS(y) = argmin ∥x − y∥2, l≤x≤u li yi ≤ li with the close form solution [PS(y)]i = yi li < yi < ui ui yi ≥ ui An example li = 1, ui = 2 and a point at (5, 5) 4 / 13 Projection onto a polyhedron ▶ projnonnegative orthant and projbox are special cases of projpolyhedron Polyhedron S = x | Ax ≤ b where A ∈ Rm×n is full rank, and b ∈ Rm ▶ Note that it does not matter Ax ≥ b or Ax ≤ b: we can absorb the negative sign into A or b 5 / 13 Example 1 : m = n = 2 orange point y = [2, −2] blue point xˆ = [12/5, −6/5] Red region x + 2y ≥ 0 Blue region 2x + y ≥ 0 a1 [2, 1] a2 [1, 2] A= 2 1 b= 0 12 0 ▶ y fulfill Ax ≥ b for the first row of A, but not the second row ▶ The optimal cost value = 45 , which is the distance between xˆ and y, and it is also the radius of the blue circle ▶ We can see in this example that, proj to nonnegative orthant is simply the vector a1, a2 are rotated to the direction of the standard basis 6 / 13 Example 2 : m = 3, n = 2 orange point y = [2, −2] blue point xˆ = [2, −1] Red region x + 2y ≥ 0 Blue region 2x + y ≥ 0 Green region x−y ≤ 3 2 1 0 A = 1 2 b= 0 −1 1 −3 ▶ The boundary of the feasible region is highlighted in yellow 7 / 13 Solving projection onto polyhedron ▶ Unlike the projection onto nonnegative orthant or projection onto a box, the problem argmin ∥x − y∥2, Ax≤b has no simple close form solution in general ▶ To solve it, we use an iterative optimization algorithms, such as 1 Alternating projection algorithm 2 Active set methods 3 Interior point methods 4 (Dual) Proximal gradient method 5 Douglas-Rachford splitting algorithm on the dual We talk very briefly on the first four approaches 8 / 13 Alternating projection algorithm ▶ The constraint Ax ≤ b describe a (non-empty) polyhedron as an intersection of m inequality constraint, each of the constraint is a problem pf projection onto halfspace argmin 1 ∥x − y∥22, ⟨a,x⟩≤b 2 with the close form solution as y ⟨a, x⟩ ≤ b ⟨a, x⟩ > b xˆ = ⟨a, x⟩ − b y − 2a ∥a∥2 ▶ Alternating projection algorithm: cycle through the projection onto halfspace from i = 1 to r and repeats a few times gives the projection onto Ax ≤ b ▶ Drawback of this approach: possibly slow if m is large 9 / 13 Active set ▶ Assume we know a set I : {i ∈ [m] | ⟨ai, x⟩ = bi}1 that at the solution the equality is satisfied Let AI be the submatrix of A with rows in I Then we can solve the following problem instead argmin 1 ∥x − y∥22 s.t AIx = bI x2 The optimality conditions can be expressed as a matrix-vector equation as I A⊤I x= 0 AI 0 z bI ▶ The idea of active set method is to construct a set J to approximate I In each iteration, it solves the above equation, and correct J Eventually J approach to I and the sol of the matrix-vector equation gives the sol of the original problem 1[n] means the set {1, 2, · · · , n} and ai is the i-th row of A 10 / 13 Interior point method ▶ First, transform Ax ≤ b to Ax + s = b, where s ≥ 0 is a slack variable ▶ Construct a log barrier on s with parameter µ > 0, and put it into the cost function, we obtain a typical problem in the interior point method argmin 1 ∥x − y∥22 − µ log |si| s.t Ax + s = b x2 i This problem is convex ▶ Starting from (x0, µ0), compute an approximate solution x, then decrease µ As µ → 0, xk → x∗ ▶ Drawback of interior point method: not scalable, not for problem with large size 11 / 13 (Dual) proximal gradient method ▶ The dual of the problem is min 1 ∥A⊤z − A⊤y∥22 + ⟨b, z⟩ s.t z ≥ 0 z2 The key is that nonnegative constraint is easy to handle, so you keep the nonnegative constraint ▶ Proximal update of this problem is simple See here for details of (unaccelerated) proximal gradient ▶ In fact, dual proximal gradient method in this case is exactly the dual projected gradient method ▶ Nesterov’s acceleration can be used ▶ See here for the discussion of dual proximal gradient method, and also how the dual problem is constructed 12 / 13 Last page - summary Discussed: solving projection onto a polyhedron ▶ Polyhedron set : Ax ≤ b, A full rank ▶ Brief overview of solving projection problem by alternating projection algorithm, active set method, interior point method and dual proximal gradient method ▶ Projection onto nonnegative orthant and box as special cases with simple close form solution Not discussed ▶ If A not full rank ▶ Details of the methods ▶ Solving the projection problem using Douglas-Rachford splitting End of document 13 / 13