PROJECTION ONTO NONNEGATIVE ORTHANT, RECTANGULAR BOX AND POLYHEDRON

Công Nghệ Thông Tin, it, phầm mềm, website, web, mobile app, trí tuệ nhân tạo, blockchain, AI, machine learning - Khoa học tự nhiên - Công nghệ thông tin Projection onto nonnegative orthant, rectangular box and polyhedron Andersen Ang ECS, Uni. Southampton, UK andersen.angsoton.ac.uk Homepage angms.science Version: April 2, 2023 First draft: March 19, 2020 Content proj onto nonnegative orthant proj onto box alternating projection algorithm active set method interior point method dual proximal gradient method Projection onto nonnegative orthant ▶ The Euclidean projection of a given point y ∈ Rn onto a (non-empty compact) set S ⊆ Rn, denoted as projS (y), is a function Rn → Rn that output a point ˆx by solving ˆx = projS (y) = argmin x∈S ∥x − y∥2. Such optimization always has a unique solution, details here. ▶ Question: What if S is the nonnegative orthant? projS (y) = argmin x≥0 ∥x − y∥2, where x ≥ 0 means x is inside the nonnegative orthant S =  x xi ≥ 0 ∀i . ▶ This problem has a trivial solution ˆx = y+, where ˆxi = yi+ with · + = max{ · , 0 }. 2 13 y+ = argmin x≥0 ∥x − y∥2 Among all the feasible value in the nonnegative orthant, 0 is the closest to negative value. Example ▶ The orange point y =  2 −2  is projected to the blue point  2 0  as the blue point is where the blue circle just touch the nonnegative orthant (violet region). ▶ Here the radius of the circle (=2) is the optimal cost value. 3 13 Projection onto a box A generalization of the nonnegativity is the box constraint PS (y) = argmin l≤x≤u ∥x − y∥2, with the close form solution PS (y)i =    li yi ≤ li yi li < yi < ui ui yi ≥ ui An example li = 1, ui = 2 and a point at (5, 5). 4 13 Projection onto a polyhedron ▶ projnonnegative orthant and projbox are special cases of projpolyhedron Polyhedron S = n x Ax ≤ b o . where A ∈ Rm×n is full rank, and b ∈ Rm. ▶ Note that it does not matter Ax ≥ b or Ax ≤ b: we can absorb the negative sign into A or b. 5 13 Example 1 : m = n = 2 orange point y = 2, −2 blue point ˆx = 125, −65 Red region x + 2y ≥ 0 Blue region 2x + y ≥ 0 a1 2, 1 a2 1, 2 A = 2 1 1 2  b =  0 0  ▶ y fulfill Ax ≥ b for the first row of A, but not the second row. ▶ The optimal cost value = 4 5 , which is the distance between ˆx and y , and it is also the radius of the blue circle. ▶ We can see in this example that, proj to nonnegative orthant is simply the vector a1, a2 are rotated to the direction of the standard basis. 6 13 Example 2 : m = 3, n = 2 orange point y = 2, −2 blue point ˆx = 2, −1 Red region x + 2y ≥ 0 Blue region 2x + y ≥ 0 Green region x − y ≤ 3 A =   2 1 1 2 −1 1   b =   0 0 −3   ▶ The boundary of the feasible region is highlighted in yellow. 7 13 Solving projection onto polyhedron ▶ Unlike the projection onto nonnegative orthant or projection onto a box, the problem argmin Ax≤b ∥x − y∥2, has no simple close form solution in general. ▶ To solve it, we use an iterative optimization algorithms, such as 1. Alternating projection algorithm 2. Active set methods 3. Interior point methods 4. (Dual) Proximal gra...

Projection onto nonnegative orthant, rectangular box and polyhedron Andersen Ang

ECS, Uni Southampton, UK

andersen.ang@soton.ac.uk

Homepage angms.science

Version: April 2, 2023

First draft: March 19, 2020

Content proj onto nonnegative orthant proj onto box

alternating projection algorithm active set method

interior point method dual proximal gradient method

Projection onto nonnegative orthant

▶ The Euclidean projection of a given point y ∈ Rn onto a (non-empty & compact) set

S ⊆ Rn, denoted as projS(y), is a function Rn→ Rn that output a point ˆx by solving

ˆ

x = projS(y) = argmin

x∈S

∥x − y∥2 Such optimization always has a unique solution, details here

▶ Question: What if S is the nonnegative orthant?

projS(y) = argmin

x≥0

∥x − y∥2, where x ≥ 0 means x is inside the nonnegative orthant S = x | xi ≥ 0 ∀i

▶ This problem has a trivial solution ˆx = [y]+, where ˆxi= [yi]+ with [ · ]+= max{ · , 0 }

2 / 13

[y]+= argmin

x≥0

∥x − y∥2

Among all the feasible value in the nonnegative orthant, 0 is the closest to negative value

Example

▶ The orange point y = 2

−2



is projected to the

blue point 2

0



as the blue point is where the blue circle just touch the nonnegative orthant

(violet region)

▶ Here the radius of the circle (=2) is the optimal

cost value

3 / 13

Projection onto a box

A generalization of the nonnegativity is the box constraint

PS(y) = argmin

l≤x≤u

∥x − y∥2,

with the close form solution [PS(y)]i=







li yi ≤ li

yi li < yi < ui

ui yi ≥ ui

An example li= 1, ui = 2 and a point at (5, 5)

4 / 13

Projection onto a polyhedron

▶ projnonnegative orthant and projbox are special cases of projpolyhedron

Polyhedron S =nx | Ax ≤ bo where A ∈ Rm×n is full rank, and b ∈ Rm

▶ Note that it does not matter Ax ≥ b or Ax ≤ b: we can absorb the negative sign into A

or b

5 / 13

Example 1 : m = n = 2

orange point y = [2, −2]

blue point x = [12/5, −6/5]ˆ

Red region x + 2y ≥ 0

Blue region 2x + y ≥ 0

A =2 1

1 2



b =0 0



▶ y fulfill Ax ≥ b for the first row of A, but not the second row

▶ The optimal cost value = 4

5, which is the distance between ˆx and y, and it is also the radius of the blue circle

▶ We can see in this example that, proj to nonnegative orthant is simply the vector a1, a2 are rotated to the direction of the standard basis

6 / 13

Example 2 : m = 3, n = 2

orange point y = [2, −2]

blue point x = [2, −1]ˆ

Red region x + 2y ≥ 0

Blue region 2x + y ≥ 0

Green region x − y ≤ 3

A =





2 1

1 2

−1 1



 b =





0 0

−3





▶ The boundary of the feasible region is highlighted in yellow

7 / 13

Solving projection onto polyhedron

▶ Unlike the projection onto nonnegative orthant or projection onto a box, the problem

argmin

Ax≤b

∥x − y∥2, has no simple close form solution in general

▶ To solve it, we use an iterative optimization algorithms, such as

1 Alternating projection algorithm

2 Active set methods

3 Interior point methods

4 (Dual) Proximal gradient method

5 Douglas-Rachford splitting algorithm on the dual

We talk very briefly on the first four approaches

8 / 13

Alternating projection algorithm

▶ The constraint Ax ≤ b describe a (non-empty) polyhedron as an intersection of m

inequality constraint, each of the constraint is a problem pf projection onto halfspace

argmin

⟨a,x⟩≤b

1

2∥x − y∥

2

2, with the close form solution as

ˆ

x =







y ⟨a, x⟩ ≤ b

y −⟨a, x⟩ − b

∥a∥2 2

a ⟨a, x⟩ > b

▶ Alternating projection algorithm: cycle through the projection onto halfspace from i = 1

to r and repeats a few times gives the projection onto Ax ≤ b

▶ Drawback of this approach: possibly slow if m is large

9 / 13

Active set

▶ Assume we know a set I : {i ∈ [m] | ⟨ai, x⟩ = bi}1 that at the solution the equality is satisfied Let AI be the submatrix of A with rows in I Then we can solve the following problem instead

argmin

x

1

2∥x − y∥

2

2 s.t AIx = bI The optimality conditions can be expressed as a matrix-vector equation as

 I A⊤I

AI 0

 x z



= 0

bI



▶ The idea of active set method is to construct a set J to approximate I In each iteration,

it solves the above equation, and correct J Eventually J approach to I and the sol of the matrix-vector equation gives the sol of the original problem

1 [n] means the set {1, 2, · · · , n} and a i is the i-th row of A

10 / 13

Interior point method

▶ First, transform Ax ≤ b to Ax + s = b, where s ≥ 0 is a slack variable

▶ Construct a log barrier on s with parameter µ > 0, and put it into the cost function, we obtain a typical problem in the interior point method

argmin

x

1

2∥x − y∥

2

2− µX

i

log |si| s.t Ax + s = b

This problem is convex

▶ Starting from (x0, µ0), compute an approximate solution x, then decrease µ As µ → 0,

xk→ x∗

▶ Drawback of interior point method: not scalable, not for problem with large size

11 / 13

(Dual) proximal gradient method

▶ The dual of the problem is

min

z

1

2∥A

⊤

z − A⊤y∥22+ ⟨b, z⟩ s.t z ≥ 0

The key is that nonnegative constraint is easy to handle, so you keep the nonnegative constraint

▶ Proximal update of this problem is simple See herefor details of (unaccelerated) proximal gradient

▶ In fact, dual proximal gradient method in this case is exactly the dual projected gradient method

▶ Nesterov’s acceleration can be used

▶ Seeherefor the discussion of dual proximal gradient method, and also how the dual

problem is constructed

12 / 13

Last page - summary

Discussed: solving projection onto a polyhedron

▶ Polyhedron set : Ax ≤ b, A full rank

▶ Brief overview of solving projection problem by alternating projection algorithm, active set method, interior point method and dual proximal gradient method

▶ Projection onto nonnegative orthant and box as special cases with simple close form

solution

Not discussed

▶ If A not full rank

▶ Details of the methods

▶ Solving the projection problem using Douglas-Rachford splitting

End of document

13 / 13