1. Notation

||$x$||: Euclidean norm ($l_2$ norm),

$$
||x||^2 = x^Tx= \sum_{i=1}^dxi^2
$$
$\mathbb{R
+} = { x\in \mathbb{R}: x\geq 0 }$

2. Cauchy-Schwarz inequality

Let $u,v\in \mathbb{R^d}$, then
$$
|u^Tv|\leq ||u||\cdot ||v||
$$

3. Spectral Norm

Let $A$ be a matrix, $A\in \mathbb{R^{m\times d}}$, then
$$
||A|| = \max_{||v||=1}||Av||
$$
where $v\in \mathbb{R^d}$. Besides,
$$
||Av|| \leq ||A||\cdot ||v||
$$
Furthermore, we have triangle inequality:
$$
||A+B||\leq ||A||+||B||
$$

4. Mean Value Theorem

$\large{\text{Theorem}\ 1.3}$: Let $a<b$ be real numbers, $h:[a,b]\rightarrow \mathbb{R}$ be a continuous function which is differential on $(a,b)$. Then there exists $c\in (a,b)$ such that:
$$
h'(c) = \frac{h(b)-h(a)}{b-a}
$$

5. Differentiability

$\large{\text{Definition}\ 1.5}$: $f: dom(f)\rightarrow \mathbb{R^m}, dom(f)\rightarrow \mathbb{R^d}. f \text{ is called differentiable at } x \text{ in the interior of } dom(f) \text{ if there exists a matrix } A\in \mathbb{R^{m\times d}} \text{ and an error function }r:\mathbb{R^d} \rightarrow \mathbb{R^m}\text{ defined in some neighborhood of } 0\in \mathbb{R^d} \text{such that for all }y \text{ in some neighborhood of }x,$
$$
f(y) = f(x)+A(y-x)+r(y-x)
$$
where
$$
\lim{v\rightarrow 0} \frac{||r(v)||}{||v||}=0
$$
Besides, $Df(x)
{ij} = \frac{\partial f_i}{\partial x_j}(x)$

6. Convex Set

$\large{\text{Theorem}\ 1.9}$: $f:dom(f)\rightarrow \mathbb{R^m} \text{ be differentiable}, X\in dom(f) \text{ a convex set}, B\in \mathbb{R_+}. \text{ If } X\in dom(f) \text{ is non-empty and open the following statements are equivalent:}$
$$
\begin{align}
&(i)\large{||f(x)-f(y)||\leq B||x-y||}(\large{\bf{\text{B-Lipschitz}}}) \
&(ii)\large{||Df(x)||} \leq B, \forall x\in X
\end{align}
$$

7. Epigraph

$$
\begin{equation}
{\bf{epi}}(f)={(x,\alpha)\in \mathbb{R^{d+1}}:x\in dom(f), \alpha\geq f(x) }
\end{equation}
$$
$\large{\text{Lemma}\ 1.11}$: $f\text{ is a convex function if and only if } {\bf epi}(f) \text{ is a convex set}$

$\large{\text{Lemma}\ 1.15}$: $f\text{ is convex if and only if } dom(f) \text{ is convex and} $:

$$
\begin{equation}
f(y)\geq f(x)+\nabla f(x)^T(y-x)
\end{equation}
$$
$\text{holds for all }x,y\in dom(f)$.

$\large{\bf{\text{Lemma}}\ 1.16}$: $\text{Suppose that }dom(f) \text{ is open and } f\text{ is differentiable. Then } f\text{ is convex if and only if } dom(f) \text{ is convex and}$
$$
\begin{equation}
\large{{(\nabla f(y)-\nabla f(x))^T(y-x)\geq 0}}
\end{equation}
$$
$\text{holds for all }x,y\in dom(f)$.
$\bf{Proof}: \text{If }f \text{ is convex,from first-order convex property we have}:$
$$
\begin{align}
f(y)\geq f(x)+\nabla f(x)^T(y-x)\
f(x)\geq f(y)+\nabla f(y)^T(x-y)
\end{align}
$$
$\text{for all } x,y\in dom(f). \text{ After adding up these two inequalities, then we get:}$
$$
\nabla f(x)^T(y-x)+\nabla f(y)^T(x-y) = (\nabla f(y)-\nabla f(x))^T(x-y)\leq 0
$$

8. Second-Order Characterization of Convexity

$\large{\text{Lemma}\ 1.17}$: $f\text{ is convex if and only if } dom(f) \text{ is convex and} $:

$$
\begin{equation}
\large\nabla^2 f(x)\geq 0
\end{equation}
$$
$\text{holds for all }x,y\in dom(f) \text{ Positive Semidefinite: } M\geq 0, x^TMx\geq 0 \text{ for all }x\neq 0$.

$\large{\text{Lemma}\ 1.25}$: $f:dom(f)\rightarrow \mathbb{R}\text{ be strictly convex. Then }f \text{ has just at most one global minimum.}$

$\large{\text{Lemma}\ 1.27}$: $\text{Suppose }f:dom(f)\rightarrow \mathbb{R}\text{ is convex and differentiable.} X\in dom(f) \text{ be a convex set. Point } x^\in X \text{ is a } {\bf minimizer} \text{ of } f \text{ if and only if}$
$$
\large\nabla f(x^
)^T(x-x^*)\geq 0, \forall x\in X
$$

9. log-sum-exp function

${\bf \large Statement}: \text{log-sum-exp function is a convex function}$
$\bf Proof$: $f(x) = \log(\sum_i e^{x_i})$
$$
\begin{align}
f(\theta x+(1-\theta)y)&=\log(\sum_i e^{\theta x_i+(1-\theta)y_i})\
&=\log(\sum_i e^{\theta x_i}e^{(1-\theta)y_i})\
&=\log(\sum_i ui^{\theta}v{i}^{(1-\theta)})
\end{align}
$$
$\text{By Inequality}:$
$$
\begin{align}
\sum_i x_iy_i\leq (\sum_i |x_i|^p)^{1/p}(\sum_i |y_i|^q)^{1/q}
\end{align}
$$
$\text{Where }1/p+1/q=1. \text{ Therefore,}$
$$
\begin{align}
\log(\sum_i ui^{\theta}v{i}^{(1-\theta)}) &\leq \log[(\sum_i |u_i|^{\theta\cdot 1/\theta})^{\theta}(\sum_i |v_i|^{(1-\theta)\cdot(1/(1-\theta))})^{1-\theta}]\
&= \theta\log(\sum_i u_i)+(1-\theta)\log(\sum_i v_i)
\end{align}
$$