矩阵论在深度学习领域的应用

矩阵分解

矩阵的三角分解

$\begin{aligned} A \in C^{n \times n}_n,\Delta_k \neq 0 &\iff A=LR \\ &\iff 有唯一解 \begin{cases} \text{Doolittle , L is unit lower triangle}\\ \text{Crout , R is unit upper triangle } \\ \text{LDR }\\ \end{cases} \end{aligned}$

$A \in C^{n \times n}_r,\Delta_k \neq 0 \Rightarrow A = LR$

$A \in C^{n \times n},A是Hermite正定矩阵 \Rightarrow A = GG^H \tag{Cholesky}$

QR分解

Houseloder Matrix

$H = I - 2uu^H,u^Hu = 1 \tag{Householder Matrix}$
$y = Hx ,\text{H is Householder Matrix} \tag{Householder change}$
$\begin{aligned} & H^H = H ,\text{ Hermite Matrix} \\ &H^HH = I ,\text{ Uintary Matrix} \\ &H^2 = I \\ & H^{-1} = H \\ & \begin{bmatrix} I_r &O \\ O &H\end{bmatrix} \text{ is n+r Householder Matrix} \\ &det H = -1 \end{aligned}$

$\forall x \in C^n,\exists H \text{ is Householder Matrix}, Hx = \alpha z,\alpha = \parallel x \parallel_2,z^Hz = 1$
$H = I-2uu^H,u = \frac{x-\alpha z}{\parallel x - \alpha z \parallel_2}$
$\text{In particular}$
$\forall x \in C^n,\exists H \text{ is Householder Matrix}, Hx = \alpha e_1,\alpha = \parallel x \parallel_2$
$\forall x \in R^n,\exists H \text{ is Householder Matrix}, Hx = \alpha e_1,\alpha = \pm \parallel x \parallel_2$

Givens Matrix

$T_{pq} = \begin{bmatrix} 1 &&&&& &&&&\\ &\ddots &&&& &&&& \\ &&1 &&& &&&&\\ &&& \overline c &&&& \overline s &&&\\ &&&& 1 &&&&&&\\ &&&&& \ddots &&&&&& \\ &&&&&& 1 &&&&& \\ &&& -s &&&& c &&& \\ &&&&&&&& 1 &&& \\ &&&&&&&&& \ddots && \\ &&&&&&&&&& 1 &\\ \end{bmatrix}$
$y = T_{pq} x \tag{Givens change}$
$\begin{aligned} T_{pq}^HT_{pq} = I\\ det T_{pq} = 1 \end{aligned}$
$\forall x = (\xi_1,\xi_2,\cdots,\xi_n)^T,\exists T_{pq},y=T_{pq} x,y_{[p]}>0,y_{[q]} = 0$
$c = \frac{\xi_p}{\sqrt{\mid \xi_p \mid^2 + \mid \xi_q \mid^2}},s = \frac{\xi_q}{\sqrt{\mid \xi_p \mid^2 + \mid \xi_q \mid^2}}$
$\forall x \in C^n,\exists T_{12},T_{13},\cdots,T_{1n}, T_{1n}\cdots T_{13}T_{12} x = \parallel x \parallel_2 e_1$

矩阵的QR分解

$\forall A \in C^{n \times n},A = QR,Q^HQ=I,\text{R is upper triangle matrix}$

• $\text{Householder trans}$
  $\begin{aligned} &A = (a_1,a_2,\cdots,a_n) \\ &H_1a_1 = \alpha e_1 \\ &H_1 A = \left(\begin{array}{c:ccc} \alpha_1 & * & \cdots & * \\ \hdashline 0 & & \\ \vdots & & \boldsymbol{B}_{n-1} \\ 0 & & \end{array}\right) \\ & \tilde H_2b_2 = \alpha_2 \tilde e_1\\ & H_2 =\begin{bmatrix} 1 &0^T \\ 0 & \tilde H_2\end{bmatrix}\\ & H_2(H_1A) = \cdots\\ &\cdots\\ &H_{n-1}\cdots H_2H_1A = R\\ & A = H_1H_2\cdots H_{n-1}R = QR \\ \end{aligned}$
• $\text{Givens trans}$
  $\begin{aligned} &A = (a_1,a_2,\cdots,a_n) \\ &T_{1n}\cdots T_{12}a_1 = \parallel a_1 \parallel_2e_1 \\ &\cdots \\ & T_{n-1,n} \cdots T_{2n}\cdots T_{23} T_{1n} \cdots T_{12} A = R \\ &A = T_{12}^H \cdots T_{1n}^H T_{23}^H \cdots T_{2n}^H \cdots T_{n-1,n}^HR = QR \end{aligned}$
• $\text{Schmidt trans}$
  $A \in C_{n}^{n \times n}$，$A$可唯一地分解为
  $A=QR$
  $\begin{aligned} A =&(a_1,a_2,\cdots,a_n) \\ \Rightarrow& (p_1,p_2,\cdots,p_n) \text{ Schmidt} \\ \Rightarrow & (q_1,q_2,\cdots,q_n) \text{ Unit} \\ A = &(q_1,q_2,\cdots,q_n) \begin{bmatrix} \parallel p_1 \parallel_2 &\lambda_{21}\parallel p_1 \parallel_2 & \cdots &\lambda_{n1} \parallel p_1 \parallel_2 \\ & \parallel p_2 \parallel_2 &\cdots &\lambda_{n2} \parallel p_2 \parallel_2\\ &&\ddots &\vdots \\ &&& \parallel p_n \parallel_2 \end{bmatrix} = QR\\ &\lambda_{ij} = \frac{(a_i,p_j)}{(p_j,p_j)} \end{aligned}$

矩阵酉相似于Hessenberg矩阵

$A = (a_{ij})_{n \times n} \in C^{n \times n},a_{ij} = 0(i>j+1) \tag{Hessenberg matrix}$
使用$\text{Householder,Givens}$变换，将矩阵逐步化简成$\text{Hessenberg}$型，例如$H_1AH_1 = A_1,\cdots, \text{ or } T_{12}AT_{12}^T= A_1$
最后得到
$Q^HAQ = H$

• $A\in C^{n \times n},\exists Q,Q^HAQ \text{ is Hessenberg matrix}$
• $A\in R^{n \times n},\exists Q,QQ^T=I,Q^TAQ \text{ is Hessenberg matrix}$
• $A \in C^{n \times n},A^H=A,\exists Q,Q^HAQ \text{ is tridoginal }$

矩阵的满秩分解

Hermite标准形

如果$A$经过有限次初等变换变为矩阵$B$，则$A$与$B$等价，其充要条件是
$\begin{aligned} &rank A = rank B \\ &\exists S \in C^{m\times m}_m,T \in C_n^{n\times n},SAT= B \end{aligned}$
设$A \in C_r^{m \times n}$，则$A$可通过初等行变换化为如下条件的矩阵$H$，即$\exists S\in C^{m \times m}_m,SA=H$
$\boldsymbol{H}=\left(\begin{array}{cccccccccccccccccc} 0 & \cdots & 0 & 1 & * & \cdots & * & 0 & * & \cdots & 0 & * & \cdots & * \\ 0 & \cdots & 0 & 0 & 0 & \cdots & 0 & 1 & * & \cdots & \vdots & \vdots & & \vdots \\ \vdots & & \vdots & \vdots & \vdots & & \vdots & \vdots & \vdots & & 0 & * & \cdots & * \\ 0 & \cdots & 0 & 0 & 0 & \cdots & 0 & 0 & 0 & \cdots & 1 & * & \cdots & * \\ 0 & \cdots & 0 & 0 & 0 & \cdots & 0 & 0 & 0 & \cdots & 0 & 0 & \cdots & 0 \\ \vdots & & \vdots & \vdots & \vdots & & \vdots & \vdots & \vdots & & \vdots & \vdots & & \vdots \\ 0 & \cdots & 0 & 0 & 0 & \cdots & 0 & 0 & 0 & \cdots & 0 & 0 & \cdots & 0 \end{array}\right\}$

• $H$的前$r$行中每一行至少含一个非零元素，且第一个非零元素是$1$， 其所在列元素均为$0$（除去自身）
• 若$H$中第$i$行的第一个非零元素$1$位于$j_i$列，则
  $j_1 < j_2 < \cdots j_r$
• $H$的第$j_1,j_2,\cdots,j_r$为$I_m$的前$r$列

这样的矩阵为$\text{Hermite}$标准形
可通过
$S(A,I_m) = (SA,S) = (H,S)$
的方法求出
以$n$阶单位矩阵$I_n$的$n$个列向量$e_1,e_2,\cdots,e_n$为列构成的$n$阶方阵
$P = (e_{i_1},e_{i_2},\cdots,e_{i_n})$
称为$n$阶置换矩阵，$i_{1},i_{2},\cdots,i_{n}$是$n$的一个全排列

• $P$是正交矩阵
• $AP$是将$A$的列按照全排列顺序（$Hermiter$标准形每一行第一个非零元素列的顺序）重新排列的矩阵
  设$A\in C_r^{m\times n},\exists S\in S_m^{m \times m}$和$n$阶置换矩阵$P$，使得
  $SAP = \begin{bmatrix} I_r &K \\ O &O\end{bmatrix}$
  如果继续进行初等列变换，可得到
  $SAT = \begin{bmatrix} I_r&O\\ O&O \end{bmatrix}$
  求$T$的方法类似于$S$的求法

矩阵的满秩分解

$\begin{aligned} A\in C_r^{m \times n},\exists F \in C_r^{m \times r},G \in C_r^{r\times n},\\ A = FG \\ SAT = \begin{bmatrix} I_r &O \\ O &O\end{bmatrix}\\ F = S^{-1}[前r行]，G = T^{-1}[前r列] \end{aligned}$
简便求$FG$的方法如下
$\begin{aligned} SA = H \\ F = A[前j_1,j_2,\cdots,j_r列]\\ G = H[前r行] \end{aligned}$

矩阵的奇异值分解

设$A,B \in C^{m \times n}$,若存在$m$阶酉矩阵$U$，$n$阶酉矩阵$V$，使得$U^HAV=B$，则称$A$与$B$酉等价。
设$A \in C_r^{m \times n}$,$A^HA$的特征值为
$\lambda_1 \geq \lambda_2 \geq \cdots \geq \lambda_r >\lambda_{r+1} = \cdots = 0$
则称$\sigma_i=\sqrt{\lambda_i}$为$A$的奇异值
酉等价矩阵有相同的奇异值
设$A\in C_r^{m \times n}$，则存在$m$阶酉矩阵$U$和$n$阶酉矩阵$V$，使
$U^HAV = \begin{bmatrix} \Sigma &O\\ O&O \end{bmatrix},\Sigma = diag(\sigma_1,\sigma_2,\cdots,\sigma_r)$
$A = U \begin{bmatrix} \Sigma & O \\ O&O \end{bmatrix} V^H$
称为$A$的奇异值分解
$\begin{bmatrix} V^HA^HAV = diag(\lambda_1,\lambda_2,\cdots,\lambda_n)\\ V = (V_1,V_2) (V_1 \in C^{n \times r}) \\ U_1 = AV_1\Sigma^{-1} \\ U=(U_1,U_2),U^HU=I\\ \end{bmatrix}$
设$A \in C_r^{m\times n},b \in C^m$,对$A$进行奇异分解，
$x^{(0)} = V \begin{bmatrix} \Sigma^{-1} &O \\ O& O \end{bmatrix} U^Hb$
是矛盾方程组$Ax = b$的最小二乘解，如果最小二乘解不唯一，则$x^{(0)}$是其中具有最小$2$范数的向量，称其为极小范数最小二乘解。

特征值的估计与表示

特征值的包含区域

Gerschgorin定理

设$A = (a_{ij})_{n \times n}$，记
$R_i = \sum_{j=1,j\neq i}^n \mid a_{ij} \mid$
称复平面上的圆域
$G_i = \{z \mid \mid z - a_{ii} \mid \leq R_i,z\in C \} (i=1,2,\cdots,n)$
为矩阵$A$的第$i$个$\text{Gerschgorin}$圆，称$R_i$为盖尔圆$G_i$的半径
$\lambda_1,\lambda_2,\cdots,\lambda_n \in \bigcup_{i=1}^n G_i \tag{Gerschgorin1}$
$A$的特征值与$A^T$的特征值相同，$A^T$的盖尔圆叫做$A$的列盖尔圆

• $\text{Gerschgorin2}$：若矩阵$A$的某一连通部分由$A$的$k$个盖尔圆构成，则其中有且仅有$A$的$k$个特征值
• 若$A$按行（列）严格对角占优，则$det A \neq 0$
• 实矩阵的复特征值成对共轭出现，如果其盖尔圆各自独立且均关于实轴对称，则特征值均为实数

特征值的隔离

选取正数，令$d_i$
$D = diag(d_1,d_2,\cdots,d_n)$
$B = DAD^{-1} = (a_{ij}\frac{d_i}{d_j})_{n \times n}$
$A$与$B$有相同的特征值，通过设某一个$d_i>1 \text{ or }d_i<1$,其他$d$设为$1$，可放大或缩小$G_i$，同时缩小放大其它盖尔圆。

$D$中不为$1$的位置为$d_i$，$DAD^{-1}=B$，$B$等于对$A$的第$i$行元素乘以$d_i$，第$i$列元素乘以$\cfrac 1{d_i}$，不对$a_{ii}$操作。

Rayleign商

$A$是$n$阶实对称矩阵，$x \in R^n$,
$R(x) = \frac{x^TAx}{ x^Tx} = \frac{(Ax,x)}{(x,x)}$
为$A$的$\text{Rayleign}$商

• $R(\lambda x) = R(x)$
• $\lambda_1 = \max_{x\neq 0} R(x)$
• $\lambda_n = \min_{x\neq 0}R(x)$
• $R(x) = \frac{x^TAx}{x^Tx} = y^TAy,\parallel y \parallel_2=1$