020. Multivariate Nomral

Multivariate Nomral (wk2)

Overview

let rvec $Z=(Z_1,\cdots ,Z_p{)}^{\prime },Z_i\stackrel{iid}{\sim }N(0,1)$.

then $X=(X_1,\cdots ,X_p{)}^{\prime }=A_{k\times p}Z+{\mu }_{k\times 1}$ follows MVN.

at here, if $rank(A)=p(\le k),A{A}^{\prime }=\Sigma$, then $X\sim N_p(0,\Sigma )$.

---

notation: $y\sim N_p(\mu ,\Sigma )$

rvec $y^{\prime }=[y_1,\cdots ,y_p]$ have Multivariate Normal Distribution, if ${\sum }_{i=1}^p{a}_i{y}_u=a^{\prime }y$ has Univariate Normal Distribution, for every possible set of values for the elements in $a$.

pdf for $f(y)=\frac{1}{(2\pi {)}^p{|\Sigma |}^{1/2}}\exp \left\{-\frac{1}{2}(y-\mu {)}^{\prime }{\Sigma }^{-1}(y-\mu )\right\}$.

---

Ellipsoid:

• path of $y$ values yielding a constant height for the density,
  i.e., all $y$ s.t. $\{(y-\mu {)}^{\prime }{\Sigma }^{-1}(y-\mu )=c^2\}$.

Standard Normal Distribtion:

• $z={\left({\Sigma }^{1/2}\right)}^{-1}(y-\mu )\sim N_p(0,I_p)$,
  where ${\left({\Sigma }^{1/2}\right)}^{-1}$ satiesfy $\Sigma ={\left({\Sigma }^{1/2}\right)}^{-1}\ast {\left({\Sigma }^{1/2}\right)}^{-1}$.

Property of $\Sigma$:

1. symmetric Matrix
2. positive definite Matrix
3. $Cov(Ay+b)=A\Sigma A^{\prime }$.

※ if $A$ is symmetric and non-singular, then $A=C{C}^{\prime }$, where $C$ is lower triangular Matrix. This is called Cholesky Decomposition of $A$.

---

1. $E(X)=\mu ,Cov(X)=A{A}^{\prime }=\Sigma$
2. $M_X(t)=\exp (t^{\prime }\mu +\frac{1}{2}{t}^{\prime }\Sigma t$
3. if $\Sigma =A{A}^{\prime }$ is non-singular Matrix $⟺rank(A)=p$
4. $\Sigma =Cov(X)$는 symmetric, n.n.d.

이상의 $X$에 대해 이하는 TFAE.

1. $X\sim N_p(0,\Sigma )$.

Spectral Decomposition

if $A$ is symmetric, non-singular, then $A=E\Lambda E^{\prime }$, where ${\lambda }_i$ are ev (${\lambda }_1\ge \cdots \ge {\lambda }_n$), and $e_i$ are evec ($E^{\prime }E=I_p)$. This is called Spectral Decomposition of $A$.

$$\Lambda =\left[\begin{array}{ccc}{\lambda }_1& & 0\\ & \ddots & \\ 0& & {\lambda }_n\end{array}\right],E=\left[e_1,\cdots ,e_p\right]$$

이때 $\Sigma =E\Lambda E^{\prime }=E\Lambda {\Lambda }^{1/2}{E}^{\prime }=E\Lambda E^{\prime }E{\Lambda }^{1/2}{E}^{\prime }={\Sigma }^{1/2}{\Sigma }^{1/2}$.

Center & Axis of ellipsoids of $\{(y-\mu {)}^{\prime }{\Sigma }^{-1}(y-\mu )=c^2\}$:

• center: $\mu$
• axis : $\pm c\sqrt{{\lambda }_i{e}_i}$

Square root Matrix:

let symmetric non-negative Matrix $A_{p\times p}$. the square root matrix of $A$ is defined as $A^{1/2}=E{\Lambda }^{1/2}{E}^{\prime }$, where

$${\Lambda }^{1/2}=\left[\begin{array}{ccc}\sqrt{{\lambda }_1}& & 0\\ & \ddots & \\ 0& & \sqrt{{\lambda }_n}\end{array}\right]$$

Negative Square Root Matrix:

Let $A$ be of full rank and all of its ${\lambda }_i$ are positive, in addition to symmetry. $A^{-1/2}=E{\Lambda }^{-1/2}{E}^{\prime }$, where

$${\Lambda }^{-1/2}=\left[\begin{array}{ccc}\frac{1}{\sqrt{{\lambda }_1}}& & 0\\ & \ddots & \\ 0& & \frac{1}{\sqrt{{\lambda }_n}}\end{array}\right]$$

Generalized Inverse:

let $A$ be a non-negative M. if ${\lambda }_1>{\lambda }_2>\cdots >{\lambda }_r>0={\lambda }_{r+1}=\cdots ={\lambda }_p$, i.e., not full rank, then the Moore-Penrose generalized inverse of $A$ is given by

$$A^{-}=\frac{1}{{\lambda }_1}{e}_1{e}_1^{\prime }+\cdots +\frac{1}{{\lambda }_r}{e}_r{e}_r^{\prime }$$

where

$${\Lambda }^{-}=\left[\begin{array}{cccccc}\frac{1}{{\lambda }_1}& & & & 0& \\ & \ddots & & & & \\ & & \frac{1}{{\lambda }_n}& & & \\ & & & 0& & \\ & & & & \ddots & \\ 0& & & & & 0\end{array}\right]$$

Marginal Distribtion:

$$\begin{array}{rl}y\sim N_p(\mu ,\Sigma )& ⟹y_i\sim N({\mu }_i,{\sigma }^{ii}),i=1,\cdots ,p\\ & /⟸\end{array}$$

Properties of MVN

1. linear combination of the components of $y$ are normally distributed.
2. any subset of $y$ have MVN.
3. conditional distribution of the components of $y$ are MVN:

$$y\sim N_p(\mu ,\Sigma )⟺a^{\prime }y\sim N(a^{\prime }\mu ,a^{\prime }\Sigma a)$$ $$y\sim N_p(\mu ,\Sigma ),A_{ntimesp}=\left[\begin{array}{ccc}{a}_{11}& \cdots & a_{1p}\\ ⋮& \ddots & ⋮\\ a_{n1}& \cdots & a_{np}\end{array}\right]⟹Ay\sim N_n(A\mu ,A\Sigma A^{\prime })$$

즉슨 dimension 변화

if $y\sim N_p(\mu ,\Sigma )$, and cvec $d$, then $y+d\sim N_p(\mu +d,\Sigma )$.

2. If we partition y, μ, S ! ! as follows

Let 1 11 2 ~ ( , ) p y y N y μ é ù ê ú = ê ú S ê ú ë û !” ! ! ! with

${\chi }^2$ distribution

if $z\sim N_p(0,I_p)$, then $z^{\prime }z={\sum }_{i=1}^p{z}_i^2\sim {\chi }_p^2$.

if $y\sim N_p(\mu ,\Sigma )$, then $(y-\mu {)}^{\prime }{\Sigma }^{-1}(y-\mu )\sim {\chi }_p^2$

the $N_p(\mu ,\Sigma )$ distribution assigns probability $1-\alpha$ to the solid ellipsoid $\left\{y:(y-\mu {)}^{\prime }{\Sigma }^{-1}(y-\mu )\le {\chi }_p^2(\alpha )\right\}$, where ${\chi }_p^2(\alpha )$ denotes upper $(100\ast \alpha )$ th percentile of the ${\chi }_p^2$ distribution.

Linear Combination of Random Vectors

Multivariate Normal Likelihood

Sampling Distribtion of $\bar{y},S$

let rvec $y_1,\cdots y_n\sim N_p(\mu ,\Sigma )$.

$\bar{y}\sim N_p(\mu ,\frac{1}{n}\Sigma )$

(n-1) \ast S \sim $Wishartdistribution,with$df=n-1$

• $S$ is random Matrix, e.g., Wishart is distribution of rM.

$\bar{y}\perp S$.

Wishart Distribtion

※ $\frac{\sum (x_i-\text{barx}{)}^2}{{\sigma }^2}=\frac{S^2}{\frac{{\sigma }^2}{n-1}}\sim {\chi }_{n-1}^2$, i.e., \sum (x_i - \barx )^2 = (n-1)S^2 \sim \sigma^2 \ast \chi_{n-1}^

for let rvec $y_1,\cdots y_n\sim N_p(\mu ,\Sigma )$,

$$\begin{array}{rl}\sum _{i=1}^n(y-\mu )(y-\mu {)}^{\prime }& \sim W_p(n,\Sigma )\\ & \\ (n-1)S^2=\sum _{i=1}^n(y-\bar{y})(y-\bar{y}{)}^{\prime }& \sim W_p(n-1,\Sigma )\end{array}$$

if $A\sim W_p(n,\Sigma ),B\sim W_p(m,\Sigma )$, and $A\perp B:$ $A+B\sim W_p(n+m,\Sigma )$

if $A\sim W_p(n,\Sigma )$, then $CA{C}^{\prime }\sim W_p(n,C\Sigma C^{\prime })$

if $A\sim W_p(n-1,\Sigma )$, $f(A)$, where gamma function.

MV t-Distribtion

※ univariate t-Distribtion $t=\frac{\frac{U}{\sigma }}{\sqrt{\frac{V}{nu}}}\sim t_{\nu }$, where $U\sim N(0,{\sigma }^2),V\sim {\chi }_{\nu }^2$, and $U\perp V$.

let $y=(y_1,\cdots ,y_n{)}^{\prime }\sim N_p(\mu ,\Sigma )$, and $V\sim {\chi }_{\nu }^2$, and $y\perp V$.

assume rvec $t=(t_1,\cdots ,t_p{)}^{\prime }$, $t_i=\frac{\frac{y_i-{\mu }_I}{{\sigma }_i}}{\sqrt{V/\nu }},i=1,\cdots ,p$ * Note that each $t_i\sim t$.

at here, joint distribution of $t$ is called MV t-distribution, with $df=\nu$ and matrix parameter $\Sigma$.

***denote this distribution by ***

Dirichlet Distribution

※ is MV generalization of $BETA$.

let $y\sim D_p({\nu }_1\cdots ,{\nu }_{p+1})$

• parameters: $\{{\nu }_i,i=1,\cdots ,p+1\}$
• pdf: $f(y)=\frac{1}{Beta({\nu }_1\cdots ,{\nu }_{p+1})}{\prod }_{i=1}^p{y}_i^{v_i-1}$

?????????????????????????????????????????????????

CLT

let

$y_1,\cdots ,y_n\stackrel{iid}{\sim }\mu ,\Sigma <\infty$. then

$$\begin{array}{rl}\sqrt{n}(\hat{y}-\mu )& \stackrel{d}{\to }{N}_p(0,\Sigma )\\ n(\hat{y}-\mu {)}^{\prime }{S}^{-1}(\hat{y}-\mu )& \stackrel{d}{\to }{\chi }_p^2\end{array}$$

Assessing Normality

1. Univariate Marginal Distribtion

a. Q-Q Plot

※ Sample quantile vs. quantile of N distribution

let order statitics, or sample quantiles $x_{(1)}\le \cdots \le x_{(n)}$.

the proportion of sample below $x_{(j)}$ is approximated by $\frac{j-\frac{1}{2}}{n}$.

the quantiles $q_{(j)}$ for std. N are defined as

$$P(z\le q_{j)})={\int }_{-\infty }^{q_{(j)}}\frac{1}{\sqrt{2\pi }}\exp \left(-\frac{1}{2}{z}^2\right)dz\stackrel{△}{=}\frac{j-\frac{1}{2}}{n}$$

if the data arise from a N population, then $(\sigma \ast q_{(j)}+\mu \text{congruent}{x}_{(j)}$.

Similarly, the pairs $(q_{(j)},x_{(j)})$ will be linearly related.

Proceeds:

1. get $x_{(1)}\le \cdots \le x_{(n)}$ from original obs.
2. calculate probability values $\frac{j-1/2}{n},j=1,\cdots ,n$
3. calculate standard normal quantities $q_{(1)},\cdots ,q_{(n)}$
4. plot the pairs of observations $(q_{(1)},x_{(1)}),\cdots ,$(q_{(n)}, x_{(n)})$

Checking the straightness of Q-Q plot:

• using corr coef
• Hypothesis tesiting: $H_0:\rho =0$, $T=\tfrac {r\sqrt{n-2}}{\sqrt{1-r^2}} \overset {H_0}{\sim} t_{n-2}

b. others

• 1. Shapiro-Wilks Test:

Test of correlation coefficient b/w $x_{(j)},r_{(j)}$. $r_{(j)}$ is function of the expected value of standard normal order statistics, and their $Cov$.

• 2. Kolmogorov-Smirnov Test

Compare cdf’s:

If the data arise from a normal population, the differences are small.

$$T=\underset{x}{\sup }\left|F(x)-S(x)\right|$$

where cdf $F(x)$, empirical cdf $S(x)$.

• 3. Skewness Test

skewness $\sqrt{b_1}=\frac{\sqrt{n}{\sum }_{i=1}^n(x_i-\bar{x}{)}^3}{{\left[{\sum }_{i=1}^n(x_i-\bar{x}{)}^2\right]}^{\frac{3}{2}}}$

When the population is normal, the skewness = 0.

• 4. Kurtosis Test:

kurtosis $b_2=\frac{n{\sum }_{i=1}^n(x_i-\bar{x}{)}^4}{{\left[{\sum }_{i=1}^n(x_i-\bar{x}{)}^2\right]}^3}$

When the population is normal, the kurtosis is 3.

• 5. Lin and Mudholkar (1980):

$$Z={\tanh }^{-1}(r)=\frac{1}{2}\ln \left(\frac{1+r}{1-r}\right)$$

where $r$ is the sample $corr$ of $n$ pair $(x_i,q_i),i=1,\cdots ,n$ with $q_i=\frac{1}{n}{\left({\sum }_{i\ne j}{x}_j^2-\frac{1}{n-1}{\left({\sum }_{i\ne j}{x}_j\right)}^2\right)}^{\frac{1}{3}}$.

if the data arise from a normal population, $Z\sim N(0,\frac{3}{n})$.

2. Bivariate Normality

※ If the data are generated from a multivariate normal, each bivariate distribution would be normal.

• 1. Scatter Plot

the contours of bivariate normal density are ellipses. The pattern of the scatter plot must be near elliptical.

• 2. Squared Generalized Distances

※ $y\sim N_p(\mu ,\Sigma )⟹(y-\mu {)}^{\prime }{\Sigma }^{-1}(y-\mu )\sim {\chi }_p^2$.

it means, for bivariate cases, Squared Generalized Distances $d_j^2=(x_j-\hat{x}{)}^{\prime }{S}^{-1}(x_j-\hat{x})\sim {\chi }_2^2$.

• 3. Chi2 Plot (Gamma Plot)

$d_1^2,\cdots ,d_n^2$ should behave like ${\chi }_2^2$ rv.

1. order the squared distances $d_{(1)}^2\le \cdots \le d_{(n)}^2$
2. calculate the probabilitt values $\frac{j-1/2}{n}$, $j=1,\cdots ,n$
3. Calculate quantiles of ${\chi }_2^2$ distribution $q_{(1)},\cdots ,q_{(n)}$.
4. Plot the pairs $(q_{(j)},d_{(j)}^2),j=1,\cdots ,n$ where $q_{(j)}={\chi }_2^2\left(\frac{j-1/2}{n}\right)$

The plot should resemble a straight line through the origin having slope 1.

2. Multivariate Normality

Practically, it is usually sufficient to investigate the univariate and bivariate distributions.

Chi-square plot is still useful. When the parent population is multivariate normal, and both $n$ and $n-p$ are greater than 25 or 30, the squared generalized distance $d_1^2\le \cdots \le d_n^2$ should behave like ${\chi }_p^2$.

Power Transformation

$$x^{\lambda }=\{\begin{array}{ll}\frac{1}{x},& \lambda =-1\text{(Reciprocal transformation)}\\ \frac{1}{\sqrt{x}},& \lambda =-\frac{1}{2}\\ \ln x,& \lambda =0\\ \sqrt{x},& \lambda =\frac{1}{2}\\ x,& \lambda =1\text{(No transformation)}\end{array}$$

Examine Q-Q plot to see whether the normal assumption is satisfactory after power transformation.

Power Transformation

$$x^{(}\lambda )=\{\begin{array}{ll}\frac{x^{\lambda }-1}{\lambda },& \lambda \ne 0\\ \ln (x),& \lambda =0\end{array}$$

at here, find $\lambda$ that maximizes

$$l(\lambda )=-\frac{n}{2}ln\left[\frac{1}{n}\sum _{j=1}^n{\left(x_j^{(\lambda )}-{\widehat{x_j}}^{(\lambda )}\right)}^2\right]+(\lambda -1)\sum _{j=1}^n\ln x_j$$

where ${\widehat{x_j}}^{(\lambda )}=\frac{1}{n}{\sum }_{j=1}^n{x}_j^{(\lambda )}$

$x^{(\lambda )}$ is the most feasible values for normal distribution, but not guaranteed to follow normal distribution.

• Transformation (Box-Cox) usually improves the approximation to normality.
• Trial-and-error calculations may be necessary to find $\lambda$ that maximizes $l(\lambda )$
• Usually, change $\lambda$ values from -1 to 1 with increment 0.1.
• Examine Q-Q plot after the Box-Cox transformation.

nqplot, contour plot, cqplot, cqplot and box-cox plot