070. Multivariate Multiple Regression

Multivariate Multiple Regression (wk6)

Overview

Recall:

univariate Linear Regression:

repsponse variable $Y$, $r$ predictor variables $Z_1,\cdots ,Z_r$.

• model:

$$\begin{array}{rlr}{Y}_j& ={\beta }_0+{\beta }_1{Z}_{j1}+\cdots +{\beta }_j{Z}_{jr}+{ϵ}_j,& E({ϵ}_j)=0,Var({ϵ}_j)={\sigma }^2\\ Y_{n\times 1}& =Z_{n\times (r+1)}{\beta }_{(r+1)\times 1}+{ϵ}_{n\times 1},& E(ϵ)=0,Var(ϵ)={\sigma }^{2}I\end{array}$$

• estimation:

$$\begin{array}{rl}\hat{\beta }& =(Z^{\prime }Z{)}^{-1}{Z}^{\prime }Y\\ \widehat{ϵ}& =(Y-Z\hat{\beta })=Y-Z(Z^{\prime }Z{)}^{-1}{Z}^{\prime }Y=(I-Z(Z^{\prime }Z{)}^{-1}{Z}^{\prime })Y\\ & =(I-H)Y\end{array}$$

• inference:

let $ϵ\sim N_n(0,{\sigma }^{2}I)$. then

$$\begin{array}{rl}\hat{\beta }& \sim N_{r+1}(\beta ,{\sigma }^2(Z^{\prime }Z{)}^{-1})\\ {\widehat{ϵ}}^{\prime }\widehat{ϵ}& \sim {\sigma }^2{\chi }_{n-r-1}^2\\ & \\ E(\widehat{ϵ})& =0\\ Cov(\widehat{ϵ})& ={\sigma }^2(I-Z(Z^{\prime }Z{)}^{-1}{Z}^{\prime })\\ E\left(\frac{{\widehat{ϵ}}^{\prime }\widehat{ϵ}}{n-r-1}\right)& ={\sigma }^2\end{array}$$

---

Multivariate Multiple Regression

• Notation

• Model

$$Y_{n\times m}=Z_{n\times (r+1)}{\beta }_{(r+1)\times m}+{ϵ}_{n\times m},E({ϵ}_{(i)})=0,Cov({ϵ}_{(i)},{ϵ}_{(j)})={\sigma }_{ik}I,i,k=1,\cdots ,m$$

• • Cov of $m$ responses:

$$\begin{array}{rlrlrl}& \Sigma =\left[\begin{array}{ccc}{\sigma }_{11}& & {\sigma }_{1m}\\ & \ddots & \\ {\sigma }_{m1}& & {\sigma }_{mm}\end{array}\right],& & Var({ϵ}_{(i)})={\sigma }_{ii}I,& & Cov({ϵ}_{(i)},{ϵ}_{(j)})=\left[\begin{array}{ccc}{\sigma }_{ik}& & 0\\ & \ddots & \\ 0& & {\sigma }_{ik}\end{array}\right]\end{array}$$

• the meaning of

  • $0$: observations from different trials, are uncorrelated
  • ${\sigma }_{ik}$: errors for different responses on the same trial are correlated

• $i$th response $Y_{(i)}$:

$$Y_{(i)}=Z{\beta }_{(i)}+{ϵ}_{(i)},\text{with }Corr({ϵ}_{(i)})={\sigma }_{ii}I,\widehat{{\beta }_{(i)}}=(Z^{\prime }Z{)}^{-1}{Z}^{\prime }{Y}_{(i)}$$

---

Least Square

• Collecting Univariate Least Squares Estimates (LSE)
• Errors

$$Y-Z\hat{\beta }=$$

• Error Sum of Squares (SSE)

  • diagonal elements: Error SS for univariate least squares $(Y_{(i)}-Z{\beta }_{(i)}{)}^{\prime }(Y_{(i)}-Z{\beta }_{(i)})$ is minimized.
  • the generalized $Var$ $|(Y-Z\beta {)}^{\prime }(Y-Z\beta )|$ is also minimized.

• Properties

$$\begin{array}{rlr}\hat{Y}& =Z\hat{\beta }=Z(Z^{\prime }Z{)}^{-1}{Z}^{\prime }Y& \\ & =HY& \\ \widehat{ϵ}& =Y-\hat{Y}=\left[I-Z(Z^{\prime }Z{)}^{-1}{Z}^{\prime }\right]Y& \\ & =(I-H)Y& \\ Z^{\prime }\widehat{ϵ}& =Z^{\prime }\left[I-Z(Z^{\prime }Z{)}^{-1}{Z}^{\prime }\right]Y& \\ & =[Z-Z^{\prime }]Y=0{\hat{Y}}^{\prime }\widehat{ϵ}& ={\hat{\beta }}^{\prime }{Z}^{\prime }\left[I-Z(Z^{\prime }Z{)}^{-1}{Z}^{\prime }\right]Y\\ & =[{\hat{\beta }}^{\prime }Z-{\hat{\beta }}^{\prime }{Z}^{\prime }]Y=0& \end{array}\text{\hspace{1em}(Predicted Values)(residuals)(3)(4)}$$

• • by (3), residuals are orthogonal to $Z$
  • by (4), residuals are orthogonal to $\hat{Y}$

• Error Sum of Squares

$$\begin{array}{rl}{Y}^{\prime }Y& =(\hat{Y}\widehat{ϵ}{)}^{\prime }(\hat{Y}\widehat{ϵ})\\ & ={\hat{Y}}^{\prime }\hat{Y}+{\widehat{ϵ}}^{\prime }\widehat{ϵ}\\ & \\ {\widehat{ϵ}}^{\prime }\widehat{ϵ}& =Y^{\prime }Y-{\hat{Y}}^{\prime }\hat{Y}\\ & ={\hat{Y}}^{\prime }\hat{Y}-{\hat{\beta }}^{\prime }{Z}^{\prime }Z\hat{\beta }\end{array}$$

• Results 1

$$\begin{array}{rlr}E(\hat{\beta })& =\beta ,Cov(\widehat{{\beta }_{(i)}},\widehat{{\beta }_{(j)}})& ={\sigma }_{il}(Z^{\prime }Z{)}^{-1}\\ & & \\ E(\widehat{ϵ})=0,E\left(\frac{1}{n-r-1}{\widehat{ϵ}}^{\prime }\widehat{ϵ}\right)=\Sigma & & \end{array}$$

• • at here, $\widehat{ϵ}$ and $\hat{\beta }$ are correlated.

• Results 2

  • If ${ϵ}_j$ has a $N_m(0,\Sigma )$, then $\hat{\beta }=(Z^{\prime }Z{)}^{-1}{Z}^{\prime }Y$ is MLE of $\beta$

$$\begin{array}{rl}\widehat{{\beta }_{(i)}}& \sim N_{r+1}({\beta }_{(i)},{\sigma }_{ii}(Z^{\prime }Z{)}^{-1})\\ \hat{\Sigma }& =\frac{1}{n}{\widehat{ϵ}}^{\prime }\widehat{ϵ}\\ & =\frac{1}{n}(Y-Z\hat{\beta }{)}^{\prime }(Y-Z\hat{\beta })\end{array}\text{\hspace{1em}(5)}$$

• • (5) is MLE of $\Sigma$
  • $n\hat{\Sigma }\sim W_{p,n-r-1}(\Sigma )$.

• Comment

  • Multivariate regression requires no new computational problems.
  • Univariate least squares $\widehat{{\beta }_{(i)}}$ are computed individually for each response variable.
  • Diagnostics check must be done as in univariate regression.
  • Residual vectors $[{ϵ}_{j1},\cdots ,{ϵ}_{jm}]$ can be examined for multivariate normality.

Hypothesis Testing

• Note:

$$\begin{array}{rl}& H_0:\text{ responses do not depend on }{Z}_{q+1},Z_{q+2},\cdots ,Z_r\\ ⟺& H_0:\left[\begin{array}{cccc}{\beta }_{(q+1)1}& {\beta }_{(q+2)1}& \cdots & {\beta }_{(q+1)m}\\ ⋮& & & ⋮\\ {\beta }_{r1}& {\beta }_{r1}& \cdots & {\beta }_{rm}\end{array}\right]=0\\ ⟺& H_0:{\beta }_{(2)}=0,\beta =\left[\begin{array}{c}{{\beta }_{(1)}}_{(q+1)\times m}\\ cdots\\ {{\beta }_{(2)}}_{(r-q)\times m}\end{array}\right]\end{array}$$

Full Model vs. Reduced Model

let $Z=\left[\begin{array}{cc}{Z}_1& ⋮Z_2\end{array}\right]$, then $Z\beta =Z_1{\beta }_{(1)}+Z_2{\beta }_{(2)}$.

under $H_0$, $Y=Z{\beta }_{(1)}+ϵ$,

let

$$\begin{array}{rlr}E& =n\hat{\Sigma }& \\ & =(Y-Z\hat{\beta }{)}^{\prime }(Y-Z\hat{\beta })& \\ & & \\ H& =n({\hat{\Sigma }}_1-\hat{\Sigma }),& \text{ where }{E}_1=n({\hat{\Sigma }}_1)=(Y-Z\widehat{{\beta }_{(1)}}{)}^{\prime }(Y-Z\widehat{{\beta }_{(1)}})\end{array}\text{\hspace{1em}(Full Model)(under H0)}$$

• $E=n\hat{\Sigma }$. 여기서 E라는 것은 오차행렬이기 때문에, 즉 univariate 를 4번 반복해서 나온 오차를 모은 것이 바로 이 $E$라는 행렬.

let ${\lambda }_1\ge \cdots \ge {\lambda }_s$ be non-zero ev of $H{E}^{-1}$, $s=min(m,r-q)$.

---

• Four Test Stat:

1. Wilk’s Lambda:

$$\frac{|E|}{|E+H|}=\prod _{i=1}^s\frac{1}{1+{\lambda }_i}$$

2. Pillai Trace:

$$tr\left[H(H+E{)}^{-1}\right]=\sum _{i=1}^s\frac{{\lambda }_i}{1+{\lambda }_i}$$

3. Lawley-Hotelling’s Trace:

$$tr(H{E}^{-1})=\sum _{i=1}^s{\lambda }_i$$

4. Roy’s Largest Root:
   • maximum ev of $H(H+E{)}^{-1}={\lambda }_1$.

---

Example)

fit FM $Y=Z\beta +ϵ$.

fit $Y_1,Y_2,Y_3,Y_4=X_1,X_2,X_3$, then we acquire $E=n\hat{\Sigma }$.

1. $~H_0: \begin{bmatrix} \beta_{31},\beta_{32},\beta_{33},\beta_{34} \end{bmatrix} =0~$,

2. $H_0:\left[\begin{array}{c}{\beta }_{21},{\beta }_{22},{\beta }_{23},{\beta }_{24}\\ {\beta }_{31},{\beta }_{32},{\beta }_{33},{\beta }_{34}\end{array}\right]=0$,

under $H_0$, $Y=Z{\beta }_{(1)}+ϵ$

$$Z_1={\left[\begin{array}{cc}1& X_{11}\\ \cdots & \cdots \\ 1& X_{n1}\end{array}\right]}_{n\times 2},{\beta }_{(1)}={\left[\begin{array}{ccc}{\beta }_{01}& \cdots & {\beta }_{0m}\\ {\beta }_{11}& \cdots & {\beta }_{1m}\end{array}\right]}_{2\times m}$$

now, fit $Y_1,Y_2,Y_3,Y_4=X_1$ (X_2, X_3 excluded), then we acquire $E_1=n{\hat{\Sigma }}_1,H=n{\hat{\Sigma }}_1-n\hat{\Sigma }=E_1-E$.

let’s calculate ev of $H{E}^{-1}$, and compute Wilk’s Lambda ${\Lambda }^{\ast }=\frac{|E|}{|E+H|}$.

---

Sampling Distribution of the Wilk’s Lambda

let Z be full rank of $r+1$, and $(r+1)+m\le n$.

let $ϵ$ be normally distributed.

under $H_0$, $-\left[n-r-1-\frac{1}{2}(m-r+q+q)\right]\ln ({\Lambda }^{\ast })\sim {\chi }_{m(r-q)}^2$.

---

Prediction

$${\hat{Y}}_{n\times m}=Z{\hat{\beta }}_{(r+1)\times m}$$

assume fixed values ${Z_0}_{(r+1)\times 1}$ of the predictor variables. then ${\hat{\beta }}_{m\times (r+1)}^{\prime }{Z}_0\sim N_m({\beta }^{\prime }{Z}_0,Z_0^{\prime }(Z^{\prime }Z{)}^{-1}{Z}_0\Sigma )$.

• $100(1-\alpha )\%$ simultaneous CI for $E(Y_i)=Z_0^{\prime }{\beta }_{(i)}$:

$$Z_0^{\prime }{\beta }_{(i)}\pm \sqrt{(n-r-1)\frac{m}{n-r-m}{F}_{m,n-r-m}(\alpha )}\sqrt{Z_0^{\prime }(Z^{\prime }Z{)}^{-1}{Z}_0\left(\frac{n}{n-r-1}{\hat{\sigma }}_{ii}\right)},i=1,\cdots ,m$$

• • where ${\beta }_{(i)}$ is the $i$th column of $\beta$.
  • ${\hat{\sigma }}_{ii}$ is the $i$th diagonal element of $\hat{\Sigma }$.

• $100(1-\alpha )\%$ simultaneous C.I. for the individual responses $Y_{0i}=Z_0^{\prime }{\beta }_{(i)}+{ϵ}_{0i}$:

$$Z_0^{\prime }{\beta }_{(i)}\pm \sqrt{(n-r-1)\frac{m}{n-r-m}{F}_{m,n-r-m}(\alpha )}\sqrt{\left(1+Z_0^{\prime }(Z^{\prime }Z{)}^{-1}{Z}_0\right)\left(\frac{n}{n-r-1}{\hat{\sigma }}_{ii}\right)},i=1,\cdots ,m$$