060. Generalized Least Squares

Generalized Least Squares

Consider a full rank parameterization

$$Y=X\beta +ϵE(ϵ)=0,Cov(ϵ)={\sigma }^2\Sigma >0$$

by SVD of $\Sigma$,

$$\begin{array}{rl}\Sigma & ={\Gamma }^{\prime }\Lambda \Gamma ={\Gamma }^{\prime }{\Lambda }^{\frac{1}{2}}{\Lambda }^{\frac{1}{2}}\Gamma ={\Gamma }^{\prime }{\Lambda }^{\frac{1}{2}}{\Gamma }^{\prime }\Gamma {\Lambda }^{\frac{1}{2}}\Gamma ={\Lambda }^{\frac{1}{2}}\\ & \\ Z& \equiv {\Lambda }^{-\frac{1}{2}}Y={\Lambda }^{-\frac{1}{2}}(X\beta +ϵ)={\Lambda }^{-\frac{1}{2}}X\beta +{\Lambda }^{-\frac{1}{2}}ϵ=W\beta +{ϵ}^{\ast }\end{array}$$ $$\begin{array}{rl}\hat{\beta }& =(W^{\prime }W{)}^{-1}{W}^{\prime }Z=(X^{\prime }{\Sigma }^{-1}X{)}^{-1}{X}^{\prime }{\Sigma }^{-1}Y\\ E(\hat{\beta })& =(X^{\prime }{\Sigma }^{-1}X{)}^{-1}{X}^{\prime }{\Sigma }^{-1}X\beta =\beta \\ Cov(\hat{\beta })& ={\sigma }^2(X^{\prime }{\Sigma }^{-1}X{)}^{-1}\\ {\hat{\sigma }}^2& =\frac{\Vert Z-{\mu }_Z{\Vert }^2}{n-p}=\frac{(Y-\hat{\mu }{)}^{\prime }{\Sigma }^{-1}(Y-\hat{\mu })}{n-p}\end{array}$$

the projection Matrix is ${\Sigma }^{-\frac{1}{2}}X(X^{\prime }{\Sigma }^{-1}X{)}^{-1}{X}^{\prime }{\Sigma }^{-\frac{1}{2}}$, which is symmetric, and hence is an orthogonal projection.

Now all computations have been done in the $z$ coordinates, so in particular $x^{\prime }\beta$ estimates ${\mu }_Z={\Sigma }^{-\frac{1}{2}}\mu$.

Since linear combinations of Gauss-Markov estimates are Gauss-Markov, it follows immediately that ${\hat{\mu }}_Z={\Sigma }^{-\frac{1}{2}}\hat{\mu }$.

A direct solution via inner products

We can approach the problem of determining the Generalized Least Squares estimators in a different way by viewing $\Sigma$ as determining an intter product.

We do this by returning to first principles, carefully defining means and covariances in a general inner product space.

let $x,y\in {\mathbb{R}}^n$ and $(x,y)=x^{\prime }y$ be the usual innter product.

choose a basis $\{e_1,\cdots ,e_n\}$, the usual coordinate vectors. then a rvec $x$ has coordinates $(e_i,x)=x_i$.

• Definition 1.

$E(x)=\mu =\left(\begin{array}{c}{\mu }_i\end{array}\right)$ where ${\mu }_i=E(e_i,x)$. For any $a\in {\mathbb{R}}^n$,

$$E((a,x))=E\left(\right(\sum _{i=1}^n{a}_i{e}_i,x\left)\right)=E(\sum _{i=1}^n{a}_i(e_i,x))=\sum _{i=1}^n{a}_i{\mu }_i=(a,\mu )$$

thus, another characterization of $\mu$ is: $\mu$ is the unique vector that satisfies $E((a,x))=(a,\mu )$ for all $a\in {\mathbb{R}}^n$.

Now, turn to Cov. use the same set-up as above. if $E(x_i^2)<\infty$, then $Cov(x_i,x_j)=(x_i={\mu }_i)(x_j-{\mu }_j)={\sigma }_{ij}={\sigma }_{ji}$ exists for all $i,j$, and defines $\Sigma =({\sigma }_{ij})$.

For any $a,b\in {\mathbb{R}}^n$,

$$Cov((a,x),(b,x))=E\left(\right(\sum _{i=1}^n{a}_i{x}_i,\sum _{j=1}^n{b}_j{x}_j\left)\right)=\sum _{i=1}^n\sum _{j=1}^n{a}_i{b}_j\ast Cov(x_i,x_j)=\sum _{i=1}^n\sum _{j=1}^n{a}_i{b}_j\ast {\sigma }_{ij}=(a,\Sigma b)$$

• Definition 2

Assume $E((a,x{)}^2)<\infty$. The unique non-negative definite linear transformation $\Sigma :V\to V$ that satisfies $Cov((a,x),(b,x))=(a,\Sigma b)$ for all $a,b\in V$ is called the covariance of $X$ and is denoted $Cov(x)$.

• Theorem 1

let $Y\in V$ with innerproduct $(\cdot ,\cdot )$, $Cov(Y)=\Sigma$. Define another inner product $(\cdot ,\cdot )$ on $V$ by $[x,y]-(x,Ay)$ for some positive definite $A$. Then the covariance of $X$ in the inner product sapce $V,[\cdot ,\cdot ])$ is $\Sigma A$.

• Note 1: This shows that if $Cov(X)$ exists in one inner product, it exists in all inner products.

If $Cov(X)=\Sigma$ in $\left(\begin{array}{cc}V& (\cdot ,\cdot )\end{array}\right)$, then if $\Sigma >0$ in the inner product $[x,y]=(x,{\Sigma }^{-1}y)$, the covariance is ${\Sigma }^{-1}\Sigma =I$.

• Theorem 2

Suppose $Cov(X)=\Sigma$ in $\left(\begin{array}{cc}V& (\cdot ,\cdot )\end{array}\right)$. If ${\Sigma }_1$ is symmetric on $\left(\begin{array}{cc}V& (\cdot ,\cdot )\end{array}\right)$, and $Cov((a,x))=(a,{\Sigma }_{1}a)$ for all $a\in V$, then ${\Sigma }_1=\Sigma$. This implies that the covariance is unique.

Consider the inner product sapce given by $\left(\begin{array}{cc}{\mathbb{R}}^n& (\cdot ,\cdot )\end{array}\right)$, where $[x,y]=(x,{\Sigma }^{-1}y)$, $E(Y)=\mu \in \mathcal{E}$ and $Cov(Y)={\sigma }^2\Sigma$.

Let $P_{\Sigma }$ be the projection on $\mathcal{E}$ in this inner product space, and let $Q_{\Sigma }=I-P_{\Sigma }$, so $y=P_{\Sigma }y+Q_{\Sigma }y$.

• Theorem 3

with $[x,y]=(x,{\Sigma }^{-1}y)$, $P_{\Sigma }=X(X^{\prime }{\Sigma }^{-1}X{)}^{-1}{X}^{\prime }{\Sigma }^{-1}$ is an orthogonal projection.

• Theorem 4

let the OLS estimate $\hat{\beta }=(X^{\prime }X{)}^{-1}{X}^{\prime }Y$ and the GLS estimate $\tilde{\beta }=(X^{\prime }{\Sigma }^{-1}X{)}^{-1}{X}^{\prime }{\Sigma }^{-1}Y$. then

$$\hat{\beta }=\tilde{\beta }⟺\mathcal{C}({\Sigma }^{-1}X)=\mathcal{C}(X)$$

• Corollary 1

$\mathcal{C}({\Sigma }^{-1}X)=\mathcal{C}(X)=\mathcal{C}(\Sigma X)$

So $\Sigma$ need not be inverted to apply the theory.

To use this equivalence theorem (due to W. Kruskal), we usually characterize the $\Sigma$‘s for a given $X$ for which $\hat{\beta }=\tilde{\beta }$.

if $X$ is completely arbitrary, then only $\Sigma ={\sigma }^{2}I$ works.

• Intra-class correlation model:

let $J_n\in \mathcal{C}(X)$. then any $\Sigma$ of the form

$$\Sigma ={\sigma }^2(1-\rho )I+{\sigma }^2\rho J_n{J}_n^{\prime }$$

with $-\frac{1}{n-1}<\rho <1$ will work.

to apply the theorem, we write,

$$\Sigma X={\sigma }^2(1-\rho )X+{\sigma }^2\rho J_n{J}_n^{\prime }X$$

so for $i>1$, the i-th coluimn of $\Sigma X$ is

$$(\Sigma X{)}_i={\sigma }^2(1-\rho )X_i+{\sigma }^2\rho J_n{a}_i$$

with $a_i=J_n^{\prime }X$.

Thus, the i-th column of $\Sigma X$ is a linear combination of the i-th column of $X$ and the column of $1$‘s.

For the first column of $\Sigma X$, we compute $a_1=J_n$ and $(\Sigma X{)}_1={\sigma }^2(1-\rho )J_n+n{\sigma }^2\rho J_n={\sigma }^2(1+\rho (n-1))J_n$, So $\mathcal{C}(\Sigma X)=\mathcal{C}(X)$ as required, provided that $1+\rho (n-1)\ne 0$ or $\rho >-\frac{1}{n-1}$.