030. Estimation

Estimation

이하와 같은 linear model 고려. 이때 $x_i^{\prime }$는 $X$의 i번째 row vector이며, $E(ϵ)=0,Cov(ϵ)={\sigma }^{2}I={\sigma }^2\Sigma$.

$$Y_{n\times 1}=X_{n\times p}{\beta }_{p\times 1}+{ϵ}_{n\times 1}=\left(\begin{array}{c}{x}_i^{\prime }\beta \end{array}\right)+ϵ$$

Identifiability and Estimability

Identifiable

모델에서의 무한한 갯수의 관측치를 보유한다면, 모델의 underlying 패러미터의 참값을 획득하는 것이 가능한 성질.

A general linear model is a parameterization

$$\begin{array}{rl}E(Y)& =f(X)\\ & =E(X\beta +ϵ)\\ & =X\beta +E(ϵ)\\ & =X\beta +0\\ & =X\beta \end{array}$$

The parameter $\beta$ is identifiable if for any ${\beta }_1$ and ${\beta }_2$ $f({\beta }_1)=f({\beta }_2)$ implies ${\beta }_1={\beta }_2$. If $\beta$ is identifiable, we say that the parameterization $f(\beta )$ is identifiable. (패러미터 $\beta$가 identifiable하다면, 우리는 해당 패러미터의 parameterization $f(\beta )$ 또한 identifiable 하다) Moreover, a vector-valued function $g(\beta )$ is identifiable if $f({\beta }_1)=f({\beta }_2)$ implies $g({\beta }_1)=g({\beta }_2)$.

For regression models for which $r(X)=p$, the parameters are identifiable: $X^{\prime }X$ is nonsingular, so if $X{\beta }_1=X{\beta }_2$, then

$${\beta }_1=(X^{\prime }X{)}^{-1}{X}^{\prime }X{\beta }_1=(X^{\prime }X{)}^{-1}{X}^{\prime }X{\beta }_2={\beta }_2$$

A function $g(\beta )$ is identifiable $⟺$ $g(\beta )$ is a function of $f(\beta )$.

Estimable

The results in the last section suggest that some linear combinations of $\beta$ in the less than full rank case will not be estimable.

The linear parametric function $c^{\prime }\beta$ is an estimable function if there exists a vector $a\in {\mathbb{R}}^n$ such that $\forall \beta :E(a^{\prime }y)=c^{\prime }\beta$.

A vector-valued linear function of $\beta$, ${\Lambda }^{\prime }\beta$ is estimable if ${\Lambda }^{\prime }\beta =P^{\prime }X\beta$ for some matrix P; In other words, ${\Lambda }^{\prime }\beta$ is estimable if $\Lambda =X^{\prime }P\in \mathcal{C}(X^{\prime })$.

Clearly, if ${\Lambda }^{\prime }\beta$ is estimable, it is identifiable and therefore it is a reasonable thing to estimate.

• estimable $\to$ identifiable

For estimable functions ${\Lambda }^{\prime }\beta =P^{\prime }X\beta$, although $P$ need not be unique, its perpendicular projection (columnwise) onto $\mathcal{C}(X)$ is unique: let $P_1,P_2$ be matrices with ${\Lambda }^{\prime }=P_1^{\prime }X=P_2^{\prime }X$, then

$$M{P}_1=X(X^{\prime }X{)}^{-}{X}^{\prime }{P}_1=X(X^{\prime }X{)}^{-}\Lambda =X(X^{\prime }X{)}^{-}{X}^{\prime }{P}_2=M{P}_2$$

• Example 2.1.4 and 2.1.5

$g(\beta )$‘s estimate, $f(Y)$, is unbiased if $\forall \beta :E[f(Y)]=g(\beta )$.

if $f(Y)=a_0+a^{\prime }Y$ for some scalar $a_0$ and vector $a$, $f(Y)$ is a linear estimate of ${\Lambda }^{\prime }\beta$.

if ${\Lambda }^{\prime }\beta$ $⟺$ $a_0=0$ and $a^{\prime }X={\Lambda }^{\prime }$; say, $\Lambda =X^{\prime }a\in \mathcal{C}(X^{\prime })$, then a linear estimate $a_0+a^{\prime }Y$ is unbiased

${\Lambda }^{\prime }\beta$ is estimable $⟺$ there exists $\rho$ such that $E({\rho }^{\prime }Y)={\Lambda }^{\prime }\beta$ for any $\beta$.

---

Estimation: Least Squares

Estimating $E(Y)$ is to take a vector in $\mathcal{C}(X)$ closest to $Y$;

$$\begin{array}{rlrl}E(Y)& =X\beta & & \in \mathcal{C}(X)\\ & & & \\ \hat{\beta }& =\underset{\beta }{\min }\left\{(Y-X\beta {)}^{\prime }(Y-X\beta )\right\}& & \\ & =\underset{\beta }{\min }\left\{\Vert Y-X\beta {\Vert }^2\right\}& & \end{array}\text{\hspace{1em}(Least Squares Estimate of beta)}$$

for any Least Squares Estimate $\hat{\beta }$, LSE of ${\Lambda }^{\prime }\beta is{\Lambda }^{\prime }\hat{\beta }$, e.g., ${\widehat{{\Lambda }^{\prime }\beta }}_{LSE}={\Lambda }^{\prime }\hat{\beta }$.

• Theorem 2.2.1

where $M$ is the perpendicular projection operator onto $\mathcal{C}(X)$, then

$\hat{\beta }$ is a LSE of $\beta$ $⟺$ $X\hat{\beta }=MY$

• Corollary 2.2.2

${\hat{\beta }}_{LSE}=X(X^{\prime }X{)}^{-}{X}^{\prime }Y$

• Corollary 2.2.3

The unique LSE of ${\rho }^{\prime }X\beta ={\rho }^{\prime }MY$.

※ Note: the unique LSE of ${\Lambda }^{\prime }\beta ={\Lambda }^{\prime }\hat{\beta }=P^{\prime }MY$.

• Theorem 2.2.4

the LSE of ${\Lambda }^{\prime }\beta$ is unique only if ${\Lambda }^{\prime }\beta$ is estimable: $\Lambda =X^{\prime }\rho$ if ${\Lambda }^{\prime }{\hat{\beta }}_1={\Lambda }^{\prime }{\hat{\beta }}_2$, so that $X{\hat{\beta }}_1=X{\hat{\beta }}_2=MY$.

※ Note: When $\beta$ is not identifiable, we need side conditions imposed on the parameters to estimate nonidentifiable parameters.

※ Note: With $r=r(X)<p$ (overparameterized model), we need $p-r$ individual side conditions to identify and estimate the parameters.

• Proposition 2.2.5

If $\Lambda =X^{\prime }\rho$, then $E({\rho }^{\prime }MY)={\Lambda }^{\prime }\beta$.

let’s decompose

$$\begin{array}{rlrl}Y& =X\hat{\beta }& & +Y-X\hat{\beta }\\ & =MY& & +(I-M)Y\\ & =\hat{Y}& & +e\end{array}$$

이때

$$\begin{array}{rl}\hat{Y}& \in \mathcal{C}(X)\\ e& \in \mathcal{C}(X{)}^{\perp }\end{array}\text{\hspace{1em}(fitted values of Y)(residuals)}$$

• Theorem 2.2.6

Let $r(X)=r$ and $Cov(ϵ)={\sigma }^{2}I$. At below formula, denominator is degrees of freedom for error.

Then an UE of ${\sigma }^2$, MSE, is as below.

$${\hat{\sigma }}^2=\frac{Y^{\prime }(I-M)Y}{rank(I-M)}=\frac{Y^{\prime }(I-M)Y}{n-r}\text{\hspace{1em}(MSE)}$$

---

Estimation: Best Linear Unbiased

• Definition 2.3.1

$a^{\prime }Y$ is a Best Linear Unbiased Estimate(BLUE) of ${\lambda }^{\prime }\beta$ if $a^{\prime }Y$ is unbiased.

e.g., $E(a^{\prime }Y)={\lambda }^{\prime }\beta$ and if for any other linear unbiased estimate $b^{\prime }Y$, $Var(a^{\prime }Y)\le Var(b^{\prime }Y)$.

• Theorem 2.3.2: Gauss-Markov thm

Consider $Y=X\beta +ϵ$ with $E(ϵ)=0$, $Cov(ϵ)={\sigma }^{2}I$. Let ${\lambda }^{\prime }\beta$ be estimable.

Then LSE of ${\lambda }^{\prime }\beta =$ BLUE of ${\lambda }^{\prime }\beta$.

• Corollary 2.3.3

Let ${\sigma }^2>0$. Then there exists a unique BLUE for any estimable function ${\lambda }^{\prime }\beta$.

---

Estimation: Maximum Likelihood

Assume that $Y\sim N_n(X\beta ,{\sigma }^2{I}_n)$. Then the Maximum Likelihood Estimates (MLEs) of $\beta$ and ${\sigma }^2$ are obtained by maximizing the log of the likelihood so that

$$\begin{array}{rl}\left(\hat{\beta },{\hat{\sigma }}^2\right)& =\text{ MLE of }\left(\beta ,{\sigma }^2\right)\\ & =\underset{\left(\beta ,{\sigma }^2\right)}{\max }\left\{-\frac{n}{2}log(2\pi )-\frac{1}{2}\log \left[({\sigma }^2{)}^n\right]-\frac{(Y-X\beta {)}^{\prime }(Y-X\beta )}{2{\sigma }^2}\right\}\end{array}$$ $$\begin{array}{rl}\hat{\beta }& =\text{ LSE of }\beta \\ & \\ {\hat{\sigma }}^2& =\frac{1}{n}\left\{Y^{\prime }(I-M)Y\right\}\end{array}$$

---

Estimation: Minimum Variance Unbiased

Assume that $Y=X\beta +ϵ$ with $ϵ\sim N_n(0,{\sigma }^2{I}_n)$.

if $\forall \beta ,{\sigma }^2:E\left\{h[T(Y)]\right\}=0$ implies that $Pr[h(T(Y))=0]=1$, A vector-valued sufficient statistic $T(Y)$ is said to be complete

If $T(Y)$ is a complete sufficient statistic, then $f(T(Y))$ is a Minimum Variance Unbiased Estimate (MVUE) of $E[f(T(Y))]$.

• Theorem 2.5.3

let $\theta =({\theta }_1,\cdots ,{\theta }_s{)}^{\prime }$ and let $Y$ be a rvec with pdf as below. then $T(Y)=(T_1(Y),\cdots ,T_s(Y){)}^{\prime }$ is a complete sufficient statistics provided that neither $\theta$ nor $T(Y)$ satisfies any linear constraints.

$$f(Y)=c(\theta )\exp \left[\sum _{i=1}^s{\theta }_i{T}_i(Y)\right]h(Y)$$

• Theorem 2.5.4

MSE is a ${\widehat{{\sigma }^2}}_{MVUE}$, and ${\widehat{{\rho }^{\prime }X\beta }}_{MVUE}={\rho }^{\prime }MY$ whenever $ϵ\sim N(0,I)$.

---

Sampling Distributions of Estimates

Assume that $Y=X\beta +ϵ$ with $ϵ\sim N_n(0,{\sigma }^2{I}_n)$. Then $Y\sim N_n(X\beta ,{\sigma }^2{I}_n)$. then

\begin{alignat}{4} \Lambda ' \hat \beta &= P' M Y &&\sim N(\Lambda ' \beta , \; &&\sigma^2 P'MP&&\; \; \; ) && \; \; \; \; \; \; \; \; \; \;&& && && \\ & &&\sim N(\Lambda ' \beta , \; &&\sigma^2 \Lambda ' (X'X)^{-} \Lambda&&\; \; \; ) && && \because && \;M && =X(X'X)^- X' \\ & && && && && && && \; \hat Y && = MY &&\sim N(X\beta, \sigma^2 M) \\ \hat \beta &= (X'X)^- X'Y &&\sim N(\beta , \; &&\sigma^2 (X'X)^{-1}) && && && && && && (\text{if X is of full rank}) \end{alignat}

Do Exercise 2.1. Show that

$$\frac{Y^{\prime }(I-M)Y}{{\sigma }^2}\sim {\chi }^2(r(I-M),\frac{{\beta }^{\prime }{X}^{\prime }(I-M)X\beta }{2{\sigma }^2})$$

---

Generalized Least Squares(GLS)

Assume that for some known positive definite $\Sigma$,

$$Y=X\beta +ϵ,$$ \begin{alignat}{3} Y &= X \beta &&+ \epsilon && \; \; \; \; \; \; \; \; \; \; && E(\epsilon)&&=0, \; \; &&\; Cov(\epsilon) &&= \sigma^2 \Sigma \tag{1} \\ \Sigma^{-\tfrac{1}{2}}Y &= \Sigma^{-\tfrac{1}{2}} X \beta &&+ \Sigma^{-\tfrac{1}{2}} \epsilon && \; \; \; \; \; \; \; \; \; \; && E(\Sigma^{-\tfrac{1}{2}} \epsilon)&&=0, &&\; Cov(\Sigma^{-\tfrac{1}{2}} \epsilon) &&= \sigma^2 I \tag{2, by SVD} \\ Y_\ast &= X_\ast \beta &&+ \epsilon_\ast && \; \; \; \; \; \; \; \; \; \; && E( \epsilon_\ast)&&=0, &&\; Cov( \epsilon_\ast) &&= \sigma^2 I \end{alignat} $$\begin{array}{rl}{\hat{\beta }}_{GLS}& =\underset{\beta }{\min }(Y_{\ast }-X_{\ast }\beta {)}^{\prime }(Y_{\ast }-X_{\ast }\beta )\\ & =\underset{\beta }{\min }\Vert Y_{\ast }-X_{\ast }\beta {\Vert }^2\\ & =\underset{\beta }{\min }(Y-X\beta {)}^{\prime }{\Sigma }^{-1}(Y-X\beta )\end{array}\text{\hspace{1em}(Generalized LSE (GLSE) of β)}$$

• Theorem 2.7.1

1. ${\lambda }^{\prime }\beta$ estimable in model (1) $⟺$ if ${\lambda }^{\prime }\beta$ is estimable in model (2).
2. $\hat{\beta }$ is GLSE of $\beta$ $⟺$ $X(X^{\prime }{\Sigma }^{-1}X{)}^{-}{X}^{\prime }{\Sigma }^{-1}Y=X\hat{\beta }$, which is Normal Equation of GLS.

• For any estimable function, there exists a unique GLSE.

3. GLSE estimate of estimable ${\lambda }^{\prime }\beta$, is BLUE of ${\lambda }^{\prime }\beta$.
4. let $ϵ\sim N(0,{\Sigma }^2\Sigma )$. then, GLSE of estimable ${\lambda }^{\prime }\beta$, is MVUE.
5. let $ϵ\sim N(0,{\Sigma }^2\Sigma )$. then, ${\hat{\beta }}_{GLS}={\hat{\beta }}_{MLE}$.

Normal Equation of GLS can be rewritten as

$$\begin{array}{rl}X(X^{\prime }{\Sigma }^{-1}X{)}^{-}{X}^{\prime }{\Sigma }^{-1}Y& =X\hat{\beta }\\ AY& =\end{array}$$

$A$ is a projection operator onto $\mathcal{C}(X)$.

$Cov(X{\hat{\beta }}_{GLS})={\sigma }^2\ast X(X^{\prime }{\Sigma }^{-1}X{)}^{-}{X}^{\prime }$ Let ${\lambda }^{\prime }\beta$ be estimable. Then $Var({\lambda }^{\prime }{\hat{\beta }}_{GLS})={\sigma }^2\ast {\lambda }^{\prime }(X^{\prime }{\Sigma }^{-1}X{)}^{-}\lambda$.

• Note: $(I-A)Y$ is residual vector of GLSE.

$$\begin{array}{rl}SS{E}_{GLS}& =(Y_{\ast }-{\hat{Y}}_{\ast }{)}^{\prime }(Y_{\ast }-{\hat{Y}}_{\ast })\\ & ⋮\\ & =Y^{\prime }(I-A{)}^{\prime }{\Sigma }^{-1}(I-A)Y\\ & \\ MS{E}_{GLS}& ={\hat{\sigma }}^2\\ & =\frac{1}{n-r(X)}\ast SS{E}_{GLS}\\ & \\ \frac{1}{{\hat{\sigma }}^2}\frac{{\lambda }^{\prime }({\hat{\beta }}_{GLS}-{\beta }_{GLS})}{{\lambda }^{\prime }(X^{\prime }{\Sigma }^{-1}X{)}^{-}\lambda }& \sim t(n-r(x))\end{array}$$

denominator는 $Var({\lambda }^{\prime }{\hat{\beta }}_{GLS})={\sigma }^2\ast {\lambda }^{\prime }(X^{\prime }{\Sigma }^{-1}X{)}^{-}\lambda$.

Let $\Sigma$ be nonsingular and $\mathcal{C}(\Sigma X)\subset \mathcal{C}(X)$. Then least squares estimates are BLUEs.

• Note: for diagonal $\Sigma$, GLS is referred to as Weighted Least Squares (WLS).

• Exercise 2.5.

Show that $A$ is the perpendicular projection operator onto $\mathcal{C}(X)$ when the inner product between two vectors $x$ and $y$ is defined as $(x,y{)}_{\Sigma }\equiv x^{\prime }{\Sigma }^{-1}y$.