Partially linear model

A partially linear model is a form of semiparametric model, since it contains parametric and nonparametric elements. Application of the least squares estimators is available to partially linear model, if the hypothesis of the known of nonparametric element is valid. Partially linear equations were first used in the analysis of the relationship between temperature and usage of electricity by Engle, Granger, Rice and Weiss (1986). Typical application of partially linear model in the field of Microeconomics is presented by Tripathi in the case of profitability of firm’s production in 1997. Also, partially linear model applied successfully in some other academic field. In 1994, Zeger and Diggle introduced partially linear model into biometrics. In environmental science, Parda-Sanchez et al used partially linear model to analysis collected data in 2000. So far, partially linear model was optimized in many other statistic methods. In 1988, Robinson applied Nadaraya-Waston kernel estimator to test the nonparametric element to build a least-squares estimator  After that, in 1997, local linear method was found by Truong.

Partially Linear Model

Main article: Econometrics

Synopsis

Algebra equation

The algebra expression of partially linear model is written as:

${\displaystyle y_{i}=\delta _{T}^{i}\beta +f(T_{i})+\mu _{i}}$[1]

Equation components outline

${\displaystyle \delta _{T}^{i}}$ and ${\displaystyle T_{i}}$: Vectors of explanatory variables. Independently random or fixed distributed variables.

${\displaystyle \beta }$: To be measured Parameter.

${\displaystyle \mu _{i}}$: The random error in statistics with 0 mean.

${\displaystyle f(T_{i})}$: To be measured part in partially linear model. 

Assumption[1]

Wolfgang, Hua Liang and Jiti Gao consider the assumptions and remarks of partially linear model under fixed and random design conditions.

When randomly distributed, introduce

${\displaystyle L_{j}(T_{i})=E(\delta _{i,j}|T_{i})}$ and ${\displaystyle \mu _{i,j}=\delta _{i,j}-E(\delta _{i,j}|T_{i})}$ (1)

${\displaystyle E(||\delta _{1}||^{3})|T=t)}$is smaller than positive infinity when t value between 0 and 1, and the sum of covariance of ${\displaystyle \delta _{1}-E(\delta _{1}|T_{1})}$ is positive. The random errors µ are independent of ${\displaystyle (\delta _{i},T_{i})}$,

When ${\displaystyle \delta _{i}}$ and Ti are fixed distributed, ${\displaystyle L_{j}}$valued between 0 and 1, and ${\displaystyle \delta _{i}}$satisfies ${\displaystyle \delta _{ij}=L_{j}(T_{i})+\mu _{ij}}$, where factor i values between 1 and n, and factor j value between 1 and p, Error factor ${\displaystyle \mu _{ij}}$satisfies, ${\displaystyle \lim _{n\to \infty }1/n\sum _{i=1}^{n}\mu _{i}\mu _{i}^{T}=\sum }$.

The least square (LS) estimators[1]

The precondition of application of the least squares estimators is the existence of nonparametric component, and running at random distributed and fixed distributed cases.

Engle, Granger, Rice and Weiss’s (1986) smoothing model should be first introduced, before applying the least squares estimators. The algebra function of their model is expressed as ${\displaystyle Y=\delta ^{T}\beta +f(t)}$ (2).

Wolfgang, Liang and Gao (1988) make an assumption that the pair (ß,g) satisfies ${\displaystyle 1/n\textstyle \sum _{i=1}^{n}\displaystyle E\{Y_{i}-\delta _{i}^{T}\beta -f(T_{i})\}^{2}=min1/n\textstyle \sum _{i=1}^{n}\displaystyle E\{Y_{i}-\delta _{i}^{T}-f(T_{i})\}^{2}}$ (3).

This means that for all 1≤i≤n, ${\displaystyle \delta _{i}^{T}\beta _{1}+f_{1}(T_{i})=\delta _{i}^{T}\beta _{2}+f_{2}(T_{i})}$.

So, ${\displaystyle f_{1}=f_{2}and\beta _{1}=\beta _{2}}$.

Under random distributed case, Wolfgang, Hua Liang and Jiti Gao assume that for all 1 ≤ i ≤ n, ${\displaystyle E[Y_{i}|(\delta _{i},T_{i})]=\delta _{i}^{T}\beta 1+f_{1}(T_{i})=\delta _{i}^{T}\beta _{2}+f_{2}(T_{i})}$ (4)

so, ${\displaystyle E\{Y_{i}-\delta _{i}^{T}\beta _{1}-f_{1}(T_{i})\}^{2}=E\{Y_{i}-\delta _{i}^{T}\beta _{2}-f_{2}(T_{i})\}^{2}+(\beta _{1}-\beta _{2})^{T}E\{(\delta _{i}-E[\delta _{i}|T_{i}])(\delta _{i}-E[\delta _{i}|T_{i}]^{T})\}(\beta _{1}-\beta _{2})}$${\displaystyle \beta _{1}=\beta _{2}}$, due to ${\displaystyle E\{(\delta _{i}-E[\delta _{i}|T_{i}])(\delta _{i}-E[\delta _{i}|T_{i}]^{T})\}}$is a positive number, as function (1) proved.So, ${\displaystyle f_{j}(T_{i})=E[Y_{i}|T_{i}]-E[\delta _{i}^{T}\beta _{j}|T_{i}]}$established for all 1≤i≤n and j equals to 1and 2 when ${\displaystyle f_{1}=f_{2}}$.

Under fixed distributed case, By parameterizing factor  from smoothing model (2) as ${\displaystyle \{f(T_{1}),.........,f(T_{n})\}^{T}=\omega _{r}and\omega _{r}=Q(Y-x\beta )}$where ${\displaystyle Q=\omega (\omega ^{T}\omega )^{-1}\omega ^{T}}$.

By making same assumption as (4), which follows from assumption (1), ${\displaystyle \beta _{1}=\beta _{2}}$and ${\displaystyle f_{1}=f_{2}}$under the fact of ${\displaystyle 1/nE\{(Y-X\beta _{1}-\omega _{r1})^{T}(Y-X\beta _{1}-\omega _{r1})\}=1/nE{(Y-X\beta _{2}-\omega _{r2})^{T}(Y-X\beta _{2}-\omega _{r2})}+1/n(\beta _{1}-\beta _{2})^{T}X^{T}(1-Q)X(\beta _{1}-\beta _{2})}$.

Assuming factors ${\displaystyle \delta _{i},T_{i},Y_{i}}$(i here are positive integers) satisfies ${\displaystyle y_{i}=\delta _{i}^{T}\beta +f(T_{i})+\mu _{i}}$and establish positive weight functions ${\displaystyle \psi _{ni}(t)}$. Any estimators of ${\displaystyle f(t)}$, for every ${\displaystyle \beta }$, we have ${\displaystyle f_{n}(t;\beta )=\sum _{i=1}^{n}\psi _{ni}(t)(Y_{i}-\delta _{i}^{T}\beta )}$. By applying LS criterion, the LS estimator of ${\displaystyle \beta _{LS}=\{({\tilde {\delta }}^{T}{\tilde {\delta }})\}^{-1}{\tilde {\delta }}^{T}{\tilde {Y}}}$. The nonparametric estimator of ${\displaystyle f(n)}$  is expressed as ${\displaystyle {\hat {f_{n}}}(t)=\sum _{i=1}^{n}\psi _{ni}(t)(Y_{i}-\delta _{i}^{T}\beta _{LS})}$. So, When the random errors are identically distributed, the estimators of variance ${\displaystyle \sigma ^{2}}$ is expressed as, ${\displaystyle {\hat {\sigma }}_{n}^{2}=1/n\sum _{i=1}^{n}({\tilde {Y_{i}}}-{\tilde {\delta _{i}^{T}}}\beta _{LS})}$.

History and Applications of partially linear model

The real-world application of partially linear model was first considered for analyzing data by Engle, Granger, Rice and Weiss in 1986.[1]

In their point of view, the relevance between temperature and the consumption of electricity cannot be expressed in a linear model, because there are massive of confounding factors, such as average income, goods price, consumer purchase ability and some other economic activities. Some of the factors are relevance with each other and might influence the observed result. Therefore, they introduced partially linear model, which contained both with parametric and nonparametric factors. The partially linear model enables and simplifies the linear transformation of data (Engle, Granger, Rice and Weiss, 1986). They also applied the smoothing spline technique for their research.

There was a case of application of partially linear model in biometrics by Zeger and Diggle in 1994. The research objective of their paper is the evolution period cycle of CD4 cell amounts in HIV (Human immune-deficiency virus) seroconverters (Zeger and Diggle, 1994).[2] CD4 cell plays a significant role in immune function in human body. Zeger and Diggle aimed to assess the proceed of disease by measuring the changing amount of CD4 cells. The number of CD4 cell is associated with body age and smoking behavior and so on. To clear the group of observation data in their experiment, Zeger and Diggle applied partially linear model for their work. Partially linear model primarily contributes to the estimation of average loss time of CD4 cells and adjusts the time dependence of some other covariables in order to simplify the proceed of data comparison, and also, the partially linear model characterizes the deviation of typical curve for their observed group to estimate the progression curve of the changing amount of CD4 cell. The deviation, granted by partially linear model, potentially helps to recognize the observe targets who had a slow progression on the amounting change of CD4 cells. 

In 1999, Schmalensee and Stoker (1999) have used partially linear model in the field of economics. The independent variable of their research is the demand of gasoline in The United State. The primary research target in their paper is the relationship between gasoline consumption and long-run income elasticity in the U.S. Similarly, there are also massive of confounding variables, which might mutually affect. Hence, Schmalemsee and Stoker chose to deal with the issues of linear transformation of data between parametric and nonparametric by applying partially linear model.[3]

In the field of environment science, Prada-Sanchez used partially linear model to predict the sulfur dioxide pollution in 2000 (Prada-Sanchez, 2000)[4], and in the next year, Lin and Carroll applied partially linear model for clustered data (Lin and Carroall, 2001).[5]

Development of partially linear model

According to Liang’s paper in 2010 (Liang, 2010), The smoothing spline technique was introduced in partially linear model by Engle, Heckman and Rice in 1986. After that, Robinson found an available LS estimator for nonparametric factors in partially linear model in 1988. At the same year, profile LS method was recommended by Speckman.[6]

Other econometrics tools in partially linear model

Kernel regression also was introduced in partially linear model. The local constant method, which is developed by Speckman, and local linear techniques, which was found by Hamilton and Truong in 1997 and was revised by Opsomer and Ruppert in 1997, are all included in kernel regression. Green et al, Opsomer and Ruppert found that one of the significant characteristic of kernel-based methods is that under-smoothing has been taken in order to find root-n estimator of beta. However, Speckman’s research in 1988 and Severini’s and Staniswalis’s research in 1994 proved that those restriction might be canceled.

Bandwidth selection in partially linear model[7]

Bandwidth selection in partially linear model is a confusing issue. Liang addressed a possible solution for this bandwidth selection in his literature by applying profile-kernel based method and backfitting methods. Also the necessity of undersmoothing for backfitting method and the reason why profile-kernel based method can work out the optimal bandwidth selection were justified by Liang. The general computation strategy is applied in Liang’s literature for estimating nonparametric function. Moreover, the penalized spline method for partially linear models and intensive simulation experiments were introduced to discover the numerical feature of the penalized spline method, profile and backfitting methods.

Kernel-Based Profile and BackFitting method[7]

By introducing ${\displaystyle E(Y|T)={E(X|T)}^{T}\beta +g(T)}$

Following with ${\displaystyle Y-E(Y|T)={X-E(X|T)}^{T}\beta +\epsilon }$

The intuitive estimator of ß can be defined as the LS estimator after appropriately estimating ${\displaystyle E(Y|T)andE(X|T)}$.

Then, for all random vector variable ${\displaystyle \xi }$, assume ${\displaystyle {\hat {E}}(\xi |T)}$is a kernel regression estimator of ${\displaystyle E(\xi |T)}$. Let ${\displaystyle {\tilde {\xi }}=\xi -E(\xi |T),\textstyle \sum _{X|T}\displaystyle =cov{X-E(X|T)}}$. For example, ${\displaystyle {\tilde {X}}_{i}=X_{i}-E(X_{i}|T_{i})}$. Denote ${\displaystyle Y=(Y_{1},...,Y_{n})^{T}}$X,g and T similarly. Let ${\displaystyle m_{x}(t)=E(X|T=t),m_{y}(t)=E(Y|T=t)}$. So ${\displaystyle \psi (m_{x},m_{y},\beta ,Y,X,T)={X-m_{x}(T)}[Y-m_{y}(T)-{X-m_{x}(T)^{T}\beta }]}$

The profile-kernel based estimators ${\displaystyle {\hat {\beta _{p}}}}$sovles,

${\displaystyle 0=\sum _{i=1}^{n}\psi ({\hat {m_{x}}},{\hat {m_{y}}},\beta ,Y_{i},X_{i},T_{i})}$

where ${\displaystyle {\hat {m_{x}}},{\hat {m_{y}}}}$are kernel estimators of mx and my.

The Penalized Spline method[7]

The penalized spline method was developed by Eilers and Marx in 1996. Ruppert and Carroll in 2000 and Brumback, Ruppert and Wand in 1999 employed this method in LME framework.

Assuming function ${\displaystyle g(t)}$can be estimated by ${\displaystyle g(t,\tau )=\tau _{0}+\tau _{1}t+...+\tau _{p}t^{p}+\textstyle \sum _{k=1}^{K}\displaystyle b_{k}(t-\xi _{k})^{p}}$

where ${\displaystyle p\geqslant 1}$is an integer, and ${\displaystyle \xi _{1}<...<\xi _{k}}$are fixed knots, ${\displaystyle a_{+}=max(a,0).}$Denote ${\displaystyle \tau =(tau_{0},...,\tau _{p})^{T}}$Consider ${\displaystyle Y=X^{T}\beta +g(T,\tau )+\epsilon }$. The penalized spline estimator ${\displaystyle ({\hat {\beta _{ps}^{T}}},{\hat {\tau _{ps}^{T}}})^{T}of(\beta ^{T},\tau ^{T})^{T}}$is defined as follow

${\displaystyle \sum _{i=1}^{n}[Y_{i}-X_{i}^{T}\beta _{i}-g(T_{i},\tau )]^{2}+\alpha \sum _{k=1}^{K}b_{k}^{2}}$

Where ${\displaystyle \alpha }$is a smoothing parameter.

As Brumback et al mentioned in 1999[8], the estimator ${\displaystyle ({\hat {\beta _{ps}^{T}}},{\hat {\tau _{ps}^{T}}})^{T}}$is same as the estimator of ${\displaystyle \beta }$ based on LME model.

${\displaystyle y=\Lambda (\beta ^{T},\tau ^{T})^{T}+Zb+\epsilon }$,

where ${\displaystyle \Lambda ={\begin{pmatrix}x_{11}&...&x_{1d}&1&T_{1}&...&T_{1}^{p}\\x_{21}&...&x_{2d}&1&T_{2}&...&T_{2}^{p}\\.&...&.&.&.&...&.\\.&...&.&.&.&...&.\\.&...&.&.&.&...&.\\x_{n1}&...&x_{nd}&1&T_{n}&...&T_{n}^{p}\end{pmatrix}}}$, ${\displaystyle Z={\begin{pmatrix}(T_{1}-\xi _{1})^{p}&...&(T_{1}-\xi _{K})_{+}^{p}\\(T_{2}-\xi _{1})^{p}&...&(T_{2}-\xi _{K})_{+}^{p}\\.&...&.\\.&...&.\\.&...&.\\(T_{n}-\xi _{1})^{p}&...&(T_{n}-\xi _{K})_{+}^{p}\end{pmatrix}}}$

Where ${\displaystyle b=(b_{1},...,b_{k})^{T}\backsim (0,\sigma _{b}^{2}),\epsilon =(\epsilon _{1},...,\epsilon _{n})^{T}\sim (0,\sigma _{\epsilon }^{2})}$, and ${\displaystyle \alpha =\sigma _{\epsilon }^{2}/\sigma _{b}^{2}}$. The matrix shows the penalized spline smoother for up above framework.   

References

1. Hardle, Liang, JiTi, WolfGang, Hua, Gao (2000). Partially linear model. PHYSICA-VERLAG.CS1 maint: multiple names: authors list (link)
2. Zeger, Scott L.; Diggle, Peter J. (1994). "Semiparametric Models for Longitudinal Data with Application to CD4 Cell Numbers in HIV Seroconverters". Biometrics. 50 (3): 689–699. doi:10.2307/2532783. ISSN 0006-341X. JSTOR 2532783.
3. Schmalensee, Richard; Stoker, Thomas M. (1999). "Household Gasoline Demand in the United States" (PDF). Econometrica. 67 (3): 645–662. doi:10.1111/1468-0262.00041. ISSN 1468-0262.
4. Prada‐Sánchez, J. M.; Febrero‐Bande, M.; Cotos‐Yáñez, T.; González‐Manteiga, W.; Bermúdez‐Cela, J. L.; Lucas‐Domínguez, T. (2000). "Prediction of SO2 pollution incidents near a power station using partially linear models and an historical matrix of predictor-response vectors". Environmetrics. 11 (2): 209–225. doi:10.1002/(SICI)1099-095X(200003/04)11:2<209::AID-ENV403>3.0.CO;2-Z. ISSN 1099-095X.
5. Carroll, Raymond J.; Lin, Xihong (2001-12-01). "Semiparametric regression for clustered data". Biometrika. 88 (4): 1179–1185. doi:10.1093/biomet/88.4.1179. ISSN 0006-3444.
6. Liang, Hua (2006-02-10). "Estimation in Partially Linear Models and Numerical Comparisons". Computational Statistics & Data Analysis. 50 (3): 675–687. doi:10.1016/j.csda.2004.10.007. ISSN 0167-9473. PMC 2824448. PMID 20174596.
7. Liang, Hua (2006-02-10). "Estimation in Partially Linear Models and Numerical Comparisons". Computational Statistics & Data Analysis. 50 (3): 675–687. doi:10.1016/j.csda.2004.10.007. ISSN 0167-9473. PMC 2824448. PMID 20174596.
8. Brumback, Babette A.; Ruppert, David; Wand, M. P. (1999). "Variable Selection and Function Estimation in Additive Nonparametric Regression Using a Data-Based Prior: Comment". Journal of the American Statistical Association. 94 (447): 794–797. doi:10.2307/2669991. ISSN 0162-1459. JSTOR 2669991.