E C O N S T A T S     About EconStatsTM     Calendar     Data     Regression Examples     Links  
      EXAMPLE:   Three-Independent Variables Regression Example   |   With Intercept

I. Data and Summary Stats 
Three-Independent Variables Regression Example
Observations :  n=10
Independent Variables :  k=3
With Intercept 


Data Table 
obsyi intcptxi,1xi,2xi,3
1. 51454
2. 41453
3. 91498
4. 31587
5. 51559
6. 518108
7. 819713
8. 5151414
9. 5114612
10. 121999
sum 6110677887
mean 6.10016.7007.8008.700
StD ≡ σ 2.7260 3.268 2.860 3.592
Means and Standard Deviations
 Mean 
 Var 
 StD 
 Mx= Σxi/n   
 Varx≡σx2 = Σ(x-Mx)2 / n-1 
 StDx≡σx=Varx1/2
y 6.100 7.433 2.726
x1 6.700 10.68 3.268
x2 7.800 8.178 2.860
x3 8.700 12.90 3.592
Covariance Matrix -- Cov(xi,xj)=Σ[(xi-Mxi)(xj-Mxj)] / n-1
NOTE: be careful of MS Excel's COVAR() function,
which divides by n instead of n-1.
y x1 x2 x3
y7.4332.0331.4672.367
x12.03310.68-0.40006.344
x21.467-0.40008.1785.489
x32.3676.3445.48912.90
Correlation Matrix -- Corr(xi,xj)=Σ[(xi-Mxi)(xj-Mxj)] / (n-1)σiσj
y x1 x2 x3
y 1.000 0.228 0.188 0.242
x1 0.228 1.000 -0.043 0.541
x2 0.188 -0.043 1.000 0.534
x3 0.242 0.541 0.534 1.000
The basic input matrices are:
  y =  
(10x1)  
5
4
9
3
5
5
8
5
5
12
 
  X = 
(10x4)
1454
1453
1498
1587
1559
18108
19713
151414
114612
1999
 
   X' = 
(4x10)  
1111111111
44455895149
559851071469
4387981314129



II. Regression Calculations yi = alpha + b1 xi,1 + b2 xi,2 + b3 xi,3 + ui The q.c.e. basic equation in matrix form is: y = Xb + e where y (dependent variable) is (nx1) or (10x1) X (independent vars) is (nxk) or (10x4) b (betas) is (kx1) or (4x1) e (errors) is (nx1) or (10x1) Minimizing sum or squared errors using calculus results in the OLS eqn:
b=(X'X)-1.X'y To minimize the sum of squared errors of a k dimensional line that describes the relationship between the k independent variables and y we find the set of slopes (betas) that minimizes Σi=1 to nei2 Re-written in linear algebra we seek to min e'e Rearranging the regression model equation, we get e = y - Xb So e'e = (y-Xb)'(y-Xb) = y'y - 2b'X'y + b'X'Xb (see Judge et al (1985) p14 ) Differentiating by b we get 0 = - 2X'y + X'Xb -> 2X'Xb=2X'y Rearranging, dividing both sides by 2 -> b = X'X-1X'y So to obtain the elements of the (kx1) vector b we need the elements of the (kxk) matrix X'X-1 and of the (kx1) matrix X'y. Caclulating X'y is easy (see (1) below) but X'X-1 requires first calculation of X'X then finding cofactors -- see (4) -- and the deteminant - see (3) - in order to invert.
(1) X'y Matrix (4x1)
    61
    427
    489
    552

(2) X'X Matrix (4x4)
    10 67 78 87
    67 545 519 640
    78 519 682 728
    87 640 728 873
(3) Determinant Det(X'X)≡|X'X|
i.e. the determinant of matrix of X'X Det(X'X) = 3248719 Det(X'X) = 10*545*682*873 - 10*545*728*728 - 10*519*519*873 ... + 10*519*728*640 + 10*640*519*728 - 10*640*682*640 ... - 67*67*682*873 + 67*67*728*728 + 67*519*78*873 ... - 67*519*728*87 - 67*640*78*728 + 67*640*682*87 ... + 78*67*519*873 - 78*67*728*640 - 78*545*78*873 ... + 78*545*728*87 + 78*640*78*640 - 78*640*519*87 ... - 87*67*519*728 + 87*67*682*640 + 87*545*78*728 ... - 87*545*682*87 - 87*519*78*640 + 87*519*519*87 (4) Cofactors(X'X) i.e. cofactor matrix of X 'X (4x4)
    4769697
    -280532
    -402041
    65593
    
    -280532
    61046
    32387
    -43804
    
    -402041
    32387
    78968
    -49529
    
    65593
    -43804
    -49529
    70600
    
(5) Adj(X'X) i.e. adjugate matrix of X'X, this is just the
transpose of the cofactor matrix. (4x4)
For a symmetric matrix, will be same as cofactor matrix.
    4769697
    -280532
    -402041
    65593
    
    -280532
    61046
    32387
    -43804
    
    -402041
    32387
    78968
    -49529
    
    65593
    -43804
    -49529
    70600
    

(6) Inverse Matrix, inv(X'X)≡(X'X)-1
= adj(X'X)/|X'X| = adj(X'X)/3248719 (4x4)
    1.468
    -0.08635
    -0.1238
    0.02019
    
    -0.08635
    0.01879
    0.009969
    -0.01348
    
    -0.1238
    0.009969
    0.02431
    -0.01525
    
    0.02019
    -0.01348
    -0.01525
    0.02173
    
(7) Beta Matrix (β) b = [X'X-1].[X'y] , this is (4x1). Finally we can calculate b through matrix multiplication.
    Betas
    alpha 3.316
    β 1 0.1883
    β 2 0.1786
    β 3 0.01489
  =   X'X-1
1.468 -0.08635 -0.1238 0.02019
-0.08635 0.01879 0.009969 -0.01348
-0.1238 0.009969 0.02431 -0.01525
0.02019 -0.01348 -0.01525 0.02173
  X   X'y
61
427
489
552

Yhat1= + 3.316x1 + 0.1883x4 + 0.1786x5 + 0.01489x4 = 5.0217
Yhat2= + 3.316x1 + 0.1883x4 + 0.1786x5 + 0.01489x3 = 5.0068
Yhat3= + 3.316x1 + 0.1883x4 + 0.1786x9 + 0.01489x8 = 5.7955
Yhat4= + 3.316x1 + 0.1883x5 + 0.1786x8 + 0.01489x7 = 5.7903
Yhat5= + 3.316x1 + 0.1883x5 + 0.1786x5 + 0.01489x9 = 5.2844
Yhat6= + 3.316x1 + 0.1883x8 + 0.1786x10 + 0.01489x8 = 6.7272
Yhat7= + 3.316x1 + 0.1883x9 + 0.1786x7 + 0.01489x13 = 6.4542
Yhat8= + 3.316x1 + 0.1883x5 + 0.1786x14 + 0.01489x14 = 6.9660
Yhat9= + 3.316x1 + 0.1883x14 + 0.1786x6 + 0.01489x12 = 7.2021
Yhat10= + 3.316x1 + 0.1883x9 + 0.1786x9 + 0.01489x9 = 6.7518


ESS
=(5.022 - 6.100)^2
=(5.007 - 6.100)^2
=(5.796 - 6.100)^2
=(5.790 - 6.100)^2
=(5.284 - 6.100)^2
=(6.727 - 6.100)^2
=(6.454 - 6.100)^2
=(6.966 - 6.100)^2
=(7.202 - 6.100)^2
=(6.752 - 6.100)^2
=6.11949851618433

REPORT 
obs calculation of yhatobs
yhatobs = Σβixi,obs
yhatobs a yobs
(data)
Meany (yobs - yhatobs)2 (yhatobs - My)2 (yobs - My)2 a eobs=yobs-yhatobs eobs2
1 Yhat1 = Σβixi,13.316x1 = 0.1883x4 + 0.1786x5 + 0.01489x4 = 5.02256.1000.0004711.1631.210 e1 = 5 - 5.022 = -0.021710.000471
2 Yhat2 = Σβixi,2 + 3.316x1 = 0.1883x4 + 0.1786x5 + 0.01489x3 = 5.00746.1001.0141.1954.410 e2 = 4 - 5.007 = -1.0071.014
3 Yhat3 = Σβixi,3 + 3.316x1 = 0.1883x4 + 0.1786x9 + 0.01489x8 = 5.79696.10010.270.092708.410 e3 = 9 - 5.796 = 3.20410.27
4 Yhat4 = Σβixi,4 + 3.316x1 = 0.1883x5 + 0.1786x8 + 0.01489x7 = 5.79036.1007.7860.095899.610 e4 = 3 - 5.790 = -2.7907.786
5 Yhat5 = Σβixi,5 + 3.316x1 = 0.1883x5 + 0.1786x5 + 0.01489x9 = 5.28456.1000.080910.66511.210 e5 = 5 - 5.284 = -0.28440.08091
6 Yhat6 = Σβixi,6 + 3.316x1 = 0.1883x8 + 0.1786x10 + 0.01489x8 = 6.72756.1002.9830.39331.210 e6 = 5 - 6.727 = -1.7272.983
7 Yhat7 = Σβixi,7 + 3.316x1 = 0.1883x9 + 0.1786x7 + 0.01489x13 = 6.45486.1002.3890.12553.610 e7 = 8 - 6.454 = 1.5462.389
8 Yhat8 = Σβixi,8 + 3.316x1 = 0.1883x5 + 0.1786x14 + 0.01489x14 = 6.96656.1003.8650.74991.210 e8 = 5 - 6.966 = -1.9663.865
9 Yhat9 = Σβixi,9 + 3.316x1 = 0.1883x14 + 0.1786x6 + 0.01489x12 = 7.20256.1004.8491.2151.210 e9 = 5 - 7.202 = -2.2024.849
10 Yhat10 = Σβixi,10 + 3.316x1 = 0.1883x9 + 0.1786x9 + 0.01489x9 = 6.752126.10027.540.424834.81 e10 = 12 - 6.752 = 5.24827.54
RSS =
Σ(yobs - yhatobs)2
ESS =
Σ(yhatobs - My)2
TSS =
Σ(yobs - My)2
e'e=Σeobs2
sum->60.786.11966.9060.78


(11) Betas and their t-Stats
     from the covar matrix of b=σ2(X'X)-1
     the var(βi) = σ2vii where vii is the ith diag element of X'X-1
     where σ2 = e'e / n-k  (k=num of ind vars plus 1 for the intercept if present).
     and where vii is the ith diag element of X'X-1
     Std(βi) = sqr root of Var(βi) 
     TStat(βi) = βi / Std(βi)
     Estimate of σ2 = 10.1300835806359
    Coef value StD(β) tStat(β)
    alpha = 1.468 * 61
    + -0.08635 * 427
    + -0.1238 * 489
    + 0.02019 * 552
    = 3.316
    (10.13 * 1.468)1/2
    = 3.857
    3.316 / 3.857
    = 0.8599
    β1 = -0.08635 * 61
    + 0.01879 * 427
    + 0.009969 * 489
    + -0.01348 * 552
    = 0.1883
    (10.13 * 0.01879)1/2
    = 0.4363
    0.1883 / 0.4363
    = 0.4315
    β2 = -0.1238 * 61
    + 0.009969 * 427
    + 0.02431 * 489
    + -0.01525 * 552
    = 0.1786
    (10.13 * 0.02431)1/2
    = 0.4962
    0.1786 / 0.4962
    = 0.3598
    β3 = 0.02019 * 61
    + -0.01348 * 427
    + -0.01525 * 489
    + 0.02173 * 552
    = 0.01489
    (10.13 * 0.02173)1/2
    = 0.4692
    0.01489 / 0.4692
    = 0.03174
(12) Table of Outputs:
    yobs = alpha + β1 X obs,1 + β2 X obs,2 + β3 X obs,3 + eobs
    3.316 0.1883 0.1786 0.01489
    (0.8599)(0.4315)(0.3598)(0.03174) <- tstats
    r2 = 0.091472 | adj r2 = -0.362792
(13) RSS = Sum{y - y_hat }^2 = 60.7805014838156 TSS = Sum{y - y_avg }^2 = 66.9 ESS(a)= Sum{y_hat - y_avg }^2 = 6.11949851618433 we use the ESSb (below) cuz smthn wrng w ESS when no intercept. ESS(b)= TSS-RSS 6.11949851618439 note: TSS = ESS + RSS (14) r2 = ESS/TSS = 9.14723246066426E-02 (15) adjusted r2 = ESS/TSS = -0.362791513090036 (16) F-stat = [ESS/(k-1)] / [RSS/(n-k)] = 0.201363870543709 see Johnston(1984) p186 F measures the joint significance of all explanatory variables. Alternatively: F-stat = r2/(k-1) / (1-r2)/(n-k) (17) Durbin-Watson Statistic (DW or d) measures autocorrelation. DW = 2.35318810450481 ________________________________________________________ Note, RSS, ESS and TSS stand for ... Residual Sum of Squares (RSS), Explained Sum of Squares (ESS), and Total Sum of Squares (TSS). However ESS is sometimes referred to as the Regression Sum of Squares. and RSS is sometimes referred to as the Sum of Squares Rresidual. Note, an alternative way of calculating TSS, ESS is... TSS = y'Ay ESS = bv'Xv'Ay where bv' Xv' are b' & X' wo intercept row col RSS = TSS-ESS Bibliography J. Johnston (1984) Econometric Methods, 3rd ed. Judge et al (1985) The Theory and Practice of Econometrics 2rd ed, Wiley, New York. Donald F. Morrison (1990) Multivariate Statistical Methods, 3rd edition, McGraw Hill, New York. A. H. Studenmund (1997) Using Econometrics: A Practical Guide, 3rd edition. Addison-Wesley, Reading.