E C O N S T A T S     About EconStatsTM     Calendar     Data     Regression Examples     Links  
      EXAMPLE:   Three-variables, 20 observations   |   With Intercept

I. Data and Summary Stats 
Three-variables, 20 observations
Observations :  n=20
Independent Variables :  k=3
With Intercept 


Data Table 
obsyi intcptxi,1xi,2xi,3
1. 51454
2. 41453
3. 91498
4. 31587
5. 51559
6. 518108
7. 819713
8. 5151414
9. 5114612
10. 121999
11. 416127
12. 81957
13. 101111119
14. 1618614
15. 2191214
16. 9112717
17. 911295
18. 315810
19. 11112107
20. 211172223
sum 15420168180210
mean 7.70018.400910.50
StD ≡ σ 4.7360 3.719 4.039 5.186
Means and Standard Deviations
 Mean 
 Var 
 StD 
 Mx= Σxi/n   
 Varx≡σx2 = Σ(x-Mx)2 / n-1 
 StDx≡σx=Varx1/2
y 7.700 22.43 4.736
x1 8.400 13.83 3.719
x2 9 16.32 4.039
x3 10.50 26.89 5.186
Covariance Matrix -- Cov(xi,xj)=Σ[(xi-Mxi)(xj-Mxj)] / n-1
NOTE: be careful of MS Excel's COVAR() function,
which divides by n instead of n-1.
y x1 x2 x3
y22.4310.818.47413.11
x110.8113.836.68411.42
x28.4746.68416.3212.53
x313.1111.4212.5326.89
Correlation Matrix -- Corr(xi,xj)=Σ[(xi-Mxi)(xj-Mxj)] / (n-1)σiσj
y x1 x2 x3
y 1.000 0.614 0.443 0.534
x1 0.614 1.000 0.445 0.592
x2 0.443 0.445 1.000 0.598
x3 0.534 0.592 0.598 1.000
The basic input matrices are:
  y =  
(20x1)  
5
4
9
3
5
5
8
5
5
12
4
8
10
16
2
9
9
3
11
21
 
  X = 
(20x4)
1454
1453
1498
1587
1559
18108
19713
151414
114612
1999
16127
1957
1111119
18614
191214
112717
11295
15810
112107
1172223
 
   X' = 
(4x20)  
11111111111111111111
44455895149691189121251217
559851071469125116127981022
43879813141297719141417510723



II. Regression Calculations yi = alpha + b1 xi,1 + b2 xi,2 + b3 xi,3 + ui The q.c.e. basic equation in matrix form is: y = Xb + e where y (dependent variable) is (nx1) or (20x1) X (independent vars) is (nxk) or (20x4) b (betas) is (kx1) or (4x1) e (errors) is (nx1) or (20x1) Minimizing sum or squared errors using calculus results in the OLS eqn:
b=(X'X)-1.X'y To minimize the sum of squared errors of a k dimensional line that describes the relationship between the k independent variables and y we find the set of slopes (betas) that minimizes Σi=1 to nei2 Re-written in linear algebra we seek to min e'e Rearranging the regression model equation, we get e = y - Xb So e'e = (y-Xb)'(y-Xb) = y'y - 2b'X'y + b'X'Xb (see Judge et al (1985) p14 ) Differentiating by b we get 0 = - 2X'y + X'Xb -> 2X'Xb=2X'y Rearranging, dividing both sides by 2 -> b = X'X-1X'y So to obtain the elements of the (kx1) vector b we need the elements of the (kxk) matrix X'X-1 and of the (kx1) matrix X'y. Caclulating X'y is easy (see (1) below) but X'X-1 requires first calculation of X'X then finding cofactors -- see (4) -- and the deteminant - see (3) - in order to invert.
(1) X'y Matrix (4x1)
    154
    1499
    1547
    1866

(2) X'X Matrix (4x4)
    20 168 180 210
    168 1674 1639 1981
    180 1639 1930 2128
    210 1981 2128 2716
(3) Determinant Det(X'X)≡|X'X|
i.e. the determinant of matrix of X'X Det(X'X) = 340453596 Det(X'X) = 20*1674*1930*2716 - 20*1674*2128*2128 - 20*1639*1639*2716 ... + 20*1639*2128*1981 + 20*1981*1639*2128 - 20*1981*1930*1981 ... - 168*168*1930*2716 + 168*168*2128*2128 + 168*1639*180*2716 ... - 168*1639*2128*210 - 168*1981*180*2128 + 168*1981*1930*210 ... + 180*168*1639*2716 - 180*168*2128*1981 - 180*1674*180*2716 ... + 180*1674*2128*210 + 180*1981*180*1981 - 180*1981*1639*210 ... - 210*168*1639*2128 + 210*168*1930*1981 + 210*1674*180*2128 ... - 210*1674*1930*210 - 210*1639*180*1981 + 210*1639*1639*210 (4) Cofactors(X'X) i.e. cofactor matrix of X 'X (4x4)
    142959642
    -6932268
    -6122802
    -1200066
    
    -6932268
    2035320
    -265020
    -740880
    
    -6122802
    -265020
    1744036
    -699748
    
    -1200066
    -740880
    -699748
    1306780
    
(5) Adj(X'X) i.e. adjugate matrix of X'X, this is just the
transpose of the cofactor matrix. (4x4)
For a symmetric matrix, will be same as cofactor matrix.
    142959642
    -6932268
    -6122802
    -1200066
    
    -6932268
    2035320
    -265020
    -740880
    
    -6122802
    -265020
    1744036
    -699748
    
    -1200066
    -740880
    -699748
    1306780
    

(6) Inverse Matrix, inv(X'X)≡(X'X)-1
= adj(X'X)/|X'X| = adj(X'X)/340453596 (4x4)
    0.4199
    -0.02036
    -0.01798
    -0.003525
    
    -0.02036
    0.005978
    -0.000778
    -0.002176
    
    -0.01798
    -0.000778
    0.005123
    -0.002055
    
    -0.003525
    -0.002176
    -0.002055
    0.003838
    
(7) Beta Matrix (β) b = [X'X-1].[X'y] , this is (4x1). Finally we can calculate b through matrix multiplication.
    Betas
    alpha -0.2555
    β 1 0.5607
    β 2 0.1531
    β 3 0.1779
  =   X'X-1
0.4199 -0.02036 -0.01798 -0.003525
-0.02036 0.005978 -0.000778 -0.002176
-0.01798 -0.000778 0.005123 -0.002055
-0.003525 -0.002176 -0.002055 0.003838
  X   X'y
154
1499
1547
1866

Yhat1= + -0.2555x1 + 0.5607x4 + 0.1531x5 + 0.1779x4 = 3.4643
Yhat2= + -0.2555x1 + 0.5607x4 + 0.1531x5 + 0.1779x3 = 3.2865
Yhat3= + -0.2555x1 + 0.5607x4 + 0.1531x9 + 0.1779x8 = 4.7881
Yhat4= + -0.2555x1 + 0.5607x5 + 0.1531x8 + 0.1779x7 = 5.0179
Yhat5= + -0.2555x1 + 0.5607x5 + 0.1531x5 + 0.1779x9 = 4.9144
Yhat6= + -0.2555x1 + 0.5607x8 + 0.1531x10 + 0.1779x8 = 7.1841
Yhat7= + -0.2555x1 + 0.5607x9 + 0.1531x7 + 0.1779x13 = 8.1749
Yhat8= + -0.2555x1 + 0.5607x5 + 0.1531x14 + 0.1779x14 = 7.1814
Yhat9= + -0.2555x1 + 0.5607x14 + 0.1531x6 + 0.1779x12 = 10.6477
Yhat10= + -0.2555x1 + 0.5607x9 + 0.1531x9 + 0.1779x9 = 7.7697
Yhat11= + -0.2555x1 + 0.5607x6 + 0.1531x12 + 0.1779x7 = 6.1910
Yhat12= + -0.2555x1 + 0.5607x9 + 0.1531x5 + 0.1779x7 = 6.8016
Yhat13= + -0.2555x1 + 0.5607x11 + 0.1531x11 + 0.1779x19 = 10.9759
Yhat14= + -0.2555x1 + 0.5607x8 + 0.1531x6 + 0.1779x14 = 7.6390
Yhat15= + -0.2555x1 + 0.5607x9 + 0.1531x12 + 0.1779x14 = 9.1182
Yhat16= + -0.2555x1 + 0.5607x12 + 0.1531x7 + 0.1779x17 = 10.5686
Yhat17= + -0.2555x1 + 0.5607x12 + 0.1531x9 + 0.1779x5 = 8.7405
Yhat18= + -0.2555x1 + 0.5607x5 + 0.1531x8 + 0.1779x10 = 5.5515
Yhat19= + -0.2555x1 + 0.5607x12 + 0.1531x10 + 0.1779x7 = 9.2493
Yhat20= + -0.2555x1 + 0.5607x17 + 0.1531x22 + 0.1779x23 = 16.7357


ESS
=(3.464 - 7.700)^2
=(3.286 - 7.700)^2
=(4.788 - 7.700)^2
=(5.018 - 7.700)^2
=(4.914 - 7.700)^2
=(7.184 - 7.700)^2
=(8.175 - 7.700)^2
=(7.181 - 7.700)^2
=(10.65 - 7.700)^2
=(7.770 - 7.700)^2
=(6.191 - 7.700)^2
=(6.802 - 7.700)^2
=(10.98 - 7.700)^2
=(7.639 - 7.700)^2
=(9.118 - 7.700)^2
=(10.57 - 7.700)^2
=(8.740 - 7.700)^2
=(5.551 - 7.700)^2
=(9.249 - 7.700)^2
=(16.74 - 7.700)^2
=184.109503872592

REPORT 
obs calculation of yhatobs
yhatobs = Σβixi,obs
yhatobs a yobs
(data)
Meany (yobs - yhatobs)2 (yhatobs - My)2 (yobs - My)2 a eobs=yobs-yhatobs eobs2
1 Yhat1 = Σβixi,1-0.2555x1 = 0.5607x4 + 0.1531x5 + 0.1779x4 = 3.46457.7002.35817.947.290 e1 = 5 - 3.464 = 1.5362.358
2 Yhat2 = Σβixi,2 + -0.2555x1 = 0.5607x4 + 0.1531x5 + 0.1779x3 = 3.28647.7000.509119.4813.69 e2 = 4 - 3.286 = 0.71350.5091
3 Yhat3 = Σβixi,3 + -0.2555x1 = 0.5607x4 + 0.1531x9 + 0.1779x8 = 4.78897.70017.748.4791.690 e3 = 9 - 4.788 = 4.21217.74
4 Yhat4 = Σβixi,4 + -0.2555x1 = 0.5607x5 + 0.1531x8 + 0.1779x7 = 5.01837.7004.0727.19422.09 e4 = 3 - 5.018 = -2.0184.072
5 Yhat5 = Σβixi,5 + -0.2555x1 = 0.5607x5 + 0.1531x5 + 0.1779x9 = 4.91457.7000.0073357.7607.290 e5 = 5 - 4.914 = 0.085650.007335
6 Yhat6 = Σβixi,6 + -0.2555x1 = 0.5607x8 + 0.1531x10 + 0.1779x8 = 7.18457.7004.7700.26617.290 e6 = 5 - 7.184 = -2.1844.770
7 Yhat7 = Σβixi,7 + -0.2555x1 = 0.5607x9 + 0.1531x7 + 0.1779x13 = 8.17587.7000.030600.22560.09000 e7 = 8 - 8.175 = -0.17490.03060
8 Yhat8 = Σβixi,8 + -0.2555x1 = 0.5607x5 + 0.1531x14 + 0.1779x14 = 7.18157.7004.7580.26907.290 e8 = 5 - 7.181 = -2.1814.758
9 Yhat9 = Σβixi,9 + -0.2555x1 = 0.5607x14 + 0.1531x6 + 0.1779x12 = 10.6557.70031.908.6897.290 e9 = 5 - 10.65 = -5.64831.90
10 Yhat10 = Σβixi,10 + -0.2555x1 = 0.5607x9 + 0.1531x9 + 0.1779x9 = 7.770127.70017.900.00485318.49 e10 = 12 - 7.770 = 4.23017.90
11 Yhat11 = Σβixi,11 + -0.2555x1 = 0.5607x6 + 0.1531x12 + 0.1779x7 = 6.19147.7004.8002.27713.69 e11 = 4 - 6.191 = -2.1914.800
12 Yhat12 = Σβixi,12 + -0.2555x1 = 0.5607x9 + 0.1531x5 + 0.1779x7 = 6.80287.7001.4360.80710.09000 e12 = 8 - 6.802 = 1.1981.436
13 Yhat13 = Σβixi,13 + -0.2555x1 = 0.5607x11 + 0.1531x11 + 0.1779x19 = 10.98107.7000.952310.735.290 e13 = 10 - 10.98 = -0.97590.9523
14 Yhat14 = Σβixi,14 + -0.2555x1 = 0.5607x8 + 0.1531x6 + 0.1779x14 = 7.639167.70069.910.00372768.89 e14 = 16 - 7.639 = 8.36169.91
15 Yhat15 = Σβixi,15 + -0.2555x1 = 0.5607x9 + 0.1531x12 + 0.1779x14 = 9.11827.70050.672.01132.49 e15 = 2 - 9.118 = -7.11850.67
16 Yhat16 = Σβixi,16 + -0.2555x1 = 0.5607x12 + 0.1531x7 + 0.1779x17 = 10.5797.7002.4608.2291.690 e16 = 9 - 10.57 = -1.5692.460
17 Yhat17 = Σβixi,17 + -0.2555x1 = 0.5607x12 + 0.1531x9 + 0.1779x5 = 8.74097.7000.067361.0831.690 e17 = 9 - 8.740 = 0.25950.06736
18 Yhat18 = Σβixi,18 + -0.2555x1 = 0.5607x5 + 0.1531x8 + 0.1779x10 = 5.55137.7006.5104.61622.09 e18 = 3 - 5.551 = -2.5516.510
19 Yhat19 = Σβixi,19 + -0.2555x1 = 0.5607x12 + 0.1531x10 + 0.1779x7 = 9.249117.7003.0652.40010.89 e19 = 11 - 9.249 = 1.7513.065
20 Yhat20 = Σβixi,20 + -0.2555x1 = 0.5607x17 + 0.1531x22 + 0.1779x23 = 16.74217.70018.1881.64176.9 e20 = 21 - 16.74 = 4.26418.18
RSS =
Σ(yobs - yhatobs)2
ESS =
Σ(yhatobs - My)2
TSS =
Σ(yobs - My)2
e'e=Σeobs2
sum->242.1184.1426.2242.1


(11) Betas and their t-Stats
     from the covar matrix of b=σ2(X'X)-1
     the var(βi) = σ2vii where vii is the ith diag element of X'X-1
     where σ2 = e'e / n-k  (k=num of ind vars plus 1 for the intercept if present).
     and where vii is the ith diag element of X'X-1
     Std(βi) = sqr root of Var(βi) 
     TStat(βi) = βi / Std(βi)
     Estimate of σ2 = 15.130656007963
    Coef value StD(β) tStat(β)
    alpha = 0.4199 * 154
    + -0.02036 * 1499
    + -0.01798 * 1547
    + -0.003525 * 1866
    = -0.2555
    (15.13 * 0.4199)1/2
    = 2.521
    -0.2555 / 2.521
    = -0.1014
    β1 = -0.02036 * 154
    + 0.005978 * 1499
    + -0.000778 * 1547
    + -0.002176 * 1866
    = 0.5607
    (15.13 * 0.005978)1/2
    = 0.3008
    0.5607 / 0.3008
    = 1.864
    β2 = -0.01798 * 154
    + -0.000778 * 1499
    + 0.005123 * 1547
    + -0.002055 * 1866
    = 0.1531
    (15.13 * 0.005123)1/2
    = 0.2784
    0.1531 / 0.2784
    = 0.5499
    β3 = -0.003525 * 154
    + -0.002176 * 1499
    + -0.002055 * 1547
    + 0.003838 * 1866
    = 0.1779
    (15.13 * 0.003838)1/2
    = 0.2410
    0.1779 / 0.2410
    = 0.7380
(12) Table of Outputs:
    yobs = alpha + β1 X obs,1 + β2 X obs,2 + β3 X obs,3 + eobs
    -0.2555 0.5607 0.1531 0.1779
    (-0.1014)(1.864)(0.5499)(0.7380) <- tstats
    r2 = 0.431979 | adj r2 = 0.325475
(13) RSS = Sum{y - y_hat }^2 = 242.090496127408 TSS = Sum{y - y_avg }^2 = 426.2 ESS(a)= Sum{y_hat - y_avg }^2 = 184.109503872592 we use the ESSb (below) cuz smthn wrng w ESS when no intercept. ESS(b)= TSS-RSS 184.109503872592 note: TSS = ESS + RSS (14) r2 = ESS/TSS = 0.431979126871404 (15) adjusted r2 = ESS/TSS = 0.325475213159792 (16) F-stat = [ESS/(k-1)] / [RSS/(n-k)] = 4.05599298483157 see Johnston(1984) p186 F measures the joint significance of all explanatory variables. Alternatively: F-stat = r2/(k-1) / (1-r2)/(n-k) (17) Durbin-Watson Statistic (DW or d) measures autocorrelation. DW = 2.63094204405623 ________________________________________________________ Note, RSS, ESS and TSS stand for ... Residual Sum of Squares (RSS), Explained Sum of Squares (ESS), and Total Sum of Squares (TSS). However ESS is sometimes referred to as the Regression Sum of Squares. and RSS is sometimes referred to as the Sum of Squares Rresidual. Note, an alternative way of calculating TSS, ESS is... TSS = y'Ay ESS = bv'Xv'Ay where bv' Xv' are b' & X' wo intercept row col RSS = TSS-ESS Bibliography J. Johnston (1984) Econometric Methods, 3rd ed. Judge et al (1985) The Theory and Practice of Econometrics 2rd ed, Wiley, New York. Donald F. Morrison (1990) Multivariate Statistical Methods, 3rd edition, McGraw Hill, New York. A. H. Studenmund (1997) Using Econometrics: A Practical Guide, 3rd edition. Addison-Wesley, Reading.