Regression Examples

Example : alpha=1 alpha=0 One-Variable Regression Example From J. Johnston (1984) p19. With Intercept No Intercept Two-Variable Regression Example From J. Johnston (1984) p178. With Intercept No Intercept Three-Independent Variables Regression Example With Intercept No Intercept Three-variables, 20 observations With Intercept No Intercept Three-variables, 10 observations With Intercept No Intercept

         



EXAMPLE:   Three-Independent Variables Regression Example
     |     
No Intercept





I. Data and Summary Stats 
Three-Independent Variables Regression Example
Observations :  n=10
Independent Variables :  k=3
No Intercept


Data Table 

 obs
yi
xi,1
xi,2
xi,3
1. 
5
4
5
4
2. 
4
4
5
3
3. 
9
4
9
8
4. 
3
5
8
7
5. 
5
5
5
9
6. 
5
8
10
8
7. 
8
9
7
13
8. 
5
5
14
14
9. 
5
14
6
12
10. 
12
9
9
9

sum 
61
67
78
87
mean 
6.100
6.700
7.800
8.700
StD ≡ σ 
2.726
3.268
2.860
3.592





Means and Standard Deviations 



 Mean 
 Var 
 StD 


 Mx= Σxi/n   
 Varx≡σx2 = Σ(x-Mx)2 / n-1 
 StDx≡σx=Varx1/2

y
6.100
7.433
2.726

x1
6.700
 10.68
 3.268

x2
7.800
 8.178
 2.860

x3
8.700
 12.90
 3.592



Covariance Matrix  -- Cov(xi,xj)=Σ[(xi-Mxi)(xj-Mxj)] / n-1 
     NOTE: be careful of MS Excel's COVAR() function,
     which divides by n instead of n-1.

  y
  x1
  x2
  x3
y
7.433
2.033
1.467
2.367
x1
2.033
10.68
-0.4000
6.344
x2
1.467
-0.4000
8.178
5.489
x3
2.367
6.344
5.489
12.90



Correlation Matrix  -- Corr(xi,xj)=Σ[(xi-Mxi)(xj-Mxj)] / (n-1)σiσj

  y
  x1
  x2
  x3
y
 1.000
 0.228
 0.188
 0.242
x1
 0.228
 1.000
 -0.043
 0.541
x2
 0.188
 -0.043
 1.000
 0.534
x3
 0.242
 0.541
 0.534
 1.000



The basic input matrices are: 



 
  y =  
 (10x1)   

5
4
9
3
5
5
8
5
5
12


   


  X = 
 (10x3)

4
5
4
4
5
3
4
9
8
5
8
7
5
5
9
8
10
8
9
7
13
5
14
14
14
6
12
9
9
9



   

 

   X' = 
 (3x10)   


4
4
4
5
5
8
9
5
14
9
5
5
9
8
5
10
7
14
6
9
4
3
8
7
9
8
13
14
12
9









II. Regression Calculations

yi =  b1 xi,1 +  b2 xi,2 +  b3 xi,3 +  ui

The q.c.e. basic equation in matrix form is: 
  y = Xb + e
  where y (dependent variable) is (nx1) or  (10x1)
        X (independent vars) is (nxk) or  (10x3)
        b (betas) is (kx1) or  (3x1)
        e (errors) is (nx1) or  (10x1)
Minimizing sum or squared errors using calculus results in the OLS eqn:
  b=(X'X)-1.X'y
To minimize the sum of squared errors of
a k dimensional line that describes the relationship 
between the k independent variables and y we
find the set of slopes (betas) that minimizes
Σi=1 to nei2
Re-written in linear algebra we seek to min e'e
Rearranging the regression model equation, we get e = y - Xb
So e'e = (y-Xb)'(y-Xb) = y'y - 2b'X'y + b'X'Xb   (see Judge et al (1985) p14 )
Differentiating by b we get 0 = - 2X'y + X'Xb -> 2X'Xb=2X'y
Rearranging, dividing both sides by 2 -> b = X'X-1X'y
So to obtain the elements of the (kx1) vector b we need the elements 
of the (kxk) matrix X'X-1 and of the (kx1) matrix X'y. 
Caclulating X'y is easy (see (1) below) but X'X-1 requires 
first calculation of X'X then finding cofactors -- see (4) -- and 
the deteminant - see (3) - in order to invert.



(1) X'y Matrix  (3x1)


427
489
552



(2) X'X Matrix  (3x3)



545	
519	
640	

519	
682	
728	

640	
728	
873	


(3) Determinant   Det(X'X)≡|X'X|  
    i.e. the determinant of matrix of X'X 

Det(X'X) = 4769697
Det(X'X) =    545*682*873 - 545*728*728
        ... - 519*519*873 + 519*728*640
        ... + 640*519*728 - 640*682*640
        

(4) Cofactors(X'X) i.e. cofactor matrix of X 'X  (3x3)

65402
12833
-58648
12833
66185
-64600
-58648
-64600
102329


(5) Adj(X'X) i.e. adjugate matrix of X'X, this is just the 
    transpose of the cofactor matrix.  (3x3)
    For a symmetric matrix, will be same as cofactor matrix.

65402
12833
-58648
12833
66185
-64600
-58648
-64600
102329



(6) Inverse Matrix, inv(X'X)≡(X'X)-1
    = adj(X'X)/|X'X| = adj(X'X)/4769697   (3x3)

0.01371
0.002691
-0.01230
0.002691
0.01388
-0.01354
-0.01230
-0.01354
0.02145


(7) Beta Matrix (β)
    b = [X'X-1].[X'y] , this is  (3x1).
    Finally we can calculate b through matrix multiplication.

 Betas 

 β 1 
 0.3833 
 β 2 
 0.4581 
 β 3 
 -0.03071 

   =    
X'X-1 

 0.01371 
 0.002691 
 -0.01230 
 0.002691 
 0.01388 
 -0.01354 
 -0.01230 
 -0.01354 
 0.02145 

   X   
 X'y 
 427 
 489 
 552 



Yhat1= + 0.3833x4 + 0.4581x5 + -0.03071x4 = 3.7009
Yhat2= + 0.3833x4 + 0.4581x5 + -0.03071x3 = 3.7316
Yhat3= + 0.3833x4 + 0.4581x9 + -0.03071x8 = 5.4104
Yhat4= + 0.3833x5 + 0.4581x8 + -0.03071x7 = 5.3663
Yhat5= + 0.3833x5 + 0.4581x5 + -0.03071x9 = 3.9306
Yhat6= + 0.3833x8 + 0.4581x10 + -0.03071x8 = 7.4017
Yhat7= + 0.3833x9 + 0.4581x7 + -0.03071x13 = 6.2572
Yhat8= + 0.3833x5 + 0.4581x14 + -0.03071x14 = 7.8999
Yhat9= + 0.3833x14 + 0.4581x6 + -0.03071x12 = 7.7464
Yhat10= + 0.3833x9 + 0.4581x9 + -0.03071x9 = 7.2962


ESS
=(3.701 - 6.100)^2
=(3.732 - 6.100)^2
=(5.410 - 6.100)^2
=(5.366 - 6.100)^2
=(3.931 - 6.100)^2
=(7.402 - 6.100)^2
=(6.257 - 6.100)^2
=(7.900 - 6.100)^2
=(7.746 - 6.100)^2
=(7.296 - 6.100)^2
=26.1857781112721

REPORT 


 obs 
 calculation of yhatobs
yhatobs = Σβixi,obs
 yhatobs
a
 yobs
 (data)
 Meany
 (yobs - yhatobs)2
 (yhatobs - My)2
 (yobs - My)2
a
eobs=yobs-yhatobs
eobs2

1
Yhat1 = Σβixi,1 = 0.3833x4 + 0.4581x5 + -0.03071x4
 = 3.701
5
6.100
1.688
5.756
1.210
 e1 = 5 - 3.701 = 1.299
1.688

2
Yhat2 = Σβixi,2 = 0.3833x4 + 0.4581x5 + -0.03071x3
 = 3.732
4
6.100
0.07205
5.609
4.410
 e2 = 4 - 3.732 = 0.2684
0.07205

3
Yhat3 = Σβixi,3 = 0.3833x4 + 0.4581x9 + -0.03071x8
 = 5.410
9
6.100
12.89
0.4756
8.410
 e3 = 9 - 5.410 = 3.590
12.89

4
Yhat4 = Σβixi,4 = 0.3833x5 + 0.4581x8 + -0.03071x7
 = 5.366
3
6.100
5.599
0.5383
9.610
 e4 = 3 - 5.366 = -2.366
5.599

5
Yhat5 = Σβixi,5 = 0.3833x5 + 0.4581x5 + -0.03071x9
 = 3.931
5
6.100
1.144
4.706
1.210
 e5 = 5 - 3.931 = 1.069
1.144

6
Yhat6 = Σβixi,6 = 0.3833x8 + 0.4581x10 + -0.03071x8
 = 7.402
5
6.100
5.768
1.695
1.210
 e6 = 5 - 7.402 = -2.402
5.768

7
Yhat7 = Σβixi,7 = 0.3833x9 + 0.4581x7 + -0.03071x13
 = 6.257
8
6.100
3.037
0.02472
3.610
 e7 = 8 - 6.257 = 1.743
3.037

8
Yhat8 = Σβixi,8 = 0.3833x5 + 0.4581x14 + -0.03071x14
 = 7.900
5
6.100
8.409
3.240
1.210
 e8 = 5 - 7.900 = -2.900
8.409

9
Yhat9 = Σβixi,9 = 0.3833x14 + 0.4581x6 + -0.03071x12
 = 7.746
5
6.100
7.543
2.711
1.210
 e9 = 5 - 7.746 = -2.746
7.543

10
Yhat10 = Σβixi,10 = 0.3833x9 + 0.4581x9 + -0.03071x9
 = 7.296
12
6.100
22.13
1.431
34.81
 e10 = 12 - 7.296 = 4.704
22.13
RSS = 
Σ(yobs - yhatobs)2
ESS = 
Σ(yhatobs - My)2
TSS = 
Σ(yobs - My)2
e'e=Σeobs2
sum->
68.27
26.19
66.90
68.27




(11) Betas and their t-Stats
     from the covar matrix of b=σ2(X'X)-1
     the var(βi) = σ2vii where vii is the ith diag element of X'X-1
     where σ2 = e'e / n-k  (k=num of ind vars plus 1 for the intercept if present).
     and where vii is the ith diag element of X'X-1
     Std(βi) = sqr root of Var(βi) 
     TStat(βi) = βi / Std(βi)
     Estimate of σ2 = 9.75302773800037



 Coef value 
 StD(β)
 tStat(β)

β1 = 
   0.01371 * 427
 + 0.002691 * 489
 + -0.01230 * 552
 = 0.3833
(9.753 * 0.01371)1/2 
 = 0.3657
0.3833 / 0.3657
 = 1.048

β2 = 
   0.002691 * 427
 + 0.01388 * 489
 + -0.01354 * 552
 = 0.4581
(9.753 * 0.01388)1/2 
 = 0.3679
0.4581 / 0.3679
 = 1.245

β3 = 
   -0.01230 * 427
 + -0.01354 * 489
 + 0.02145 * 552
 = -0.03071
(9.753 * 0.02145)1/2 
 = 0.4574
-0.03071 / 0.4574
 = -0.06714


(12) Table of Outputs:
 yobs =
 β1
 X obs,1 + 
 β2
 X obs,2 + 
 β3
 X obs,3 + 
 eobs
 0.3833
 
 0.4581
 
 -0.03071
 
(1.048)
(1.245)
(-0.06714)
 <- tstats 
 r2 = -0.020496    |    adj r2 = -0.530744


(13)  RSS   = Sum{y     - y_hat }^2  = 68.2711941660026
      TSS   = Sum{y     - y_avg }^2  = 66.9
      ESS(a)= Sum{y_hat - y_avg }^2  = 26.1857781112721
           we use the ESSb (below) cuz smthn wrng w ESS when no intercept.
      ESS(b)= TSS-RSS -1.37119416600258
      note:  TSS = ESS + RSS 
(14)  r2 = ESS/TSS = -0.020496175874478
(15)  adjusted r2 = ESS/TSS = -0.530744263811717
(16)  F-stat = [ESS/(k-1)] / [RSS/(n-k)] = -4.68638800461949E-02
               see Johnston(1984) p186
               F measures the joint significance of 
               all explanatory variables.
      Alternatively:  F-stat = r2/(k-1) / (1-r2)/(n-k)
(17)  Durbin-Watson Statistic (DW or d) measures autocorrelation.
      DW = 2.47500522305087
 ________________________________________________________

Note, RSS, ESS and TSS stand for ... 
      Residual Sum of Squares (RSS),
      Explained Sum of Squares (ESS), and 
      Total Sum of Squares (TSS).
      However ESS is sometimes referred to as the Regression Sum of Squares.
      and     RSS is sometimes referred to as the Sum of Squares Rresidual.
Note, an alternative way of calculating TSS, ESS is...
      TSS = y'Ay   
      ESS = bv'Xv'Ay  where bv' Xv' are b' & X' wo intercept row col
      RSS = TSS-ESS 

Bibliography 

J. Johnston (1984) Econometric Methods, 3rd ed.
Judge et al (1985) The Theory and Practice of Econometrics 2rd ed, Wiley, New York.
Donald F. Morrison (1990) Multivariate Statistical Methods, 3rd edition, McGraw Hill, New York. 
A. H. Studenmund (1997) Using Econometrics: A Practical Guide, 3rd edition.  Addison-Wesley, Reading.