E C O N S T A T S     About EconStatsTM     Calendar     Data     Regression Examples     Links   



         



EXAMPLE:   Three-variables, 10 observations 
     |     
No Intercept






I. Data and Summary Stats 
Three-variables, 10 observations 
Observations :  n=10
Independent Variables :  k=3
No Intercept


Data Table 



 obsyixi,1xi,2xi,3

1. 1731127

2. 8015217090

3. 5614113262

4. 23533145

5. 7399172152

6. 74133108158

7. 39449784

8. 38438698

9. 3813349100

10. 4650114126



sum 468921970942

mean 46.8092.109794.20

StD ≡ σ 24.71
44.43
54.53
42.90






Means and Standard Deviations 






 Mean 

 Var 

 StD 




 Mx= Σxi/n   

 Varx≡σx2 = Σ(x-Mx)2 / n-1 

 StDx≡σx=Varx1/2


y
46.80
610.4
24.71


x1
92.10
 1974
 44.43


x2
97
 2974
 54.53


x3
94.20
 1841
 42.90




Covariance Matrix  -- Cov(xi,xj)=Σ[(xi-Mxi)(xj-Mxj)] / n-1 
     NOTE: be careful of MS Excel's COVAR() function,
     which divides by n instead of n-1.



  y  x1  x2  x3

y610.4668.71225781.4

x1668.719741015399.1

x21225101529741428

x3781.4399.114281841




Correlation Matrix  -- Corr(xi,xj)=Σ[(xi-Mxi)(xj-Mxj)] / (n-1)σiσj



  y  x1  x2  x3

y 1.000 0.609 0.909 0.737

x1 0.609 1.000 0.419 0.209

x2 0.909 0.419 1.000 0.610

x3 0.737 0.209 0.610 1.000




The basic input matrices are: 







 

  y =  
 (10x1)   



1
80
56
23
73
74
39
38
38
46




   




  X = 
 (10x3)



731127

15217090

14113262

533145

99172152

133108158

449784

438698

13349100

50114126






   


 



   X' = 
 (3x10)   





731521415399133444313350

1117013231172108978649114

279062451521588498100126















II. Regression Calculations

yi =  b1 xi,1 +  b2 xi,2 +  b3 xi,3 +  ui

The q.c.e. basic equation in matrix form is: 
  y = Xb + e
  where y (dependent variable) is (nx1) or  (10x1)
        X (independent vars) is (nxk) or  (10x3)
        b (betas) is (kx1) or  (3x1)
        e (errors) is (nx1) or  (10x1)
Minimizing sum or squared errors using calculus results in the OLS eqn:
  b=(X'X)-1.X'y
To minimize the sum of squared errors of
a k dimensional line that describes the relationship 
between the k independent variables and y we
find the set of slopes (betas) that minimizes
Σi=1 to nei2
Re-written in linear algebra we seek to min e'e
Rearranging the regression model equation, we get e = y - Xb
So e'e = (y-Xb)'(y-Xb) = y'y - 2b'X'y + b'X'Xb   (see Judge et al (1985) p14 )
Differentiating by b we get 0 = - 2X'y + X'Xb -> 2X'Xb=2X'y
Rearranging, dividing both sides by 2 -> b = X'X-1X'y
So to obtain the elements of the (kx1) vector b we need the elements 
of the (kxk) matrix X'X-1 and of the (kx1) matrix X'y. 
Caclulating X'y is easy (see (1) below) but X'X-1 requires 
first calculation of X'X then finding cofactors -- see (4) -- and 
the deteminant - see (3) - in order to invert.



(1) X'y Matrix  (3x1)


    

    49121

    56421

    51118




(2) X'X Matrix  (3x3)


    

    
102587	
98473	
90350	

    
98473	
120856	
104224	

    
90350	
104224	
105302	



(3) Determinant   Det(X'X)≡|X'X|  
    i.e. the determinant of matrix of X'X 

Det(X'X) = 38095748040474
Det(X'X) =    102587*120856*105302 - 102587*104224*104224
        ... - 98473*98473*105302 + 98473*104224*90350
        ... + 90350*98473*104224 - 90350*120856*90350
        

(4) Cofactors(X'X) i.e. cofactor matrix of X 'X  (3x3)
    

    1863736336
    -952765446
    -656089648

    -952765446
    2639493774
    -1794991938

    -656089648
    -1794991938
    2701322743



(5) Adj(X'X) i.e. adjugate matrix of X'X, this is just the 
    transpose of the cofactor matrix.  (3x3)
    For a symmetric matrix, will be same as cofactor matrix.
    

    1863736336
    -952765446
    -656089648

    -952765446
    2639493774
    -1794991938

    -656089648
    -1794991938
    2701322743




(6) Inverse Matrix, inv(X'X)≡(X'X)-1
    = adj(X'X)/|X'X| = adj(X'X)/38095748040474   (3x3)

    

    0.000049
    -0.000025
    -0.000017

    -0.000025
    0.000069
    -0.000047

    -0.000017
    -0.000047
    0.000071



(7) Beta Matrix (β)
    b = [X'X-1].[X'y] , this is  (3x1).
    Finally we can calculate b through matrix multiplication.



     Betas 

    

     β 1  0.1117 

     β 2  0.2721 

     β 3  0.1203 


   =    
X'X-1 



 0.000049  -0.000025  -0.000017 

 -0.000025  0.000069  -0.000047 

 -0.000017  -0.000047  0.000071 


   X   
 X'y 

 49121 
 56421 
 51118 






Yhat1= + 0.1117x73 + 0.2721x11 + 0.1203x27 = 14.3942
Yhat2= + 0.1117x152 + 0.2721x170 + 0.1203x90 = 74.0596
Yhat3= + 0.1117x141 + 0.2721x132 + 0.1203x62 = 59.1229
Yhat4= + 0.1117x53 + 0.2721x31 + 0.1203x45 = 19.7680
Yhat5= + 0.1117x99 + 0.2721x172 + 0.1203x152 = 76.1437
Yhat6= + 0.1117x133 + 0.2721x108 + 0.1203x158 = 63.2486
Yhat7= + 0.1117x44 + 0.2721x97 + 0.1203x84 = 41.4131
Yhat8= + 0.1117x43 + 0.2721x86 + 0.1203x98 = 39.9927
Yhat9= + 0.1117x133 + 0.2721x49 + 0.1203x100 = 40.2172
Yhat10= + 0.1117x50 + 0.2721x114 + 0.1203x126 = 51.7617


ESS
=(14.39 - 46.80)^2
=(74.06 - 46.80)^2
=(59.12 - 46.80)^2
=(19.77 - 46.80)^2
=(76.14 - 46.80)^2
=(63.25 - 46.80)^2
=(41.41 - 46.80)^2
=(39.99 - 46.80)^2
=(40.22 - 46.80)^2
=(51.76 - 46.80)^2
=3950.73359835819

REPORT 




 obs 
 calculation of yhatobs
yhatobs = Σβixi,obs
 yhatobs
a
 yobs
 (data)
 Meany
 (yobs - yhatobs)2
 (yhatobs - My)2
 (yobs - My)2
a
eobs=yobs-yhatobs
eobs2


1
Yhat1 = Σβixi,1 = 0.1117x73 + 0.2721x11 + 0.1203x27 = 14.39146.80179.410502098 e1 = 1 - 14.39 = -13.39179.4


2
Yhat2 = Σβixi,2 = 0.1117x152 + 0.2721x170 + 0.1203x90 = 74.068046.8035.29743.11102 e2 = 80 - 74.06 = 5.94035.29


3
Yhat3 = Σβixi,3 = 0.1117x141 + 0.2721x132 + 0.1203x62 = 59.125646.809.752151.984.64 e3 = 56 - 59.12 = -3.1239.752


4
Yhat4 = Σβixi,4 = 0.1117x53 + 0.2721x31 + 0.1203x45 = 19.772346.8010.45730.7566.4 e4 = 23 - 19.77 = 3.23210.45


5
Yhat5 = Σβixi,5 = 0.1117x99 + 0.2721x172 + 0.1203x152 = 76.147346.809.883861.1686.4 e5 = 73 - 76.14 = -3.1449.883


6
Yhat6 = Σβixi,6 = 0.1117x133 + 0.2721x108 + 0.1203x158 = 63.257446.80115.6270.6739.8 e6 = 74 - 63.25 = 10.75115.6


7
Yhat7 = Σβixi,7 = 0.1117x44 + 0.2721x97 + 0.1203x84 = 41.413946.805.82329.0260.84 e7 = 39 - 41.41 = -2.4135.823


8
Yhat8 = Σβixi,8 = 0.1117x43 + 0.2721x86 + 0.1203x98 = 39.993846.803.97146.3477.44 e8 = 38 - 39.99 = -1.9933.971


9
Yhat9 = Σβixi,9 = 0.1117x133 + 0.2721x49 + 0.1203x100 = 40.223846.804.91643.3377.44 e9 = 38 - 40.22 = -2.2174.916


10
Yhat10 = Σβixi,10 = 0.1117x50 + 0.2721x114 + 0.1203x126 = 51.764646.8033.2024.620.6400 e10 = 46 - 51.76 = -5.76233.20
RSS = 
Σ(yobs - yhatobs)2		ESS = 
Σ(yhatobs - My)2		TSS = 
Σ(yobs - My)2		e'e=Σeobs2
sum->408.339515494408.3








(11) Betas and their t-Stats
     from the covar matrix of b=σ2(X'X)-1
     the var(βi) = σ2vii where vii is the ith diag element of X'X-1
     where σ2 = e'e / n-k  (k=num of ind vars plus 1 for the intercept if present).
     and where vii is the ith diag element of X'X-1
     Std(βi) = sqr root of Var(βi) 
     TStat(βi) = βi / Std(βi)
     Estimate of σ2 = 58.3245985793197

    

    

 Coef value 
 StD(β)
 tStat(β)

    
β1 = 
   0.000049 * 49121
     + -0.000025 * 56421
     + -0.000017 * 51118
     = 0.1117
    		(58.32 * 0.000049)1/2 
     = 0.05342
    		0.1117 / 0.05342
     = 2.091

    
β2 = 
   -0.000025 * 49121
     + 0.000069 * 56421
     + -0.000047 * 51118
     = 0.2721
    		(58.32 * 0.000069)1/2 
     = 0.06357
    		0.2721 / 0.06357
     = 4.280

    
β3 = 
   -0.000017 * 49121
     + -0.000047 * 56421
     + 0.000071 * 51118
     = 0.1203
    		(58.32 * 0.000071)1/2 
     = 0.06431
    		0.1203 / 0.06431
     = 1.871



(12) Table of Outputs:
     yobs = β1 X obs,1 +  β2 X obs,2 +  β3 X obs,3 +  eobs
     0.1117  0.2721  0.1203 
    (2.091)(4.280)(1.871) <- tstats 
     r2 = 0.925682    |    adj r2 = 0.888523


(13)  RSS   = Sum{y     - y_hat }^2  = 408.272190055238
      TSS   = Sum{y     - y_avg }^2  = 5493.6
      ESS(a)= Sum{y_hat - y_avg }^2  = 3950.73359835819
           we use the ESSb (below) cuz smthn wrng w ESS when no intercept.
      ESS(b)= TSS-RSS 5085.32780994476
      note:  TSS = ESS + RSS 
(14)  r2 = ESS/TSS = 0.925682213838787
(15)  adjusted r2 = ESS/TSS = 0.888523320758181
(16)  F-stat = [ESS/(k-1)] / [RSS/(n-k)] = 29.0633679660271
               see Johnston(1984) p186
               F measures the joint significance of 
               all explanatory variables.
      Alternatively:  F-stat = r2/(k-1) / (1-r2)/(n-k)
(17)  Durbin-Watson Statistic (DW or d) measures autocorrelation.
      DW = 2.80165232500866
 ________________________________________________________

Note, RSS, ESS and TSS stand for ... 
      Residual Sum of Squares (RSS),
      Explained Sum of Squares (ESS), and 
      Total Sum of Squares (TSS).
      However ESS is sometimes referred to as the Regression Sum of Squares.
      and     RSS is sometimes referred to as the Sum of Squares Rresidual.
Note, an alternative way of calculating TSS, ESS is...
      TSS = y'Ay   
      ESS = bv'Xv'Ay  where bv' Xv' are b' & X' wo intercept row col
      RSS = TSS-ESS 

Bibliography 

J. Johnston (1984) Econometric Methods, 3rd ed.
Judge et al (1985) The Theory and Practice of Econometrics 2rd ed, Wiley, New York.
Donald F. Morrison (1990) Multivariate Statistical Methods, 3rd edition, McGraw Hill, New York. 
A. H. Studenmund (1997) Using Econometrics: A Practical Guide, 3rd edition.  Addison-Wesley, Reading.