A NOTE ON EFFECT OF OFF-DIAGONAL ENTRIES IN A VARIANCE-COVARIANCE MATRIX ON EIGEN ANALYSIS

Bushra Shamshad. Department of Statistics, University of Karachi. ...................................................................................................................... Manuscript Info Abstract ......................... ........................................................................ Manuscript History Received: 05 February 2019 Final Accepted: 07 March 2019 Published: April 2019

A positive definite symmetric variance covariance matrix with nonzero diagonal entries-plays an important role in multivariate analysis. In this paper we will try to explain the basic algorithm needed to calculate the Eigen values and Eigen vector using such variance covariance matrix (or standardized variance covariance matrix). It is to be noted that coefficients of Principal Components are simply Eigen vectors and Eigen values are there variances (See, Anderson (1984), Johnson & Wichern (1988) & Hotelling (1936)). These eigen values are calculated from variance covariance matrix (or standardized variance covariance matrix). In this paper we will be discussing the effect on eigen analysis when entries in variance covariance matrix are changed. This eigen value problem will be studied on 2 2  and 3 3  variance covariance matrix, which then be carried out in higher order matrices (i.e. on 4×4) in order to generalize the coefficients of characteristic polynomial. With some proportion, eigen values vary, when off diagonal entries in variance covariance matrix are changed. This direct or indirect relationship between covariance terms of a variance covariance matrix and its respective eigen values then be deeply studied by regression analysis.

Eigenvalues And Eigenvectors
The eigenvalues, also called characteristic or latent roots of a square p p  matrix A, the p solutions for of the characteristic equation, given as (1) The roots can be complex-valued and some roots can have the same value, since is the root of polynomial of the degree p. Associated with each eigen value is a (right) eigenvector, which is a non-zero vector x with the property   Ax x, and a left eigenvector, which is a non-zero vector y with the property a a a a (2) In order to obtain the eigenvalues and eigenvector we have to use the characteristic equation, that is, (A -λ I) x = 0 (3) Where λ are the unknown constants or Eigen values which has to be determined and I is the identity matrix of order 2×2. After taking the determinant, we get This is a second degree polynomial or the Second order characteristic polynomial of variance covariance matrix A. The behavior is obvious from the equation (4), when the off diagonal entries in a variance covariance matrix are changed, there will be no effect on the second part of equation (4), because it is the sum of the diagonal entries. Only the last term of polynomial equation of variance covariance matrix is effected by the change, in the off diagonal entries. For example, consider a symmetric variance covariance matrix of order 2×2, The corresponding characteristic polynomial of matrix A is λ 2 -25 λ + 150 = 0 (6) The eigen roots of the variance covariance matrix are λ = {15, 10}. These eigen values are exactly equal to the diagonal entries of the variance covariance matrix. The reason is that, in calculating the determinant (that is, the third term in equation (6)), we subtract the product of diagonal entries with the off diagonal entries. Here the off diagonal entries of the variance covariance matrix are zero. So, the negative part of the determinant is eliminated. That is, [(10 -λ) (15 -λ) -0] = 0 In other words, we are not subtracting any value form (10 -λ) (15 -λ). Consider the same variance covariance matrix of order 2×2, with a change in off diagonal entries. That is, in place of zero covariance term we use covariance term equal to 1, the characteristic polynomial becomes λ 2 -25 λ + 149 = 0 (7) with the corresponding eigenvalues. 9.8074} , {15.1925 λ  As there is a unit increase in the covariance term, the largest eigen values also increases with proportion 0.1925. This indicates that as the covariance term increases it accumulates with the maximum variation present in the data and this mount up of maximum variation is represented by highest eigen values. The non-zero off diagonal entries plays an important role. Whereas, both the properties (given in section (1)) are also verified. Consider variance covariance matrix A, given below. Where, 'z' indicates the covariance term.
The behavior of eigen values are studied and presented in Table ( (1), the first column shows the z values, i.e. at the positions 12 and 21 of a variance covariance matrix (2.7), the eigenvalues are only affected by the determinant of matrix A. So, when the covariance term changes in matrix A, the determinant cause the change in the eigen values. It is clear that, when the off diagonal entries increase, determinant of a variance covariance matrix decreases, which further causes an increase in the magnitude of the largest eigen value, with some proportion. Another important point, that the eigen values are not affected by the sign of covariance term, can be seen in table (1), which is also presented in Figure (1). That is, the eigen values corresponds to covariance term equals to +1 is exactly the same as the eigen values corresponds to covariance term equal to -1. This is due to the fact that in a variance covariance matrix of order 2×2, the determinant is the value obtained by subtracting the product of two covariance terms from the product of two variances. At that point when two covariances are multiplied with each other the negative sign is eliminated and we get the same value of determinant as of the positive signed covariance term. The behavior of the negative and positive covariance's can be further explained by the Figure (1 We may conclude that eigen values are not effected by the sign of the covariance term in matrix of order 2×2. But have a slight change in the eigenvector. The negative sign with the first entry of the first eigenvector corresponds to the largest eigen value, will now transferred to the second entry of the other eigenvector. Figure (2), shows the increment in the first eigenvalues, with an increase in covariance term, shown in second last column of the table (1). Plotting these increment against off diagonal entries of matrix A, we observe an "S" shaped graph. For both negative and positive signed covariance term, the increment is similar but the direction is different. If we neglect the sign, we get a symmetric plot. We know that solution of equation (4), mainly depends on the third term. That is, the determinant of matrix A, which is directly depending on the covariance term in the variance covariance matrix. In spite of varying the covariance term in matrix A and observe the change in the eigen values, we constructed a table (2) in which we considered (unit) change in the determinant of a variance covariance matrix to observe the change in eigen values. The purpose is to take into account with what increment, change in eigen values occurs with a change in covariance term. The F-ratio for the quadratic regression is also very high as compare to the F value for linear model. Where as, the fitted plot shows almost all points are on the curve. Therefore, it may be concluded that there is a quadratic relationship between the increment in covariance term and the increment in the magnitude of highest eigen value.

Behavior Of Eigen Value In Variance Covariance Matrix Of Order 3×3
In the previous section we studied the behavior of variance covariance matrix of order 2×2, and observe the effects on the eigen roots, for different values of covariance term. In variance covariance matrix of order 3×3, it is quite complicated to observe the change in the eigen values, as the covariance terms of a variance covariance matrix are changed. Consider a variance covariance matrix of order 3×3, in its general form;    (11), the coefficient of  3 is unity, where as the coefficient of  2 is the trace of variance covariance matrix (9). The coefficient of  can be partitioned into two parts, one containing the three combination of pairs of diagonal entries and the other part containing the negative effect of the cross product term of the off diagonal entries ( that is, product of entries at position 12 and 21, at position 13 and 31 and so on). Hence, in general we can say that we are subtracting the effect of product of covariances form the possible pairs of product of diagonal element.
Here the point should be noted that the negative sign in the covariance term does not affect the coefficient of  . So, whether the value of covariance terms are negative or positive, the coefficient of  remains the same. But this negative sign of covariance terms affect the constant term in the equation (11). It is also applicable when standardized variance covariance matrix (i.e. correlation matrix) is used for eigen analysis. Consider the following correlation matrix, (Everitt and Dunn, 1991), of order 3×3. Characteristic equation for a standardized variance covariance matrix is same as that of a variance covariance matrix. From equation (11), we can easily write the characteristic polynomial of order 3×3.
with the following roots.  1 = 2.521912,  2 = 0.334108 and  3 = 0.143979 Characteristic polynomial for a correlation matrix can be more easily explained as compare to variance covariance matrix. From equation (13), it is clear that coefficients of λ 3 and λ 2 , of any standardized variance covariance matrix of order 3×3, will always be the same as in (13), i.e., 1 and 3 respectively. Whereas, the coefficient of λ is the sum of coefficients of non-determination and the last term (i.e. the constant term) will always be the determinant of the standardized variance covariance matrix.
The behavior of eigen values for higher order of matrix A is similar to that of variance covariance matrix of order 2×2. That is, the largest eigenvalue is formed by accumulating some proportion of covariance into the variance of the variable containing the maximum variation. Thus the largest eigenvalue explain the maximum variation of the data set. In case of variance covariance matrix (or standardized variance covariance matrix) of order 3×3 the behavior is similar but quite complicated. If we carry out eigen analysis of correlation matrix (12), by changing the correlation at different positions one by one, we can observe the behavior of eigenvalues, as given in Table (3). Table (3), shows the change (by 0.1) in the value of correlation, starting from -1. Whereas, rest of the columns represent the highest eigen values, when correlation at different position are being swaped. For example, the second column entitled "at position 13", "at position 12" and "at position 23", represent the largest eigen value (i.e. 2.02711, 2.01235 and 2.00351 respectively), when the entity at position 13 in a correlation matrix has been replaced by -1 and so on.  (4) indicates the similar pattern for any position (say, 12, 13 or 23) in a correlation matrix. It is also clear that, when a positive correlation (say, at position 13) is replaced by a strong negative (say -1 to -0.7), there will be a decline in the magnitude of largest eigen value. As, it move towards moderate to week negative correlation, the other two positive correlations (i.e., at 12 and 23) pulls the magnitude of largest eigen value, and thus increase its magnitude. If we take the correlation matrix similar to an identity matrix and then we change the correlation at different position one by one, the behavior of eigen values at either position will exactly the same, making a Vshaped graph that is symmetric at correlation equals to zero. Behavior of eigen value in higher order variance-covariance matrix.

280
In previous sections we have deeply observe the behavior of eigen values with variance covariance matrix (or standardized variance covariance matrix) of order 2×2 and 3×3. The characteristic polynomial, given in equation (15), of a variance covariance matrix, given in (14) The pattern is clear from equation (15), that the coefficient of the highest power of λ (i.e. λ 4 ) will always be unity. With the second highest power of λ, (i.e. λ 3 ), the coefficient will be the trace of the corresponding matrix. The coefficient of third highest power of λ will contain possible pairs of variances and a negative effect of their squared covariances. The coefficients of the next higher power of λ will be having possible pairs of entities, of size three, of a variance covariance matrix. Where as, the last term will always be the determinant of the corresponding matrix (having pairs of entries of size 4). In general we can write the characteristic polynomial as,

Conclusion:-
From the above analysis we can conclude that, there is a direct relationship between covariance term of a variance covariance matrix and its respective eigen values. Now, one can understand the phenomenon of eigen value problem, which is the base of principal component analysis (See, Anderson (1984)