EXPLORATORY FACTOR ANALYSIS WITH A COMMON FACTOR WITH TWO INDICATORS

Yutaka Kano Institute of Mathematics, University of Tsukuba Tsukuba, Ibaraki 305, JAPAN EXPLORATORY FACTOR ANALYSIS WITH A COMMON FACTOR WITH TWO INDICATORS

Abstract

Any exploratory factor analysis model requires at least three indicators (observed variables) for each common factor to ensure model identifiability. If one makes exploratory factor analysis for a data set in which one of common factors would have only two indicators in its population, one would encounter difficulties such as improper solutions and nonconvergence of iterative process in calculating estimates. In this paper, we first develop conditions for partial identifiability of the remaining factor loadings except for a factor loading vector which relates to a common factor with only two indicators. Two models for analyzing such data sets are then proposed with the help of confirmatory factor analysis and covariance structure analysis. The first model is an exploratory factor analysis model that permits correlation between unique factors; the second model is a kind of confirmatory factor model with equal factor loadings. Two real data sets are analyzed to illustrate usefulness of these models.

Keywords: Anderson-Rubin’s condition, estimable, exploratory and confirmatory factor analysis, identification, indicator.

1. Introduction

Factor analysis is one of multivariate statistical analysis techniques useful particularly in analyzing intercorrelations among a set of observed variables. Correlations of observed variables are assumed to be explained by hypothesized latent common factors in factor analysis. The model with k common factors for a p × 1 observed random vector x is represented as

x = μ + Λf + u,

where μ = E[x] is a general mean vector, Λ = (λ_{ij}) is a factor loading (or pattern) matrix of order p × k and f = [F_{1},…, F_{k}]′ and u = [U_{1},…, U_{p}]′ are vectors of common and unique factors satisfying E[f] = 0, E[u] = 0, Cov(f, u) = 0, Var(f) = Φ and Var(u) = Ψ, a p × p diagonal matrix. In this article, we consider an orthogonal factor analysis model, that is, Φ = I_{k}. In the case, the covariance matrix of observed vector x is expressed in the form:

Var(x) = ΛΛ′ + Ψ(= Σ, say).  (1)

The expression (1) is called the fundamental equation in factor analysis. The model (1) is said to be identifiable if and only if the parameter (Λ, Ψ) is determined by Σ up to orthogonal rotation to Λ.

There are two different versions in factor analysis, namely exploratory analysis and confirmatory analysis. The purpose of exploratory factor analysis is to explore latent structure of observed variables, that is, to find how many common factors should be taken to account for the intercorrelations, which common factors mainly influence on which observed variables, and to make interpretations of factor analysis results, particularly to clarify what the common factors would be. Exploratory factor analysis makes a hypothesis on latent structures of observed variables. In contrast, confirmatory factor analysis confirms researcher’s hypotheses on latent structure of observed variables. To implement confirmatory analysis, one needs to know how many factors would account for the intercorrelations, what the common factors mean, which variables are influenced by which common factors. Thus, the two versions of factor analysis apply to completely different situations.

There is a serious drawback in exploratory factor analysis. Anderson and Rubin (1956) showed that existence of (at least) three nonzero elements in each of the column vectors in Λ is a necessary condition for model identifiability. The condition means that every common factor must influence at least three observed variables, that is, every common factor must have at least three indicators. One can not make exploratory factor analysis of a data set such that one of common factors has only two indicators in an assumed model. If one would do it for such a data set, he or she would encounter difficulties in estimation such as improper solutions, nonconvergence of iteration process, and estimates depending on initial values and/or optimization (estimation) methods. It is because the model is not identifiable.

Bechtoldt (1961) randomly divided a data set on primary mental abilities into two sets, and made exploratory factor analysis separately for each data set. He found that the two results of factor analysis are rather different. This finding greatly lost reliability of factor analysis. His solutions stay in the proper parameter space, because fast and useful convergent algorithms such as the Gauss-Newton method had not been developed at that time. It has been found that both solutions are improper. van Driel (1978) studied several causes of improper solutions, and found that the cause of improper solutions of Bechtoldt’s data sets is indefiniteness (nonidentifiability) of the factor analysis model, that is, the sixth common factor influences only two variables X_{1} and X_{2}.

In this paper, we propose a modified exploratory factor analysis model to analyze such data sets. Two particular models are proposed in Section 3. Section 2 discusses the identification problem. In Section 4, we apply the developed procedure to a Japanese WISC-R data set as well as Bechtoldt’s data set.

2. Identification

Consider the following parameter (matrices) in an orthogonal exploratory factor analysis model (1):

Λ = [ \begin{matrix}λ_{1} & λ_{1k} \\ λ_{2} & λ_{2k} \\ Λ_{3} & 0 \end{matrix} ] , Ψ = [ \begin{matrix}ψ_{1} \\ & ψ_{2} \\ & & ψ_{3} \\ & & & . & . & . \\ & & & & & & ψ_{p} \end{matrix} ] ,  (2)

where λ_{i}’s are row (k − 1)-vectors, Λ_{3} is of (p − 2) × (k − 1) and λ_{1k}, λ_{2k} are scalars. The parameter (Λ, Ψ) is not identified as mentioned in Introduction. Actually, if a and b are arbitrary real numbers with ab = λ_{1k}λ_{2k}, then a different parameter

Λ̃ = [ \begin{matrix}λ_{1} & a \\ λ_{2} & b \\ Λ_{3} & 0 \end{matrix} ] Ψ̃ = [ \begin{matrix}ψ_{1} − a^{2} + λ_{1k}^{2} \\ & ψ_{2} − b^{2} + λ_{2k}^{2} \\ & & ψ_{3} \\ & & & . & . & . \\ & & & & & & ψ_{p} \end{matrix} ]  (3)

also produces the same covariance matrix Σ in (1) as (2) does, provided that Ψ̃ is positive definite.

The problem that we address in this section is to study whether a parameter different from (3) can also yield the same covariance matrix Σ, in other words, whether or

not [ \begin{matrix}λ & 1 \\ λ & 2 \\ Λ_{3} \end{matrix} ] and ψ_{3},…, ψ_{p} are identifiable (up to rotation). If so, these parameters are

estimable (a, b, ψ_{1} and ψ_{2} are not). We will call it partial identifiability because some of the parameters are identifiable.

The partial identification of [ \begin{matrix}λ & 1 \\ λ & 2 \\ Λ_{3} \end{matrix} ] and ψ_{3},…, ψ_{p} is not always true. In fact, let

Λ = [ \begin{matrix}1 & 1/4 \\ 1 & 1/2 \\ 1 & 0 \\ 1 & 0 \end{matrix} ] and Λ̃ = [ \begin{matrix}3/4 & 3/4 \\ 3/4 & 3/4 \\ 3/4 & 7/12 \\ 4/3 & 0 \end{matrix} ] ,  (4)

and then ΛΛ′ and Λ̃Λ̃′ give exactly the same off-diagonal elements and their diagonal elements are all different. More generally, when (p, k) = (4, 2), not only λ_{1k}, λ_{2k} but also the first column in Λ and Ψ in (2) are not identifiable.

Anderson and Rubin (1956) gave a very useful and concise condition on identifiability in the exploratory factor analysis model (1):

Let Λ be a factor loading matrix of order p×k. There remain two nonsingular k × k submatrices of Λ even when any one row vector of Λ is deleted.

The condition determines uniquely the communalities, or (Λ, Ψ) is identifiable up to orthogonal rotation to Λ. We will call it Anderson and Rubin’s condition on Λ, or, it will be said that the Λ meets Anderson and Rubin’s condition. The condition requires that p ≥ 2k + 1.

The next theorem gives a sufficient condition for partial identifiability of the model (2). We denote by O(k) the set of orthogonal matrices of order k.

Theorem 1. Let Σ = ΛΛ′+Ψ, where Λ = [ \begin{matrix}λ_{1} & λ_{1k} \\ λ_{2} & λ_{2k} \\ Λ_{3} & 0 \end{matrix} ] with λ_{1k}λ_{2k} ≠ 0 and Ψ is diagonal

(as in (2)). Assume that Λ_{3} satisfies Anderson and Rubin’s condition. If

ΛΛ′ + Ψ = Σ = LL′ + D with L of p × k and D = diag(d_{1},…, d_{p}),

then there exist T ∈ O(k) and scalars a, b such that

LT = [ \begin{matrix}λ_{1} & a \\ λ_{2} & b \\ Λ_{3} & 0 \end{matrix} ] , ab = λ_{1k}λ_{2k};

D = [ \begin{matrix}ψ_{1} − a^{2} + λ_{1k}^{2} \\ & ψ_{2} − b^{2} + λ_{2k}^{2} \\ & & ψ_{3} \\ & & & . & . & . \\ & & & & & & ψ_{p} \end{matrix} ] .

Proof. Without any loss of generality we assume that p = 2k + 3. Let λ_{3} be any row vector of Λ_{3} and let Λ_{4} and Λ_{5} be (k − 1) × (k − 1) nonsingular submatrices of Λ_{3} when

λ_{3} is deleted from Λ_{3}. After suitable rearrangement of rows of Λ_{3}, we write Λ_{3} = [ \begin{matrix}λ & 3 \\ Λ_{4} \\ Λ_{5} \end{matrix} ],

so that

Λ = [ \begin{matrix}λ_{1} & λ_{1k} \\ λ_{2} & λ_{2k} \\ λ_{3} & 0 \\ Λ_{4} & 0 \\ Λ_{5} & 0 \end{matrix} ] .

Let the corresponding partition of L be expressed as

L = [ \begin{matrix}ℓ & 1 \\ ℓ & 2 \\ ℓ & 3 \\ L_{4} \\ L_{5} \end{matrix} ] ,

and Σ is also partitioned as Σ = [Σ_{ij}] correspondingly. It should be noted that every off-diagonal element of ΛΛ′ and LL′ coincides. It follows that

[ \begin{matrix}λ_{1} & λ_{1k} \\ Λ_{4} & 0 \end{matrix} ] [ \begin{matrix}λ_{2} & λ_{2k} \\ Λ_{5} & 0 \end{matrix} ]′ = [ \begin{matrix}ℓ & 1 \\ L_{4} \end{matrix} ] [ \begin{matrix}ℓ & 2 \\ L_{5} \end{matrix} ]′ .

Note that the four matrices above are all of order k. Since Λ_{4} and Λ_{5} are nonsingular and λ_{1k}λ_{2k} ≠ 0, the matrices in LHS above are of rank k, and hence

rank(L_{4}) = k − 1  (5)
rank [ \begin{matrix}ℓ & 2 \\ L_{5} \end{matrix} ] = k, (a nonsingular matrix).  (6)

Then we have

rank ([ \begin{matrix}ℓ & 2 \\ L_{5} \end{matrix} ] [ \begin{matrix}ℓ & 3 \\ L_{4} \end{matrix} ]′) = rank ([ \begin{matrix}λ_{2} & λ_{2k} \\ Λ_{5} & 0 \end{matrix} ] [ \begin{matrix}λ_{3} & 0 \\ Λ_{4} & 0 \end{matrix} ]′)
= rank ([ \begin{matrix}λ & 2 \\ Λ_{5} \end{matrix} ] [ \begin{matrix}λ & 3 \\ Λ_{4} \end{matrix} ]′)
= k − 1,

which, along with (6), implies that rank [ \begin{matrix}ℓ & 3 \\ L_{4} \end{matrix} ] = k − 1, or equivalently, there is a x ∈ R^{k} such that

ℓ_{3} = x′L_{4}  (7)

in view of (5). We thus obtain

λ_{3}λ′_{3} = λ_{3}Λ′_{4}(Λ_{5}Λ′_{4})^{−1}Λ_{5}λ′_{3} = Σ_{34}Σ_{54}^{−1}Σ_{53}
= ℓ_{3}L′_{4}(L_{5}L′_{4})^{−1}L_{5}ℓ′_{3} = ℓ_{3}L′_{4}(L_{5}L′_{4})^{−1}L_{5}L′_{4}x (by (7))
= ℓ_{3}ℓ′_{3}.

This holds for any rows of Λ_{3}, and hence

L_{3}L′_{3} = Λ_{3}Λ′_{3}.  (8)

There exists T ∈ O(k) such that

L_{3}T = [Λ_{3}, 0]

and then

LT = [ \begin{matrix}ℓ̃ & 1 \\ ℓ̃ & 2 \\ Λ_{3} 0 \end{matrix} ] ,  (9)

where ℓ̃_{i} = ℓ_{i}T. Since

[Λ_{3}, 0] [ \begin{matrix}ℓ̃ & 1 \\ ℓ̃ & 2 \end{matrix} ]′ = [Λ_{3}, 0] [ \begin{matrix}λ_{1} & λ_{1k} \\ λ_{2} & λ_{2k} \end{matrix} ]′

and Λ_{3} is of full-column rank, we have

[ \begin{matrix}ℓ̃ & 1 \\ ℓ̃ & 2 \end{matrix} ] = [ \begin{matrix}λ_{1} & a \\ λ_{2} & b \end{matrix} ]  (10)

for some scalars a and b. Substitution of Equation (9) with (10) into LL′ + D = ΛΛ′ + Ψ and use of (8) results in

[ \begin{matrix}a \\ b \\ 0 \end{matrix} ] [ \begin{matrix}a \\ b \\ 0 \end{matrix} ]′ + D = [ \begin{matrix}λ_{1k} \\ λ_{2k} \\ 0 \end{matrix} ] [ \begin{matrix}λ_{1k} \\ λ_{2k} \\ 0 \end{matrix} ]′ + Ψ,

from which we finally obtain

ab = λ_{1k}λ_{2k},
a^{2} + d_{1} = λ_{1k}^{2} + ψ_{1},
b^{2} + d_{2} = λ_{2k}^{2} + ψ_{2},
d_{i} = ψ_{i} (i = 3,…, p).

This type of partial identifiability in factor analysis was first studied by Sato (1992, Theorem 3.9). He showed that λ_{i}’s are identifiable if the uniqueness of [Λ_{3}, 0] is assumed. The derivation of (10) from (8) is an adaptation of Sato (1992).

The identification condition described in Theorem 1 is the one for the situation where one does not know that λ_{ik} = 0 (i = 3,…, p) while they are assumed to be zero in the population. The next theorem gives a condition when it is known to the researcher that λ_{ik} = 0 (i = 3,…, p). The condition is similar to Anderson and Rubin’s but is weaker than that of Theorem 1.

Theorem 2. Let Σ = ΛΛ′+Ψ, where Λ = [ \begin{matrix}λ_{1} & λ_{1k} \\ λ_{2} & λ_{2k} \\ Λ_{3} & 0 \end{matrix} ] with λ_{1k}λ_{2k} ≠ 0 and Ψ is diagonal

(as in (2)). Assume that there remain two nonsingular submatrices of order k − 1 either in [ \begin{matrix}λ & 1 \\ Λ_{3} \end{matrix} ] or [ \begin{matrix}λ & 2 \\ Λ_{3} \end{matrix} ] even when any one row vector of Λ_{3} is deleted. If

ΛΛ′ + Ψ = Σ = LL′ + D with L = [ \begin{matrix}ℓ_{1} & a \\ ℓ_{2} & b \\ L_{3} & 0 \end{matrix} ] of p × k and D = diag(d_{1},…, d_{p}),

then the same conclusion as in Theorem 1 holds.

Proof. Without any loss of generality we assume that p = 2k + 2. Let λ_{3} be any row vector of Λ_{3} and let [ \begin{matrix}λ & 2 \\ Λ_{4} \end{matrix} ] and Λ_{5} be (k − 1) × (k − 1) nonsingular submatrices when λ_{3} is deleted from Λ_{3}. After suitable rearrangement of rows of Λ_{3}, we can write

Λ = [ \begin{matrix}λ_{1} & λ_{1k} \\ λ_{2} & λ_{2k} \\ λ_{3} & 0 \\ Λ_{4} & 0 \\ Λ_{5} & 0 \end{matrix} ] .

Let the corresponding partition of L be expressed as

L = [ \begin{matrix}ℓ_{1} & a \\ ℓ_{2} & b \\ ℓ_{3} & 0 \\ L_{4} & 0 \\ L_{5} & 0 \end{matrix} ] .

We then have

λ_{3}λ′_{3} = λ_{3} [ \begin{matrix}λ & 2 \\ Λ_{4} \end{matrix} ]′ (Λ_{5} [ \begin{matrix}λ & 2 \\ Λ_{4} \end{matrix} ]′)^{−1} Λ_{5}λ′_{3}
= [λ_{3} 0] [ \begin{matrix}λ_{2} & λ_{2k} \\ Λ_{4} & 0 \end{matrix} ]′ ([Λ_{5} 0] [ \begin{matrix}λ_{2} & λ_{2k} \\ Λ_{4} & 0 \end{matrix} ]′)^{−1} [Λ_{5} 0][λ_{3} 0]′
= [ℓ_{3} 0] [ \begin{matrix}ℓ_{2} & b \\ L_{4} & 0 \end{matrix} ]′ ([L_{5} 0] [ \begin{matrix}ℓ_{2} & b \\ L_{4} & 0 \end{matrix} ]′)^{−1} [L_{5} 0][ℓ_{3} 0]′
= ℓ_{3} [ \begin{matrix}ℓ & 2 \\ L_{4} \end{matrix} ]′ (L_{5} [ \begin{matrix}ℓ & 2 \\ L_{4} \end{matrix} ]′)^{−1} L_{5}ℓ′_{3},

and then it is seen that L_{5} [ \begin{matrix}ℓ & 2 \\ L_{4} \end{matrix} ]′, L_{5} and [ \begin{matrix}ℓ & 2 \\ L_{4} \end{matrix} ] are all nonsingular matrices of order

k − 1. We thus obtain λ_{3}λ′_{3} = ℓ_{3}ℓ′_{3}. We can repeat the same proof as that in Theorem 1 to complete the proof.

The basic idea of these proofs comes from Ihara and Kano (1986) and Kano (1989), where a simple alternative proof of Anderson and Rubin’s identifiability condition is given.

We shall end this section by noting the following two things: (i) The condition of Theorem 1 requires p ≥ 2p + 3, whereas that of Theorem 2 requires p ≥ 2p + 2; (ii) the factor loading matrix Λ in (3) satisfies the condition of Theorem 2, but it does not meet that of Theorem 1.

3. Suggested models

We suggest two models to perform factor analysis for such data sets that one of the common factors has only two indicators, X_{1} and X_{2}, say, in the population.

Consider a factor analysis model Σ = ΛΛ′ + Ψ as in (1) with

Λ = [ Λ_{1} | \begin{matrix}λ_{1k} \\ λ_{2k} \\ 0 \end{matrix} ] ,  (11)

where Λ_{1} is of p × (k − 1), and λ_{1k} and λ_{2k} are scalars. It will be a model between exploratory and confirmatory analyses in that λ_{ik} (i = 3,…, p) are specified a priori to be zero as in confirmatory analysis and Λ_{1} can be rotated as in exploratory analysis. We need to impose a further constraint to achieve identifiability.

One model we suggest is a model in which λ_{1k} and λ_{2k} are equated. In the other model, we remove the k-th common factor and allows for correlation between unique factors U_{1} and U_{2}. These two models are equivalent to each other.

The basic idea for the second model is as follows: In the first model, the observed variables X_{1} and X_{2} have the form:

X_{1} = λ_{11}F_{1} + ⋯ + λ_{1,k−1}F_{k−1} + λ_{1k}F_{k} + U_{1},
X_{2} = λ_{21}F_{1} + ⋯ + λ_{2,k−1}F_{k−1} + λ_{2k}F_{k} + U_{2}.

We put Ũ_{1} = λ_{1k}F_{k} + U_{1} and Ũ_{2} = λ_{2k}F_{k} + U_{2}, so that Ũ_{1} and Ũ_{2} are correlated to each other through F_{k}. But they are uncorrelated with F_{1},…, F_{k−1} and U_{3},…, U_{p}.

This kind of models have been used in the context of confirmatory factor analysis or covariance structure analysis. Surprisingly none has emphasized to use these simple models in the exploratory context.

Existence of a two-indicator factor as in (2) or (11) has been recognized as a typical cause of improper solutions in exploratory factor analysis (van Driel 1978). If an improper solution is obtained with such cause, we usually give up making exploratory factor analysis of the data set. Even in the case, some of the parameters are identifiable, as ensured by Theorems in Section 2 and can be estimated at least consistently. If one takes the models suggested here, one can perform statistical analysis properly, that is, the standard errors computed are correct, and a goodness-of-fit statistic has an asymptotic chi-square distribution.

Interpretation should be made carefully. In the first model, the estimates for λ_{1k} and λ_{2k} are identical due to the constraint. One should not conclude that the k-th factor influences on the X_{1} and X_{2} equally. From the model, we can not mention at all how much each of the observed variables is affected by the F_{k}.

One needs to know a priori which two observed variables are influenced by the kth common factor, or which two unique factors are correlated. Previous exploratory factor analyses before considering these two models will give a good insight to the choice. Typically, one has done exploratory factor analysis with k − 1 common factors, and the model has been rejected (and then factor analysis with one more factor was performed and as a result the model encountered a trouble). In the result of k−1 common factor analysis, one can find the largest standardized residual in absolute value, the (i, j)-element, say, in D_{S}^{−1/2}{S − (Λ̂_{k−1}Λ̂′_{k−1} + Ψ̂_{k−1})}D_{S}^{−1/2}, where S is the sample covariance matrix, D_{S} is a diagonal matrix with diagonal elements of S and (Λ̂_{k−1}, Ψ̂_{k−1}) is the solution in factor analysis model with k−1 common factors. Then one can introduce an additional common factor which loads equally on X_{i} and X_{j}, or arrow U_{i} and U_{j} to be correlated. A more reasonable way is to perform Lagrange Multiplier tests to examine whether Cov(U_{i}, U_{j}) = 0.

4. Applications

Here, we shall term, Model 1 and Model 2, the way of equating appropriate factor loadings and the way of introducing correlation between two unique factors, respectively.

The following analyses of two real data sets will illustrate importance of the theoretical result on identification and usefulness of the models suggested in the previous section. They are Bechtoldt’s data set and a data set of the Wechsler Intelligence Scale for Children-Revised (WISC-R), as noted in Introduction.

Bechtoldt (1961) made exploratory factor analysis of some portion (p = 17) of the data set collected and analyzed originally by Thurstone and Thurstone (1941), to study stability of factor analysis results against sample fluctuations. He randomly divided the data set into two sets of sample size N = 212 (Sample I) and N = 213 (Sample II), and did factor analysis of these samples separately. See page 412 in Bechtoldt’s paper for the sample correlation matrices. The six factors were found to be appropriate for the both data sets. He compared seven estimation methods and found that some methods including the maximum likelihood give rather different estimates for the communalities and factor loadings of the variables X_{1} and X_{2} between the two samples whereas the other methods such as the centroid method provide fairly close estimates. All the estimates are proper. Since then, many authors including van Driel (1978), Tumura and Sato (1981), Okamoto (1987) and Sato (1992) have studied the data sets by taking recent standard estimation methods and optimization algorithm such as maximum likelihood and least squares with types of Gauss-Newton algorithm (e.g. Jöreskog 1967; Jennrich and Robinson 1969; Okamoto and Ihara 1984 for estimation methods). We can summarize these studies as (i) the statistical examination on the number of factors shows that for the both samples, the five-factor model is definitely rejected and the six-factor model is acceptable; (ii) the estimates depend on the samples, estimation methods, optimization algorithms and starting values, and they are improper in almost all the cases; (iii) only the estimates for the loadings and uniqueness related to X_{1} and X_{2} and the “Memory factor” are unstable and the other estimates are rather stable across the samples, estimation methods, optimization algorithms and starting values. To see these, we calculated the maximum likelihood estimate (MLE) and least squares estimate (LSE) for each of the samples along with statistics for goodness-of-fit, where we used EQS version 5 (Bentler 1995) to obtain these estimates and statistics with the starting value of 0.5 for each parameter. See Browne (1982, 84) for the goodness-of-fit statistics associated with the LSE. We impose that ψ̂_{i} ≥ 0. Four sets of estimates of the uniqueness ψ_{1},…, ψ_{17} are given in Table 1. The estimates of ψ_{3},…, ψ_{17} are rather close across the four sets, whereas those of ψ_{1} and ψ_{2} are very different.

< Table 1 should be about here >

The instability and occurrence of improper solutions have been known to come from the fact that there is a common factor (Memorizing factor) which only influences on X_{1} and X_{2} (van Driel 1978). We can verify that Λ_{3} in the notation (2) meets Anderson-Rubin’s condition since the solution can be rotated to be very close to simple structure with each factor having three indicators, see Table 3b. Thus our theory can apply.

Let ψ_{i,j} = Cov(U_{i}, U_{j}) for i ≠ j. To illustrate how effectively Lagrange multiplier (LM) tests work to see which factor loadings are to be equated, or which unique factors are to be correlated, we perform exploratory factor analysis along with (univariate) LM tests to test each hypothesis that

H_{0} : ψ_{i,j} = 0 versus H_{1} : ψ_{i,j} ≠ 0

for each pair (i, j) with i > j. Some results of the LM tests and standardized residuals D_{S}^{−1/2}{S − (Λ̂Λ̂′ + Ψ̂)}D_{S}^{−1/2} are given in Table 2. It is concluded that λ_{16} and λ_{26} should be equated or ψ_{2,1} should be allowed to correlate.

< Table 2 should be about here >

Results based on the two Models developed in this paper are provided in the following tables. Table 3a gives results of goodness-of-fit chi-square statistics and estimates for equated loadings and the uniqueness in Samples I and II, where ML and LS methods in estimation are employed. Note that the estimates of ψ_{i} for i = 3,…, 17 are identical for Model 1 and Model 2 because of their equivalence. Table 3b gives factor loading ML estimates for Samples I and II based on Model 1 after VARIMAX rotation for the 17 × 5 factor loading matrix for the common factors 1 to 5 is made. No improper solutions are obtained. The MLE’s for Samples I and II give quite close values, and each of the factors from F_{2} to F_{6} has three good indictors. We do not have a reasonable interpretation of a rather large value of λ̂_{16} = .404 for Sample II at present, though. Nevertheless, We would say that the factor analysis results are fairly stable against the sample fluctuations.

< Tables 3a and 3b should be about here >

The next example is a data set of Japanese version of the WISC-R that was conducted for six years old children, reported in Kodama et al. (1978). See e.g., Kaufman (1975), Ueno (1982) and Watanabe (1993) for the WISC-R. The sample correlation matrix is given in Table 4 (N = 100). For several of the numbers of common factors that are less than 6, Watanabe (1992) performed principal factor analysis with the SMC communality estimates, so that proper solutions are obtained for each of the numbers of factors. Kaufman (1975) also conducted principal factor analysis, to get proper solutions for k = 3.

< Table 4 should be about here >

Table 5 shows results of exploratory factor analysis of the WISC-R data for the case when the constraints ψ_{i} ≥ 0 are imposed and for the case with no such constraints, where ML and LS methods in estimation are employed. As you see, rather different solutions are obtained.

< Table 5 should be about here >

The MLE reported in Table 5 under no constraints on ψ_{i} obviously gives a local minimum. We reported it to emphasize that such anomaly easily takes place in nonidentifiable models. Careful choice of starting values can result in solutions similar to but different from the LSE listed just below.

To specify appropriate variables to have equated loadings or to be correlated, we perform exploratory factor analysis with two common factors along with LM tests. The standardized residuals and the LM test result are presented in Table 6. If one takes the approaches developed in this paper, it is suggested that the third factor F_{3} should equally loads only on X_{3} and X_{6} in Model 1 and that U_{3} and U_{6} should be correlated in Model 2.

< Table 6 should be about here >

Table 7a gives final results mainly on uniqueness estimates; Table 7b provides the VARIMAX rotated factor loading matrix of the first two factors. It is seen that the observed variables X_{1}, X_{2}, X_{4} and X_{5} are good indicators of the F_{1}, Verbal Comprehension; the observed variables X_{7} and X_{9}-X_{12} are good indicators of the F_{2}, Perceptual Organization. The factor loading submatrix Λ_{3} in (3) is obtained by deleting the third and sixth rows and the third column from the entire matrix, and we can say that the resultant matrix meets Anderson and Rubin’s condition.

< Table 7a and 7b should be about here >

No improper solutions take place at all, and the MLE and LSE are very close to each other.

5. Final remarks The models proposed in this paper is mediated between exploratory and confirmatory

models. The use of the models successfully makes factor analysis of the two data sets.

The reader may claim that confirmatory factor analysis can be done for these data sets. Actually, the Bechtoldt data were collected to measure the six primary mental abilities, so it would be confirmatory data; Toyoda (1992) successfully made a confirmatory analysis of a WISC-R data set. A typical context in which one can apply the models developed in this article is not in confirmatory analysis but in exploratory analysis. We offered a methodology when an exploratory factor analysis model with a certain number of factors is rejected and an exploratory factor analysis with an additional factor gives an improper solution. A typical cause of the improper solution is that there is a common factor with two indicators. We developed models that can handle these situations.

Once analyses based on these models are performed and these kinds of results are obtained lik Bechtoldt’s or WISC-R data, one could move on to the next stage of analysis, namely confirmatory analysis.

For alternative treatments of improper solutions, see Kano (1990) and Kano and Ihara (1994), where use of non-iterative estimation method and identification of inconsistent variables are discussed.

As the reader may notice, the models proposed here are those corresponding to the identification result in Theorem 2. We can, of course, consider models which reflect Theorem 1, for example, the model in which λ_{1k} = λ_{2k} and λ_{ik} (i = 3,…, p) are all free parameters. One can examine how close to zero λ̂_{ik}’s (i = 3,…, p) are, under the condition in Theorem 1. Note that the restriction λ_{1k} = λ_{2k} is arbitrary, and one can take λ_{1k}/\sqrt{σ_{11}} = λ_{2k}/\sqrt{σ_{22}} for instance. Rotated estimates of factor loadings do depend on restrictions, and thus, interpretations can be difficult. We leave it an open problem in the future. The methods proposed in this paper can be implemented by usual softwares of covariance structure analysis and exploratory factor analysis.

Acknowlegment

The author would like to thank Dr. Manabu Sato of Hiroshima University for his discussion on partial identifiability in factor analysis.

References

Anderson, T. W. and Rubin, H. (1956). Statistical inference in factor analysis. Proceedings of the Third Berkeley Symposium on Mathematical Statistics and Probability, 5, 111-150. Berkeley: University of California Press.

Bechtoldt, H. P. (1961). An empirical study of the factor analysis stability hypothesis. Psychometrika, 26, 405-432.

Bentler, P. M. (1995). EQS structural equations program manual. Multivariate Software,

Inc.: CA. Browne, M. W. (1982). Covariance structures. In Topics in Applied Multivariate Analysis,

(D.M. Hawkins, ED.) pp.72-141. Cambridge University Press, Cambridge, England. Browne, M. W. (1984). Asymptotically distribution-free methods for the analysis of

covariance structures. British Journal of Mathematical and Statistical Psychology, 37, 62-83.

Ihara, M. and Kano, Y. (1986). A new estimator of the uniqueness in factor analysis.

Psychometrika, 51, 563-566. Jennrich, R. I. and Robinson, S. M. (1969). A Newton-Raphson algorithm for maximum

likelihood factor analysis. Psychometrika, 34, 111-123. Jöreskog K. G. (1967). Some contributions to maximum likelihood factor analysis. Psy-

chometrika, 32, 443-482. Kano, Y. (1989). A new estimation procedure using G-inverse matrix in factor analysis.

Mathematica Japonica, 34, 43-52. Kano, Y. (1990). Noniterative estimation and the choice of the number of factors in

exploratory factor analysis. Psychometrika, 55, 277-291. Kano, Y. and Ihara, M. (1994). Identification of inconsistent variates in factor analysis.

Psychometrika, 59, 5-20. Kaufman, A. S. (1975). Factor analysis of the WISC-R at 11 age levels between 6\frac{1}{2} and

16\frac{1}{2} years. Journal of Consulting and Clinical Psychology, 43, 135-147. Kodama, S., Shinagawa, F. and Mogi, S. (1978). Nihonban WICS-R chinou kensahou

(Japanese version of the WISC-R). Nihon Bunka Kagakusha (In Japanese). Okamoto, M. (1987). Inshibunseki no saikin no hatten (Recent developments in factor

analysis). In Shakaikagaku no keiryoubunseki (Measurement Analysis in Social Sciences) (Eds. Suzuki, S. and Takeuchi, K.), pp. 7-26. University of Tokyo Press: Tokyo (In Japanese).

Okamoto, M. and Ihara, M. (1984). Partial Gauss-Newton algorithm for least-squares and maximum likelihood methods in factor analysis. Journal of the Japan Statistical Society, 14, 137-144.

Sato, M. (1992). A study of an identification problem and substitute use of principal component analysis in factor analysis. Hiroshima Mathematical Journal, 22, 479-524.

Thurstone, L. L. and Thurstone, T. G. (1941). Factorial studies of intelligence. Psycho-

metric Monograph No. 2. Chicago: University of Chicago Press. Toyoda, H. (1992). SAS niyoru kyoubunsan kouzou bunseki (Covariance structure analysis

with SAS). University of Tokyo Press: Tokyo (In Japanese).

Tumura, Y. and Sato, M. (1981). On the convergence of iterative procedures in factor

analysis. TRU Mathematics, 17, 159-168. Ueno, K. (1982). WISC-R no hyoujunka dehta no inshibunseki (Factor analysis of stan-

dardized WISC-R data). Shinri Sokutei Jahnaru (Psychological Measurement Journal). 18, 19-21 (In Japanese).

van Driel, O. P. (1978). On various causes of improper solutions in maximum likelihood

factor analysis. Psychometrika, 43, 225-243. Watanabe, H. (1992). Tahenryou kaiseki nyuumon: Jireihen (Introduction to multivariate

analysis: Applications). Fukumura Shuppan: Tokyo (In Japanese). Watanabe, H. (1993). Shinri kensahou nyuumon (Introduction to methods of psychological

measurements). Fukumura Shuppan: Tokyo (In Japanese).

Table 1 Bechtoldt’s data: Exploratory factor analysis (k = 6)

Results of goodness-of-fit chi-square statistics and uniqueness estimates (multiplied by 1000).

Sample Estimation Goodness-of-fit

Uniqueness estimates

Model  χ_{49}^{2}-value P-value ψ_{1} ψ_{2} ψ_{3} ψ_{4}
Sample I  MLE  59.590  0.143 358 636 158 131
LSE  56.869  0.205 673 000 184 132
Sample II  MLE  54.803  0.264 000 698 178 160
LSE  55.206  0.252 195 660 187 164

Table 1 (continued)

Uniqueness estimates

ψ_{5} ψ_{6} ψ_{7} ψ_{8} ψ_{9} ψ_{10} ψ_{11} ψ_{12} ψ_{13} ψ_{14} ψ_{15} ψ_{16} ψ_{17}
230 371 337 563 367 234 314 447 178 488 273 356 464
214 370 351 559 369 227 319 440 191 475 257 362 476
256 262 358 499 412 346 279 183 422 490 354 408 457
245 274 352 489 408 356 272 203 406 481 363 387 464

Table 2

Bechtoldt’s data: LM tests and standardized residual.

Sample  Variables  LM test (P-value) Standardized Residual
Sample I U_{2}, U_{1} (ψ_{2,1})  34.449 (0.000)  .335
U_{11}, U_{10} (ψ_{11,10})  9.869 (0.002)  .013
U_{9}, U_{5} (ψ_{9,5})  8.078 (0.004)  .043
U_{11}, U_{9} (ψ_{11,9})  7.356 (0.007)  -.026
U_{4}, U_{1} (ψ_{4,1})  7.331 (0.007)  .043
Sample II U_{2}, U_{1} (ψ_{2,1})  31.912 (0.000)  .278
U_{17}, U_{15} (ψ_{17,15})  21.642 (0.000)  .113
U_{14}, U_{13} (ψ_{14,13})  6.915 (0.009)  .042
U_{11}, U_{5} (ψ_{11,5})  6.798 (0.009)  -.032
U_{10}, U_{3} (ψ_{10,3})  5.465 (0.019)  -.026

Table 3a Bechtoldt’s data: New models

Results of goodness-of-fit chi-square statistics, and estimates of uniqueness, equal factor loadings λ_{16} = λ_{26} (Model 1) and covariance ψ_{2,1} between U_{1} and U_{2} (Model 2), multiplied by 1000.

Sample Estimation Goodness-of-fit Model 1

Model 2

Model  χ_{60}^{2}-value P-value λ_{16} = λ_{26} ψ_{1} ψ_{2} ψ_{2,1} ψ_{1} ψ_{2}
Sample I  MLE  71.947  0.139  589  461 552 346 807 899
LSE  72.086  0.136  590  460 547 348 808 895
Sample II  MLE  71.393  0.149  559  415 519 313 728 832
LSE  71.366  0.150  558  429 523 311 740 834

Table 3a (continued)

Uniqueness estimates (Models 1 and 2)

ψ_{3} ψ_{4} ψ_{5} ψ_{6} ψ_{7} ψ_{8} ψ_{9} ψ_{10} ψ_{11} ψ_{12} ψ_{13} ψ_{14} ψ_{15} ψ_{16} ψ_{17}
171 134 229 401 309 568 374 232 317 451 170 495 269 358 466
193 130 210 404 318 557 378 221 328 434 220 484 247 362 476
180 160 258 270 353 501 415 346 278 161 432 474 441 336 522
193 160 250 276 365 485 413 354 275 201 405 473 431 312 524

Table 3b Bechtoldt’s data: Model 1 Estimates of factor loadings, multiplied by 1000. The loadings of over .4 in value are typed in boldface.

Factor

Variable Sample I Sample II
F_{1} F_{2} F_{3} F_{4} F_{5} F_{6} F_{1} F_{2} F_{3} F_{4} F_{5} F_{6}
X_{1} First Name  255 224 014 198 194 589 150 276 000 103 404 559
X_{2} Word-Number  135 238 074 052 136 589 197 176 107 246 163 559
X_{3} Sentences  817 194 -025 138 323 000 816 238 027 152 270 000
X_{4} Vocabulary  839 299 -001 191 190 000 813 302 027 195 223 000
X_{5} Completion  771 215 252 120 230 000 768 258 120 135 230 000
X_{6} First Letters  210 691 057 205 180 000 224 783 060 167 190 000
X_{7} Four Letter Words 073 799 064 181 098 000 204 702 170 092 274 000
X_{8} Suffixes  304 551 -033 149 112 000 305 608 009 112 153 000
X_{9} Flags  045 075 748 179 166 000 037 112 740 144 063 000
X_{10} Figures  005 074 872 039 029 000 030 002 802 032 093 000
X_{11} Cards  094 -006 816 020 088 000 066 074 830 128 089 000
X_{12} Addition  149 231 146 649 184 000 122 054 089 898 078 000
X_{13} Multiplication  112 191 -009 866 153 000 131 178 089 683 214 000
X_{14} Three-Higher  206 196 245 511 321 000 160 137 272 512 382 000
X_{15} Letter Series  287 224 126 295 704 000 217 199 173 283 602 000
X_{16} Pedigrees  362 195 141 134 660 000 394 131 041 106 692 000
X_{17} Letter Grouping  227 210 153 339 549 000 135 306 211 236 516 000

F_{1}: Verbal Comprehension, F_{2}: Word Fluency, F_{3}: Space, F_{4}: Number, F_{5}: Reasoning or Inductive, F_{6}: Memorizing.

Table 4

Sample correlation matrix of WISC-R data for six years old children (N = 100) (Kodama et.al. 1978; Watanabe 1992).

Verbal
X1  Information          1.00
X2  Similarities         0.39 1.00
X3  Arithmetic           0.23 0.33 1.00
X4  Vocabulary           0.43 0.44 0.14 1.00
X5  Comprehension        0.43 0.43 0.17 0.56 1.00
X6  Digit Span           0.25 0.24 0.45 0.24 0.22 1.00
Performance
X7  Picture Completion   0.15 0.28 0.35 0.17 0.17 0.13 1.00
X8  Picture Arrangement  0.12 0.24 0.01 0.22 0.13 0.08 0.32 1.00
X9  Block Design         0.14 0.25 0.32 0.14 0.10 0.30 0.33 0.28 1.00
X10 Object Assembly      0.20 0.34 0.16 0.27 0.28 0.18 0.16 0.07 0.40 1.00
X11 Coding               0.07 0.21 0.09 0.00 0.08 0.11 0.11 0.19 0.27 0.12 1.00
X12 Maze                 0.11 0.14 0.16 0.07 0.08 0.05 0.23 0.17 0.33 0.33 0.37 1.00

Table 5

WISC-R: Exploratory factor analysis (k = 3)

Results of goodness-of-fit chi-square statistics and uniqueness estimates, multiplied by 1000.

Constraint Estimation Goodness-of-fit

Uniqueness estimates

Model  χ_{33}^{2}-value P-value ψ_{1}  ψ_{2}  ψ_{3} ψ_{4}
0 ≤ ψ_{i} < ∞  MLE  30.758  0.579 654  564  000 415
LSE  30.716  0.581 653  559  000 409
−∞ < ψ_{i} < ∞  MLE  47.977  0.045 660 -663.718^{∗} 788 431
LSE  29.593  0.638 653  560  -492 415

Note: ^{∗} The value is not multiplied by 1000.

Table 5 (continued)

Uniqueness estimates

ψ_{5} ψ_{6} ψ_{7} ψ_{8} ψ_{9} ψ_{10} ψ_{11} ψ_{12}
466 745 754 808 528 709 786 681
471 747 769 823 534 736 778 642
450 805 775 872 436 733 846 752
472 786 774 826 533 734 782 656

Table 6 WISC-R data: LM tests and standardized residual.

Variables  LM test (P-value) Standardized Residual
U_{6}, U_{3} (ψ_{6,3})  11.353 (0.001)  .249
U_{12}, U_{11} (ψ_{12,11})  6.009 (0.014)  .167
U_{8}, U_{3} (ψ_{8,3})  4.618 (0.032)  -.164
U_{8}, U_{7} (ψ_{8,7})  3.622 (0.057)  -.144
U_{7}, U_{3} (ψ_{7,3})  2.956 (0.086)  .120

Table 7a WISC-R: New models

Results of goodness-of-fit chi-square statistics, and estimates of uniqueness, equal factor loadings λ_{16} = λ_{26} (Model 1) and covariance ψ_{2,1} between U_{1} and U_{2} (Model 2), multiplied by 1000.

Estimation Goodness-of-fit Model 1

Model 2

Model  χ_{42}^{2}-value P-value λ_{33} = λ_{63} ψ_{3} ψ_{6} ψ_{6,3} ψ_{3} ψ_{6}
MLE  43.280  0.417  538  520 559 289 809 849
LSE  43.229  0.419  546  522 558 298 819 856

Table 7a (continued)

Uniqueness estimates (Models 1 and 2)

ψ_{1} ψ_{2} ψ_{4} ψ_{5} ψ_{7} ψ_{8} ψ_{9} ψ_{10} ψ_{11} ψ_{12}
653 564 433 459 763 856 493 719 823 723
644 544 442 476 759 865 481 745 814 703

Table 7b WISC-R data: Model 1 Estimates of factor loadings, multiplied by 1000. The loadings of over .4 in value are typed in boldface.

Factor

Variable MLE LSE
F_{1} F_{2} F_{3} F_{1} F_{2} F_{3}
X_{1} Information  574 134 000 583 127 000
X_{2} Similarities  571 334 000 587 334 000
X_{3} Arithmetic  209 374 538 238 353 546
X_{4} Vocabulary  749 074 000 744 061 000
X_{5} Comprehension  732 068 000 721 065 000
X_{6} Digit Span  290 245 538 305 227 546
X_{7} Picture Completion 189 449 000 204 446 000
X_{8} Picture Arrangement 184 332 000 170 326 000
X_{9} Block Design  100 706 000 121 709 000
X_{10} Object Assembly  306 434 000 303 404 000
X_{11} Coding  017 422 000 014 431 000
X_{12} Maze  034 526 000 025 545 000

F_{1}: Verbal Comprehension, F_{2}: Perceptual Organization, F_{3}: Freedom from Distractibility.