DELTA METHOD APPROACH IN A CERTAIN IRREGULAR CONDITION

Dedicated to the retirement of Professor Nariaki Sugiura from the University of Tsukuba

Yutaka Kano

Faculty of Human Sciences, Osaka University Suita, Osaka 565-0871, Japan; kano@hus.osaka-u.ac.jp

Key Words: Asymptotic distribution; Differentiability; Factor analysis; The number of components;

ABSTRACT

Let X_{n} be a sequence of random p-vectors such that a_{n}(X_{n} −b) ⟶^{L} Z, where a_{n} ↗ ∞, b ∈ R^{p} and Z is a continuously distributed random p-vector. Let f(·) be a measurable mapping from a domain of R^{p} to R^{q}, where the domain may not include b, i.e., f(b) may not be defined. Under this setup, we study the asymptotic distribution of f(X_{n}). Two theorems are developed to obtain the asymptotic distribution. Comprehensive examples are provided to show when and where such an irregular situation takes place and to illustrate the usefulness of these theorems. The examples include the problem of choosing the number of components and noniterative estimation in factor analysis.

1. INTRODUCTION AND MOTIVE EXAMPLES

Let \{X_{n}\} be a sequence of random p-vectors such that \sqrt{n}(X_{n}−b) ⟶^{L} Z, where b ∈ R^{p} and Z is typically normally distributed. The delta method approach is particularly useful in obtaining asymptotic distributions of statistics f(X_{n}), where f(·) is a measurable mapping from a domain of R^{p} to R^{q}. When f(x) is (totally) differentiable at x = b, the theorem shows that

\sqrt{n}(f(X_{n}) − f(b)) ⟶^{L} \frac{∂f(b)}{∂x′}Z as n → ∞.

See e.g. Rao (1973, page 388) for a proof. We denote convergence in probability and in distribution by ⟶^{P} and ⟶^{L}, respectively. What happens if the important regularity condition — differentiability — fails? More irregularly what if f(x) can not be defined at x = b? This paper discusses asymptotic theory in such situations.

In multivariate analysis, a typical example of cases where the differentiability condition is violated is asymptotic theory for latent roots or singular values of a random matrix with multiplicity. In the case, sample roots or singular values are continuous but not differentiable. A large number of significant works on the topic have been published (e.g., Anderson 1963; Chikuse 1976; Eaton and Tyler 1991, 1994; Fujikoshi 1977, 1980; Sugiura 1973, 1976, 1990; Tyler 1983; Waternaux 1976). We shall take the simplest case where nS = (ns_{ij}) follows W_{2}(n, I_{2}), the Wishart distribution with I_{2} the parameter matrix and n degrees of freedom. The eigenvalues, (λ̂_{1}, λ̂_{2}), of S are expressed as

\frac{1}{2} (s_{11} + s_{22} ± \sqrt{(s_{11} − s_{22})^{2} + 4s_{12}^{2}}) .  (1)

It is easily seen that λ̂_{i}’s are not differentiable at S = I_{2}. The asymptotic distribution of (λ̂_{1}, λ̂_{2}) and its relation with our theorem will be noted in Section 2. Notice that the eigenvalues are defined at S = I_{2}. In this paper, we discuss a more difficult situation where the function f(x) may not be defined at x = b.

Asymptotics in many types of nonregular cases have been studied: density functions are not differentiable; support of density functions depends on parameters; rather strong dependency is introduced between observations and so forth (e.g., Akahira and Takeuchi 1995; Roussas 1972; Basawa and Scott 1983; Tanaka 1996). The situation we discuss here takes place when the Fisher information matrix is singular or the probability model is not identified.

Now we shall briefly discuss the problem of choosing the number of components in finite normal mixture distribution as a simple but motive example.

Let Y_{1}, …, Y_{n} be a random sample from a population distributed as

πN(μ_{1}, σ^{2}) + (1 − π)N(μ_{2}, σ^{2}),  (2)

where μ_{1} ≠ μ_{2}. One approach to choosing the number of components is to test the following hypothesis:

H_{0} : π = 0 or 1 versus H_{1} : 0 < π < 1.  (3)

Many problems of the method of likelihood ratio (LR) have been reported, among which Hartigan (1985) showed that the LR test statistic diverges to infinity in probability as n → ∞ and Romano and Siegel (1986) showed the MLE does not exist. These anomalies occur partially because the density is independent of the parameter μ_{2} for π = 1 or μ_{1} for π = 0 in H_{0}, and then the Fisher information matrix is singular.

We shall consider the method of moments for the testing problem. First, we assume that μ_{1} and σ^{2} are known and the other two parameters π and μ_{2} are to be estimated. A more general case will be discussed in Section 2. Let

\{ \begin{matrix}X_{1}^{(n)} & = & \frac{1}{n} ∑_{i=1}^{n} Y_{i} (= Ȳ , say) \\ X_{2}^{(n)} & = & \frac{1}{n−1} ∑_{i=1}^{n}(Y_{i} − Ȳ)^{2}, \end{matrix}  (4)

and calculate E[Y_{i}] = (1−π)(μ_{2}−μ_{1})+μ_{1} and Var[Y_{i}] = π(1−π)(μ_{2}−μ_{1})^{2}+σ^{2}. The estimator based on the moment method is a solution to the simultaneous equations:

\{ \begin{matrix}X_{1}^{(n)} & = & (1 − π)(μ_{2} − μ_{1}) + μ_{1} \\ X_{2}^{(n)} & = & π(1 − π)(μ_{2} − μ_{1})^{2} + σ^{2}. \end{matrix}  (5)

The estimator for π can be then expressed as

π̂ = π̂(X_{n}) = \frac{X_{2}^{(n)} − σ^{2}}{(X_{1}^{(n)} − μ_{1})^{2} + (X_{2}^{(n)} − σ^{2})},  (6)

where X_{n} = [X_{1}^{(n)}, X_{2}^{(n)}]′.

Consider separately the cases where π = 1 and π = 0 in H_{0}. First, assume that π = 1 holds. We then have

X_{n} = [ \begin{matrix}X_{1}^{(n)} \\ X_{2}^{(n)} \end{matrix} ] ⟶^{P} [ \begin{matrix}E[X_{1}^{(n)}] \\ E[X_{2}^{(n)}] \end{matrix} ] = [ \begin{matrix}μ_{1} \\ σ^{2} \end{matrix} ] = b, say,
\sqrt{n}(X_{n} − b) ⟶^{L} Z = [ \begin{matrix}Z_{1} \\ Z_{2} \end{matrix} ] ∼ N_{2} (0, [ \begin{matrix}σ^{2} & 0 \\ 0 & 2σ^{4} \end{matrix} ]) .  (7)

It is seen from (6) that π̂(X_{n}) can not be defined at X_{n} = b, the limiting point of X_{n}.

Even though the difficulty about the definition arises, consistency of π̂ holds and its asymptotic distribution can be easily derived. These asymptotic results are obtained by direct application of Theorem 1, as described in Section 2. A rough proof is stated in this section to help the reader understand the statement and the proof of Theorem 1 more easily.

It follows from (6) and (7) that

π̂ = \frac{\sqrt{n}(X_{2}^{(n)} − σ^{2})}{\sqrt{n}(X_{1}^{(n)} − μ_{1})^{2} + \sqrt{n}(X_{2}^{(n)} − σ^{2})}
= \frac{\sqrt{n}(X_{2}^{(n)} − σ^{2})}{O_{p}(n^{−1/2}) + \sqrt{n}(X_{2}^{(n)} − σ^{2})}
⟶^{P} 1
\sqrt{n}(π̂ − 1) = −\frac{(\sqrt{n}(X_{1}^{(n)} − σ^{2}))^{2}}{\sqrt{n}(X_{1}^{(n)} − μ_{1})^{2} + \sqrt{n}(X_{2}^{(n)} − σ^{2})}
= −\frac{(\sqrt{n}(X_{1}^{(n)} − σ^{2}))^{2}}{O_{p}(n^{−1/2}) + \sqrt{n}(X_{2}^{(n)} − σ^{2})}
⟶^{L} −\frac{Z_{1}^{2}}{Z_{2}} .

Since the asymptotic distribution of π̂ does not depend on the unknown parameters, we can use π̂ to test the hypothesis in (3).

Next, consider the case π = 0. It can be easily verified that π̂ defined in (6) converges in probability to zero as n tends to infinity, and that

\sqrt{n}π̂ = \frac{\sqrt{n}(X_{2}^{(n)} − σ^{2})}{(μ_{2} − μ_{1})^{2}} + o_{p}(1)
⟶^{L} \frac{Z_{2}}{(μ_{2} − μ_{1})^{2}} ∼ N (0, \frac{2σ^{4}}{(μ_{2} − μ_{1})^{4}} ) .

This is a typical consequence of the usual delta theorem. No anomaly occurs because the Fisher information matrix (of (π, μ_{2})) is not singular.

In Sections 2 and 3 two different theorems will be given on asymptotic distributions of such statistics. In the both theorems, we assume that each component of f(x) is written as F_{i}(x)/h_{i}(x). Theorem 1 requires F_{i}(x) and h_{i}(x) to have a particular structure, whereas Theorem 2 requires differentiability of F_{i}(x) and h_{i}(x).

Examples on eigenvalues, the problem of choosing the number of components and noniterative statistics in factor analysis are provided to illustrate usefulness of those theorems.

2. THEOREM 1 AND EXAMPLES

The following theorem is obtained.

Theorem 1. Let D be an open domain in R^{p}, let b ∈ D, and let a_{n} be a positive sequence which tends to infinity as n → ∞. Let X_{n} be a sequence of random p-vectors with P (X_{n} ∈ D \ {b}) = 1 for each n and let Z be a nondegenerate continuous random p-vector. Suppose that a_{n}(X_{n} − b) ⟶^{L} Z as n → ∞. Let f(x) = [f_{1}(x), …, f_{q}(x)]′ be a continuous function from D \ {b} to R^{q}. Assume that there exists a continuous function h(x) = [h_{1}(x), …, h_{q}(x)]′ from D to R^{q} such that for each i (i = 1, …, p), lim_{x→b} f_{i}(x)h_{i}(x) = c_{i} < ∞ and h_{i}(X_{n}) ≠ 0 with probability one. Define

F_{i}(x) = \{ \begin{matrix}f_{i}(x)h_{i}(x) & if x ≠ b \\ c_{i} & if x = b \end{matrix} .

Assume further that there exist nonnegative numbers s_{i}, t_{i} and continuous functions F̄_{i}(·, ·) and h̄_{i}(·, ·) such that

\{ \begin{matrix}a^{s_{i}}F_{i}(x) & = & F̄_{i}(a(x − b), x − b) \\ a^{t_{i}}h_{i}(x) & = & h̄_{i}(a(x − b), x − b) \end{matrix}  (8)

for any a > 0 and that h̄_{i}(Z, 0) ≠ 0 with probability one. Define G_{i}(Z) =

\frac{F̄_{i}(Z, 0)}{h̄_{i}(Z, 0)}. Then, we have

a_{n}^{s_{i}−t_{i}}f_{i}(X_{n}) ⟶^{L} G_{i}(Z) as n → ∞.  (9)

If s_{i} = t_{i} and G_{i}(Z) = β_{i} a nonstochastic constant with probability one, we then have

f_{i}(X_{n}) ⟶^{P} G_{i}(Z) = β_{i} as n → ∞.  (10)

Assume that there exist a nonnegative number u_{i} and a continuous function F̃_{i}(·, ·) such that

a^{u_{i}}(F_{i}(x) − β_{i}h_{i}(x)) = F̃_{i}(a(x − b), x − b)  (11)

for any a > 0. Then we have

a_{n}^{u_{i}−t_{i}}(f_{i}(X_{n}) − β_{i}) ⟶^{L} \frac{F̃_{i}(Z, 0)}{h̄_{i}(Z, 0)}  as n → ∞.  (12)

Proof. Note that f_{i}(X_{n}) = F_{i}(X_{n})/h_{i}(X_{n}) with probability one, and evaluate directly the distribution of the statistics as

Pr ( ⋂_{i=1}^{q} a_{n}^{s_{i}−t_{i}}f_{i}(X_{n}) < x_{i}) = Pr ( ⋂_{i=1}^{q} \frac{a_{n}^{s_{i}}F_{i}(X_{n})}{a_{n}^{t_{i}}h_{i}(X_{n})} < x_{i})

= Pr( ⋂_{i=1}^{q}{a_{n}^{s_{i}}F_{i}(X_{n}) − x_{i}a_{n}^{t_{i}}h_{i}(X_{n}) < 0, a_{n}^{t_{i}}h_{i}(X_{n}) > 0}

∪ {a_{n}^{s_{i}}F_{i}(X_{n}) − x_{i}a_{n}^{t_{i}}h_{i}(X_{n}) > 0, a_{n}^{t_{i}}h_{i}(X_{n}) < 0}) (13)

= Pr( ⋂_{i=1}^{q}{F̄_{i}(ξ_{n}, η_{n}) − x_{i}h̄_{i}(ξ_{n}, η_{n}) < 0, h̄_{i}(ξ_{n}, η_{n}) > 0}

∪ {F̄_{i}(ξ_{n}, η_{n}) − x_{i}h̄_{i}(ξ_{n}, η_{n}) > 0, h̄_{i}(ξ_{n}, η_{n}) < 0}) (14)

with ξ_{n} = a_{n}(X_{n} − b) and η_{n} = X_{n} − b. Assumption in (8) has been used here. Because of the continuity of F̄_{i}(·, ·) and h̄_{i}(·, ·) and the convergence

(ξ_{n}, η_{n}) ⟶^{L} (Z, 0), we have

[ \begin{matrix}F̄_{i}(ξ_{n}, η_{n}) \\ h̄_{i}(ξ_{n}, η_{n}) \end{matrix} ] ⟶^{L} [ \begin{matrix}F̄_{i}(Z, 0) \\ h̄_{i}(Z, 0) \end{matrix} ] .  (15)

Use of (14) and (15) results in

Pr ( ⋂_{i=1}^{q} a_{n}^{s_{i}−t_{i}}f_{i}(X_{n}) < x_{i})

→ Pr( ⋂_{i=1}^{q}{F̄_{i}(Z, 0) − x_{i}h̄_{i}(Z, 0) < 0, h̄_{i}(Z, 0) > 0}

∪ {F̄_{i}(Z, 0) − x_{i}h̄_{i}(Z, 0) > 0, h̄_{i}(Z, 0) < 0})

= Pr ( ⋂_{i=1}^{q} \frac{F̄_{i}(Z, 0)}{h̄_{i}(Z, 0)} < x_{i}) .

Here we have used Pr(h̄_{i}(Z, 0) = 0) = 0. The result in (12) follows from a similar argument.

Q.E.D.

We shall return to the example of the problem of choosing the number of components described in Section 1. Consider the case π = 1. The estimator for μ_{2} based on the moment method is jointly considered here. Solving equations in (5) for μ_{2}, we have

μ̂_{2} = \frac{(X_{1}^{(n)} − μ_{1})^{2} + μ_{1}(X_{1}^{(n)} − μ_{1}) + (X_{2}^{(n)} − σ^{2})}{X_{1}^{(n)} − μ_{1}}.

Let

f(x) = [ \begin{matrix}f_{1}(x) \\ f_{2}(x) \end{matrix} ] = [ \frac{x_{2} − σ^{2}}{(x_{1} − μ_{1})^{2} + (x_{2} − σ^{2})} ; \frac{(x_{1} − μ_{1})^{2} + μ_{1}(x_{1} − μ_{1}) + (x_{2} − σ^{2})}{x_{1} − μ_{1}} ] .

We define h_{1}(x) = (x_{1} − μ_{1})^{2} + (x_{2} − σ^{2}) and h_{2}(x) = x_{1} − μ_{1}, and then F_{1}(x) = x_{2} − σ^{2} and F_{2}(x) = (x_{1} − μ_{1})^{2} + μ_{1}(x_{1} − μ_{1}) + (x_{2} − σ^{2}). Recall that ξ_{n} = (ξ_{1}^{(n)}, ξ_{2}^{(n)})′ = (\sqrt{n}(X_{1}^{(n)} − μ_{1}), \sqrt{n}(X_{2}^{(n)} − σ^{2}))′ and η_{n} = (η_{1}^{(n)}, η_{2}^{(n)})′ = (X_{1}^{(n)} − μ_{1}, X_{2}^{(n)} − σ^{2})′. Thus, we shall choose h̄_{i} and F̄_{i} as below and we have

\sqrt{n}h_{1}(x) = ξ_{1}^{(n)}η_{1}^{(n)} + ξ_{2}^{(n)} = h̄_{1}(ξ_{n}, η_{n}) ⟶^{L} Z_{2}
\sqrt{n}h_{2}(x) = ξ_{1}^{(n)} = h̄_{2}(ξ_{n}, η_{n}) ⟶^{L} Z_{1}
\sqrt{n}F_{1}(x) = ξ_{2}^{(n)} = F̄_{1}(ξ_{n}, η_{n}) ⟶^{L} Z_{2}
\sqrt{n}F_{2}(x) = ξ_{1}^{(n)}η_{1}^{(n)} + μ_{1}ξ_{1}^{(n)} + ξ_{2}^{(n)} = F̄_{2}(ξ_{n}, η_{n}) ⟶^{L} μ_{1}Z_{1} + Z_{2}.

By (9) and (10) we have

π̂ = f_{1}(X_{n}) ⟶^{L} \frac{Z_{2}}{Z_{2}} = 1 (or ⟶^{P} 1)
μ̂_{2} = f_{2}(X_{n}) ⟶^{L} \frac{μ_{1}Z_{1} + Z_{2}}{Z_{1}} = μ_{1} + \frac{Z_{2}}{Z_{1}}

as n → ∞. It follows from (11) and (12) that

\sqrt{n}(π̂ − 1) = −\sqrt{n}\frac{(X_{1}^{(n)} − μ_{1})^{2}}{(X_{1}^{(n)} − μ_{1})^{2} + (X_{2}^{(n)} − σ^{2})}

= −\frac{(ξ_{1}^{(n)})^{2}}{ξ_{1}^{(n)}η_{1}^{(n)} + ξ_{2}^{(n)}} = −\frac{F̃_{1}(ξ_{n}, η_{n})}{h̄_{1}(ξ_{n}, η_{n})}
⟶^{L} −\frac{Z_{1}^{2}}{Z_{2}} .

In sum, the joint asymptotic distribution of [π̂, μ̂_{2}]′ can be characterized as

[ \begin{matrix}\sqrt{n} & 0 \\ 0 & 1 \end{matrix} ] [ \begin{matrix}π̂ − 1 \\ μ̂_{2} − μ_{1} \end{matrix} ] ⟶^{L} [ \begin{matrix}−Z_{1}^{2}/Z_{2} \\ Z_{2}/Z_{1} \end{matrix} ] as n → ∞.

Next we shall move to a slightly more general case where μ_{1} is unknown in addition to π and μ_{2}. Assume that μ_{1} and μ_{2} are distinct. Since three unknown parameters are involved, one more statistic and equation are required in addition to (X_{1}^{(n)}, X_{2}^{(n)}) in (4) and equations in (5). We shall take as a statistic

X_{3}^{(n)} = \frac{1}{n} ∑_{i=1}^{n}(Y_{i} − Ȳ)^{3}

and the equation is derived as

X_{3}^{(n)} = π(1 − π)(1 − 2π)(μ_{1} − μ_{2})^{3}.  (16)

The expectation of the left-hand side in (16) is asymptotically the same as in the right-hand side. When π = 1, we have

X_{n} = [ \begin{matrix}X_{1}^{(n)} \\ X_{2}^{(n)} \\ X_{3}^{(n)} \end{matrix} ] ⟶^{P} [ \begin{matrix}E[X_{1}^{(n)}] \\ E[X_{2}^{(n)}] \\ E[X_{3}^{(n)}] \end{matrix} ] = [ \begin{matrix}μ_{1} \\ σ^{2} \\ 0 \end{matrix} ] = b, say,

\sqrt{n}(X_{n} − b) ⟶^{L} Z = [ \begin{matrix}Z_{1} \\ Z_{2} \\ Z_{3} \end{matrix} ] ∼ N_{3} (0, [ \begin{matrix}σ^{2} & 0 & 3σ^{4} \\ 0 & 2σ^{4} & 0 \\ 3σ^{4} & 0 & 15σ^{6} \end{matrix} ]) .

If π̂ is a solution to the equations in (5) and (16), then 1 − π̂ is also a solution. Existence of two solutions comes from luck of identification of the model (2) when μ_{1} and μ_{2} are both unknown. In fact, we can not distinguish between πN(μ_{1}, σ) + (1 − π)N(μ_{2}, σ) and (1 − π)N(μ_{2}, σ) + πN(μ_{1}, σ) (cf. Teicher 1963). To achieve identification we restrict the parameter space of π to be the interval [\frac{1}{2}, 1], and choose a larger solution as an estimate for π.

Here we suppress the superscript (n) for simplicity. Solving (5) and (16) for π, we have

π̂ − 1 = −\frac{4(X_{2} − σ^{2})^{3}X_{3}^{2} + 16(X_{2} − σ^{2})^{6}}{2 (4(X_{2} − σ^{2})^{3} + X_{3}^{2}) (4(X_{2} − σ^{2})^{3} + X_{3}^{2} + \sqrt{4(X_{2} − σ^{2})^{3}X_{3}^{2} + X_{3}^{4}})}

(17)

and hence

\sqrt{n}(π̂ − 1) = −\frac{4ξ_{2}^{3}ξ_{3}^{2} + 16ξ_{2}^{5}η_{2}}{2 (4ξ_{2}^{2}η_{2} + ξ_{3}^{2}) (4ξ_{2}^{2}η_{2} + ξ_{3}^{2} + \sqrt{4ξ_{2}^{2}η_{2}ξ_{3}^{2} + ξ_{3}^{4}})} = \frac{F̃_{1}(ξ, η)}{h̄_{1}(ξ, η)}, say,

with ξ_{i} = \sqrt{n}(X_{i} − b_{i}) and η_{i} = X_{i} − b_{i}. Using Theorem 1, this quantity converges in distribution to

\frac{F̃_{1}(Z, 0)}{h̄_{1}(Z, 0)} = −\frac{Z_{2}^{3}}{Z_{3}^{2}} .

The asymptotic distribution is independent of unknown parameters, and hence one can use it to test the hypothesis in (3).

A similar derivation leads to asymptotic distributions of μ̂_{1} and μ̂_{2}. We report here the final result:

[ \begin{matrix}\sqrt{n} & 0 & 0 \\ 0 & \sqrt{n} & 0 \\ 0 & 0 & 1 \end{matrix} ] [ \begin{matrix}π̂ − 1 \\ μ̂_{1} − μ_{1} \\ μ̂_{2} − μ_{1} \end{matrix} ] ⟶^{L} [ \begin{matrix}−Z_{2}^{3}/Z_{3}^{2} \\ Z_{1} + Z_{2}^{2}/Z_{3} \\ Z_{3}/Z_{2} \end{matrix} ] as n → ∞.

It would be interesting to note that when n goes to infinity, the estimator for the indeterminate parameter μ_{2} does not concentrate at any specific value and converges in law to the Cauchy distribution whether for the two-parameter case or for the three-parameter case.

It would be Day (1969) who first described the moment method in the normal mixture. He did not recommend its use for the reason of efficiency in estimation, however. We suppose that he did not realize the difficulties of the MLE and such beautiful results of the moment method as consistency and easy evaluation of the asymptotic distribution.

We have not taken the parameter space into account when we construct π̂. The estimator can be outside [0, 1] or [\frac{1}{2}, 1] in the two examples above. It is possible to form modified estimators π̂^{∗} to be in these parameter spaces. Testing procedures based on π̂^{∗} and π̂ are equivalent to each other whereas these estimators have different asymptotic distributions. See Hatsuyama (1995) for details.

Under alternative distributions, π̂ is asymptotically normal, and use of the asymptotic distribution enables us to evaluate statistical power of the test based on π̂. See again Hatsuyama (1995).

To end the example, we note alternative approaches to the problem of choosing the number of components: Henna (1985)’s nonparametric method, penalty method by Leroux (1992) and Chen and Kalbfleisch (1993), Bayes approach with MCMC due to Richardson and Green (in press) and Roeder and Wasserman (1997).

We shall finalize this section with the eigenvalue problem noted in Section 1. The problem can be dealt with via Theorem 1. Assume that

\sqrt{n}(X_{n} − b) = \sqrt{n} ([ \begin{matrix}s_{11} \\ s_{21} \\ s_{22} \end{matrix} ] − [ \begin{matrix}1 \\ 0 \\ 1 \end{matrix} ]) ⟶^{L} [ \begin{matrix}Z_{1} \\ Z_{2} \\ Z_{3} \end{matrix} ] ∼ N_{3}(0, Γ),

where Γ is the asymptotic covariance matrix. When nS follows the Wishart

distribution W_{2}(n, I_{2}), Γ = [ \begin{matrix}2 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 2 \end{matrix} ]. We do not have to restrict the distri-

bution to the Wishart one, however.

We take h_{i}(·) = 1 because the eigenvalues are defined at S = I_{2} (or X_{n} = b). It should be noted that \sqrt{n}(λ̂_{i} − 1) can be expressed exactly as functions of \sqrt{n}(S − I_{2}) (or of ξ_{n} = \sqrt{n}(X_{n} − b)). In fact,

\frac{1}{2} (\sqrt{n}(s_{11} − 1) + \sqrt{n}(s_{22} − 1) ± \sqrt{\{\sqrt{n}(s_{11} − 1) − \sqrt{n}(s_{22} − 1)\}^{2} + 4(\sqrt{n}s_{12})^{2}})
= F̄(\sqrt{n}(X_{n} − b), 0),

which converges in distribution to

F̄(Z, 0) = \frac{1}{2}(Z_{1} + Z_{3}) ± \sqrt{(Z_{1} − Z_{3})^{2} + 4Z_{2}^{2}}.

A more general case can be discussed where S is a p×p symmetric matrix and a_{n}(S −I_{p}) ⟶^{L} Z, a random matrix of order p. Let λ̂_{i}(S) be the i-th eigenvalue of S, and then

a_{n}(λ̂_{i}(S) − 1) = λ̂_{i}{a_{n}(S − I_{p})} ⟶^{L} λ̂_{i}(Z)

because of the continuity of eigenvalues. The essence of the derivation is to express the eigenvalues as functions of a_{n}(S−I_{p}) (or of ξ_{n}) as in the assumption of Theorem 1. Eaton and Tyler (1991) used this idea to derive the asymptotic distribution of eigenvalues in a more general setting.

3. THEOREM 2 AND EXAMPLES

There are some cases where the assumption on F_{i}(x) and h_{i}(x) in (8) are not met. Such an example follows Theorem 2 below. In this section, we shall assume differentiability of F_{i}(x) and h_{i}(x) rather than the assumption in (8). Obviously, the estimators in (1) and (17) are not differentiable at X_{n} = b for example, so one theorem does not include the other.

For the sake of notational simplicity, we shall write

A^{<s>} = \overbrace{A ⊗ ⋯ ⊗ A}^{s times} and d^{s}F (b) = \overbrace{(\frac{∂}{∂x′} ) ⊗ ⋯ ⊗ (\frac{∂}{∂x′} )}^{s times} F (x)|_{x=b}

for a differentiable function F (·). Here the symbol ⊗ denotes the (right) Kronecker product. The formal Taylor expansion of F (x) about x = b is then expressible in the form:

F (x) = ∑_{s=0}^{∞} \frac{d^{s}F (b)}{s!}(x − b)^{<s>} (= ∑_{s=0}^{∞} \frac{1}{s!} [∑_{j=1}^{p}(x_{j} − b_{j})\frac{∂}{∂x_{j}} ]^{s} F (b)) .

The following theorem is obtained.

Theorem 2. Let D be an open domain in R^{p}, let b ∈ D, and let a_{n} be a positive sequence which tends to infinity as n → ∞. Let X_{n} be a sequence of random p-vectors with P (X_{n} ∈ D \ {b}) = 1 for each n and let Z be a nondegenerate continuous random p-vector. Suppose that a_{n}(X_{n} − b) ⟶^{L} Z as n → ∞. Let f(x) = [f_{1}(x), …, f_{q}(x)]′ be a measurable function from D \ {b} to R^{q}. Assume that there exists a measurable function h(x) = [h_{1}(x), …, h_{q}(x)]′ from D to R^{q} such that for each i (i = 1, …, p),

lim_{x→b} f_{i}(x)h_{i}(x) = c_{i} < ∞.  (18)

Define

F_{i}(x) = \{ \begin{matrix}f_{i}(x)h_{i}(x) & if x ≠ b \\ c_{i} & if x = b \end{matrix} .

Assume that F_{i}(x) and h_{i}(x) are differentiable around x = b.

Let s_{i} and t_{i} be minimum nonnegative integers such that d^{s_{i}}F_{i}(b) ≠ 0 and

d^{t_{i}}h_{i}(b) ≠ 0, respectively. Define G_{i}(Z) = \frac{d^{s_{i}}F_{i}(b)Z^{<s_{i}>}/s_{i}!}{d^{t_{i}}h_{i}(b)Z^{<t_{i}>}/t_{i}!}. Then, we have

a_{n}^{s_{i}−t_{i}}f_{i}(X_{n}) ⟶^{L} G_{i}(Z) as n → ∞.  (19)

Assume further that s_{i} = t_{i} and that d^{t_{i}}F_{i}(b) = β_{i}d^{t_{i}}h_{i}(b) for some β_{i} ∈ R^{1}. Then,

f_{i}(X_{n}) ⟶^{P} G_{i}(Z) = β_{i} as n → ∞.  (20)

Let u_{i} be a minimum nonnegative integer such that d^{u_{i}}F_{i}(b) − β_{i}d^{u_{i}}h_{i}(b) ≠ 0. Then we have

a_{n}^{u_{i}−t_{i}}(f_{i}(X_{n}) − β_{i}) ⟶^{L} \frac{(d^{u_{i}}F_{i}(b) − β_{i}d^{u_{i}}h_{i}(b))Z^{<u_{i}>}/u_{i}!}{d^{t_{i}}h_{i}(b)Z^{<t_{i}>}/t_{i}!}.  (21)

Note 1: Orders of differentiability required are not specified in the theorem. Stating strictly, differentiability of orders lower than or equal to the order where a nonzero derivative appears at first is required for this theorem.

Note 2: When the limit c_{i} in (18) is nonzero, then s_{i} = 0.

Note 3: The scalar β_{i} is nonzero since d^{t_{i}}F_{i}(b) ≠ 0 and d^{t_{i}}h_{i}(b) ≠ 0 by definition.

Note 4: The scalar u_{i} is greater than t_{i} since d^{t_{i}}F_{i}(b) − β_{i}d^{t_{i}}h_{i}(b) = 0 by assumption.

Proof. The proof is similar to that of Theorem 1. It follows from the differentiability that

[ \begin{matrix}a_{n}^{s_{i}}F_{i}(X_{n}) \\ a_{n}^{t_{i}}h_{i}(X_{n}) \end{matrix} ] = [ \begin{matrix}d^{s_{i}}F_{i}(b)\{a_{n}(X_{n} − b)\}^{<s_{i}>}/s_{i}! \\ d^{t_{i}}h_{i}(b)\{a_{n}(X_{n} − b)\}^{<t_{i}>}/t_{i}! \end{matrix} ] + o_{p}(1)
⟶^{L} [ \begin{matrix}d^{s_{i}}F_{i}(b)Z^{<s_{i}>}/s_{i}! \\ d^{t_{i}}h_{i}(b)Z^{<t_{i}>}/t_{i}! \end{matrix} ] .  (22)

Use of (13) and (22) results in

Pr ( ⋂_{i=1}^{q} a_{n}^{s_{i}−t_{i}}f_{i}(X_{n}) < x_{i})

→ Pr ( ⋂_{i=1}^{q} {d^{s_{i}}F_{i}(b)Z^{<s_{i}>}/s_{i}! − x_{i}d^{<t_{i}>}h_{i}(b)Z^{<t_{i}>}/t_{i}! < 0,

d^{<t_{i}>}h_{i}(b)Z^{<t_{i}>}/t_{i}! > 0}

∪ {d^{s_{i}}F_{i}(b)Z^{<s_{i}>}/s_{i}! − x_{i}d^{t_{i}}h_{i}(b)Z^{t_{i}}/t_{i}! > 0,

d^{t_{i}}h_{i}(b)Z^{<t_{i}>}/t_{i}! < 0 ^{} )}

= Pr ( ⋂_{i=1}^{q} \frac{d^{s_{i}}F_{i}(b)Z^{<s_{i}>}/s_{i}!}{d^{t_{i}}h_{i}(b)Z^{<t_{i}>}/t_{i}!} < x_{i}) .

Note that Pr(d^{t_{i}}h_{i}(b)Z^{<t_{i}>} = 0) = 0. The result in (21) follows from a similar argument.

Q.E.D.

Let us return to the example of choosing of the number of components described in Section 1. Here we consider the case π = 1 as before. We can easily verify that

d^{1}h_{1}(b) = [0, 1] and d^{1}F_{1}(b) = [0, 1], so that s_{1} = t_{1} = 1,
d^{1}h_{2}(b) = [1, 0] and d^{1}F_{2}(b) = [μ_{1}, 1], so that s_{2} = t_{2} = 1.

Since d^{1}h_{1}(b) = d^{1}F_{1}(b) and so β_{1} = 1, application of (20) leads to

π̂ = f_{1}(X_{n}) ⟶^{P} 1.

From (19), we have

μ̂_{2} = f_{2}(X_{n}) ⟶^{L} \frac{d^{1}F_{2}(b)Z}{d^{1}h_{2}(b)Z} = μ_{1} + \frac{Z_{2}}{Z_{1}} ,

which is exacly the cauchy distribution. Next, the asymptotic distribution of f_{1}(X_{n}) will be evaluated. Since F_{1}(x) − β_{1}h_{1}(x) = −(x_{1} − μ_{1})^{2}, d^{2}F_{1}(b) − β_{1}d^{2}h_{1}(b) = −[2, 0, 0, 0] ≠ 0, and hence u_{1} = 2. Application of (21) leads to

\sqrt{n}(π̂ − 1) = (\sqrt{n})^{u_{1}−s_{1}}(f_{1}(X_{n}) − β_{1}) ⟶^{L} \frac{(d^{2}F_{1}(b) − β_{1}d^{2}h_{1}(b))Z/2}{d^{1}h_{1}(b)Z} = −\frac{Z_{1}^{2}}{Z_{2}} .

A more interesting example is given where Theorem 1 can not apply but Theorem 2 can do. The (exploratory) factor analysis model can be characterized as a covariance structure of an observed random vector Y . The covariance structure of the factor analysis model is in the form:

Var(Y ) = Σ = ΛΛ′ + Ψ,  (23)

where Λ = (λ_{ij}) is a factor loading matrix of p × k and Ψ = diag(ψ_{1}, …, ψ_{p}) is a diagonal matrix of order p representing unique variances. The k is said to be the number of factors. The parameters (Λ, Ψ) are unknown and estimated from a sample variance matrix S, based on a sample of size n. Assume that the S has a continuous distribution and that

a_{n}(X_{n} − b) = a_{n}(v(S) − v(Σ)) ⟶^{L} Z,

where a_{n} ↗ ∞ and Z is a random vector continuously distributed. The so-called vec-operator v(S) for a symmetric matrix S of order p denotes a p(p + 1)/2-vector of the elements in the lower triangular part of S (see e.g., Magnus and Neudecker 1988). In practice, the number k of factors is unknown. Let m denote the number of assumed (or extracted) factors. In many cases, m may not be equal to k. When the number of factors is underestimated, i.e. m < k, the estimators are biased and the fit of the model will be poor. When it is overestimated, i.e. m > k, estimation encounters serious difficulties, such as nonconvergence in iteration. The anomaly occurs because the model is not identifiable. Take m = k+1 for example. There are infinitely many parameters for such models when the k-factor model holds as in (23). In fact, it holds that

Σ = [ c ; Λ 0 ; ⋮ ; 0 ] [ c ; Λ 0 ; ⋮ ; 0 ]′ + [ \begin{matrix}ψ_{1} − c^{2} \\ & ψ_{2} \\ & & . & . & . \\ & & & & & ψ_{p} \end{matrix} ]  (24)

for any number c. The direct noniterative estimator of Ihara and Kano (1986) and Kano (1990b) overcomes the identification problem. Assume that p ≥ 2m + 1, and partition the p variables into four groups as Y_{1}, Y_{2}-Y_{m+1}, Y_{m+2}- Y_{2m+1} and Y_{2m+2}-Y_{p}, where the last group can be empty. In this paper, we assume that m = k + 1 (so that p ≥ 2k + 3), and partition further each of the two groups of m variables into two subgroups of k variables and one variable. The corresponding partition of a factor loading matrix is made as

Λ = [ \begin{matrix}λ & 1 \\ Λ_{2} \\ λ & 3 \\ Λ_{4} \\ λ & 5 \\ Λ_{6} \end{matrix} ] \begin{matrix}\} & 1 \\ \} & k \\ \} & 1 \\ \} & k \\ \} & 1 \\ \} & p − (2k + 3) \end{matrix} .

Assume that the k × k matrices Λ_{2} and Λ_{4} are nonsingular. It is a famous sufficient condition for identification for the k-factor model (Anderson and Rubin 1956). Partition Σ, Ψ and S correspondingly. The noniterative estimator for ψ_{1} is expressed as

ψ̂_{1}^{(k+1)} = s_{11} − [s_{12}, s_{13}] [ \begin{matrix}S_{42} & s_{43} \\ s_{52} & s_{53} \end{matrix} ]^{−1} [ \begin{matrix}s & 41 \\ s_{51} \end{matrix} ] .

The inverse matrix involving the estimator above can be taken with probability one because the distribution of S is continuous. Since

[ \begin{matrix}S_{42} & s_{43} \\ s_{52} & s_{53} \end{matrix} ] ⟶^{P} [ \begin{matrix}Σ_{42} & σ_{43} \\ σ_{52} & σ_{53} \end{matrix} ] = [ \begin{matrix}Λ_{4} \\ λ & 5 \end{matrix} ] [ \begin{matrix}Λ_{2} \\ λ & 3 \end{matrix} ]′ ,

the limiting matrix is singular (its rank is k whereas the order is k + 1), so that the estimator ψ̂_{1}^{(k+1)} can not be defined at S = Σ. Does the estimator have a limiting point? If so, is the limiting point the true unique variance ψ_{1}? What kind of asymptotic distribution does it have? Theorem 2 answers these questions.

By use of a formula of the inversion of a partitioned matrix (see e.g., Anderson 1984, Theorem A.3.3), we have

ψ̂_{1}^{(k+1)}

= \frac{(s_{11} − s_{12}S_{42}^{−1}s_{41})(s_{53} − s_{52}S_{42}^{−1}s_{43}) − (s_{13} − s_{12}S_{42}^{−1}s_{43})(s_{51} − s_{52}S_{42}^{−1}s_{41})}{s_{53} − s_{52}S_{42}^{−1}s_{43}}

= \frac{F (X_{n})}{h(X_{n})}, say.

It should be noted that F (x) and h(x) do not satisfy the assumption (8) in Theorem 1, so we can not appy Theorem 1 to this problem.

When S = Σ (or X_{n} = b), we have
s_{11} − s_{12}S_{42}^{−1}s_{41} = λ_{1}λ′_{1} + ψ_{1} − λ_{1}Λ′_{2}(Λ_{4}Λ′_{2})^{−1}Λ_{4}λ′_{1} = ψ_{1}
s_{ij} − s_{i2}S_{42}^{−1}s_{4j} = λ_{i}λ′_{j} − λ_{i}Λ′_{2}(Λ_{4}Λ′_{2})^{−1}Λ_{4}λ′_{j} = 0
(for i, j = 1, 3, 5 and i ≠ j).

in view of the fact that Λ_{2} and Λ_{4} are nonsingular matrices. It follows that d^{1}h(b) ≠ 0 and d^{1}F (b) = ψ_{1}d^{1}h(b), which implies from (20) in Theorem 2 that

ψ̂_{1}^{(k+1)} ⟶^{P} \frac{d^{1}F (b)Z}{d^{1}h(b)Z} = ψ_{1}.

This shows consistency of the noniterative estimator for the case where m = k + 1, the number of factors is overestimated.

By usual delta theorem, we have

a_{n}(s_{11} − s_{12}S_{42}^{−1}s_{41} − ψ_{1}) ⟶^{L} d_{11}Z
a_{n}(s_{ij} − s_{i2}S_{42}^{−1}s_{4j}) ⟶^{L} d_{ij}Z

where d_{ij} = \frac{∂(s_{ij} − s_{i2}S_{42}^{−1}s_{4j})}{∂v(S)′} |_{S=Σ}. The asymptotic distribution of ψ̂_{1}^{(k+1)} is

obtained from (21) as

a_{n}(ψ̂_{1}^{(k+1)} − ψ_{1}) = \frac{a_{n}^{2}(F (X_{n}) − ψ_{1}h(X_{n}))}{a_{n}h(X_{n})}
⟶^{L} \frac{(d^{2}F (b) − ψ_{1}d^{2}h(b))Z^{<2>}/2}{d^{1}h(b)Z}
= \frac{(d_{11}Z)(d_{53}Z) − (d_{13}Z)(d_{51}Z)}{d_{53}Z}
= d_{11}Z − \frac{(d_{13}Z)(d_{51}Z)}{d_{53}Z}.

Equivalents to the derivatives d_{ij}’s were given in Kano (1990a).

We can take the same derivation to find the asymptotic distribution of ψ̂_{1}^{(m)} for a general m(≥ k + 1) as well.

Kano (1991) assumed normality of observations to derive the asymptotic distribution of ψ̂_{1}^{(m)} for an overestimated number of factors. The proof depends heavily on the Bartlett decomposition of the Wishart matrix nS. In this paper, we have taken a completely different derivation to obtain the asymptotic distribution under general distribution assumption.

Finally we would like to make a comment on the examples employed in this paper. The examples of selecting the number of components and factor analysis seem to differ instinctively. Identification is not achieved in both of the examples, that is, μ_{2} and ψ_{1} are indeterminate. It appears natural to us that μ̂_{2} does have a distribution even in the limit because of the luck of identification. How should we interpret the fact that ψ̂_{1} is consistent ? Although there are many parameter values that perfectly reproduce the same population covariance matrix Σ in (23), the most reasonable value is unique, and it is the value when c = 0 in (24) because the factor corresponding to the last column of the factor loading matrix in (24) is not a common factor but a unique factor. The noniterative estimator approaches the most reasonable value as the sample size goes to infinity. In contrast, usual estimators including the MLE in factor analysis do not have this property.

Inconsistency of estimators for indeterminate parameters could not be regarded as a result of luck of identification only, and alternative factors may have to be introduced to account for it. We can not yet find any solution to the problem as to when estimators are consistent or inconsistent for indeterminate parameters.

ACKNOWLEDGEMENTS

The author expresses his thanks to Professor Masanobu Taniguchi of Osaka University for his valuable comments on this work. He also thanks reviewers for their comments that help improve the original manuscript greatly.

BIBLIOGRAPHY

Akahira, M. and Takeuchi, K. (1995). Non-regular Statistical Estimation. Lec-

ture Notes in Statistics, Vol.107. Springer-Verlag: New York. Anderson, T. W. (1963). Asymptotic theory of principal component analysis.

Ann. Math. Statist., 34, 122-148. Anderson, T. W. (1984). An Introduction to Multivariate Statistical Analysis,

2nd ed. Wiley: New York.

Anderson, T. W. and Rubin, H. (1956). Statistical inference in factor analysis. Proc. 3rd Berkeley Symp. Math. Statist. Probab., 5, 111-150. University of California Press: Berkeley.

Basawa, I. V. and Scott, D. J. (1983). Asymptotic Optimal Inference for Nonergodic Models. Lecture Notes in Statistics, Vol.17. Springer-Verlag: New York.

Chen, J. and Kalbfleisch, J. D. (1993). Penalized minimum distance estimates in finite mixture models. Technical Report Series 94-08, University of Waterloo.

Chikuse, Y. (1976). Asymptotic distributions of the latent roots of the covariance matrix with multiple population roots. Jour. Multi. Anal., 6, 237-249.

Day, N. E. (1969). Estimating the components of a mixture of normal distributions. Biometrika, 56, 463-474.

Eaton, M. L. and Tyler, D. E. (1991). On Wielandt’s inequality and its application to the asymptotic distribution of the eigenvalues of a random symmetric matrix Ann. Statist., 19, 260-271.

Eaton, M. L. and Tyler, D. E. (1994). The asymptotic distribution of singular values with applications to canonical correlations and corresponding analysis. Jour. Multi. Anal., 50, 238-264.

Fujikoshi, Y. (1977). An asymptotic expansion of the distributions of the latent roots of the Wishart matrix with multiple population roots. Ann. Inst. Math. Statist., 29A, 379-387.

Fujikoshi, Y. (1980). Asymptotic expansions of the distributions of the sample roots under nonnormality. Biometrika, 67, 45-51.

Hartigan, J. A. (1985). A failure of likelihood asymptotics for normal mixture. In Proceedings of the Berkeley Conference in Honor of Jerzy Neyman and Jack Kiefer, Vol. 2. edited by L. LeCam and R. A. Olshen, pp. 807-810.

Hatsuyama, M. (1995). Selection of the number of components in a finite mixture distribution. Master’s Thesis (In Japanese). Osaka Prefecture University: Sakai, Osaka, Japan.

Henna, J.(1985). On estimating of the number of constituents of a finite mixture of continuous distributions. Ann. Inst. Statist. Math., 37, 235-240.

Ihara, M. and Kano, Y. (1986). A new estimator of the uniqueness in factor analysis. Psychometrika, 51, 563-566.

Kano, Y. (1990a). Comparative studies of non-iterative estimators based on Ihara and Kano’s method in exploratory factor analysis. Commun. Statist. Theo. Meth., 19, 431-444.

Kano, Y. (1990b). Noniterative estimation and the choice of the number of factors in exploratory factor analysis. Psychometrika, 55, 277-291.

Kano, Y. (1991). The asymptotic distribution of a noniterative estimator in exploratory factor analysis. Ann. Statist., 19, 272-282.

Leroux, B. (1992). Consistent estimation of a mixture distribution. Ann. Statist., 20, 1350-1360.

Magnus, J. R. and Neudecker, H. (1988). Matrix Differential Calculus with Applications in Statistics and Econometrics. John Wiley: Chichester.

Rao, C. R. (1973). Linear Statistical Inference and its Applications, 2nd ed. Wiley: New York.

Richardson, S. and Green P. J. (in press). On Baysian analysis of mixtures with an unknown number of components. Jour. Royal Statist. Soc., Series B.

Roeder K. and Wasserman, L. (1997). Practical Baysian density estimation using mixture of normals. Jour. Amer. Statist. Assoc., 92, 894-902.

Romano, J. P. and Siegel, A. F. (1986). Counterexamples in Probability and Statistics. Wadsdorth & Brooks/Cole Advanced Books & Software: Belmont, CA.

Roussas, G. G. (1972). Contiguity of Probability Measures: Some Applications in Statistics. Cambridge At The University Press: Cambridge.

Sugiura, N. (1973). Derivatives of the characteristic root of a symmetric or Hermitian matrix with two applications in multivariate analysis. Commun. Statist., 1, 393-417.

Sugiura, N. (1976). Asymptotic expansions of the distributions of the latent roots and latent vector of the Wishart and multivariate F matrices. Jour. Multi. Anal., 6, 500-525.

Sugiura, N. (1990). Graphs of the distribution of bivariate Wishart roots and their cumulants. Jour. Japan Statist. Soc., 20, 117-136.

Tanaka, K. (1996). Time Series Analysis: Nonstationary and Noninvertible Distribution Theory. John Wiley & Sons, Inc.: New York.

Teicher, H. (1963). Identifiability of finite mixtures. Ann. Math. Statist., 34, 1265-1269.

Tyler, D. E. (1983). The asymptotic distribution of principal component roots under local alternatives to multiple roots. Ann. Statist., 11, 1232-1242.

Waternaux, C. M. (1976). Asymptotic distribution of the sample roots for a nonnormal population. Biometrika, 63, 639-645.