Sankhyā : The Indian Journal of Statistics 1997, Volume 59, Series A, Pt. 2, pp.179-197

BEYOND THIRD-ORDER EFFICIENCY

By YUTAKA KANO University of Tsukuba

SUMMARY. Fifth-order (asymptotic) efficiency of the second-order bias-corrected MLE, minimizing the n^{−3} term of an expansion of the quadratic risk of Fisher-consistent estimators bias-corrected similarly, is established in a general curved exponential family with a structural parameter vector. A characterization theorem of the MLE in terms of its higher-order derivatives is provided, and an alternative bias correction factor is proposed. Both the characterization and the new bias correction play an important role in proving the fifth-order efficiency. The matrix form symmetric tensor and higher-order derivatives are utilized, rather than usual elementwise tensors with Einstein’s convention, to derive all the results of this article.

1. Introduction

The Fisher-Rao theorem [Fisher (1925) and Rao (1961, 1962)] states that the information loss of the MLE is minimal among an appropriate class of best asymptotic normal estimators in a curved multinomial model with a single parameter, and they say that the MLE is second-order efficient. A new terminology recently more often used for this efficiency is third-order efficiency, see e.g., Pfanzagl and Wefelmeyer [(1978), Note 1, page 2] and Ghosh (1994). We shall follow this terminology here. Rao (1963) points out the equivalence between two measures of third-order efficiency, namely information loss and quadratic risk. Since then, a lot of literature generalizes the basic Fisher-Rao theorem to many cases such as more general loss functions, general multi-parameter curved exponential family, regression models, and some time series models [see e.g., Ghosh and Subramanyam (1974), Efron (1975), Eguchi (1983), Amari (1985), Hosoya (1990), and Taniguchi and Watanabe (1994)]. In particular, we appreciate Efron’s introduction of statistical curvature, Amari’s geometrical methods in asymptotic theory, and developments on concentration probability by Pfanzagl and Wefelmeyer (1978), Ghosh, Sinha and Subramanyam (1979), Ghosh, Sinha and Wieand (1980), Akahira and Takeuchi (1981), Pfanzagl (1982), Takeuchi and Morimune (1985), and Taniguchi (1986, 1987, 1991). Note that Ghosh,

Paper received. July 1995; revised August 1996. AMS (1991) subject classifications. 62F12.

Key words and phrases. Curved exponential family, Fisher-consistency, higher-order matrix derivatives, maximum likelihood estimator, quadratic risk, symmetric tensor.

Sinha and Subramanyam (1979) gave a proof of the third-order efficiency of the MLE without Cramer’s condition on population distributions. Ghosh (1994) gives an excellent review on higher-order asymptotics.

Except for a few works, all the research on higher-order efficiency has aimed at studying to what extent - in what kind of models or under what kind of distribution assumption - the third-order efficiency holds, but has not proceed to more higher-order efficiency such as fifth-order. Efron [(1975), page 1210] mentions that the fifth-order term of Pitman’s estimator may be smaller than that of the MLE, so that the MLE may not be fifth-order efficient. Rao, Sinha and Subramanyam (1982) show, by very complicated computation, that the fifthorder efficiency of the MLE holds among a relatively small class of estimators in the multinomial model with a single parameter. On the contrary, Ghosh and Sinha (1982) present a counterexample that the MLE is not fifth-order efficient. Akahira (1995), recently, shows the fourth-order efficiency of the third-order efficient estimators, based on concentration probability.

Suppose that x_{1},…, x_{n} are independent and identically distributed pdimensional random sample drawn from a population with a curved exponential family of distributions defined as

exp{θ(u)′x − ψ(θ(u))}μ(dx),

where θ is a p-vector of natural parameters with structure θ(u), u is a q-vector of structural parameters (q < p), ψ(θ) is a normalizing factor, and μ(dx) is a carrier measure [see e.g., Amari (1985), Section 4.1]. Assume that θ and u are in certain open subsets of R^{p} and R^{q} and that ψ(θ) and θ(u) are continuously differentiable up to suitable order. Let \overline{x} = \frac{1}{n} ∑_{j=1}^{n} x_{j}, and let g(\overline{x}) be an estimator for u, where g(·) is continuously differentiable. Let E[\overline{x}] = η(u). Fisher-consistency requires

g(η(u)) = u  … (1.1)

for all possible u ∈ R^{q}. Let g^{∗}(\overline{x}) be a suitably bias-corrected version of g(\overline{x}), and let F be a class of such estimators g^{∗}(\overline{x}). It is known that the quadratic risk of g^{∗}(x̄) can be expanded (formally) as

E[(g^{∗}(\overline{x}) − u)(g^{∗}(\overline{x}) − u)′] = \frac{1}{n}V_{1}(u) + \frac{1}{n^{2}} V_{2}(u) + \frac{1}{n^{3}} V_{3}(u) + …

An estimator g^{∗}(\overline{x}) is said to be first-order efficient in a class F if and only if V_{1}(u) is minimal at all possible u among all estimators in F. A first-order efficient estimator is said to be third-order efficient in a class F if and only if V_{2}(u) is minimal at all possible u among all first-order efficient estimators in F. In the same manner, we define fifth-order efficiency [cf. Amari (1985), Section 5.1].

The paper makes two main contributions to the estimation problem in the curved exponential family. One is to give a characterization theorem to the matrices \overline{G}_{k} of k-th order derivatives of the MLE û(\overline{x}) (Theorem 3.1), which states that for each k, \overline{G}_{k} is a solution to a certain minimization problem under some linear constraints (see Section 2.2 for the precise definition of \overline{G}_{k}). The second is to establish the fifth-order efficiency of the bias-corrected MLE with a certain bias correction factor among the class of all third-order efficient Fisherconsistent estimators bias-corrected in a similar way, employing quadratic risk as a measure of efficiency (Theorem 4.2).

We shall end this section by stating the idea of this article. Let Cov(\overline{x}, \overline{x}) = \frac{1}{n}Ψ_{11}, and let G_{k} be a q × p^{k} matrix of the k-th order derivatives of g(\overline{x}). The delta theorem shows

V_{1}(u) = G_{1}Ψ_{11}G′_{1}.

It is well-known that the (bias-corrected) MLE minimizes V_{1}(u) under the condition (1.1), implying the first-order efficiency. We will extend the derivation to higher-order efficiency. That is, we will show that the bias-corrected MLE minimizes

V_{2}(u) = \frac{1}{2!}G_{2}(Ψ_{11} ⊗ Ψ_{11})G′_{2}

under the condition that g^{∗}(\overline{x}) be first-order efficient and bias-corrected. As a result, the bias-corrected MLE is third-order efficient. Looking at these facts, one will expect that the bias-corrected MLE again minimizes

V_{3}(u) = \frac{1}{3!}G_{3}(Ψ_{11} ⊗ Ψ_{11} ⊗ Ψ_{11})G′_{3} (+ terms free from estimators)

under the condition that g^{∗}(\overline{x}) be third-order efficient and bias-corrected suitably. This establishes the fifth-order efficiency of the MLE. One of the results of this article is to prove the above expectation if one makes a particular bias correction.

The derivation described above relies upon the two facts : for any (k − 1)-th order efficient estimator, i) V_{k}(u) is essentially representable as G_{k}(Ψ_{11} ⊗… ⊗ Ψ_{11})G′_{k}, ii) the MLE minimizes the quadratic form G_{k}(Ψ_{11} ⊗… ⊗ Ψ_{11})G′_{k}. In Section 3 we prove the statement ii) for any k, which is a sort of characterization of the MLE. The statement i) is shown for k = 1, 2, 3 in Section 4. Section 2 describes some preliminary results on special matrices, higher-order matrix derivatives, moments, and the MLE.

2. Preliminaries

In this section, we summarize many useful formulas that help reduce computation very much. In the later sections, the formulas will often be used without specific citations. Elementary proofs of the formulas are given in Kano (1993).

2.1 Commutation matrix and symmetric tensor. The commutation matrix K_{pq} is defined by the relation :

K_{pq}(b ⊗ a) = (a ⊗ b)

for all p-vector a and q-vector b, where the symbol ⊗ denotes the (right) Kronecker product. For properties of the Kronecker product, see Rao [(1973), Section 1b.8]. Henderson and Searle (1981) present a good review of K_{pq}. The following basic results hold. See Magnus and Neudecker [(1979); (1988), page 47] for proofs.

Proposition 2.1. Let A, B and b be of order p_{1} × p_{2}, q_{1} × q_{2} and q_{1} × 1.

i) K_{pq}^{−1} = K′_{pq} = K_{qp}, K_{p1} = K_{1p} = I_{p}, and K_{pp} is symmetric;

ii) K_{q_{1}p_{1}}(A ⊗ B)K_{p_{2}q_{2}} = B ⊗ A, and K_{q_{1}p_{1}}(A ⊗ b) = b ⊗ A;

iii) K_{p,qr}(I_{q} ⊗ K_{rp}) = K_{pq} ⊗ I_{r}.

Magnus and Neudecker [(1988), page 48] also define N_{p^{<2>}} = (I_{p^{2}} + K_{pp})/2. It has the property:

N_{p^{<2>}}(a_{1} ⊗ a_{2}) = (a_{1} ⊗ a_{2} + a_{2} ⊗ a_{1})/2

for p-vectors a_{1}, a_{2}. The matrix N_{p^{<2>}} is equivalent to M_{p} defined by Browne (1974), who uses M_{p} to express the covariance matrix of a vectorized sample covariance matrix. We generalize N_{p^{<2>}} to the case of k-fold Kronecker product. Define the p^{k} × p^{k} matrix N_{p^{<k>}} by

N_{p^{<k>}}(a_{1} ⊗… ⊗ a_{k}) = ∑_{\binom{1...k}{σ(1)...σ(k)}∈S_{k}} (a_{σ(1)} ⊗… ⊗ a_{σ(k)}) /k!,

where a_{j}’s are any p-vectors and S_{k} denotes the k-th symmetric group, that is, the summation is taken over all permutations of (1,…, k). Note that N_{p^{<1>}} = I_{p} and N_{p^{<k>}} = 1 for p = 1. The matrix is known to be symmetrizer or symmetric tensor in tensor analysis. In statistics, Holmquist (1988) uses N_{p^{<k>}} to express the higher-order moments of N_{p}(0, Σ) in matrix form. We simply write k-fold Kronecker product of the same matrix A as

A^{<k>} = A ⊗… ⊗ A,

and it is convenient to define A^{<0>} = 1. For proofs of the next proposition, see Sternberg [(1964), Section 1.3] and Satake [(1974), Section 5.3].

Proposition 2.2.

i) N_{p^{<k>}} is symmetric, idempotent and rank (N_{p^{<k>}}) = \binom{p+k−1}{k}: ii) N_{p^{<k>}}a^{<k>} = a^{<k>} for p-vector a;

iii) N_{p^{<k>}}A^{<k>} = A^{<k>}N_{q^{<k>}} = N_{p^{<k>}}A^{<k>}N_{q^{<k>}} for A of p × q;

iv) (I_{p^{k_{1}}} ⊗ K_{p^{k_{2}} ,p^{k_{3}}} ⊗ I_{p^{k_{4}}} )N_{p^{<k>}} for any k_{j} ≥ 0 with ∑_{j}^{4} k_{j} = k; (I_{p^{k_{1}}} ⊗ N_{p^{<k_{2}>}} ⊗ I_{p^{k_{3}}} )N_{p^{<k>}} = N_{p^{<k>}} for any k_{j} ≥ 0 with ∑_{j}^{3} k_{j} = k.

2.2 Higher-order matrix derivatives. Let x = [x_{1},…, x_{p}]′ and A(x) = (a_{ij}(x)) be an a_{1} × a_{2} matrix-valued function. Put \frac{∂A(x)}{∂x_{k}} = (\frac{∂a_{ij}(x)}{∂x_{k}}), an a_{1} × a_{2} matrix. Then we define

\frac{dA(x)}{dx′} = \frac{d}{dx′} A(x) = [\frac{∂A(x)}{∂x_{1}},…, \frac{∂A(x)}{∂x_{p}}] ,

which is of order a_{1} × pa_{2} and

\frac{dA(x)}{dx} = \frac{d}{dx}A(x) = [ ⋮_{\frac{∂A(x)}{∂x_{p}}}^{\frac{∂A(x)}{∂x_{1}}} ] ,

which is of order pa_{1} × a_{2}. The definition coincides with that of Magnus and Neudecker [(1988, page 114] when the variate is a vector.

The higher-order derivatives are defined inductively by (\frac{d}{dx′} )^{<k>} = \frac{d}{dx′} (\frac{d}{dx′} )^{<k−1>}. Let g(x) be a q-vector valued analytic function. The Taylor series of g(x) about x_{0} is expressible in the form :

g(x) = ∑_{k=0}^{∞} \frac{G_{k}}{k!} (x − x_{0})^{<k>} with G_{k} = (\frac{d}{dx′} )^{<k>} g(x)|x=x_{0} .

The matrix of the derivatives, G_{k}, has an important property :

G_{k}N_{p^{<k>}} = G_{k},  … (2.1)

The following are basic but useful formulas on matrix differentiations [cf. Bentler and Lee (1978) and Magnus and Neudecker (1985)].

Proposition 2.3. Let x be a p-vector and let A(x) and B(x) be of order a_{1} × a_{2} and b_{1} × b_{2}. Suppose that the following matrix products in the LHS are permissible.

i) \frac{d}{dx′} [A(x)B(x)] = \frac{dA(x)}{dx′}(I_{p} ⊗ B(x)) + A(x)\frac{dB(x)}{dx′};

ii) \frac{d}{dx′} [A(x) ⊗ B(x)] = (\frac{dA(x)}{dx′} ⊗ B(x)) + (A(x) ⊗ \frac{dB(x)}{dx′}) (K_{a_{2}p} ⊗ I_{b_{2}});

iii) \frac{d}{dt′} [A(x(t))B] = \frac{dA(x)}{dx′} (\frac{dx(t)}{dt′} ⊗ B), where x(t) is a p-vector valued function of t and B is a constant matrix.

The corresponding formulas on (\frac{d}{dx})^{<k>} are easily obtained in view of the

fact that ((\frac{d}{dx})^{<k>}A(x))′ = (\frac{d}{dx′} )^{<k>} A(x)′.

2.3 Moments. Let x_{1},…, x_{n} be independent and identically distributed random p-vectors with E[x_{1}] = η, and let

Ψ_{kl} = E[(x_{1} − η)^{<k>}(x_{1} − η)^{<l>′}].

Put z_{n} = \sqrt{n}(\overline{x}−η), where \overline{x} is the sample mean vector. Using the symmetrizer N_{p^{<k>}} in Section 2.1, we can express some expectations of the sample moments in a fairly simple form :

E[z_{n}z′_{n}]  = Ψ_{11}, E[z_{n}^{<2>}] = Ψ_{20}, E[z_{n}^{<3>}] = \frac{1}{\sqrt{n}}Ψ_{30};
E[z_{n}^{<2>}z_{n}^{<2>′}] = 2N_{p^{<2>}}Ψ_{11}^{<2>}N_{p^{<2>}} + Ψ_{20}Ψ_{02} + O (\frac{1}{n});
E[z_{n}^{<3>}z′_{n}]  = 3N_{p^{<3>}}(Ψ_{20} ⊗ Ψ_{11}) + O (\frac{1}{n}) = 3N_{p^{<3>}}(Ψ_{11} ⊗ Ψ_{20}) + O (\frac{1}{n});
E[z_{n}^{<4>}]  = 3N_{p^{<4>}}Ψ_{20}^{<2>} + O (\frac{1}{n});
E[z_{n}^{<3>}z_{n}^{<3>′}] = 6N_{p^{<3>}}Ψ_{11}^{<3>}N_{p^{<3>}}

+9N_{p^{<3>}}(Ψ_{20} ⊗ Ψ_{11} ⊗ Ψ_{02})N_{p^{<3>}} + O (\frac{1}{n}) .

Notice that

N_{p^{<3>}}(Ψ_{02} ⊗ Ψ_{11} ⊗ Ψ_{20})N_{p^{<3>}} = N_{p^{<3>}}(Ψ_{20} ⊗ Ψ_{11} ⊗ Ψ_{02})N_{p^{<3>}} … (2.2)

by virtue of N_{p^{<3>}}. Moreover N_{p^{<3>}}(Ψ_{20}⊗Ψ_{11}⊗Ψ_{02})N_{p^{<3>}} remains unchanged under any permutations of Ψ_{20}, Ψ_{11}, Ψ_{02}.

Some immediate consequences involving N_{p^{<k>}} and moment matrices are ready for later use. Let G_{3} = (\frac{d}{dx′} )^{<3>} g(x)|_{x=η} as in Section 2.2. It follows that

G_{3}E[(z_{n}^{<3>} − 3(z_{n} ⊗ Ψ_{20}))z′_{n}]

= G_{3} [3N_{p<3>} (Ψ_{11} ⊗ Ψ_{20}) + O (\frac{1}{n}) − 3(Ψ_{11} ⊗ Ψ_{20})] = O (\frac{1}{n}) … (2.3a)

in view of (2.1). Noting (2.2), a similar argument leads to

G_{3}E[(z_{n}^{<3>} − 3(z_{n} ⊗ Ψ_{20}))z_{n}^{<3>′}]G′_{3} = 6G_{3}Ψ_{11}^{<3>}G′_{3} + O (\frac{1}{n}) , … (2.3b)

G_{3}E[(z_{n}^{<3>} − 3(z_{n} ⊗ Ψ_{20}))(z_{n}^{<3>} − 3(z_{n} ⊗ Ψ_{20}))′]G′_{3} = 6G_{3}Ψ_{11}^{<3>}G′_{3} + O (\frac{1}{n}) .

… (2.3c)

Let

Ψ_{kl}(θ) = (\frac{d}{dθ} )^{<k>} (\frac{d}{dθ′} )^{<l>} ψ(θ).

In particular, E[x_{1}] = Ψ_{10}(θ(u)) = η(u), being the expectation parameter. It is also known that

E[(x_{1} − η(u))^{<k>}(x_{1} − η(u))^{<l>′}] = Ψ_{kl}(θ(u)) for k + l = 2, 3.

Accordingly, the same symbol Ψ_{kl} is used as the moment matrices.

2.4 MLE Let

Θ_{1}(u) = \frac{d}{du′} θ(u).

The likelihood function L(u|\overline{x}) of u based on the sample x_{1},…, x_{n} is of the form : L(u|\overline{x}) = exp\{n(θ(u)′\overline{x} − ψ(θ(u)))\}, and the derivatives of the loglikelihood is written as

\frac{d}{du}logL(u|\overline{x}) = nΘ_{1}(u)′\{\overline{x} − η(u)\} = \sqrt{n}Θ_{1}(u)′z_{n}  … (2.4)

with z_{n} = \sqrt{n}\{\overline{x}−η(u)\}. The MLE û = û(\overline{x}) is assumed to be a unique solution to the equation

Θ_{1}(u)′\{\overline{x} − η(u)\} = 0.  … (2.5)

It is also known that the Fisher information matrix of u is given as

i_{u} = Θ_{1}(u)′Ψ_{11}(θ(u))Θ_{1}(u),  … (2.6)

and that

\frac{dû(\overline{x})}{d\overline{x}′} |_{\overline{x}=η(u)} = i_{u}^{−1}Θ_{1}(u)′.  … (2.7)

Basically, the following sections take over these definitions and notations. For simplicity, sometimes, let us write η, Θ_{l} and Ψ_{kl} for η(u), Θ_{l}(u) and Ψ_{kl}(θ(u)) respectively if there is no confusion.

3. Characterization of the MLE

When ψ(θ) and θ(u) are k+1 times continuously differentiable and the Fisher information matrix i_{u} in (2.6) is positive definite, then the implicit function theorem guarantees that the likelihood equation (2.5) uniquely determines the MLE û(\overline{x}) in a neighborhood of u for all \overline{x} in a neighborhood of η(u) and that û(\overline{x}) is k times continuously differentiable. We write

\overline{G}_{j} = \overline{G}_{j}(η(u)) = (\frac{d}{d\overline{x}′} )^{<j>} û(\overline{x})|_{\overline{x}=η(u)} (j = 1,…, k).  … (3.1)

Let us denote by g(\overline{x}) an arbitrary Fisher-consistent estimator as in Section 1. When it is differentiable, we write

G_{j} = G_{j}(η(u)) = (\frac{d}{d\overline{x}′} )^{<j>} g(\overline{x})|_{\overline{x}=η(u)} (j = 1,…, k).  … (3.2)

It is convenient to write \overline{G}_{0} = û(η(u)) = u and G_{0} = g(η(u)) = u.

The following theorem asserts that the derivative matrix \overline{G}_{k} is obtained as a solution to an optimization problem subject to some linear constraints. Let N be the set of natural numbers.

Theorem 3.1 Let k ∈ N be arbitrarily fixed. Assume that ψ(θ) and θ(u) are k + 1 times continuously differentiable and that i_{u} and Ψ_{11} are positive definite. Let \overline{G}_{k} be a matrix of the k-th order derivatives of the MLE û(\overline{x}), defined by (3.1). Then, \overline{G}_{k} is a unique solution to the minimization problem :

min_{X:q×p^{k}} XΨ_{11}^{<k>}X′

subject to

X(Ψ_{11}Θ_{1} ⊗ I_{p^{k−1}}) = \frac{d}{du′} \overline{G}_{k−1}(η(u)),  … (3.3a)
XN_{p^{<k>}} = X.  … (3.3b)

The theorem leads to the following important relation between the derivatives of the two estimators û(\overline{x}) and g(\overline{x}).

Corollary 3.2 In addition to the assumptions in Theorem 3.1, suppose that a Fisher-consistent estimator g(\overline{x}) is k times continuously differentiable, and let G_{k} be defined in (3.2). If G_{k−1} = \overline{G}_{k−1}, then G_{k}Ψ_{11}^{<k>}G′_{k} ≥ \overline{G}_{k}Ψ_{11}^{<k>}\overline{G}′_{k}. The equality holds if and only if G_{k} = \overline{G}_{k}.

Remark 3.1 For any Fisher-consistent estimator, we have G_{0}(η(u)) = u = \overline{G}_{0}(η(u)), which implies that the assumption in Corollary 3.2 is met at k = 1. Thus, G_{1}Ψ_{11}G′_{1} ≥ \overline{G}_{1}Ψ_{11}\overline{G}′_{1}. This shows that the MLE is first-order efficient. In addition, g(\overline{x}) is first-order efficient if and only if G_{1} = \overline{G}_{1}. Such logic will be used to prove third- and fifth-order efficiency in the next section.

Remark 3.2 From the proof of the corollary stated later, it will be seen that the constraint (3.3a) comes from G_{k−1} = \overline{G}_{k−1}. What does (3.3b) imply? An implication thereof can be described as follows : Some columns of the derivatives \overline{G}_{k} are functionally dependent, for example, \frac{∂^{2}û(\overline{x})}{∂x̄_{1}∂x̄_{2}} and \frac{∂^{2}ū(\overline{x})}{∂x̄_{2}∂x̄_{1}} (at x̄ = η) are identical but located in different columns in Ḡ_{2}. The constraint XN_{p^{<k>}} = X reflects such restrictions. Note that the number of functionally

independent columns of Ḡ_{k} is \binom{p+k−1}{k}, which equals the rank of N_{p^{<k>}} as shown in Proposition 2.2 i).

Remark 3.3 Eguchi (1985) uses differential-geometric terminology to present some result on Ḡ_{k}. His result may be related to the following lemma.

We state a lemma to prove Theorem 3.1. A proof thereof will be given in the appendix.

Lemma 3.3 For any k ∈ N, there exists a q × qp^{k−1} matrix B_{k} such that

Ḡ_{k} = B_{k}(Θ′_{1} ⊗ I_{p^{k−1}})N_{p^{<k>}}.  … (3.5)

Proof of Theorem 3.1. It follows that

XΨ_{11}^{<k>}X′ = (X − Ḡ_{k})Ψ_{11}^{<k>}(X − Ḡ_{k})′ + Ḡ_{k}Ψ_{11}^{<k>}Ḡ′_{k}

because

(X − Ḡ_{k})Ψ_{11}^{<k>}Ḡ′_{k} = (X − Ḡ_{k})N_{p^{<k>}}Ψ_{11}^{<k>}(Θ_{1} ⊗ I_{p^{k−1}})B′_{k}
= (X − Ḡ_{k})(Ψ_{11}Θ_{1} ⊗ I_{p^{k−1}})(I_{q} ⊗ Ψ_{11}^{<k−1>})B′_{k}
= 0.

The first equality holds from Lemma 3.3 and Proposition 2.2 iii); the second and last equalities are due to the constrains in (3.3b) and (3.3a). The uniqueness is obvious.

Q.E.D.

Proof of Corollary 3.2 Remember that G_{k−1} = G_{k−1}(η(u)) with η(u) = Ψ_{10}(θ(u)). Thus, differentiation of G_{k−1} = Ḡ_{k−1} leads to G_{k}(Ψ_{11}Θ_{1} ⊗ I_{p^{k−1}}) = \frac{d}{du′} Ḡ_{k−1}(η(u)) in view of Proposition 2.3 iii). Note that G_{k}N_{p^{<k>}} = G_{k}, as described in (2.1). Thus, the matrix G_{k} satisfies the constraints on X in (3.3). As a result, G_{k}Ψ_{11}^{<k>}G′_{k} ≥ Ḡ_{k}Ψ_{11}^{<k>}Ḡ′_{k}. The uniqueness of Ḡ_{k} in the minimization problem proves the last statement.

Q.E.D.

4. Asymptotic efficiency

In this section, we develop the theory of higher-order asymptotic efficiency. Recall that û(x̄) and g(x̄) are the MLE and an arbitrary Fisher-consistent, differentiable estimator. We expand

û(x̄) = u + \frac{1}{\sqrt{n}}Ḡ_{1}z_{n} + \frac{1}{2n}Ḡ_{2}z_{n}^{<2>} + \frac{1}{6n\sqrt{n}}Ḡ_{3}z_{n}^{<3>} + o_{p} (\frac{1}{n\sqrt{n}})

and

g(x̄) = u + \frac{1}{\sqrt{n}}G_{1}z_{n} + \frac{1}{2n}G_{2}z_{n}^{<2>} + \frac{1}{6n\sqrt{n}}G_{3}z_{n}^{<3>} + o_{p} (\frac{1}{n\sqrt{n}})

with z_{n} = \sqrt{n}(x̄ − η(u)).

The bias of g(x̄) has the form :

E[g(x̄) − u] = \frac{1}{2n}G_{2}Ψ_{20} + \frac{1}{6n^{2}} G_{3}Ψ_{30} + \frac{1}{8n^{2}} G_{4}Ψ_{20}^{<2>} + o (\frac{1}{n^{2}} ) .

The first-order bias correction is made with

\frac{1}{2n}Ĝ_{2}Ψ_{20},  … (4.1)

so that

E[g(x̄) − \frac{1}{2n}Ĝ_{2}Ψ_{20} − u]

= \frac{1}{n^{2}} [− ( \frac{1}{4} \frac{d}{dx̄′}^{<2>}Ĝ_{2}Ψ_{20}|_{x^{}¯ =η(u)}) Ψ_{20} + \frac{1}{6}G_{3}Ψ_{30} + \frac{1}{8}G_{4}Ψ_{20}^{<2>}] + o (\frac{1}{n^{2}} )
= \frac{1}{n^{2}} b_{2} + o (\frac{1}{n^{2}} ) say.

Here b_{2} is said to be the second-order bias.

Let b̂_{2} be an estimator of b_{2}. We now consider a family F of second order bias corrected Fisher-consistent estimators :

F = \{g^{∗}(x̄) = g(x̄) − \frac{1}{2n}Ĝ_{2}Ψ_{20} − \frac{1}{n^{2}} b̂_{2}|

E[g^{∗}(x̄) = u] = o (\frac{1}{n^{2}} ) , g(η(u)) = u for all possible u ∈ R^{q}\} … (4.2)

The class F depends on choice of bias correction factors. Here we take

Ĝ_{2}Ψ_{20} = G_{2}(x̄)Ψ_{20}(θ(g(x̄))).  … (4.3)

Let us refer to the bias-correction factor in some details. The only requirement for Ĝ_{2}Ψ_{20} in (4.1) is E[Ĝ_{2}Ψ_{20}] = G_{2}Ψ_{20} + o (\frac{1}{n}) in order to prove third-order efficiency. The estimation method does not matter. On the other hand, the requirement is not enough to study fifth-order optimality because the distribution of Ĝ_{2}Ψ_{20} does influence upon the n^{−3} terms. The counterexample in Ghosh and Sinha (1982) shows that the usual bias correction factor does not work, which is given by Ĝ_{2}Ψ_{20} = G_{2}(η(g(x̄)))Ψ_{20}(θ(g(x̄))). The choice for an estimator Ĝ_{2}Ψ_{20} in (4.3) enables us to establish the fifth-order efficiency of the MLE.

Theorem 4.1. Assume that ψ(θ) and θ(u) are four times continuously differentiable and i_{u} in (2.6) is positive definite. Let û(x̄) and g(x̄) be, respectively, the MLE and an arbitrary Fisher-consistent, three times continuously differentiable estimator, with Ḡ_{k} in (3.1) and G_{k} in (3.2) being matrices of the k-th order derivatives of û(x̄) and g(x̄). Let F be defined in (4.2) with (4.3). Let û^{∗}(x̄) and g^{∗}(x̄), in the F, be second-order bias corrected versions of û(x̄) and g(x̄). Then

i) for any g^{∗}(x̄) ∈ F,

\begin{aligned}E[g^{∗}(x̄) − u)(g^{∗}(x̄) − u)′] & = & \frac{1}{n}G_{1}Ψ_{11}G′_{1} − \frac{1}{n}Ḡ_{1}Ψ_{11}Ḡ′_{1} \\ & & +E[(û^{∗}(x̄) − u)(û(x̄)^{∗} − u)′] + o (\frac{1}{n}); \end{aligned}

… (4.4a)

ii) for any g^{∗}(x̄) ∈ F with G_{1} = Ḡ_{1},

\begin{aligned}E[g^{∗}(x̄) − u)(g^{∗}(x̄) − u)′] & = & \frac{1}{2n^{2}} G_{2}Ψ_{11}^{<2>}G′_{2} − \frac{1}{2n^{2}} Ḡ_{2}Ψ_{11}^{<2>}Ḡ′_{2} \\ & & +E[(û^{∗}(x̄) − u)(û(x̄) − u)′] + o (\frac{1}{n^{2}} ); \end{aligned}

… (4.4b)

iii) for any g^{∗}(x̄) ∈ F with G_{1} = Ḡ_{1} and G_{2} = Ḡ_{2},

\begin{aligned}E[g^{∗}(x̄) − u)(g^{∗}(x̄) − u)′] & = & \frac{1}{3!n^{3}} G_{3}Ψ_{11}^{<3>}G′_{3} − \frac{1}{3!n^{3}} Ḡ_{3}Ψ_{11}^{<3>}Ḡ′_{3} \\ & & +E[(û^{∗}(x̄) − u)(û(x̄) − u)′] + o (\frac{1}{n^{3}} ) . \end{aligned}

… (4.4c)

Remark 4.1 (4.4a) holds for estimators without bias correction; (4.4b) holds for those without second-order bias correction.

Remark 4.2 The terms depending on the estimator g(x̄) are G_{k}Ψ_{11}^{<k>}G′_{k} with k = 1, 2, 3; the other terms relate only to the MLE.

Remark 4.3 The second and third terms in the RHS in (4.4a) are canceled to be o (\frac{1}{n}); those in (4.4b) reduce to \frac{1}{n} Ḡ_{1}Ψ_{11}Ḡ′_{1} [see Kano (1993) for a proof].

We are now in a position to study higher-order asymptotic efficiency with the help of Corollary 3.1 and Theorem 4.1. We shall make a similar discussion to that in Remark 3.1. Assume that Ψ_{11} is positive definite. Recall that û^{∗}(x̄) and g^{∗}(x̄) belong to F. The quadratic risk of g^{∗}(x̄) can be expressed as (4.4a), and Corollary 3.1 asserts that g^{∗}(x̄) minimizes the quadratic form G_{1}Ψ_{11}G′_{1} in F if and only if G_{1} = Ḡ_{1}. Thus, the estimators g^{∗}(x̄) meeting G_{1} = Ḡ_{1} are first-order efficient in F, and particularly the (bias-corrected) MLE û^{∗}(x̄) is first-order efficient in F. The quadratic risk of a first-order efficient estimator g^{∗}(x̄) can be expressed as (4.4b), and Corollary 3.1 asserts that g^{∗}(x̄) minimizes the quadratic form G_{2}Ψ_{11}^{<2>}G′_{2} in the class of first-order efficient estimators in F if and only if G_{2} = Ḡ_{2}. Thus, the estimators g^{∗}(x̄) meeting G_{2} = Ḡ_{2} are thirdorder efficient in F, and particularly the bias-corrected MLE û^{∗}(x̄) is third-order efficient in F.

The same argument holds for fifth-order efficiency. Indeed, let g^{∗}(x̄) be third-order efficient, that is, G_{1} = Ḡ_{1} and G_{2} = Ḡ_{2}, and hence the quadratic risk of g^{∗}(x̄) can be expressed as (4.4c). Corollary 3.2 shows g^{∗}(x̄) minimizes the G_{3}Ψ_{11}^{<3>}G′_{3} in the class of third-order efficient estimators in F if and only if G_{3} = Ḡ_{3}. Thus, the second-order bias-corrected MLE û^{∗}(x̄) is fifth-order efficient in F.

We summarize the above results as the following two statements.

Theorem 4.2 In addition to the assumptions in Theorem 4.1, assume that Ψ_{11} is positive definite. Then, an estimator g^{∗}(x̄) ∈ F is first-order efficient in F if and only if G_{1} = Ḡ_{1}; a first-order efficient estimator g^{∗}(x̄) is third-order efficient in F if and only if G_{2} = Ḡ_{2}; finally a third-order efficient estimator g^{∗}(x̄) is fifth-order efficient in F if and only if G_{3} = Ḡ_{3}.

Corollary 4.3 Assume the same assumptions as in Theorem 4.2. The second-order bias-corrected MLE û^{∗}(x̄) ∈ F be first-, third- and fifth-order efficient in F.

Remark 4.4 It follows from Remark 4.1 that the first-order efficiency (the third-order efficiency) in Theorem 4.2 and Corollary 4.3 holds for a class of estimators without bias correction (without second-order bias correction).

Before proving Theorem 4.1, we state a useful formula in evaluating expectations involving the score function, cf. Akahira and Takeuchi [(1981), Lemma 5.1.1], Amari [(1985), page 124] and Taniguchi and Watanabe [(1994), formula (5.10)]. A proof of the lemma will be given in the appendix.

Lemma 4.4 Let A(z_{n}|u) be of a_{1} × a_{2}. Then

E[z′_{n}Θ_{1} ⊗ A(z_{n}|u)] = \frac{1}{\sqrt{n}} \{\frac{d}{du′} E[A(z_{n}|u)] − E [\frac{d}{du′} A(z_{n}|u)]\} …. (4.5)

Note two things : if A(z_{n}|u) is independent of u, the second term of the RHS in (4.5) disappears; for a vector-valued function a(z_{n}|u), z′_{n}Θ_{1} ⊗a(z_{n}|u) = a(z_{n}|u)z′Θ_{1}.

Proof of Theorem 4.1. The result (4.4a) is obvious. As a consequence of the bias correction, we have

\frac{1}{n}E[g^{∗}(x̄) − û^{∗}(x̄))] = o (\frac{1}{n^{2}} ) or o (\frac{1}{n^{3}} )  … (4.6)

and

E [(g^{∗}(x̄) − û^{∗}(x̄)) (\frac{1}{\sqrt{n}}Ḡ_{1}z_{n})′] = o (\frac{1}{n^{2}} ) or o (\frac{1}{n^{3}} ) ,  … (4.7)

where the RHS’s are o(\frac{1}{n^{2}} ) if the first-order bias correction, with (4.1), is made; they are o(\frac{1}{n^{3}} ) if the second-order bias correction is made. The evaluation (4.7) is obtained because the LHS is equal to

\frac{1}{\sqrt{n}}E[(g^{∗}(x̄) − û^{∗}(x̄))z′_{n}Θ_{1}]i_{u}^{−1} = \frac{1}{n} \frac{d}{du′} E[g^{∗}(x̄) − û^{∗}(x̄)]i_{u}^{−1}

in view of (2.7) and Lemma 4.4, and then (4.6) applies.

Let us show (4.4b). We have

û^{∗}(x̄) − u = \frac{1}{\sqrt{n}}Ḡ_{1}z_{n} + \frac{1}{2n}Ḡ_{2}(z_{n}^{<2>} − Ψ_{20}) + o_{p} (\frac{1}{n}) ,  … (4.8)

and g^{∗}(x̄) permits the same expansion with G_{2} for Ḡ_{2}. Therefore,

g^{∗}(x̄) − û^{∗}(x̄) = \frac{1}{2n}(G_{2} − Ḡ_{2})(z_{n}^{<2>} − Ψ_{20}) + o_{p} (\frac{1}{n}) ,

which implies

E[(g^{∗}(x̄) − û^{∗}(x̄))(g^{∗}(x̄) − û^{∗}(x̄))′] = \frac{1}{2n^{2}} (G_{2} − Ḡ_{2})Ψ_{11}^{<2>}(G_{2} − Ḡ_{2})′ + o (\frac{1}{n^{2}} )

… (4.9)

and

E [(g^{∗}(x̄) − û^{∗}(x̄)) (\frac{1}{2n}Ḡ_{2}(z_{n}^{<2>} − Ψ_{20}))′] = \frac{1}{2n^{2}} (G_{2}−Ḡ_{2})Ψ_{11}^{<2>}Ḡ′_{2}+o (\frac{1}{n^{2}} ) .

… (4.10)

Use of (4.7), (4.8) and (4.10) leads to

E[(g^{∗}(x̄) − û^{∗}(x̄))(û^{∗}(x̄) − u)′] = \frac{1}{2n^{2}} (G_{2} − Ḡ_{2})Ψ_{11}^{<2>}Ḡ′_{2} + o (\frac{1}{n^{2}} ) …. (4.11)

The result (4.4b) follows from (4.9), (4.11) and the idea that g^{∗}(x̄) − u = (g^{∗}(x̄) − û^{∗}(x̄)) + (û^{∗}(x̄) − u).

The result (4.4c) can be established on the same lines. We first note that the factor (4.3) permits an expansion as

\frac{1}{2n}G_{2}Ψ̂_{20} = \frac{1}{2n}Ḡ_{2}Ψ_{20} + \frac{1}{2n\sqrt{n}}\{G_{3}(I_{p} ⊗ Ψ_{20}) + Ḡ_{2}Ψ_{21}Θ_{1}Ḡ_{1}\}z_{n} + o_{p} (\frac{1}{n\sqrt{n}}) ,

and the corresponding factor for û(x̄) has the same expansion with Ḡ_{3} for G_{3}. Thus we have

û^{∗}(x̄) − u = \frac{1}{\sqrt{n}}(I_{q} − \frac{1}{2n}Ḡ_{2}Ψ_{21}Θ_{1})Ḡ_{1}z_{n}

+\frac{1}{2n}Ḡ_{2}(z_{n}^{<2>} − Ψ_{20}) + \frac{1}{6n\sqrt{n}}Ḡ_{3}(z_{n}^{<3>} − 3(z_{n} ⊗ Ψ_{20})) + o_{p} (\frac{1}{n\sqrt{n}})

… (4.12)

192 yutaka kano and

g^{∗}(x̄) − û^{∗}(x̄) = \frac{1}{6n\sqrt{n}}(G_{3} − Ḡ_{3})(z_{n}^{<3>} − 3(z_{n} ⊗ Ψ_{20})) + o_{p} (\frac{1}{n\sqrt{n}}) …. (4.13)

Therefore, using (2.3c), we have

E[g^{∗}(x̄) − û^{∗}(x̄))(g^{∗}(x̄) − û^{∗}(x̄))′] = \frac{1}{6n^{3}} (G_{3} − Ḡ_{3})Ψ_{11}^{<3>}(G_{3} − Ḡ_{3})′ + o (\frac{1}{n^{3}} ) .

… (4.14)

We can evaluate

E [(g^{∗}(x̄) − û^{∗}(x̄)) (\frac{1}{2n}Ḡ_{2}(z_{n}^{<2>} − Ψ_{20}))′]

= \frac{1}{2n}E[(g^{∗}(x̄) − û^{∗}(x̄))(z′_{n}Θ_{1} ⊗ z′_{n})]B′_{2} + o (\frac{1}{n^{3}} )

(by (4.6), Lem. 3.3 and Prop. 2.2 ii))

= \frac{1}{2n}E[z′_{n}Θ_{1} ⊗ \{(g^{∗}(x̄) − û^{∗}(x̄))z′_{n}\}]B′_{2} + o (\frac{1}{n^{3}} )
= \frac{1}{2n\sqrt{n}} \{\frac{d}{du′} E[g^{∗}(x̄) − û^{∗}(x̄)z′_{n}] − E [\frac{d}{du′} (g^{∗}(x̄) − û^{∗}(x̄)z′_{n}]\} B′_{2} + o (\frac{1}{n^{3}} )

(by Lem. 4.4)

= \frac{1}{12n^{3}} \frac{d}{du′} \{(G_{3} − Ḡ_{3})E[(z_{n}^{<3>} − 3(z_{n} ⊗ Ψ_{20}))z′_{n}]\}B′_{2}

+\frac{1}{2n}E[g^{∗}(x̄) − û^{∗}(x̄)]\frac{dη′}{du′} (I_{q} ⊗ B′_{2}) + o (\frac{1}{n^{3}} ) (by (4.13))

= o (\frac{1}{n^{3}} ) (by (2.3a) and (4.6)).

… (4.15)

Finally, we calculate

E [(g^{∗}(x̄) − û^{∗}(x̄)) (\frac{1}{6n\sqrt{n}}Ḡ_{3}z_{n}^{<3>})′]

= \frac{1}{36n^{3}} (G_{3} − Ḡ_{3})E[(z_{n}^{<3>} − 3(z_{n} ⊗ Ψ_{20}))z_{n}^{<3>′}]Ḡ_{3} + o (\frac{1}{n^{3}} ) (by (4.13))
= \frac{1}{6n^{3}} (G_{3} − Ḡ_{3})Ψ_{11}^{<3>}Ḡ′_{3} + o (\frac{1}{n^{3}} ) (by (2.3b)).
… (4.16)

Thus, use of (4.7), (4.12), (4.15) and (4.16) results in

E[(g^{∗}(x̄) − û^{∗}(x̄))(û^{∗}(x̄) − u)′] = \frac{1}{6n^{3}} (G_{3} − Ḡ_{3})Ψ_{11}^{<3>}Ḡ′_{3} + o (\frac{1}{n^{3}} )

which, along with (4.14), establishes (4.4c). Q.E.D.

5. Final remarks

One of the feature of our paper is that the explicit form of the derivatives Ḡ_{k} is not used to evaluate the quadratic risk, unlike in Rao, Sinha and Subramanyam (1982). Of course the explicit form, using higher-order derivatives of ψ(θ) and θ(u), can be obtained but is very complicated especially for k ≥ 3. Instead, we develop an important expression Ḡ_{k} = B_{k}(Θ′_{1} ⊗ I_{p^{k−1}})N_{p^{<k>}} in Lemma 3.3, which is the only property of Ḡ_{k} that we need in order to establish the higher-order efficiency. The idea successfully reduces the computation.

In this paper we have not discussed validity of the asymptotic expansions. All the results here are based on “approximate estimators”, obtained by Taylor expansions of estimators in F. To confirm the results will necessitate appropriate conditions as in Fuller [(1976), Theorem 5.4.3], or replacement with a truncated loss function in Ghosh and Subramanyam (1974).

There are many third-order efficient estimators. For instance, Rao, Sinha and Subramanyam (1982) and Eguchi (1983) have formed some estimators asymptotically equivalent to the MLE up to order n^{−2}; estimators obtained by iterating the scoring method twice are third-order efficient as well [see e.g., Eguchi (1983)]. Thus, the results developed here are useful in distinguishing them. Moreover, we emphasize that this is the first step to establish a beautiful property that the MLE would enjoy, namely absolute asymptotic efficiency, i.e., the MLE is asymptotically efficient at any order, see Rao, Sinha and Subramanyam [(1982), Remark 3.1]. We hope many researchers will jump in the area to establish the absolute asymptotic efficiency.

Appendix

In this section, we give proofs of Lemma 3.3 and Lemma 4.4

Proof of Lemma 3.3. Let

Θ_{11}(u) = \frac{d^{2}}{dudu′} θ(u)

Remember that

Ḡ_{k}(x) = (\frac{d}{dx̄′} )^{<k>} û(x̄) and Ḡ_{k}(u) = Ḡ_{k}(Ψ_{10}(θ(u)))

with Ψ_{10}(θ(u)) = η(u) and that the MLE satisfies the equation

Θ_{1}(û(x̄))′{x̄ − Ψ_{10}(θ(û(x̄))} = 0  … (A.1)

(see (2.5)). For simplicity we write

Θ_{1}, Ψ_{kl}, Ḡ_{k} for Θ_{1}(u), Ψ_{kl}(θ(u)), Ḡ_{k}(u)

and

Θ_{1}(x̄), Θ_{11}(x̄), Ψ_{kl}(x̄) for Θ_{1}(û(x̄)), Θ_{11}(û(x̄)), Ψ_{kl}(θ(û(x̄)).

Further, we denote i_{u}(x̄) = Θ_{1}(x̄)′Ψ_{11}(x̄)Θ_{1}(x̄). Then (A.1) becomes

Θ_{1}(x̄)′(x̄ − Ψ_{10}(x̄)) = 0.  … (A.2)

Differentiation of (A.2) with respect to x̄ and application of Proposition 2.3 i), iii) give

Θ_{11}(x̄)′{Ḡ_{1}(x̄) ⊗ (x̄ − Ψ_{10}(x̄))} + Θ_{1}(x̄)′{I_{p} − Ψ_{11}(x̄)Θ_{1}(x̄)Ḡ_{1}(x̄)} = 0,

or

i_{u}(x̄)Ḡ_{1}(x̄) = A_{0}(x̄)Ḡ_{1}(x̄) + Θ_{1}(x̄)′  … (A.3)

with A_{0}(x̄) = Θ_{11}(x̄)′{I_{q} ⊗ (x̄ − Ψ_{10}(x̄))}. Note that

A_{0}(Ψ_{10}) = 0.  … (A.4)

Using Proposition 2.1 ii), we calculate

\frac{d}{dx′^{}}¯ [i_{u}(x̄)Ḡ_{1}(x̄)] = \frac{di_{u}(x̄)}{dx̄′}K_{pq}(Ḡ_{1}(x̄) ⊗ I_{p})K_{pp} + i_{u}(x̄)Ḡ_{2}(x̄)
\frac{d}{dx′^{}}¯ [A_{0}(x̄)Ḡ_{1}(x̄)] = \frac{dA_{0}(x̄)}{dx̄′}K_{pq}(Ḡ_{1}(x̄) ⊗ I_{p})K_{pp} + A_{0}(x̄)Ḡ_{2}(x̄)
\frac{d}{dx′^{}}¯ Θ_{1}(x̄)′  = Θ_{11}(x̄)′(Ḡ_{1}(x̄) ⊗ I_{p})

and hence, differentiation of both sides in (A.3), postmultiplication by N_{p<2>}, and use of (2.1) and Proposition 2.2 iv) results in

i_{u}(x̄)Ḡ_{2}(x̄) = A_{0}(x̄)Ḡ_{2}(x̄) + A_{21}(x̄)(Ḡ_{1}(x̄) ⊗ I_{p})N_{p^{<2>}}

with A_{21}(x̄) = \frac{dA_{0}(x^{}¯ )}{dx′^{}}¯ K_{pq} + Θ_{11}(x̄)′ − \frac{di_{u}(x^{}¯ )}{dx′^{}}¯ K_{pq}, being of order q × pq. We can show by induction that for any α ≥ 2,

i_{u}(x̄)Ḡ_{α}(x̄) = A_{0}(x̄)Ḡ_{α}(x̄) + ∑_{β=1}^{α−1} A_{αβ}(x̄)(Ḡ_{β}(x̄) ⊗ I_{p^{α−β}})N_{p^{<α>}} … (A.5)

for some A_{αβ}(x̄) of order q × qp^{α−β}. The proof is obtained by differentiating both sides of (A.5) and postmultiplying them by N_{p^{<α+1>}}. Here we just notice that

\frac{d}{dx̄′} [A_{αβ}(x̄)(Ḡ_{β}(x̄) ⊗ I_{p^{α−β}})N_{p^{<α>}}]

= (\frac{dA_{αβ}(x̄)}{dx̄′}(I_{p} ⊗ Ḡ_{β}(x̄) ⊗ I_{p^{α−β}}) + A_{αβ}(x̄)(Ḡ_{β+1}(x̄) ⊗ I_{p^{α−β}})) (I_{p} ⊗ N_{p^{<α>}})
= (\frac{dA_{αβ}(x̄)}{dx̄′}K_{p,qp^{α−β}}(Ḡ_{β}(x̄) ⊗ I_{p^{α+1−β}})K_{p^{α},p}
+A_{αβ}(x̄)(Ḡ_{β+1}(x̄) ⊗ I_{p^{α−β}}))(I_{p} ⊗ N_{p^{<α>}})

and (I_{p} ⊗ N_{p^{<α>}})N_{p^{<α+1>}} = N_{p^{<α+1>}} and K_{p^{α},p}N_{p^{<α+1>}} = N_{p^{<α+1>}} [see Prop. 2.2 iv)]. Evaluating (A.5) at x̄ = Ψ_{10} and using (A.4) gives

i_{u}Ḡ_{α} = ∑_{β=1}^{α−1} A_{αβ}(Ḡ_{β} ⊗ I_{p^{α−β}})N_{p^{<α>}}  … (A.6)

for α ≥ 2, where A_{αβ} = A_{αβ}(Ψ_{10}). Recall that Ḡ_{1} = i_{u}^{−1}Θ′_{1} in (2.7), and hence we have from (A.6) with α = 2

Ḡ_{2} = B_{2}(Θ′_{1} ⊗ I_{p})N_{p^{<2>}},

where B_{2} = i_{u}^{−1}A_{21}(i_{u}^{−1}⊗I_{p}). Thus, (3.5) is seen to be true for k = 1, 2. Suppose that (3.5) holds true for k = 1, 2,…, α. Then, substitution of (3.5) into (A.6) with α + 1 for α gives

Ḡ_{α+1} = i_{u}^{−1} ∑_{β=1}^{α} A_{α+1,β}{(B_{β}(Θ′_{1} ⊗ I_{p^{β−1}})N_{p^{<β>}}) ⊗ I_{p^{α+1−β}}}N_{p^{<α+1>}}

= i_{u}^{−1} ∑_{β=1}^{α} A_{α+1,β}(B_{β} ⊗ I_{p^{α+1−β}})(Θ′_{1} ⊗ I_{p^{β−1}} ⊗ I_{p^{α+1−β}})(N_{p^{<β>}} ⊗ I_{p^{α+1−β}})N_{p^{<α+1>}}
= B_{α+1}(Θ′_{1} ⊗ I_{p^{α}})N_{p^{<α+1>}}

with B_{α+1} = i_{u}^{−1} ∑_{β=1}^{α} A_{α+1,β}(B_{β} ⊗ I_{p^{α+1−β}}). This proves Lemma 3.3 by induction Q.E.D.

Proof of Lemma 4.4. Use of (2.4) gives

E[z′_{n}Θ_{1}⊗A(z_{n}|u)] = \frac{1}{\sqrt{n}}E [\frac{dlogL(u|x̄)}{du′} ⊗ A(z_{n}|u)] = \frac{1}{\sqrt{n}} ∫ \frac{dL(u|x̄)}{du′}⊗A(z_{n}|u)μ(dX).

Proposition 2.3 ii) helps differentiate

\frac{d}{du′} [L(u|x̄) ⊗ A(z_{n}|u)] = \frac{dL(u|x̄)}{du′} ⊗ A(z_{n}|u) + L(u|x̄).\frac{dA(z_{n}|u)}{du′}.

196 yutaka kano

These equalities complete the proof, provided that suitable exchangeability in order between differentiation and integration is permitted. Q.E.D.

Acknowledgment. The author would like to express his thanks to Professor M. Akahira of University of Tsukuba, Dr. M. Taniguchi of Osaka University, Dr. S. Eguchi of the Institute of Statistical Mathematics and Dr. Y. Takagi of University of Osaka Prefecture for stimulating discussions on higher-order efficiency.

References

Akahira, M. (1995). Third order efficiency implies fourth order efficiency : A resolution of the conjecture of J.K. Ghosh. To appear in Ann. Inst. Statist. Math.

Akahira, M. and Takeuchi, K. (1981). Asymptotic Efficiency of Statistical Estimators : Concepts and Higher Order Asymptotic Efficiency, Lecture Notes in Statistics, Vol. 7. Springer-Verlag, New York.

Amari, S. Differential-Geometrical Methods in Statistics, Lecture Notes in Statistics, Vol. 28. Springer-Verlag, Berlin.

Bentler, P.M. and Lee, Sik-Yum (1978). Matrix derivatives with chain rule and rules for simple, Hadamard, and Kronecker products. J. Math. Psychol. 17, 255-262.

Browne, M.W. (1974). Generalized least squares estimators in the analysis of covariance structures. South Afr. Statist. J., 8, 1-24.

Efron, B. (1975). Defining the curvature of a statistical problem (with applications to second order efficiency). Ann. Statist., 3, 1189-1242.

Eguchi, S. (1983). Second order efficiency of minimum contrast estimators in a curved exponential family. Ann. Statist., 11, 793-803.

− − −− (1985). Model fidelity of the maximum likelihood estimator in a curved exponential family. In Statistical Theory and Data Analysis (K. Matusita, ed.) 207-223. North-Holland, Amsterdam.

Fisher, R.A. (1925). Theory of statistical estimation. Proc. Cambridge Philos. Soc. 122, 700-725.

Fuller, W.A. (1976). Introduction to Statistical Time Series. Wiley, New York.

Ghosh, J.K. (1994). Higher Order Asymptotics. NSF-CBMS Regional Conference Series in Probability and Statistics 4. IMS, Hayward CA.

Ghosh, J.K. and Sinha, B.K. (1982). Third order efficiency of the MLE − A counterexample. Calcutta Statist. Assoc. Bull., 31, 151-158.

Ghosh, J.K., Sinha, B.K. and Subramanyam, K. (1979). Edgeworth expansions for Fisher-consistent estimators and second order efficiency. Calcutta Statist. Assoc. Bull., 28, 1-18.

Ghosh, J.K., Sinha, B.K. and Wieand, H.S. (1980). Second order efficiency of the MLE with respect to any bounded bowl-shaped loss function. Ann. Statist. 8, 506-521.

Ghosh, J.K. and Subramanyam, K. (1974). Second order efficiency of maximum likelihood estimates. Sankhyā, Ser. A, 36, 325-358.

Henderson, H.V. and Searle, S.R. (1981). The vec-permutation matrix, the vec operator and Kronecker products : A review. Linear and Multilinear Algebra, 9, 271-288.

beyond third-order efficiency 197

Holmquist, B. (1988). Moments and cumulants of the multivariate normal distribution. Stoch. Anal. App. 6, 273-278.

Hosoya, Y. (1990). Information amount and higher-order efficiency in estimation. Ann. Inst. Statist. Math., 42, 37-49.

Kano, Y. (1993). Beyond third-order efficiency : Supplement. Technical Report, University of Osaka Prefecture.

Magnus, J.R. and Neudecker, H. (1979). The commutation matrix : Some properties and applications. Ann. Statist. 7, 381-394.

− − −− (1985). Matrix differential calculus with applications to simple, Hadamard, and Kronecker products. J. Math. Psychol. 29, 474-492.

− − −− (1988). Matrix Differential Calculus with Applications in Statistics and Econometrics. John Wiley, Chichester.

Pfanzagl, J. (1982). Contributions to General Asymptotic Statistical Theory, Lecture Notes in Statistics, Vol. 13, Springer-Verlag, New York.

Pfanzagl, J. and Wefelmeyer, W. (1978). A third order optimum property of the maximum likelihood estimator. J. Multivariate Anal., 8, 1-29.

Rao, C.R. (1961). Asymptotic efficiency and limiting information. Proc. Fourth Berkeley Symp. 1, 531-545.

− − −− (1962). Efficient estimates and optimum inference procedures in large samples. J. Royal Statist. Soc., Ser. B, 24, 46-72.

− − −− (1963). Criteria of estimation in large samples. Sankhyā, Ser. A, 25, 189-206.

− − −− (1973). Linear Statistical Inference and its Applications, 2nd ed. Wiley, New York.

Rao, C.R., Sinha, B.K. and Subramanyam, K. (1982). Third order efficiency of the maximum likelihood estimator in the multinomial distribution. Statistics & Decisions, 1, 1-16.

Satake, I. (1974). Linear Algebra, 2nd ed. (in Japanese). Shokabo, Tokyo.

Sternberg, S. (1964). Lectures on Differential Geometry. Prentice-Hall Inc., Englewood Cliffs, NJ.

Takeuchi, K. and Morimune, M. (1985). Third-order efficiency of the extended maximum likelihood estimators in a simultaneous equation system. Econometrica, 53, 177-200.

Taniguchi, M. (1986). Third order asymptotic properties of maximum likelihood estimators for Gaussian ARMA processes. J. Multivariate Anal. 18, 1-31.

− − −− (1987). Third order asymptotic properties of BLUE and LSE for a regression model with ARMA residual. J. Time Ser. Anal., 8, 111-114.

− − −− (1991). Higher Order Asymptotic Theory for Time Series Analysis, Lecture Notes in Statistics, Vol. 68, Springer-Verlag, Berlin.

Taniguchi, M. and Watanabe, Y. (1994). Statistical analysis of curved probability densities. J. Multivariate Anal., 48, 228-248.

Institute of Mathematics University of Tsukuba Tsukuba, Ibaraki 305 Japan