THIRD-ORDER EFFICIENCY IMPLIES FOURTH-ORDER EFFICIENCY

Yutaka KANO^{∗} (February 13, 1996)

Abstract. Takeuchi [37], Takeuchi and Akahira [38] and Pfanzagl [27] among others proved that any first-order efficient estimators are second-order efficient. Many authors e.g., Ghosh [15], have conjectured that any third-order efficient estimators are fourth-order efficient. Based on concentration probability of estimators about a true parameter, this paper gives a positive answer to the conjecture in a curved exponential family with multi-structural parameters. It is seen that choice of bias-correction factors is critical.

1. Introduction Let u ∈ ℝ^{q} be a parameter vector of interest and let an open set Ω(⊂ ℝ^{q}) be

a parameter space of u. Let u_{0} ∈ Ω be an arbitrarily fixed (inner) point of Ω. For every u ∈ N (u_{0}), a neighborhood of u_{0}, assume that random p-vectors x_{1},…, x_{n} are independent and identically distributed according to a (continuous) curved exponential family with density

(1.1)  exp {θ(u)′x − ψ(θ(u))} μ(dx),

where μ(·) is a carrier measure on ℝ^{p} (see Amari [6], Section 4.1) and θ(·) is an ℝ^{p}-valued measurable function from Ω(⊂ ℝ^{q}) smooth in u ∈ N (u_{0}). The real-valued function, ψ(θ), is defined as a normalizing constant ψ(θ) = log ∫_{ℝ^{p}} exp(θ′x)μ(dx). Write Θ_{1} = \frac{d}{du′} θ(u), Ψ_{10} = \frac{dψ}{dθ} and Ψ_{11} = \frac{d^{2}ψ}{dθdθ′}. It is well-known that

E[x_{i}] = Ψ_{10}(θ(u)) (= η(u), say) and Cov[x_{i}, x_{i}] = Ψ_{11}(θ(u)) (= Ψ_{11}, say)

and that the Fisher information matrix is expressed as

i_{u} = Θ′_{1}Ψ_{11}Θ_{1}.

Assume that Ψ_{11} and i_{u} are positive definite.

Typeset by AMS-TEX

∗Institute of Mathematics, University of Tsukuba, Ibaraki 305, Japan

Let x̄ = \frac{1}{n} ∑_{i=1}^{n} x_{i} and then E[x̄] = η(u). Let g(·), b_{g}^{(1)}(·) and b_{g}^{(2)}(·) be measurable functions from ℝ^{p} to ℝ^{q}, four times continuously differentiable in N (η(u_{0})), a neighborhood of η(u_{0}). The statistics b_{g}^{(i)}(x̄)’s are bias-correction factors, possibly to be zero. For given b_{g}^{(i)}(x̄)’s, consider a class of possibly bias-corrected Fisher consistent estimators for u, defined as

F = \{g^{∗}(x̄) = g(x̄) − \frac{1}{n}b_{g}^{(1)}(x̄) − \frac{1}{n^{2}} b_{g}^{(2)}(x̄)| g(η(u)) = u for every u ∈ N (u_{0})\} .

We say that g^{∗}(x̄) is first-order (or second-order) bias corrected if E[g^{∗}(x̄)−u] = O (\frac{1}{n^{2}} ) (or O (\frac{1}{n^{3}} )). The bias-correction factors are not unique, and the class F with one correctionfactor may be different from that with another factor. Discussing fourth-order efficiency here, we will deal with four different bias-correction factors.

We assume that the MLE û(x̄) that maximizes the likelihood function

(1.2)  L(u) = ∏_{i=1}^{n} exp{θ(u)′x_{i} − ψ(θ(u))} = exp{n(θ(u)′x̄ − ψ(θ(u))}

exists and that it is Fisher-consistent and four times continuously differentiable in N (η(u_{0})) a neighborhood of η(u_{0}), with a large probability for large n. The assumption is met under certain regularity conditions including strong identifiability (see Kano [19]).

Let C be a class of all Borel convex sets of ℝ^{q} including the origin and let C_{0} be a class of all Borel convex sets of ℝ^{q} symmetric about the origin. The class C_{0} is a subclass of C. We expand concentration probability of g^{∗}(x̄) about the true parameter vector u as

(1.3)  P_{θ(u)}\{\sqrt{n}(g^{∗}(x̄) − u) ∈ C\} = A_{1} + \frac{1}{\sqrt{n}}A_{2} + \frac{1}{n}A_{3} + \frac{1}{n\sqrt{n}}A_{4} + ⋯,

where C belongs to C or to C_{0}. The coefficients A_{k}’s depend on u, g^{∗}(·) and C. The expansion is made with the help of Edgeworth expansions.

An estimator in F is said to be first-order efficient (1OE) for the class C (or C_{0}) iff it maximizes A_{1} among all the estimators in F for every C ∈ C (or C ∈ C_{0}). A 1OE estimator is said to be second-order efficient (2OE) for the class C (or C_{0}) iff it maximizes A_{2} among all the 1OE estimators in F for every C ∈ C (or C ∈ C_{0}); 3OE and 4OE for the class C (or C_{0}) are defined consecutively in the same manner.

Whether an estimator in F is 1OE does not depend on the bias-correction factors; and the factor b_{g}^{(2)}(x̄) does not influence upon 2OE and 3OE.

The limiting distribution of \sqrt{n}(g^{∗}(x̄) − u) (or equivalently \sqrt{n}(g(x̄) − u)) is multivariate normal with mean zero and covariance matrix G_{1}Ψ_{11}G′_{1}, where G_{1} = \frac{dg(x̄)}{dx̄′}|_{x̄=η(u)}, and hence

A_{1} = ∫_{C} N_{q}(x|0, G_{1}Ψ_{11}G′_{1})dx.

It is known that

G_{1}Ψ_{11}G′_{1} ≥ i_{u}^{−1}

and that the equality holds if and only if G_{1} = i_{u}^{−1}Θ′_{1} = \frac{dû(x̄)}{dx̄′}|_{x̄=η(u)}, the matrix of the first-order derivatives of the MLE û(x̄) (see e.g., Takeuchi [37] example 6.5). As a result, we have

∫_{C} N_{q}(x|0, G_{1}Ψ_{11}G′_{1})dx ≤ ∫_{C} N_{q}(x|0, i_{u}^{−1})dx for every C ∈ C.

In other words, the MLE and estimators whose first-order derivatives are identical with the MLE are all 1OE for the class C.

Second-order efficiency has been discussed for a class of first-order bias-corrected estimators g^{∗}(x̄) which meet

E[g^{∗}(x̄) − u] = o (\frac{1}{n}) .

The 2OE of the (bias-corrected) MLE was established by showing that the second term A_{2} of the MLE attains the lower bound, which is derived by applying Neyman-Pearson’s lemma to a certain testing problem (Pfanzagl [26]; Akahira [1]). After the 2OE of the MLE, some authors including Chibisov [12], Takeuchi [37], Takeuchi and Akahira [38] and Pfanzagl [27] found that the term A_{2} is identical for any 1OE estimators, so that 1OE implies 2OE and that the notion of 2OE cannot distinguish 1OE estimators. The basic results on 2OE have been extended to more general and complicated models by many authors including Akahira and Takeuchi [3,4], Hosoya [18], Taniguchi [39] and Yoshida [42,43].

Consider Takeuchi’s and Pfanzagl’s surprising result via an alternative criterion, quadratic risk:

E[\{\sqrt{n}(g^{∗}(x̄) − u)\}\{\sqrt{n}(g^{∗}(x̄) − u)\}′] = B_{1} + \frac{1}{\sqrt{n}}B_{2} + \frac{1}{n}B_{3} + \frac{1}{n\sqrt{n}}B_{4} + · · · .

Higher-order efficiency based on the criterion is defined in the same way as that on concentration probability in (1.3). Since B_{1} = G_{1}Ψ_{11}G′_{1}, the asymptotic covariance matrix of g(x̄), we make the same conclusion on 1OE as before. We know that B_{2} = 0 for any estimators in F, which means that 1OE implies 2OE. Notice that B_{2} = 0 whether or not bias-correction is made and hence that 2OE holds for the class of 1OE estimators even without bias-correction.

Results on 3OE similar to those on 1OE hold under first-order bias-correction. That is, the MLE and estimators whose second-order derivatives coincide with those of the MLE are 3OE (e.g., Pfanzagl and Wefelmeyer [29]; Akahira and Takeuchi [4]; Amari [6]; Taniguchi [40]). Rao [30,31,32], Ghosh and Subramanyam [16], Efron [13] and Eguchi [14] have obtained 3OE of the MLE on the basis of quadratic risk or loss of information.

Here is an interesting question; does it hold that 3OE implies 4OE? On the issue, Ghosh [15](page 64) mentioned “given that 1OE implies 2OE, it is natural to conjecture that 3OE implies 4OE. The proof of that must be very messy.”

The statement is known to be true if one takes quadratic risk as a criterion, because B_{4} = 0. This fact would convince us that the conjecture be true under concentration probability as well as under quadratic risk.

In this paper, we distinguish between asymptotic efficiency for the class C and for the class C_{0}. The idea was taken by Akahira and Takeuchi [4] to study 3OE of estimators in models with general density functions. We will show that 3OE implies 4OE for C or for C_{0}, depending on a choice of bias-correction factors.

In Section 2, we give preliminary results on matrix derivatives and multivariate Hermite polynomials. Section 3 describes main results on asymptotic efficiency up to the fourth-order under several types of bias-correction factors. Proofs are given in Section 4.

The derivation of this paper is so-called formal, and we do not directly prove validity of the Edgeworth expansion (1.3). According to general theory, the formal expansion is actually valid because (i) the continuity of the curved exponential family (1.1) ensures the Cramer condition: lim sup_{||s||→∞} E[e^{is′(x−η(u))}] < 1; (ii) the smoothness of ψ(θ) and θ(u) is assumed, so that the moments of any orders exist; (iii) the class C of all measurable convex sets meets a boundary condition: sup_{C∈C} ∫_{(∂C)^{ϵ}} N_{q}(x|0, I_{q})dx = O(ϵ) (ϵ ↓ 0). For details, see Bhattacharya and Denker [10](Remark 1.4.1 and Theorem 2.1) or Bhattacharya and Rao [11] (Corollary 3.2).

2. Preliminary Results 2.1. Matrix derivatives and symmetric tensor

Let x = [x_{1},…, x_{p}]′ and A(x) = [a_{ij}(x)] be an a_{1}×a_{2} matrix-valued continuously differentiable function. Put \frac{∂A(x)}{∂x_{k}} = [\frac{∂a_{ij}(x)}{∂x_{k}}]. Define \frac{dA(x)}{dx′} = \frac{d}{dx′} A(x) = [\frac{∂A(x)}{∂x_{1}}, ⋯, \frac{∂A(x)}{∂x_{p}}] and \frac{d}{dx}A(x) = (\frac{d}{dx′} A(x)′)′. The higher-order (matrix) derivatives are defined inductively by (\frac{d}{dx′} )^{<k>} = \frac{d}{dx′} (\frac{d}{dx′} )^{<k−1>} with (\frac{d}{dx′} )^{<0>} = 1. For such matrix derivatives, see Kano [20,21] (cf. Bentler and Lee [9]; Magnus and Neudecker [25]). Here we simply note some formulas on the matrix differentiation. The proof is due to direct calculation (see Kano [21]). Let x be a p-vector, let A(x) and B(x) be of order a_{1} × a_{2} and b_{1} × b_{2} and suppose that the following matrix products in the LHS are permissible. We then have

\frac{d}{dx′} [A(x)B(x)] =\frac{dA(x)}{dx′}(I_{p} ⊗ B(x)) + A(x)\frac{dB(x)}{dx′};
\frac{d}{dx′} [A(x) ⊗ B(x)] = (\frac{dA(x)}{dx′} ⊗ B(x)) + (A(x) ⊗ \frac{dB(x)}{dx′}) (K_{a_{2}p} ⊗ I_{b_{2}}) ;
\frac{d}{dt′} [AB(x(t))C] =A\frac{dB(x)}{dx′} (\frac{dx(t)}{dt′} ⊗ C) ,

where x(t) is an ℝ^{p}-valued function of t and A and C are constant matrices in the last formula. Here the matrix K_{ab} of order ab is a commutation matrix defined by the relation:

K_{ab}(b ⊗ a) = a ⊗ b

for arbitrary a-vector a and b-vector b. See Henderson and Searle [17] and Magnus and Neudecker [25](page 47) for commutation matrices.

Let us simply write A^{<k>} = \overbrace{A ⊗ ⋯ ⊗ A}^{k-fold}, k-fold right Kronecker (tensor) product of the same matrix A. Note A^{<0>} = 1. The symmetric tensor for the Kronecker product of p-vectors a_{1},…, a_{k} is denoted by N_{p^{<k>}}, which operates as

N_{p^{<k>}}(a_{1} ⊗ ⋯ ⊗ a_{k}) = ∑ (a_{σ(1)} ⊗ ⋯ ⊗ a_{σ(k)}) /k!,

where the summation runs over all permutations (σ(1),…, σ(k)) of (1,…, k). See e.g. Satake [34](Section 5.3) and Sternberg [35](Section 1.3) for the symmetric tensor. Let g(x) be an ℝ^{q}-valued smooth function defined on a neighborhood of x_{0} ∈ ℝ^{p}. The (formal) Taylor series of g(x) about x_{0} is expressible in the form:

g(x) = ∑_{k=0}^{∞} \frac{G_{k}}{k!} (x − x_{0})^{<k>} with G_{k} = (\frac{d}{dx′} )^{<k>} g(x)|_{x=x_{0}} .

The matrix of the derivatives, G_{k}, has an important property:

(2.1)  G_{k}N_{p^{<k>}} = G_{k}.

Define

(2.2)  Ψ_{kℓ} = Ψ_{kℓ}(θ(u)) = (\frac{d}{dθ} )^{<k>} (\frac{d}{dθ′} )^{<ℓ>} ψ(θ)|_{θ=θ(u)} .

2.2 Multivariate Hermite polynomials

Let φ(s) = e^{−s′Ψs/2} with s a p-vector and Ψ p × p positive definite. The covariant Hermite polynomials are defined as

H_{k}(s|Ψ) = [(−\frac{d}{ds})^{<k>} φ(s)] /φ(s) (k = 0, 1, 2,…),

and the contravariant Hermite polynomials are

H̃_{k}(s|Ψ) = (Ψ^{−1})^{<k>}H_{k}(s|Ψ) (k = 0, 1, 2,…).

The first few polynomials are

H_{0}(s|Ψ) = 1, H_{1}(s|Ψ) = Ψs, H_{2}(s|Ψ) = (Ψs)^{<2>} − vec(Ψ);
H_{3}(s|Ψ) = (Ψs)^{<3>} − 3N_{p^{<3>}}(vec(Ψ) ⊗ Ψs),

and

H̃_{0}(s|Ψ) = 1 H̃_{1}(s|Ψ) = s, H̃_{2}(s|Ψ) = s^{<2>} − vec(Ψ^{−1});
H̃_{3}(s|Ψ) = s^{<3>} − 3N_{p^{<3>}}(vec(Ψ^{−1}) ⊗ s).

Let z have a multivariate normal distribution N_{p}(0, Ψ). We then have E[e^{is′z}] = φ(s) and

E[e^{is′z}z^{<k>}z^{<ℓ>′}] = i^{k+ℓ} (−\frac{d}{ds′} )^{<ℓ>} (−\frac{d}{ds})^{<k>} φ(s)

= i^{k+ℓ} (−\frac{d}{ds′} )^{<ℓ>} [H_{k}(s|Ψ)φ(s)]

for k, ℓ = 0, 1, 2,… According to Barndorff-Nielsen and Cox [8](Eq. (5.67)) or Takemura [36](page 239), we know that E[e^{is′z}H_{k}(z|Ψ^{−1})] = (is)^{<k>}φ(s), and hence

(2.3)  E[e^{is′z}H̃_{k}(z|Ψ^{−1})] = (iΨs)^{<k>}φ(s).

We further note that

(2.4)  E[e^{is′z}H̃_{k}(z|Ψ^{−1})z^{<ℓ>′}] = i^{k+ℓ} (−\frac{d}{ds′} )^{<ℓ>} [(Ψs)^{<k>}φ(s)] .

Recall that x_{1},…, x_{n} are an i.i.d. sample from the curved exponential family (1.1). Let us replace z with z_{n} = \sqrt{n}(x̄−η(u)), and then all the formulas above still hold provided that we substitute Ψ_{11} for Ψ and add o(1); for example, the corresponding result to (2.3) is

(2.5)  E[e^{is′z_{n}} H̃_{k}(z_{n}|Ψ_{11}^{−1})] = (iΨ_{11}s)^{<k>}φ(s) + o(1)

with φ(s) = e^{−s′Ψ_{11}s/2}.

3. Main Results We begin with second-order efficiency. Let z_{n} = \sqrt{n}(x̄ − η(u)), G_{k} = G_{k}(η(u)) =

(\frac{d}{dx̄′} )^{<k>} g(x̄)|_{x̄=η(u)} and Ḡ_{k} = Ḡ_{k}(η(u)) = (\frac{d}{dx̄′} )^{<k>} û(x̄)|_{x̄=η(u)} for k = 0, 1, 2,… We expand g(x̄) in F and the MLE û(x̄) stochastically in the form:

(3.1)

g(x̄) = u + \frac{1}{\sqrt{n}}G_{1}z_{n} + \frac{1}{2n}G_{2}z_{n}^{<2>} + \frac{1}{6n\sqrt{n}}G_{3}z_{n}^{<3>} + ⋯,
û(x̄) = u + \frac{1}{\sqrt{n}}Ḡ_{1}z_{n} + \frac{1}{2n}Ḡ_{2}z_{n}^{<2>} + \frac{1}{6n\sqrt{n}}Ḡ_{3}z_{n}^{<3>} + · · · .

The first-order bias is given as E[g(x̄) − u] = \frac{1}{2n}G_{2}Ψ_{20} + O (\frac{1}{n^{2}} ) with Ψ_{20} = vec(Ψ_{11}) = (\frac{d}{dθ})^{<2>}ψ(θ)|_{θ=η(u)}.

To discuss 2OE, we take b_{g}^{(1)}(x̄) = \frac{1}{2} {[ \atop G_{2}Ψ_{20}} and b_{g}^{(2)}(x̄) to be unspecified. Then,

(3.2)

g^{∗}(x̄) =g(x̄) − \frac{1}{2n}{[ \atop G_{2}Ψ_{20}} − \frac{1}{n^{2}} b_{g}^{(2)}(x̄),
û^{∗}(x̄) =û(x̄) − \frac{1}{2n}{[ \atop Ḡ_{2}Ψ_{20}} − \frac{1}{n^{2}} b(2) ; û(x̄),

and hence

(3.3)  E[\sqrt{n}(g^{∗}(x̄) − û^{∗}(x̄))] = O (\frac{1}{n\sqrt{n}}) .

Here we do not necessarily specify how to estimate G_{2}Ψ_{20}.

Takeuchi [37](page 185) and Pfanzagl [27](Section 7) showed the following theorem. See also Ghosh [15](Section 6.4).

Theorem 1. Let g^{∗}(x̄) ∈ F and the MLE û^{∗}(x̄) ∈ F be bias-corrected as in (3.2). Assume that g^{∗}(x̄) is 1OE. Then,

P_{θ(u)}[\sqrt{n}(g^{∗}(x̄) − u) ∈ C] = P_{θ(u)}[\sqrt{n}(û^{∗}(x̄) − u) ∈ C] + o (\frac{1}{\sqrt{n}})

for every C ∈ C and u ∈ N (u_{0}), that is, under the bias-correction in (3.2), any 1OE estimators are 2OE for the class C.

Can we say something about 2OE, without bias-correction? The following theorem gives a certain answer to the question.

Theorem 2. Let û(x̄) be the MLE. Assume that a Fisher consistent estimator g(x̄) is 1OE. Then,

P_{θ(u)}[\sqrt{n}(g(x̄) − u) ∈ C] = P_{θ(u)}[\sqrt{n}(û(x̄) − u) ∈ C] + o (\frac{1}{\sqrt{n}})

for every C ∈ C_{0} and u ∈ N (u_{0}), that is, any 1OE estimators are 2OE for the class C_{0}.

Some authors must have noticed the result of Theorem 2, but no explicit statement regarding 2OE for the class of no bias-corrected estimators has appeared. The next theorem on 3OE is well-known (e.g., Pfanzagl and Wefelmeyer [29] Theorem 1; Akahira and Takeuchi [4] Theorem 5.1.6; Ghosh [15] Chapter 6; Taniguchi and Watanabe [41]).

Theorem 3. Let g^{∗}(x̄) ∈ F and the MLE û^{∗}(x̄) ∈ F be bias-corrected as in (3.2). Assume that g^{∗}(x̄) is 1OE. Then,

P_{θ(u)}[\sqrt{n}(g^{∗}(x̄) − u) ∈ C] = P_{θ(u)}[\sqrt{n}(û^{∗}(x̄) − u) ∈ C] − \frac{1}{n}Δ_{u,g^{∗},C}^{2} + o (\frac{1}{n})

for every C ∈ C_{0} and u ∈ N (u_{0}), where Δ_{u,g^{∗},C}^{2} is nonnegative. Further, Δ_{u,g^{∗},C}^{2} = 0 for any C ∈ C_{0} if and only if G_{2} = Ḡ_{2}, that is, under the bias-correction in (3.2), a 1OE estimator is 3OE for the class C_{0} if and only if G_{2} = Ḡ_{2}. When dim(u) = q = 1, all the statements above hold for the class C.

The actual form of Δ_{u,g^{∗},C}^{2} will be given in (4.11) in the proof. According to Theorem 3, we see that 3OE of g^{∗}(x̄) means G_{1} = Ḡ_{1} and G_{2} = Ḡ_{2}.

Now we are in a position to state some results on 4OE. The distribution of \sqrt{n}(b_{g}^{(1)} − \frac{1}{2}G_{2}Ψ_{20}) = \sqrt{n}(\frac{1}{2} {[ \atop G_{2}Ψ_{20}} − \frac{1}{2}G_{2}Ψ_{20}) contributes to the terms of order O (\frac{1}{n\sqrt{n}}) of concentration probability, and hence how to estimate G_{2}Ψ_{20} is quite important to study 4OE.

Consider the following different types of bias-corrections (or different choices of b_{g}^{(1)} and b_{g}^{(2)}):

Case (I): first-order bias-correction only

g^{∗}(x̄) = g(x̄) − \frac{1}{2n} {[ \atop G_{2}Ψ_{20}};

Case (II): second-order bias-correction (1), the historical one (see e.g., Rao et.al. [33]) g^{∗}(x̄) = g(x̄) − \frac{1}{2n}G_{2}(η(g(x̄)))Ψ_{20}(θ(g(x̄))) − \frac{1}{n^{2}} b_{g}^{(2)}(x̄)

so that E[g^{∗}(x̄) − u] = O (\frac{1}{n^{3}} );

Case (III): second-order bias-correction (2), suggested by Kano [20,22] g^{∗}(x̄) = g(x̄) − \frac{1}{2n}G_{2}(x̄)Ψ_{20}(θ(g(x̄))) − \frac{1}{n^{2}} b_{g}^{(2)}(x̄)

so that E[g^{∗}(x̄) − u] = O (\frac{1}{n^{3}} );

Case (IV): second-order bias-correction (3), based on Amari [6](Eq. (4.27))

g^{∗}(x̄) = g(x̄) − \frac{1}{2n}G_{2}(x̄)Ψ_{20}(Ψ_{10}^{−1}(x̄)) − \frac{1}{n^{2}} b_{g}^{(2)}(x̄)
so that E[g^{∗}(x̄) − u] = O (\frac{1}{n^{3}} ).

Here θ = Ψ_{10}^{−1}(η) is the inverse function of η = Ψ_{10}(θ). For Cases (II)-(IV),

(3.4)  E[\sqrt{n}(g^{∗}(x̄) − û^{∗}(x̄))] = O (\frac{1}{n^{2}\sqrt{n}}) ,

whereas (3.3) holds for Case (I).

We now state the main theorem.

Theorem 4. Let g^{∗}(x̄) ∈ F and the MLE û^{∗}(x̄) ∈ F be bias-corrected as in one of the four cases above. Assume that g^{∗}(x̄) is 3OE. (i) Under Cases (I) and (II),

(3.5)  P_{θ(u)}[\sqrt{n}(g^{∗}(x̄) − u) ∈ C] = P_{θ(u)}[\sqrt{n}(û^{∗}(x̄) − u) ∈ C] + o (\frac{1}{n\sqrt{n}})

for every C ∈ C_{0} and u ∈ N (u_{0}), that is, any 3OE estimators are 4OE for the class C_{0}. (ii) Under Cases (III) and (IV), (3.5) holds for the class C, that is, any 3OE estimators are 4OE for the class C.

It is seen from Theorem 4 that important is the way of estimating the first-order bias G_{2}Ψ_{20}, more precisely estimating G_{2} (the result is independent of choice of estimators for Ψ_{20}). In Appendix A we will give explicit forms of G_{2}(x̄) and G_{2}(η(g(x̄))) for the MLE, which may help realize at what point these quantities are different.

Kano [20,22,23] and Amari [7] have established 5OE of the MLE under the bias correction (III) or (IV) on the basis of quadratic risk, concentration probability and information loss, respectively. Akahira [2], recently, proved the fourth-order efficiency of the MLE for the class C ∈ C_{0} under Case (II) in a general probability model.

4. Proofs 4.1 Auxiliary results

Differentiation of both sides in the equality G_{k}(Ψ_{10}(θ(u))) = Ḡ_{k}(Ψ_{10}(θ(u))) w.r.t. u′ and use of (2.1) shows that

(4.1a) G_{1} = Ḡ_{1} ⟹ (G_{2} − Ḡ_{2})(Ψ_{11}Θ_{1} ⊗ I_{p}) = (G_{2} − Ḡ_{2})(I_{p} ⊗ Ψ_{11}Θ_{1}) = 0,
(4.1b) G_{2} = Ḡ_{2} ⟹ (G_{3} − Ḡ_{3})(Ψ_{11}Θ_{1} ⊗ I_{p^{2}}) = (G_{2} − Ḡ_{2})(I_{p} ⊗ Ψ_{11}Θ_{1} ⊗ I_{p})
= (G_{2} − Ḡ_{2})(I_{p^{2}} ⊗ Ψ_{11}Θ_{1})
= 0.

These are key relations to our derivation here.

Let t ∈ ℝ^{q} and φ(s) = e^{−s′Ψ_{11}s/2}. Recall that Ḡ_{1} = i_{u}^{−1}Θ′_{1}. It follows from (2.5) and (4.1a) that

(G_{2} − Ḡ_{2})E[e^{it′Ḡ_{1}z_{n}}(z_{n}^{<2>} − Ψ_{20})] = −(G_{2} − Ḡ_{2})(Ψ_{11}Θ_{1}i_{u}^{−1})^{<2>}φ(Ḡ′_{1}t) + o(1)

= o(1) (4.2) Using (2.4), we have

E[e^{it′Ḡ_{1}z_{n}}(z_{n}^{<2>} − Ψ_{20})z′_{n}] = i\frac{d}{ds′} [(Ψ_{11}s)^{<2>}φ(s)]|_{s=Ḡ′_{1}t} + o(1)

(4.3) = ie^{−t′i_{u}^{−1}t/2}N_{p^{<2>}} [(2Ψ_{11} − Ψ_{11}Ḡ′_{1}tt′Ḡ_{1}Ψ_{11}) ⊗ (Ψ_{11}Ḡ′_{1}t)] + o(1). In a similar manner, we have from (2.4) and (4.1a)

(G_{2} − Ḡ_{2})E[e^{it′Ḡ_{1}z_{n}}(z_{n}^{<2>} − Ψ_{20})z_{n}^{<2>′}]

= (G_{2} − Ḡ_{2}) (\frac{d}{ds′} )^{<2>} [(Ψ_{11}s)^{<2>}φ(s)]|_{s=Ḡ′_{1}t} + o(1)

= 2e^{−t′i_{u}^{−1}t/2}(G_{2} − Ḡ_{2})Ψ_{11}^{<2>} + o(1),

from which, it follows that

(4.4a)  (G_{2} − Ḡ_{2})E[e^{it′Ḡ_{1}z_{n}}(z_{n}^{<2>} − Ψ_{20})z_{n}^{<2>′}]Ḡ′_{2} = o(1)

and

(G_{2} − Ḡ_{2})E[e^{it′Ḡ_{1}z_{n}}(z_{n}^{<2>} − Ψ_{20})(z_{n}^{<2>} − Ψ_{20})′](G_{2} − Ḡ_{2})′

= 2e^{−t′i_{u}^{−1}t/2}(G_{2} − Ḡ_{2})Ψ_{11}^{<2>}(G_{2} − Ḡ_{2})′ + o(1). (4.4b) We have used that Ḡ_{2} = B(Θ′_{1} ⊗I_{p})N_{p^{<2>}} for some B (see (A.2) for a proof and also Kano [20] Lemma 3.3 for a more general result) and have applied (4.1a) again to get (4.4a).

Using (2.4) and (4.1b), we have

(G_{3} − Ḡ_{3})E[e^{it′Ḡ_{1}z_{n}}(z_{n}^{<3>} − 3(Ψ_{20} ⊗ z_{n})z_{n}^{<ℓ>′}]

= i^{3+ℓ}(G_{3} − Ḡ_{3}) (−\frac{d}{ds′} )^{<ℓ>} [(Ψ_{11}s)^{<3>}φ(s)]|_{s=Ḡ′_{1}t} + o(1)

= o(1) (4.5) for ℓ = 0, 1, 2.

4.2 Proofs of the theorems

Here we shall prove Theorems 3 and 4. The proof of Theorem 3 is different from and simpler than the existing proofs, and it will be used to prove Theorem 4.

We begin by noting that the difference between the bias-corrected 1OE estimators g^{∗}(x̄) and û^{∗}(x̄) is stochastically expressed as

(4.6)  \sqrt{n}(g^{∗}(x̄) − û^{∗}(x̄)) = \frac{1}{2\sqrt{n}}(G_{2} − Ḡ_{2})(z_{n}^{<2>} − Ψ_{20}) + o_{p} (\frac{1}{\sqrt{n}}) = O_{p} (\frac{1}{\sqrt{n}}) ,

in view of (3.1), (3.2) and the fact that G_{1} = Ḡ_{1} for any 1OE estimators. From (4.2) and (4.6), we can easily examine

(4.7)  E[e^{it′\sqrt{n}(û^{∗}(x̄)−u)}\sqrt{n}(g^{∗}(x̄) − û^{∗}(x̄))] = o (\frac{1}{\sqrt{n}}) .

To prove Theorems 3 and 4, we need to study the expectation (4.7), a sort of a cross term, up to higher-order. There are useful lemmas to do so. The basic idea of the lemmas were originated in Kano [22], who studies fifth-order efficiency. The proof will be given in Appendix B.

Lemma 1. Assume that x_{1},…, x_{n} are an i.i.d. random sample from the curved exponential family (1.1). Let T_{n}^{(1)} = T_{n}^{(1)}(x_{1},…, x_{n}) and T_{n}^{(2)} = T_{n}^{(2)}(x_{1},…, x_{n}) be random q- and q′-vectors which may depend on the sample size n but do not depend on the parameter u ∈ Ω. Let ϕ = ϕ(u, t) = E[e^{it′\sqrt{n}(T_{n}^{(1)}−u)}T_{n}^{(2)}]. Then, ϕ satisfies the following system of partial differential equations:

(4.8)  \frac{dϕ}{dt′} = −\{(t′i_{u}^{−1}) ⊗ ϕ\} + C(u, t),

where

C(u, t) = \frac{i}{\sqrt{n}} \frac{dϕ}{du′} i_{u}^{−1} + iE[e^{it′\sqrt{n}(T_{n}^{(1)}−u)}T_{n}^{(2)}\{\sqrt{n}(T_{n}^{(1)} − u) − Ḡ_{1}z_{n}\}′].

Lemma 2. Let t_{k} = [t_{1},…, t_{k}, 0,…, 0]′ and C(u, t) = [c_{1}(u, t),…, c_{q}(u, t)]. The solution to (4.8) is expressible as

ϕ(u, t) = e^{−t′i_{u}^{−1}t/2} (E[T_{n}^{(2)}] + ∑_{k=1}^{q} ∫_{0}^{t_{k}} c_{k}(u, t_{k})e^{t′_{k}i_{u}^{−1}t_{k}/2}dt_{k}) .

Now we are in a position to give a proof of Theorem 3. In view of (4.6), the characteristic function of \sqrt{n}(g^{∗}(x̄) − u) is evaluated as

E[e^{it′\sqrt{n}(g^{∗}(x̄)−u)}] =E[e^{it′\sqrt{n}(û^{∗}(x̄)−u)} · e^{it′\sqrt{n}(g^{∗}(x̄)−û^{∗}(x̄))}]
=E [e^{it′\sqrt{n}(û^{∗}(x̄)−u)}\{1 + it′\sqrt{n}(g^{∗}(x̄) − û^{∗}(x̄))

+ \frac{1}{2}\{it′\sqrt{n}(g^{∗}(x̄) − û^{∗}(x̄))\}^{2} + O_{p} (\frac{1}{n\sqrt{n}})\}] (4.9)

from which, the difference between the characteristic functions can be expressed in the form:

(4.10)

E[e^{it′\sqrt{n}(g^{∗}(x̄)−u)}] − E[e^{it′\sqrt{n}(û^{∗}(x̄)−u)}]
= (it)′E[e^{it′\sqrt{n}(û^{∗}(x̄)−u)}\sqrt{n}(g^{∗}(x̄) − û^{∗}(x̄))]
+ \frac{1}{2}(it)′E[e^{it′\sqrt{n}(û^{∗}(x̄)−u)}\sqrt{n}(g^{∗}(x̄) − û^{∗}(x̄))\sqrt{n}(g^{∗}(x̄) − û^{∗}(x̄))′](it)
+ o (\frac{1}{n}) .

We shall show later that the first term in the RHS in (4.10) is not just of order o (\frac{1}{\sqrt{n}}) (see (4.7)) but of order o (\frac{1}{n}). As a consequence, use of (4.6) and (4.4b) results in

E[e^{it′\sqrt{n}(g^{∗}(x̄)−u)}]−E[e^{it′\sqrt{n}(û^{∗}(x̄)−u)}]

= \frac{e^{−t′i_{u}^{−1}t/2}}{4n} (it)′(G_{2} − Ḡ_{2})Ψ_{11}^{<2>}(G_{2} − Ḡ_{2})′(it) + o (\frac{1}{n}) .

Inverting the characteristic functions to distribution functions, we have

P_{θ(u)}[\sqrt{n}(g^{∗}(x̄) − u) ∈ C] = P_{θ(u)}[\sqrt{n}(û^{∗}(x̄) − u) ∈ C] − \frac{1}{n}Δ_{u,g^{∗},C}^{2} + o (\frac{1}{n})

for every Borel convex set C ⊂ ℝ^{q}, where

Δ_{u,g^{∗},C}^{2} = \frac{1}{4}vec(G_{2} − Ḡ_{2})′

(4.11) · (Ψ_{11}^{<2>} ⊗ ∫_{C} (i_{u} − i_{u}xx′i_{u}) N_{q}(x|0, i_{u}^{−1})dx) vec(G_{2} − Ḡ_{2}).

If C is symmetric about the origin (i.e., C ∈ C_{0}), the integral above is shown to be nonnegative; and it is positive, assuming further that the interior point of C is not empty (see Pfanzagl [28] Lemma 13.2.4). When q = 1, the integral in (4.11) can be shown to be positive for every possibly asymmetric interval including the origin, which belongs to C. As a consequence, Theorem 3 follows, provided that

(4.12)  ϕ = ϕ(u, t) = E[e^{it′\sqrt{n}(û^{∗}(x̄)−u)}\sqrt{n}(g^{∗}(x̄) − û^{∗}(x̄))] = o (\frac{1}{n}) .

Applying Lemma 1 [T_{n}^{(1)} = û^{∗}(x̄), T_{n}^{(2)} = \sqrt{n}(g^{∗}(x̄) − û^{∗}(x̄))] and using (4.6), we can evaluate C(u, t) in Lemma 1 as

C(u, t) = \frac{i}{\sqrt{n}} \frac{dϕ}{du′} i_{u}^{−1} + \frac{i}{4n}(G_{2} − Ḡ_{2})E[e^{it′Ḡ_{1}z_{n}}(z_{n}^{<2>} − Ψ_{20})(z_{n}^{<2>} − Ψ_{20})′]Ḡ′_{2} + o (\frac{1}{n})

= \frac{i}{\sqrt{n}} \frac{dϕ}{du′} i_{u}^{−1} + o (\frac{1}{n})

in view of (4.2) and (4.4a). Then, c_{k}(u, t) in Lemma 2 is written as \frac{i}{\sqrt{n}} \frac{dϕ}{du′} i_{u}^{−1}e_{k} + o (\frac{1}{n}) with e_{k} the k-th unit vector in ℝ^{q}. Use of Lemma 2 and (3.3) leads to

ϕ =e^{−t′i_{u}^{−1}t/2} (E[\sqrt{n}(g^{∗}(x̄) − û^{∗}(x̄))] + \frac{i}{\sqrt{n}} ∑_{i=1}^{q} ∫_{0}^{t_{k}} \frac{dϕ(u, t_{k})}{du′}i_{u}^{−1}e_{k}e^{t′_{k}i_{u}^{−1}t_{k}/2}dt_{k})
+ o (\frac{1}{n})
=\frac{ie^{−t′i_{u}^{−1}t/2}}{\sqrt{n}} ∑_{i=1}^{q} ∫_{0}^{t_{k}} \frac{dϕ(u, t_{k})}{du′}i_{u}^{−1}e_{k}e^{t′_{k}i_{u}^{−1}t_{k}/2}dt_{k} + o (\frac{1}{n}) .

Substitution of ϕ = o (\frac{1}{\sqrt{n}}) shown in (4.7) into the integrand in the RHS above leads to ϕ = o (\frac{1}{n}). The proof of Theorem 3 is complete. Q.E.D.

Now we shall give a proof of Theorem 4. Consider Cases (III) and (IV) first. The bias-correction factors can be expanded as

\frac{1}{2\sqrt{n}}G_{2}(x̄)Ψ_{20}(θ(g(x̄))) =\frac{1}{2\sqrt{n}}G_{2}Ψ_{20} + \frac{1}{2n}G_{2}Ψ_{21}Θ_{1}G_{1}z_{n}

+ \frac{1}{2n}G_{3}(Ψ_{20} ⊗ I_{p})z_{n} + o_{p} (\frac{1}{n}) ,

\frac{1}{2\sqrt{n}}G_{2}(x̄)Ψ_{20}(Ψ_{10}^{−1}(x̄)) =\frac{1}{2\sqrt{n}}G_{2}Ψ_{20} + \frac{1}{2n}G_{2}Ψ_{21}Ψ_{11}^{−1}z_{n}

+ \frac{1}{2n}G_{3}(Ψ_{20} ⊗ I_{p})z_{n} + o_{p} (\frac{1}{n})

and thus for a 3OE estimator g^{∗}(x̄) (i.e., G_{1} = Ḡ_{1} and G_{2} = Ḡ_{2}) we have

\sqrt{n}(g^{∗}(x̄) − û^{∗}(x̄)) = \frac{1}{6n}(G_{3} − Ḡ_{3})\{z_{n}^{<3>} − 3(Ψ_{20} ⊗ z_{n})\} + o_{p} (\frac{1}{n}) = O_{p} (\frac{1}{n})

for the both cases. For all the cases, we shall write

(4.13) \sqrt{n}(g^{∗}(x̄) − û^{∗}(x̄)) =\frac{1}{6n}(G_{3} − Ḡ_{3})\{z_{n}^{<3>} − 3(Ψ_{20} ⊗ z_{n})\} + \frac{1}{2n}B(u)z_{n} + o_{p} (\frac{1}{n})

=O_{p} (\frac{1}{n}) ,

where B(u) is a constant matrix of order q × p given as

B(u) = \{ \begin{matrix}(G_{3} − Ḡ_{3})(Ψ_{20} ⊗ I_{p}) − \frac{d}{dx̄′} [{[ \atop G_{2}Ψ_{20}} − {[ \atop Ḡ_{2}Ψ_{20}}]|_{x̄=η(u)} & for Case (I) \\ (G_{3} − Ḡ_{3})(Ψ_{20} ⊗ I_{p}) − (G_{3} − Ḡ_{3})(Ψ_{21}Θ_{1}Ḡ_{1} ⊗ Ψ_{20}) & for Case (II) \\ 0 & for Cases (III), (IV). \end{matrix}

Since \sqrt{n}(g^{∗}(x̄) − û^{∗}(x̄)) = O_{p} (\frac{1}{n}), the same derivation as in (4.9) and (4.10) can apply to get

E[e^{it′\sqrt{n}(g^{∗}(x̄)−u)}]−E[e^{it′\sqrt{n}(û^{∗}(x̄)−u)}]
= (it)′E[e^{it′\sqrt{n}(û^{∗}(x̄)−u)}\sqrt{n}(g^{∗}(x̄) − û^{∗}(x̄))] + o (\frac{1}{n\sqrt{n}}) .

Put

ϕ = ϕ(u, t) = E[e^{it′\sqrt{n}(û^{∗}(x̄)−u)}\sqrt{n}(g^{∗}(x̄) − û^{∗}(x̄))],

and then the inversion of the Fourier transform is made, so that

(4.14)

P_{θ(u)}\{\sqrt{n}(g^{∗}(x̄) − u) ∈ C\} − P_{θ(u)}\{\sqrt{n}(û^{∗}(x̄) − u) ∈ C\}

= ∫_{C} \frac{1}{(2π)^{q}} ∫_{ℝ^{q}} e^{−ix′t}(it)′ϕ(u, t)dtdx + o (\frac{1}{n\sqrt{n}})

for every Borel convex set C ⊂ ℝ^{q}. Let us apply Lemmas 1 and 2 to evaluate ϕ. We have

iE[e^{it′\sqrt{n}(û^{∗}(x̄)−u)}\sqrt{n}(g^{∗}(x̄) − û^{∗}(x̄))\{\sqrt{n}(û^{∗}(x̄) − u) − Ḡ_{1}z_{n}\}′]
=\frac{i}{n\sqrt{n}}E[e^{it′Ḡ_{1}z_{n}}\{\frac{1}{6}(G_{3} − Ḡ_{3})(z_{n}^{<3>} − 3(Ψ_{20} ⊗ z_{n})) + \frac{1}{2}B(u)z_{n}\}
· \{\frac{1}{2} Ḡ_{2}(z_{n}^{<2>} − Ψ_{20})\}′] + o (\frac{1}{n\sqrt{n}}) (by (4.6),(4.13))
=\frac{i}{4n\sqrt{n}}B(u)E[e^{it′Ḡ_{1}z_{n}}z_{n}(z_{n}^{<2>} − Ψ_{20})′]Ḡ′_{2} + o (\frac{1}{n\sqrt{n}}) (by (4.5))
= − \frac{e^{−t′i_{u}^{−1}t/2}}{4n\sqrt{n}} B(u) [(2Ψ_{11} − Ψ_{11}Ḡ′_{1}tt′Ḡ_{1}Ψ_{11}) ⊗ (t′Ḡ_{1}Ψ_{11})] Ḡ′_{2}
+ o (\frac{1}{n\sqrt{n}}) (by (4.3))

and \frac{dϕ}{du′} i_{u}^{−1} = o (\frac{1}{n}) from (4.12). As a result, the C(u, t) in Lemma 1 can be written as

C(u, t) = \frac{e^{−t′i_{u}^{−1}t/2}}{n\sqrt{n}} B(u)D(u, t) + o (\frac{1}{n\sqrt{n}}) ,

where

D(u, t) = −\frac{1}{4} [(2Ψ_{11} − Ψ_{11}Ḡ′_{1}tt′Ḡ_{1}Ψ_{11}) ⊗ (t′Ḡ_{1}Ψ_{11})] Ḡ′_{2}.

By Lemma 2, we finally obtain

ϕ(u, t) =e^{−t′i_{u}^{−1}t/2} \{E[\sqrt{n}(g^{∗}(x̄) − û^{∗}(x̄))] + \frac{1}{n\sqrt{n}}B(u) ∑_{i=1}^{q} ∫_{0}^{t_{k}} d_{k}(u, t_{k})dt_{k}\}

+ o (\frac{1}{n\sqrt{n}}) , (4.15) where D(u, t) = [d_{1}(u, t),…, d_{q}(u, t)].

When E[\sqrt{n}(g^{∗}(x̄) − û^{∗}(x̄))] = o (\frac{1}{n\sqrt{n}}) and B(u) = 0, which is met by Cases (III) and (IV) (see (3.4)), we have from (4.15) that ϕ = o (\frac{1}{n\sqrt{n}}). This, along with (4.14), proves the second statement in Theorem 4. We shall prove Theorem 4-(i). First the reader should note that ϕ(u, t) is even in argument t since D(u, t) is odd in t. Thus, (it)′ϕ(u, t) is odd in t and of order O (\frac{1}{n\sqrt{n}}). The second term in the RHS in (4.14) vanishes if C is symmetric about the origin. The proof is complete. Q.E.D.

Kano [22] proved 4OE of the MLE for Case (III) as a corollary of his main result on 5OE. The proof given here is direct and much simpler.

Appendix A In this appendix, we will give explicit expressions of Ḡ_{2}(x̄) and Ḡ_{2}(η(û(x̄))), estima-

tors of the second-order derivative of the MLE û(x̄). Let

Θ_{kℓ}(u) = (\frac{d}{du})^{<k>} (\frac{d}{du′} )^{<ℓ>} θ(u),  Θ̂_{kℓ} = Θ_{kℓ}(û(x̄))
Ψ_{kℓ}(θ) = (\frac{d}{dθ} )^{<k>} (\frac{d}{dθ′} )^{<ℓ>} ψ(θ),  Ψ̂_{kℓ} = Ψ_{kℓ}(θ(û(x̄)))

for nonnegative integers k, ℓ. (cf. (2.2) and Θ_{01}(u) = Θ_{1}, which is an old notation used in the text.)

The likelihood equation is written from (1.2) as

Θ̂′_{01}(x̄ − Ψ̂_{10}) = 0.

Operating \frac{d}{dx̄′} to the equality above, we have

(A.1)  Θ̂′_{11} (Ḡ_{1}(x̄) ⊗ (x̄ − Ψ̂_{10})) + Θ̂′_{01} (I_{p} − Ψ̂_{11}Θ̂_{01}Ḡ_{1}(x̄)) = 0,

which leads to

Ḡ_{1}(x̄) = {Θ̂′_{01}Ψ̂_{11}Θ̂_{01} − Θ̂′_{11} (I_{q} ⊗ (x̄ − Ψ̂_{10}))}^{−1} Θ̂′_{01}.

Operating \frac{d}{dx̄′}, again, to (A.1) and solving for Ḡ_{2}(x̄), we obtain

Ḡ_{2}(x̄) = {Θ̂′_{01}Ψ̂_{11}Θ̂_{01} − Θ̂′_{11} (I_{q} ⊗ (x̄ − Ψ̂_{10}))}^{−1}
· [Θ̂′_{21} (I_{q^{2}} ⊗ (x̄ − Ψ̂_{10})) Ḡ_{1}(x̄)^{<2>}
+ 2Θ̂′_{11} {Ḡ_{1}(x̄) ⊗ (I_{p} − Ψ̂_{11}Θ̂_{01}Ḡ_{1}(x̄))} N_{p^{<2>}}
−Θ̂′_{01}(Ψ̂_{12}Θ̂_{01}^{<2>} + Ψ̂_{11}Θ̂_{02})Ḡ_{1}(x̄)^{<2>}] .

Evaluation at x̄ = E[x̄] = Ψ_{10} reduces some of the terms above, and hence we have

Ḡ_{2}(η(û(x̄))) =(Θ̂′_{01}Ψ̂_{11}Θ̂_{01})^{−1}
· [2Θ̂′_{11} {Ḡ_{1}(η(û(x̄))) ⊗ (I_{p} − Ψ̂_{11}Θ̂_{01}Ḡ_{1}(η(û(x̄))))} N_{p^{<2>}}
−Θ̂′_{01}(Ψ̂_{12}Θ̂_{01}^{<2>} + Ψ̂_{11}Θ̂_{02})Ḡ_{1}(η(û(x̄)))^{<2>}] .

We see that all the terms involving x̄ − Ψ̂_{10} remain in Ḡ_{2}(x̄) whereas they do disappear in Ḡ_{2}(η(û(x̄))).

Finally we note that

Ḡ_{2} =i_{u}^{−1} [2Θ̂′_{11}{i_{u}^{−1} ⊗ (I_{p} − Ψ_{11}Θ_{01}Ḡ_{1})}(Θ′_{01} ⊗ I_{p})N_{p^{<2>}}
−Θ′_{01}(Ψ_{12}Θ_{01}^{<2>} + Ψ_{11}Θ_{02})(i_{u}^{−1} ⊗ Ḡ_{1})(Θ′_{01} ⊗ I_{p})N_{p^{<2>}}]
=i_{u}^{−1} [2Θ̂′_{11}{i_{u}^{−1} ⊗ (I_{p} − Ψ_{11}Θ_{01}Ḡ_{1})}
−Θ′_{01}(Ψ_{12}Θ_{01}^{<2>} + Ψ_{11}Θ_{02})(i_{u}^{−1} ⊗ Ḡ_{1})] (Θ′_{01} ⊗ I_{p})N_{p^{<2>}}
(A.2)  =B(Θ′_{01} ⊗ I_{p})N_{p^{<2>}}

with i_{u} = Θ′_{01}Ψ_{11}Θ_{01}. This is a result used to show (4.4).

Appendix B

We shall give proofs of Lemmas 1 and 2 here.

There is a useful formula in evaluating expectations involving the score function, see e.g., Akahira and Takeuchi [4](Lemma 5.1.1), Amari [6](page 124) or Kano [20](Eq.(4.5)):

(B.1)  E[z′_{n}Θ_{1} ⊗ A(z_{n}, u)] = \frac{1}{\sqrt{n}} \{\frac{d}{du′} E[A(z_{n}, u)] − E [\frac{d}{du′} A(z_{n}, u)]\} ,

where A(z, u) is a matrix-valued function measurable in z in ℝ^{p} and continuously differentiable in u.

Recall that Ḡ_{1} = i_{u}^{−1}Θ′_{1}. Differentiation of ϕ w.r.t. \frac{d}{dt′} gives

\frac{dϕ}{dt′} = iE[e^{it′\sqrt{n}(T_{n}^{(1)}−u)}T_{n}^{(2)}\sqrt{n}(T_{n}^{(1)} − u)′]
= iE[e^{it′\sqrt{n}(T_{n}^{(1)}−u)}T_{n}^{(2)}\{\sqrt{n}(T_{n}^{(1)} − u) − Ḡ_{1}z_{n}\}′]

+ iE[(z′_{n}Θ_{1}) ⊗ (e^{it′\sqrt{n}(T_{n}^{(1)}−u)}T_{n}^{(2)})]i_{u}^{−1}. (B.2)

Exchangeability of the differentiation and expectation is permitted for exponential families (see e.g., Lehmann [24] page 59). Application of the formula (B.1) to the second term in (B.2) leads to

\frac{i}{\sqrt{n}} (\frac{d}{du′} E[e^{it′\sqrt{n}(T_{n}^{(1)}−u)}T_{n}^{(2)}] − E[\frac{d}{du′} e^{it′\sqrt{n}(T_{n}^{(1)}−u)}T_{n}^{(2)}]) i_{u}^{−1}

= \frac{i}{\sqrt{n}} \frac{dϕ}{du′} i_{u}^{−1} − (t′i_{u}^{−1} ⊗ ϕ).

Substitution of this into (B.2) gives (4.8). Q.E.D.

Let us prove Lemma 2. It suffices to show that the following partial differential equations (B.3) has the solution (B.6):

(B.3)  \frac{dϕ(t)}{dt′} = −(t′i_{u}^{−1} ⊗ ϕ(t)) + C(t).

Put ϕ(t) = ϕ̃(t) · e^{−t′i_{u}^{−1}t/2}, and then we have

\frac{dϕ̃(t)}{dt′} = e^{t′i_{u}^{−1}t/2}C(t) (= e^{t′i_{u}^{−1}t/2}[c_{1}(t),…, c_{q}(t)], say) .

That is,

(B.4)  \frac{∂ϕ̃(t)}{∂t_{k}} = e^{t′i_{u}^{−1}t/2}c_{k}(t) (k = 1,…, q).

Using (B.4), we have

ϕ̃(t) = ϕ̃(0) + ∑_{k=1}^{q} {ϕ̃(t_{k}) − ϕ̃(t_{k−1})}

= ϕ̃(0) + ∑_{k=1}^{q} ∫_{0}^{t_{k}} \frac{∂ϕ̃(t_{k})}{∂t_{k}}dt_{k}

= ϕ̃(0) + ∑_{k=1}^{q} ∫_{0}^{t_{k}} c_{k}(t_{k})e^{t′_{k}i_{u}^{−1}t_{k}/2}dt_{k}. (B.5)

Conversely, the function (B.5) is actually a solution to (B.4), provided that (B.4) has a solution. Thus, we obtain

(B.6)  ϕ(t) = e^{−t′i_{u}^{−1}t/2} (ϕ(0) + ∑_{k=1}^{q} ∫_{0}^{t_{k}} c_{k}(t_{k})e^{t′_{k}i_{u}^{−1}t_{k}/2}dt_{k}) .

ACKNOWLEDGMENT The author would like to express his thanks to Professor M. Akahira of University of

Tsukuba and Dr. M. Taniguchi of Osaka University for invaluable discussion on higherorder efficiency. He also thanks Professor S. Amari of The University of Tokyo for showing his bias-correction factor (Case (IV) in Theorem 4). He is finally indebted to the reviewer for his (or her) useful comments.

REFERENCES

[1] Akahira, M. (1975). A note on the second order asymptotic efficiency of estimators in an autoregressive process. Rep. Univ. Electro-Comm., 26-1, 143-149.

[2] Akahira, M. (1995). Third order efficiency implies fourth order efficiency: A resolution of the conjecture of J. K. Ghosh. To appear in Ann. Inst. Math. Stat.

[3] Akahira, M. and Takeuchi, K. (1976). On the second order asymptotic efficiency of estimators in multiparameter cases. Rep. Univ. Electro-Comm., 26-2, 261-269.

[4] Akahira, M. and Takeuchi, K. (1981). Asymptotic Efficiency of Statistical Estimators: Concepts and Higher Order Asymptotic Efficiency, Lecture Notes in Statistics, Vol.7. Springer-Verlag, New York.

[5] Amari, S. (1982). Differential geometry of curved exponential families: Curvatures and information loss. Ann. Statist., 10, 357-387.

[6] Amari, S. (1985). Differential-Geometrical Methods in Statistics, Lecture Notes in Statistics, Vol.28. Springer-Verlag, Berlin.

[7] Amari, S. (1995). Personal communications.

[8] Barndorff-Nielsen, O. E. and Cox, D. R. (1982). Asymptotic Techniques for Use in Statistics. Chapman and Hall, London.

[9] Bentler, P. M. and Lee, Sik-Yum (1978). Matrix derivatives with chain rule and rules for simple, Hadamard, and Kronecker products. J. Math. Psychol., 17, 255-262.

[10] Bhattacharya, R. and Denker, M. (1990). Asymptotic Statistics. Birkhäuser, Basel.

[11] Bhattacharya, R. and Rao, R. R. (1976). Normal Approximation and Asymptotic Expansions. Wiley, New York.

[12] Chibisov, D. M. (1972). On the normal approximation for a certain class of statistics. Proc. Sixth Berkeley Symp. Math. Statist. Probab., 1, 153-174.

[13] Efron, B. (1975). Defining the curvature of a statistical problem (with applications to second order efficiency). Ann. Statist., 3, 1189-1242.

[14] Eguchi, S. (1983). Second order efficiency of minimum contrast estimators in a curved exponential family. Ann. Statist., 11, 793-803.

[15] Ghosh, J. K. (1994). Higher Order Asymptotics. NSF-CBMS Regional Conference Series in Probability and Statistics, Vol.4. IMS, Hayward, CA.

[16] Ghosh, J. K. and Subramanyam, K. (1974). Second order efficiency of maximum likelihood estimates. Sankhyā, Ser. A, 36, 325-358.

[17] Henderson H. V. and Searle, S. R. (1981). The vec-permutation matrix, the vec operator and Kronecker products: A review. Linear and Multilinear Algebra, 9, 271-288.

[18] Hosoya, Y. (1979). Higher-order efficiency in the estimation of linear processes. Ann. Statist., 7, 516-530.

[19] Kano, Y. (1986). Conditions on consistency of estimators in covariance structure model. J. Japan Statist. Soc., 16, 75-80.

[20] Kano, Y. (1993). Beyond third-order efficiency. (submitted)

[21] Kano, Y. (1993). Beyond third-order efficiency: Supplement. Technical Report, University of Osaka Prefecture, Osaka, Japan.

[22] Kano, Y. (1994). More higher-order efficiency: Concentration probability. Technical Reports of Statistical Group: DATA95-04, Tokyo, Japan.

[23] Kano, Y. (in press). Fourth and fifth order efficiency: Fisher information. In Proc. Japan-Russia Symp. Probab. Statist. World Scientific, Singapore.

[24] Lehmann, E. L. (1986). Testing Statistical Hypotheses (2nd Ed.). John Wiley, New York.

[25] Magnus, J. R. and Neudecker, H. (1988). Matrix Differential Calculus with Applications in Statistics and Econometrics. John Wiley, Chichester.

[26] Pfanzagl, J. (1973). Asymptotic expansions related to minimum contrast estimators. Ann. Statist., 1, 993-1026.

[27] Pfanzagl, J. (1976). First order efficiency implies second order efficiency. In Contribution to Statistics (Hajek Memorial Volume), (J. Jureckova, Ed.), pp. 167-196, Academia, Prague.

[28] Pfanzagl, J. (1985). Asymptotic Expansions for General Statistical Models. Lecture Notes in Statistics, Vol.31. Springer-Verlag, Berlin.

[29] Pfanzagl, J. and Wefelmeyer, W. (1978). A third order optimum property of the maximum likelihood estimator. J. Multivariate Anal., 8, 1-29.

[30] Rao, C. R. (1961). Asymptotic efficiency and limiting information. Proc. Fourth Berkeley Symp., 1, 531-545.

[31] Rao, C. R. (1962). Efficient estimates and optimum inference procedures in large samples. J. Royal Statist. Soc., Ser. B, 24, 46-72.

[32] Rao, C. R. (1963). Criteria of estimation in large samples. Sankhyā, Ser. A, 25, 189-206.

[33] Rao, C. R., Sinha, B. K. and Subramanyam, K. (1982). Third order efficiency of the maximum likelihood estimator in the multinomial distribution. Statist. Decisions, 1, 1-16.

[34] Satake, I. (1974). Senkei Daisu Gaku (Linear Algebra) (2nd Ed.). Shokabo, Tokyo (in Japanese).

[35] Sternberg, S. (1964). Lectures on Differential Geometry. Prentice-Hall Inc., Englewood Cliffs, NJ.

[36] Takemura, A. (1991). Tahenryo Suisoku Tokei no Kiso (The foundation of Multivariate Statistical Inference). Kyoritsu Shuppan, Tokyo (in Japanese).

[37] Takeuchi, K. (1974). Tokeiteki Suisoku no Zenkin Riron (Asymptotic Theory of Statistical Estimation). Kyoiku Shuppan, Tokyo (in Japanese).

[38] Takeuchi, K. and Akahira, M. (1976). On the second order asymptotic efficiencies of estimators. In Proc. Third Japan-USSR Symp. Probab. Theory, Lecture Notes in Mathematics 550, pp. 604-638. Springer-Verlag, Berlin.

[39] Taniguchi, M. (1983). On the second order asymptotic efficiency of estimators of Gaussian ARMA Processes. Ann. Statist., 11, 157-169.

[40] Taniguchi, M. (1991). Higher Order Asymptotic Theory for Time Series Analysis, Lecture Notes in Statistics, Vol.68. Springer-Verlag, Berlin.

[41] Taniguchi, M and Watanabe, Y. (1994). Statistical analysis of curved probability densities. J. Multivariate Anal., 48, 228-248.

[42] Yoshida, N. (1992). Asymptotic expansions of maximum likelihood estimators for small diffusions via the theory of Malliavin–Watanabe. Probab. Theory Relat. Fields, 92, 275-311.

[43] Yoshida, N. (1993). Asymptotic expansion of Bayes estimators for small diffusions. Probab. Theory Relat. Fields, 95, 429-450.