FOURTH AND FIFTH ORDER EFFICIENCY:

FISHER INFORMATION

YUTAKA KANO ^{∗} Institute of Mathematics, University of Tsukuba Tsukuba, Ibaraki 305, Japan E-mail: kano@math.tsukuba.ac.jp

ABSTRACT

Loss of (Fisher) information, whose concept was defined by Fisher, is one of important measures of asymptotic efficiency. Based on Fisher information, this paper studies fourth and fifth order asymptotic efficiency of estimators in a curved exponential family of distributions with a structural parameter vector. In particular, we show that the bias-corrected MLE with a certain bias correction factor is fourth and fifth order efficient in a class of Fisher consistent estimators bias-corrected similarly.

1. Introduction

Efficiency is one of key concepts in the estimation problems in statistics. Asymptotic efficiency is often studied to discuss efficiency of estimators for parameters of models with a non-linear structure. There are mainly three measures or criteria to study asymptotic efficiency, namely loss of information, quadratic risk and concentration probability. Historically, taking loss of information, Fisher(1925) and Rao(1961) successfully proved the third order efficiency of the MLE in a class of the BAN (best asymptotic normal) estimators in a curved one-parameter multinomial distribution, that is, the MLE is the best of the class of first order efficient estimators. This result is often called the Fisher-Rao theorem. Amari (1982) and Eguchi (1983) took a geometric approach to extend the Fisher-Rao theorem to the general curved exponential family with a structural parameter vector. The third order efficiency has been shown to hold under other criteria, quadratic risk and concentration probability as well (e.g., Ghosh and Subramanyam 1974; Efron 1975; Pfanzagl and Wefelmeyer 1978; Akahira and Takeuchi 1981; Amari 1985). Notice that the term “third order ” using in this paper means “second order ” that is called by Fisher, Rao or Eguchi.

The basic results on third order efficiency have been extended to more general cases, see Takeuchi-Morimune (1985), Taniguchi (1991) and Taniguchi and Watanabe (1994) among others. Ghosh (1994) recently published a well-written review book on the topic.

There are many third order efficient estimators, however. For instance, Rao, Sinha and Subramanyam (1982) and Eguchi (1983) have formed some estimators asymptotically equivalent to the MLE up to order n^{−2}; estimators obtained by iterating the scoring method twice are third order efficient as well (see e.g., Eguchi 1983). Thus, it is important to develop theory of more higher order efficiency to distinguish these third order efficient estimators.

∗This work is partially supported by a grant-in-aid from the Ministry of Education.

There are a few works on asymptotic efficiency beyond the third order. Kano (1993a, 94, 95) have taken quadratic risk or concentration probability as a measure of efficiency to study fourth and fifth order asymptotic efficiency. Here, we shall study the same topic under Fisher information as a measure of efficiency. The loss of information is difference in the Fisher information between an estimator and the original sample, and it requires tedious calculation to get an explicit form. To avoid it, we evaluate the difference in the Fisher information between the MLE and a competitive estimator.

The result obtained here is an extension of the classical Fisher-Rao theorem to the case at more higher order. The reason that the proof of the fourth and fifth order efficiency of the MLE under the Fisher information comes after that under quadratic risk or concentration probability is that Fisher information is more intractable than the other two.

2. Curved Exponentials and Approximate Fisher Information

Let u ∈ R^{q} be a parameter vector of interest and let an open set Ω(⊂ R^{q}) be a parameter space of u. Let u_{0} ∈ Ω be an arbitrarily fixed (inner) point of Ω. For every u ∈ N (u_{0}), a neighborhood of u_{0}, assume that random p-vectors x_{1},…, x_{n} are independent and identically distributed according to a (continuous) curved exponential family with density

exp {θ(u)′x − ψ(θ(u))} μ(dx),

where μ(·) is a carrier measure on R^{p} (see Amari 1985, Section 4.1) and θ(·) is an R^{p}- valued measurable function from Ω(⊂ R^{q}) smooth in u ∈ N (u_{0}). The real-valued function, ψ(θ), is defined as a normalizing constant ψ(θ) = log ∫_{R^{p}} exp(θ′x)μ(dx). Write Θ_{1} = \frac{d}{du′} θ(u), Ψ_{10} = \frac{dψ}{dθ} and Ψ_{11} = \frac{d^{2}ψ}{dθdθ′}. It is well-known that

E[x_{i}] = Ψ_{10}(θ(u)) (= η(u), say) and Cov[x_{i}, x_{i}] = Ψ_{11}(θ(u)) (= Ψ_{11}, say)

and that the Fisher information (matrix) of the original sample of each x_{i} is expressed as

i_{u} = Θ′_{1}Ψ_{11}Θ_{1}.

Assume that Ψ_{11} and i_{u} are positive definite.

There are two ways to evaluate higher order Fisher information of estimators, one is to make use of the representation in conditional expectations (e.g., Fisher 1925; Rao 1961; Eguchi 1983) and the other is to utilize an Edgeworth expansion (Hosoya 1988). We shall take the latter here.

Let g_{n}(x̄) be an estimator for u and let p(x|u) be a probability density function of the distribution of \sqrt{n}(g_{n}(x̄) − u) w.r.t. the Lebesgue measure on (R^{q}, B_{q}), where B_{q} denotes the q-dimensional Borel σ-field. The density function of g_{n}(x̄) is then written as p(\sqrt{n}(g_{n}(x̄) − u)|u)n^{q/2}. Let

∂ = \frac{∂}{∂x} − \frac{1}{\sqrt{n}} \frac{∂}{∂u}.

The Fisher information (matrix) of g_{n}(x̄) is expressible in the form:

F I(g_{n}(x̄))

= E [\{\frac{∂}{∂u} log p(\sqrt{n}(g_{n}(x̄) − u)|u)n^{q/2}\} \{\frac{∂}{∂u} log p(\sqrt{n}(g_{n}(x̄) − u)|u)n^{q/2}\}′]
= n ∫ \frac{\{∂p(x|u)\}\{∂p(x|u)\}′}{p(x|u)} dx.

It is too restrictive to assume that the distribution of g_{n}(x̄) permits a density function because usually g_{n}(x̄) is defined implicitly as a solution to nonlinear simultaneous equations and the differentiability is guaranteed only in a neighborhood of η(u) = E[x̄]. On the other hand, under minimal regularity conditions (see e.g. Bhattacharya and Denker 1990), the distribution of \sqrt{n}(g_{n}(x̄) − u)(= (g_{n}^{(1)}(x̄),…, g_{n}^{(q)}(x̄))′, say) can be evaluated with the help of Edgeworth expansions: For a given positive integer K,

Pr{g_{n}^{(1)}(x̄) ≤ x_{1},…, g_{n}^{(q)}(x̄) ≤ x_{q}}
= ∑_{k=0}^{K} n^{−k/2} ∫_{−∞}^{x_{1}} ⋯ ∫_{−∞}^{x_{q}} r_{k}(x, u)N(x|0, iu^{−1})dx_{1} ⋯ dx_{q} + o(n^{−K/2}), (1)

where r_{k}(x, u) is a polynomial in x with coefficients that are continuously differentiable in u ∈ N (u_{0}). Let us term

p_{n}(x|u) = ∑_{k=0}^{K} n^{−k/2}r_{k}(x, u)N(x|0, iu^{−1})

an approximate density function of the distribution of \sqrt{n}(g_{n}(x̄)−u) up to the order n^{−K/2}, and the Fisher information (matrix) based on p_{n}(x|u) approximate Fisher information (AFI) of g_{n}(x̄). We further expand

AF I(g_{n}(x̄)) = n ∫_{R_{q}} \frac{\{∂p_{n}(x|u)\}\{∂p_{n}(x|u)\}′}{p_{n}(x|u)} dx
= nA_{1} + n^{\frac{1}{2}} A_{2} + ⋯ + n^{− \frac{K}{2} +1}A_{K+1} + o(n^{− \frac{K}{2} +1}),

and then we shall define asymptotic efficiency on the basis of the expansion above (see Hosoya 1988).

Let F be a class of estimators for u under consideration. An estimator in F is said to be first order efficient (1OE) iff it maximizes A_{1} among all the estimators in F; a 1OE estimator (in F) is said to be second order efficient (2OE) in F iff it maximizes A_{2} among all the 1OE estimators (in F); 3OE, 4OE and so fourth are defined consecutively in the same manner.

Let g(·) be a measurable function from R^{p} to R^{q}, four times continuously differentiable in a neighborhood of η(u_{0}). Consider, as F, a class of second order bias-corrected version g^{∗}(x̄) of Fisher consistent estimators:

F = \{ g^{∗}(x̄) = g(x̄) − \frac{1}{n}b_{g}^{(1)}(x̄) − \frac{1}{n^{2}} b_{g}^{(2)}(x̄)|

g(η(u)) = u for every u ∈ N (u_{0}), E[g^{∗}(x̄) − u] = O(n^{−3})}.

The measurable functions b_{g}^{(1)}(x̄) and b_{g}^{(2)}(x̄) are factors for bias correction, continuously differentiable in x̄ ∈ N (η(u_{0})). The specific form of b_{g}^{(1)}(x̄) will be given in (3).

It is well known that g^{∗}(x̄) is 1OE iff

G_{1} = \frac{∂g(x̄)}{∂x̄′} |_{x^{}¯ =η(u)} = iu^{−1}Θ′_{1}

and that the MLE meets the condition. For every estimator in F, A_{2} = 0. Thus, the MLE is 1OE and 2OE. We have also known that the MLE is 3OE (Fisher-Rao theorem).

3. Fourth and Fifth Order Efficiency

Let x = [x_{1},…, x_{p}]′ and A(x) = [a_{ij}(x)] be an a_{1} × a_{2} matrix-valued continuously differentiable function. Put \frac{∂A(x)}{∂x_{k}} = [\frac{∂a_{ij}(x)}{∂x_{k}}]. Define \frac{dA(x)}{dx′} = \frac{d}{dx′} A(x) =

[\frac{∂A(x)}{∂x_{1}},…, \frac{∂A(x)}{∂x_{p}}] and \frac{d}{dx}A(x) = (\frac{d}{dx′} A(x)′)′. The higher order (matrix) deriva-

tives are defined inductively by (\frac{d}{dx′} )^{<k>} = \frac{d}{dx′} (\frac{d}{dx′} )^{<k−1>} with (\frac{d}{dx′} )^{<0>} = 1. For some properties of such matrix derivatives, see Kano (1993a,b) (cf. Bentler and

Lee 1980; Magnus and Neudecker 1988). Let us simply write A^{<k>} = \overbrace{A ⊗ ⋯ ⊗ A}^{k-fold}, k-fold right Kronecker (tensor) product of the same matrix A. Note A^{<0>} = 1. Let g^{∗}(x̄) be in F, and let û^{∗}(x̄)(∈ F) be the bias-corrected version of the MLE. Under certain minimal regularity conditions on the population distribution, the distributions of û^{∗}(x̄) ∈ F and g^{∗}(x̄) ∈ F permit Edgeworth expansions as in (1) with polynomials r̄_{k}(x, u) and r_{k}(x, u), respectively. Let p̄_{n}(x|u) and p_{n}(x|u) be the approximate densities associated with û^{∗}(x̄) and g^{∗}(x̄). We further write

\begin{matrix}Ḡ_{k}(x̄) = (\frac{d}{dx^{}¯ ′} )^{<k>} û(x̄), & Ḡ_{k} = Ḡ_{k}(η(u)) \\ G_{k}(x̄) = (\frac{d}{dx^{}¯ ′} )^{<k>} g(x̄), & G_{k} = Ḡ_{k}(η(u)) \end{matrix}

Note that G_{k}(·) and Ḡ_{k}(·) are defined on a neighborhood of η(u_{0}), and they are extended to be measurable functions on the entire R^{p}.

For every 1OE estimator g^{∗}(x̄) ∈ F, Kano (1995 formula (4.11)) derived

p̄_{n}(x|u) − p_{n}(x|u) = \frac{1}{4n}(r̄_{3}(x, u) − r_{3}(x, u))N_{q}(x|0, iu^{−1}) + o (\frac{1}{n})
= \frac{1}{4n}tr[(G_{2} − Ḡ_{2})Ψ_{11}^{<2>}(G_{2} − Ḡ_{2})′(i_{u} − i_{u}xx′i_{u})]
·N_{q}(x|0, iu^{−1}) + o (\frac{1}{n}) .  (2)

Based on this, we can show a necessary and sufficient condition that g^{∗}(x̄) be 3OE in F (in terms of Fisher information) is that G_{3} = Ḡ_{3}. The proof is exactly the same as that for 5OE which will follows. Thus, we reach a known result that the MLE is 3OE. The necessary and sufficient condition on the derivatives of estimators may not be known.

The first order bias is \frac{1}{2n}G_{2}Ψ_{20}. Here we take

b_{g}^{(1)}(x̄) = \frac{1}{2}G_{2}(x̄)Ψ_{20}(θ(g(x̄))).  (3)

In the case, Kano (1994 formula (5.9)) derived, for every 3OE estimator g^{∗}(x̄) ∈ F

p̄_{n}(x|u) − p_{n}(x|u) = \frac{1}{12n^{2}} (r̄_{5}(x, u) − r_{5}(x, u))N_{q}(x|0, iu^{−1}) + o (\frac{1}{n^{2}} )
= \frac{1}{12n^{2}} tr[(G_{3} − Ḡ_{3})Ψ_{11}^{<3>}(G_{3} − Ḡ_{3})′(i_{u} − i_{u}xx′i_{u})]
·N_{q}(x|0, iu^{−1}) + o (\frac{1}{n^{2}} ) .  (4)

Theorem. Let the MLE û^{∗}(x̄) ∈ F and g^{∗}(x̄) ∈ F be bias-corrected versions with the factors defined in (3). Assume that g^{∗}(x̄) is 3OE. Then, g^{∗}(x̄) is always 4OE in F; and g^{∗}(x̄) is 5OE in F if and only if G_{3} = Ḡ_{3}. In particular, the bias-corrected version of the MLE is 5OE in F.

Proof. We simply denote, by p̄_{n} and p_{n}, p̄_{n}(x|u) and p_{n}(x|u). Consider the difference in AF I between û^{∗}(x̄) and g^{∗}(x̄):

AF I(û^{∗}(x̄)) − AF I(g^{∗}(x̄))

= ∫ \frac{(∂p̄_{n})(∂p̄_{n})′}{p̄_{n}} dx − ∫ \frac{(∂p_{n})(∂p_{n})′}{p_{n}} dx

= ∫ \frac{(p_{n} − p̄_{n})(∂p̄_{n})(∂p̄_{n})′}{p̄_{n}p_{n}} dx + ∫ \frac{∂(p̄_{n} − p_{n})(∂p_{n})′}{p_{n}} dx

+ ∫ \frac{(∂p_{n})∂(p̄_{n} − p_{n})′}{p_{n}} dx.  (5)

We need to evaluate the terms up to O (\frac{1}{n^{2}} ). Noting (4), we can replace ∂ = \frac{∂}{∂x} − \frac{1}{\sqrt{n}} \frac{∂}{∂u} with \frac{∂}{∂x} and regard, as N_{q}(x|0, iu^{−1}), all the terms p̄_{n} and p_{n} but those in the difference p̄_{n} − p_{n}. Let A_{G} = (G_{3} − Ḡ_{3})Ψ_{11}^{<3>}(G_{3} − Ḡ_{3})′. Then, the first term in the LHS of (5) is equal to

\frac{1}{12n^{2}} ∫ tr[A_{G}(i_{u}xx′i_{u} − i_{u})](i_{u}x)(i_{u}x)′N_{q}(x|0, iu^{−1})dx + o (\frac{1}{n^{2}} )

= \frac{1}{12n^{2}} ∫ tr[A_{G}(xx′ − i_{u})]xx′N_{q}(x|0, i_{u})dx + o (\frac{1}{n^{2}} )

= \frac{1}{6n^{2}} i_{u}A_{G}i_{u} + o (\frac{1}{n^{2}} ) .

The second and third terms in (5) vanish because ∂(p̄_{n} − p_{n}) and ∂p_{n} are basically the third order and the first order multivariate Hermite polynomials, respectively, and they are orthogonal to each other. As a consequence, we obtain

AF I(û^{∗}(x̄)) − AF I(g^{∗}(x̄)) = \frac{1}{6n^{2}} i_{u}(G_{3} − Ḡ_{3})Ψ_{11}^{<3>}(G_{3} − Ḡ_{3})′i_{u} + o (\frac{1}{n^{2}} ) (6)

from which the necessary and sufficient condition on 5OE in the Theorem follows. The equation (6) also implies that the terms of O (\frac{1}{n\sqrt{n}}) coincide between AF I(û^{∗}(x̄)) and AF I(g^{∗}(x̄)) for any 3OE g^{∗}(x̄) in F. This shows that 3OE implies 4OE. Q.E.D.

Staring from (2), we can iterate exactly the same process to prove the 2OE and 3OE of 1OE estimators in F. While the proof is alternative to the existing proofs of 3OE based on the representation in conditional expectation (e.g., Rao 1961; Eguchi 1983), it requires more calculations than the traditional ones because the consequence (2) needs computation rather much. A merit of our proofs for 2OE and 3OE is easy generalizability to those of 4OE and 5OE.

Besides efficiency of the MLE, the loss of Fisher information of 5OE estimators and its geometrical structure are also interesting topics. Much more tedious calculation is involved in assessment of the loss of information itself, as compared with the calculation of the difference in Fisher information. The result will be coming soon.

ACKNOWLEDGMENT

The author would like to express his thanks to Professor S. Amari of The University of Tokyo for his suggestion.

References

Akahira, M. and Takeuchi, K. (1981). Asymptotic Efficiency of Statistical Estimators: Concepts and Higher Order Asymptotic Efficiency, Lecture Notes in Statistics, Vol.7. Springer-Verlag, New York.

Amari, S. (1982). Differential geometry of curved exponential families—Curvatures

and information loss. Ann. Statist., 10, 357-385. Amari, S. (1985). Differential-Geometrical Methods in Statistics, Lecture Notes in

Statistics, Vol.28. Springer-Verlag, Berlin. Bentler, P. M. and Lee, Sik-Yum (1978). Matrix derivatives with chain rule and

rules for simple, Hadamard, and Kronecker products. J. Math. Psychol., 17, 255-262.

Bhattacharya, R. and Denker, M. (1990). Asymptotic Statistics. Birkhäuser, Basel. Efron, B. (1975). Defining the curvature of a statistical problem (with applications

to second order efficiency). Ann. Statist., 3, 1189-1242. Eguchi, S. (1983). Second order efficiency of minimum contrast estimators in a curved

exponential family. Ann. Statist., 11, 793-803. Fisher, R. A. (1925). Theory of statistical estimation. Proc. Cambridge Philos. Soc.,

122, 700-725 Ghosh, J. K. (1994). Higher Order Asymptotics. NSF-CBMS Regional Conference

Series in Probability and Statistics, Vol.4. IMS, Hayward, CA. Ghosh, J. K. and Subramanyam, K. (1974). Second order efficiency of maximum

likelihood estimates. Sankhyā, Ser. A, 36, 325-358. Hosoya, Y. (1988). The second-order Fisher information. Biometrika, 75, 265-274. Kano, Y. (1993a). Beyond third-order efficiency. (submitted) Kano, Y. (1993b). Beyond third-order efficiency: Supplement. Technical Report,

University of Osaka Prefecture, Osaka, Japan. Kano, Y. (1994). More higher-order efficiency: Concentration probability. Technical

Reports of Statistical Group: DATA95-04, Tokyo, Japan. Kano, Y. (1995). Third-order efficiency implies fourth-order efficiency. Technical

Reports of Statistical Group: DATA95-07, Tokyo, Japan. Magnus, J. R. and Neudecker, H. (1988). Matrix Differential Calculus with Applica-

tions in Statistics and Econometrics. John Wiley, Chichester. Pfanzagl, J. and Wefelmeyer, W. (1978). A third order optimum property of the

maximum likelihood estimator. J. Multivariate Anal., 8, 1-29. Rao, C. R. (1961). Asymptotic efficiency and limiting information. Proc. Fourth

Berkeley Symp., 1, 531-545.

Rao, C. R., Sinha, B. K. and Subramanyam, K. (1982). Third order efficiency of the maximum likelihood estimator in the multinomial distribution. Statist. Decisions, 1, 1-16.

Takeuchi, K. and Morimune, K. (1985). Third-order efficiency of the extended maximum likelihood estimators in a simultaneous equation system. Econometrica, 53, 177-200.

Taniguchi, M. (1991). Higher Order Asymptotic Theory for Time Series Analysis,

Lecture Notes in Statistics, Vol.68. Springer-Verlag, Berlin. Taniguchi, M and Watanabe, Y. (1994). Statistical analysis of curved probability

densities. J. Multivariate Anal., 48, 228-248.