It is widely believed that under some fairly general conditions, MLEs are consistent, asymptotically normal, and efficient. Stephen Stigler has elegantly documented some of Fisher’s troubles when he wanted a proof. You want proof? Of course you can pile on assumptions so that the proof is easy. If checking your assumptions in any particular case is harder than checking the conclusion in that case, you will have joined a great tradition.
I used to think that efficiency was a thing for the theorists (I can live with inefficiency), that normality was a thing of the past (we can simulate), but that—in spite of Ralph Waldo Emerson—consistency is a thing we should demand of any statistical procedure. Not any more. These days we can simulate in and around the conditions of our data, and learn whether a novel procedure behaves as it should in that context. If it does, we might just believe the results of its application to our data. Other people’s data? That’s their simulation, their part of the parameter space, their problem. Maybe some theorist will take up the challenge, and study the procedure, and produce something useful. But if we’re still waiting for that with MLEs in general (canonical exponential families are in good shape), I wouldn’t hold my breath for this novel procedure. By the time a few people have tried the new procedure, each time checking its suitability by simulation in their context, we will have built up a proof by simulation. Shocking? Of course.
Some time into my career as a statistician, I noticed that I don’t check the conditions of a theorem before I use some model or method with a set of data. I think in statistics we need derivations, not proofs. That is, lines of reasoning from some assumptions to a formula, or a procedure, which may or may not have certain properties in a given context, but which, all going well, might provide some insight. The evidence that this might be the case can be mathematical, not necessarily with epsilon-delta rigour, simulation, or just verbal. Call this “a statistician’s proof ”. This is what I do these days. Should I be kicked out of the IMS?

After reading many astronomy literature, I develop a notion that astronomers like to use the maximum likelihood as a robust alternative to the chi-square minimization for fitting astrophysical models with parameters. I’m not sure it is truly robust because not many astronomy paper list assumptions and conditions for their MLEs.

Often I got confused with their target parameters. They are not parameters in statistical models. They are not necessarily satisfy the properties of probability theory. I often fail to find statistical properties of these parameters for the estimation. It is rare checking statistical modeling procedures with assumptions described by Prof. Speed. Even derivation is a bit short to be called “rigorous statistical analysis.” (At least I wish to see a sentence that “It is trivial to derive the estimator with this and that properties”).

Common phrases I confronted from astronomical literature is that authors’ strategy is statistically rigorous, superior, or powerful without showing why and how it is rigorous, superior, or powerful. I tried to convey these pitfalls and general restrictions in their employed statistical methods. Their strategy is not “statistically robust” nor “statistically powerful” nor “statistically rigorous.” Statisticians have own measures of “superiority” to discuss the improvement in their statistics, analysis strategies, and methodology.

It has not been easy since I never intend to case specific fault picking every time I see these statements. A method believed to be robust can be proven as not a robust method with your data and models. By simulations and derivations with the sufficient description of conditions, your excellent method can be presented with statistical rigors.

Within similar circumstances for statistical modeling and data analysis, there’s a trade off between robustness and conditions among statistical methodologies. Before stating a particular method adopted is robust or rigid, powerful or insensitive, efficient or inefficient, and so on; derivation, proof, or simulation studies are anticipated to be named the analysis and procedure is statistically excellent.

Before it gets too long, I’d like say that statistics have traditions for declaring working methods via proofs, simulations, or derivations. Each has their foundations: assumptions and conditions to be stated as “robust”, “efficient”, “powerful”, or “consistent.” When new statistics are introduced in astronomical literature, I hope to see some additional effort of matching statistical conditions to the properties of target data and some statistical rigor (derivations or simulations) prior to saying they are “robust”, “powerful”, or “superior.”

[arxiv:physics.data-an:0808.0012]
Lectures on Probability, Entropy, and Statistical Physics by Ariel Caticha
Abstract: These lectures deal with the problem of inductive inference, that is, the problem of reasoning under conditions of incomplete information. Is there a general method for handling uncertainty? Or, at least, are there rules that could in principle be followed by an ideally rational mind when discussing scientific matters? What makes one statement more plausible than another? How much more plausible? And then, when new information is acquired how do we change our minds? Or, to put it differently, are there rules for learning? Are there rules for processing information that are objective and consistent? Are they unique? And, come to think of it, what, after all, is information? It is clear that data contains or conveys information, but what does this precisely mean? Can information be conveyed in other ways? Is information physical? Can we measure amounts of information? Do we need to? Our goal is to develop the main tools for inductive inference–probability and entropy–from a thoroughly Bayesian point of view and to illustrate their use in physics with examples borrowed from the foundations of classical statistical physics.

Consider a simple case where you have N observations of the luminosities of a source. Let us say that all N sources have been detected and their luminosities are estimated to be L_i, i=1..N, and that they are ordered such that L_i < L_i+1 Then, it is easy to see that the fraction of sources above each L_i can be written as the sequence

{ N-1, N-2, N-3, … 2, 1, 0}/N

The K-M estimator is a generalized form that describes this sequence, and is written as a product. The probability that an object in the sample has luminosity greater than L_k is

S(L>L₁) = (N-1)/N
S(L>L₂) = (N-1)/N * ((N-1)-1)/(N-1) = (N-1)/N * (N-2)/(N-1) = (N-2)/N
S(L>L₃) = (N-1)/N * ((N-1)-1)/(N-1) * ((N-2)-1)/(N-2) = (N-3)/N
…
S(L>L_k) = Π_i=1..k (n_i-1)/n_i = (N-k)/N

where n_k are the number of objects still remaining at luminosity level L ≥ L_k, and at each stage one object is decremented to account for the drop in the sample size.

Now that was for the case when all the objects are detected. But now suppose some are not, and only upper limits to their luminosities are available. A specific value of L cannot be assigned to these objects, and the only thing we can say is that they will “drop out” of the set at some stage. In other words, the sample will be “censored”. The K-M estimator is easily altered to account for this, by changing the decrement in each term of the product to include the censored points. Thus, the general K-M estimator is

S(L>L_k) = Π_i=1..k (n_i-c_i)/n_i

where c_i are the number of objects that drop out between L_i-1 and L_i.

Note that the K-M estimator is a maximum likelihood estimator of the cumulative probability (actually one minus the cumulative probability as it is usually understood), and uncertainties on it must be estimated via Monte Carlo or bootstrap techniques [or not.. see below].

• [astro-ph:0806.0650] Kimball and Ivezi\’c
  A Unified Catalog of Radio Objects Detected by NVSS, FIRST, WENSS, GB6, and SDSS (The catalog is available HERE. I’m always fascinated with the possibilities in catalog data sets which machine learning and statistics can explore. And I do hope that the measurement error columns get recognition from non astronomers.)

• [astro-ph:0806.0820] Landau and Simeone
  A statistical analysis of the data of Delta \alpha/ alpha from quasar absorption systems (It discusses Student t-tests from which confidence intervals for unknown variances and sample size based on Type I and II errors are obtained.)

• [stat.ML:0806.0729] R. Girard
  High dimensional gaussian classification (Model based – gaussian mixture approach – classification, although it is often mentioned as clustering in astronomy, on multi- dimensional data is very popular in astronomy)

• [astro-ph:0806.0520] Vio and Andreani
  A Statistical Analysis of the “Internal Linear Combination” Method in Problems of Signal Separation as in CMB Observations (Independent component analysis, ICA is discussed)

• [astro-ph:0806.0560] Nobel and Nowak
  Beyond XSPEC: Towards Highly Configurable Analysis (The flow of spectral analysis with XSPEC and Sherpa has not been accepted smoothly; instead, it has been a personal struggle. It seems the paper considers XSPEC as a black box, which I completely agree with. The main objective of the paper is comparing XSPEC and ISIS)

• [astro-ph:0806.0113] Casandjian and Grenier
  A revised catalogue of EGRET gamma-ray sources (The maximum likelihood detection method, which I never heard from statistical literature, is utilized)

• [stat.ME:0805.2756] Fionn Murtagh
  The Remarkable Simplicity of Very High Dimensional Data: Application of Model-Based Clustering

• [astro-ph:0805.2945] Martin, de Jong, & Rix
  A comprehensive Maximum Likelihood analysis of the structural properties of faint Milky Way satellites

• [astro-ph:0805.2946] Kelly, Fan, & Vestergaard
  A Flexible Method of Estimating Luminosity Functions [my subjective comment is added at the bottom]

• [stat.ME:0805.3220] Bayarri, Berger, Datta
  Objective Bayes testing of Poisson versus inflated Poisson models (will it be of use when one is dealing with many zero background counts, underpopulated above zero background counts, and underpopulated source counts?)

[Comment] You must read it. It can serve as a very good Bayesian tutorial for astronomers. I think there’s a typo, nothing major, plus/minus sign in the likelihood, though. Tom Loredo kindly has informed through his extensive slog comments about Schechter function and this paper made me appreciate the gamma distribution more. Schechter function and the gamma density function share the same equation although the objective of their use does not have much to be shared (Forgive my Bayesian ignorance in the extensive usage of gamma distribution except the fact it’s a conjugate of Poisson or exponential distribution).

FYI, there was another recent arxiv paper on zero-inflation [stat.ME:0805.2258] by Bhattacharya, Clarke, & Datta
A Bayesian test for excess zeros in a zero-inflated power series distribution

• [astro-ph:0805.1290]R. Barnard, L. Shaw Greening, U. Kolb
  A multi-coloured survey of NGC 253 with XMM-Newton: testing the methods used for creating luminosity functions from low-count data

• [astro-ph:0805.1469] Philip J. Marshall et al.
  Automated detection of galaxy-scale gravitational lenses in high resolution imaging data

• [astro-ph:0805.1470] E. P. Kontar, E. Dickson, J. Kasparova
  Low-energy cutoffs and in electron spectra of solar flares: statistical survey (It is not statistically rigorous but the topic can be connected to dip tests or gap tests in statistics)

• [astro-ph:0805.1936] J. Yee & B. Gaudi
  Characterizing Long-Period Transiting Planets Observed by Kepler (discusses uncertainty in light curves and Fisher matrix)

• [astro-ph:0805.1994] the QUad collaboration: C. Pryke et al.
  Second and third season QUaD CMB temperature and polarization power spectra (What is jackknife tests? A brief scan of the paper does not register with my understanding of jackknifing. It looks more close to cross validation. Another slog topic shall come: bootstrap, cross validation, jackknife, and resampling.)

• [astro-ph:0805.2121] N. Cole et al.
  Maximum Likelihood Fitting of Tidal Streams With Application to the Sagittarius Dwarf Tidal Tails

• [astro-ph:0805.2155] J Yoo & M Zaldarriaga
  Improved estimation of cluster mass profiles from the cosmic microwave background

• [astro-ph:0805.2207] A.Vikhlinin et al.
  Chandra Cluster Cosmology Project II: Samples and X-ray Data Reduction (it mentions calibration uncertainty and background, can it be a reference to stacking, coadding, source detection, etc?)

• [astro-ph:0805.2325] J.M. Loh
  A valid and fast spatial bootstrap for correlation functions

• [astro-ph:0805.2326] T. Wickramasinghe, M. Struble, J. Nieusma
  Observed Bimodality of the Einstein Crossing Times of Galactic Microlensing Events

• [astro-ph:0804.2904]M. Cruz et al.
  The CMB cold spot: texture, cluster or void?

• [astro-ph:0804.2917] Z. Zhu, M. Sereno
  Testing the DGP model with gravitational lensing statistics

• [astro-ph:0804.3390] Valkenburg, Krauss, & Hamann
  Effects of Prior Assumptions on Bayesian Estimates of Inflation Parameters, and the expected Gravitational Waves Signal from Inflation

• [astro-ph:0804.3413] N.Ball et al.
  Robust Machine Learning Applied to Astronomical Datasets III: Probabilistic Photometric Redshifts for Galaxies and Quasars in the SDSS and GALEX (Another related publication [astro-ph:0804.3417])

• [astro-ph:0804.3471] M. Cirasuolo et al.
  A new measurement of the evolving near-infrared galaxy luminosity function out to z~4: a continuing challenge to theoretical models of galaxy formation

• [astro-ph:0804.3475] A.D. Mackey et al.
  Multiple stellar populations in three rich Large Magellanic Cloud star clusters

• [stat.ME:0804.3853] C. R\”over , R. Meyer, N. Christensen
  Modelling coloured noise (MCMC & sunspot data)

• [astro-ph:0804.1809] H. Khiabanian, I.P. Dell’Antonio
  A Multi-Resolution Weak Lensing Mass Reconstruction Method (Maximum likelihood approach; my naive eyes sensed a certain degree of relationship to the GREAT08 CHALLENGE)

• [astro-ph:0804.1909] A. Leccardi and S. Molendi
  Radial temperature profiles for a large sample of galaxy clusters observed with XMM-Newton

• [astro-ph:0804.1964] C. Young & P. Gallagher
  Multiscale Edge Detection in the Corona

• [astro-ph:0804.2387] C. Destri, H. J. de Vega, N. G. Sanchez
  The CMB Quadrupole depression produced by early fast-roll inflation: MCMC analysis of WMAP and SDSS data

• [astro-ph:0804.2437] P. Bielewicz, A. Riazuelo
  The study of topology of the universe using multipole vectors

• [astro-ph:0804.2494] S. Bhattacharya, A. Kosowsky
  Systematic Errors in Sunyaev-Zeldovich Surveys of Galaxy Cluster Velocities

• [astro-ph:0804.2631] M. J. Mortonson, W. Hu
  Reionization constraints from five-year WMAP data

• [astro-ph:0804.2645] R. Stompor et al.
  Maximum Likelihood algorithm for parametric component separation in CMB experiments (separate section for calibration errors)

• [astro-ph:0804.2671] Peeples, Pogge, and Stanek
  Outliers from the Mass–Metallicity Relation I: A Sample of Metal-Rich Dwarf Galaxies from SDSS

• [astro-ph:0804.2716] H. Moradi, P.S. Cally
  Time-Distance Modelling In A Simulated Sunspot Atmosphere (discusses systematic uncertainty)

• [astro-ph:0804.2761] S. Iguchi, T. Okuda
  The FFX Correlator

• [astro-ph:0804.2742] M Bazarghan
  Automated Classification of ELODIE Stellar Spectral Library Using Probabilistic Artificial Neural Networks

• [astro-ph:0804.2827]S.H. Suyu et al.
  Dissecting the Gravitational Lens B1608+656: Lens Potential Reconstruction (Bayesian)

• [astro-ph:0804.0620] Q. Wu et al.
  Late transient acceleration of the universe in string theory on $S^{1}/Z_{2}$ (MCMC)

• [astro-ph:0804.0692] Corless, Dobke & King
  The Hubble constant from galaxy lenses: impacts of triaxiality and model degeneracies (MCMC, Bayesian Modeling)

• [astro-ph:0804.0788] Zamfir, Sulentic, & Marziani
  New Insights on the QSO Radio-Loud/Radio-Quiet Dichotomy: SDSS Spectra in the Context of the 4D Eigenvector1 Parameter Space

• [astro-ph:0804.0965] Bloom, Butler, & Perley
  Gamma-ray Bursts, Classified Physically (instead of statistics, it relies on physics to grow a (classification) tree)

• [astro-ph:0804.1089] G.K.Skinner
  The sensitivity of coded mask telescopes

• [astro-ph:0804.1197] Bagla, Prasad and Khandai
  Effects of the size of cosmological N-Body simulations on physical quantities – III: Skewness

• [astro-ph:0804.1447] Marsh, Ireland, & Kucera
  Bayesian Analysis of Solar Oscillations

• [astro-ph:0804.1532] C. López-Sanjuan, C. E. García-Dabó, M. Balcells
  A maximum likelihood method for bidimensional experimental distributions, and its application to the galaxy merger fraction

• [astro-ph:0804.1536] V.J.Martinez (One of my favorite astronomers who brings in mathematics and statistics)
  The Large Scale Structure in the Universe: From Power-Laws to Acoustic Peaks

Knowing that photons follows Poission distribution, that limited numbers of non nested candidate models (one does have physics to constrain models), that generalized linear model (GLM) could expand regression models discussed in [1] to account Poisson behavior, and that there hasn’t been any collaboration between economists and astronomers, citing some papers from economics journals may help astronomers handle non-nested models and test them (and get parameter estimates and proper error bars).

References—–

[1] MacKinnon, J.G. (1983). Model Specification Tests Against Non-Nested Alternatives, Queens’ Economics Department Working Paper, No. 573

[2] Cox, D.R. (1961). Tests of separate families of hypotheses, Proceedings of the fourth Berkeley symposium on mathematical statistics and probability, 1, pp. 105-123

[3] Cox. D.R. (1962). Further results on tests of separate families of hypotheses, JRSS, B, 24, pp. 406-424

[4] White, H. (1982). Maximum likelihood estimation of misspecified models, Econometrica, 50, pp. 1-25

[5] Nishii, R. (1988). Maximum likelihood principle and model selection when the true model is unspecified, Journal of Multivariate Analysis, 27(2), pp. 392-403

[6] Vuoung, Q.H. (1989). Likelihood ratio test for model selection and non-nested hypotheses, Econometrica, 57(2), pp.307-333

[7] Sin, C. & White, H. (1996) Information criteria for selecting possibly misspecified parametric models, Journal of Econometrics, 71, pp.207-225

The non-nested hypothesis testing problem was evolved from [2]&[3]. [4],[5],[6],&[7] are well cited papers on the topic or experienced my readership. Please, advice me if you have more information regarding non-nested hypothesis tests in astronomy.