I have found that astronomers are generally unable to judge the comparative powers of different tests. This is not surprising, because we aren’t trained for it. So for instance when two tests may give apparently contradictory answers, we can’t tell which one to believe. The answer, of course, is to always go with the more powerful one. But of course, that’s easier to say than to do.

I do admit however that the concept of statistical power is not all that familiar to astronomers. We think of that usually in the context of probability of detection, and not much else. However, it is also true that most statistical techniques that exist currently are far too general and simpleminded to deal with the complexities enountered daily in astronomical data. It would be foolish to expect, say, that a local detect method would work as well as a human to tag and identify coronal loops in a Hinode image. So naturally we develop a bs detector to tell whether some test or technique is not working. But don’t mistake that for rigorous analysis!