The AstroStat Slog » Meta http://hea-www.harvard.edu/AstroStat/slog Weaving together Astronomy+Statistics+Computer Science+Engineering+Intrumentation, far beyond the growing borders Fri, 09 Sep 2011 17:05:33 +0000 en-US hourly 1 http://wordpress.org/?v=3.4 accessing data, easier than before but… http://hea-www.harvard.edu/AstroStat/slog/2009/accessing-data/ http://hea-www.harvard.edu/AstroStat/slog/2009/accessing-data/#comments Tue, 20 Jan 2009 17:59:56 +0000 hlee http://hea-www.harvard.edu/AstroStat/slog/?p=301 Someone emailed me for globular cluster data sets I used in a proceeding paper, which was about how to determine the multi-modality (multiple populations) based on well known and new information criteria without binning the luminosity functions. I spent quite time to understand the data sets with suspicious numbers of globular cluster populations. On the other hand, obtaining globular cluster data sets was easy because of available data archives such as VizieR. Most data sets in charts/tables, I acquire those data from VizieR. In order to understand science behind those data sets, I check ADS. Well, actually it happens the other way around: check scientific background first to assess whether there is room for statistics, then search for available data sets.

However, if you are interested in massive multivariate data or if you want to have a subsample from a gigantic survey project, impossible all to be documented in contrast to those individual small catalogs, one might like to learn a little about Structured Query Language (SQL). With nice examples and explanation, some Tera byte data are available from SDSS. Instead of images in fits format, one can get ascii/table data sets (variables of million objects are magnitudes and their errors; positions and their errors; classes like stars, galaxies, AGNs; types or subclasses like elliptical galaxies, spiral galaxies, type I AGN, type Ia, Ib, Ic, and II SNe, various spectral types, etc; estimated variables like photo-z, which is my keen interest; and more). Furthermore, thousands of papers related to SDSS are available to satisfy your scientific cravings. (Here are Slog postings under SDSS tag).

If you don’t want to limit yourself with ascii tables, you may like to check the quick guide/tutorial of Gator, which aggregated archives of various missions: 2MASS (Two Micron All-Sky Survey), IRAS (Infrared Astronomical Satellite), Spitzer Space Telescope Legacy Science Programs, MSX (Midcourse Space Experiment), COSMOS (Cosmic Evolution Survey), DENIS (Deep Near Infrared Survey of the Southern Sky), and USNO-B (United States Naval Observatory B1 Catalog). Probably, you also want to check NED or NASA/IPAC Extragalactic Database. As of today, the website said, 163 million objects, 170 million multiwavelength object cross-IDs, 188 thousand associations (candidate cross-IDs), 1.4 million redshifts, and 1.7 billion photometric measurements are accessible, which seem more than enough for data mining, exploring/summarizing data, and developing streaming/massive data analysis tools.

Probably, astronomers might wonder why I’m not advertising Chandra Data Archive (CDA) and its project oriented catalog/database. All I can say is that it’s not independent statistician friendly. It is very likely that I am the only statistician who tried to use data from CDA directly and bother to understand the contents. I can assure you that without astronomers’ help, the archive is just a hot potato. You don’t want to touch it. I’ve been there. Regardless of how painful it is, I’ve kept trying to touch it since It’s hard to resist after knowing what’s in there. Fortunately, there are other data scientist friendly archives that are quite less suffering compared to CDA. There are plethora things statisticians can do to improve astronomers’ a few decade old data analysis algorithms based on Gaussian distribution, iid assumption, or L2 norm; and to reflect the true nature of data and more relaxed assumptions for robust analysis strategies than for traditionally pursued parametric distribution with specific models (a distribution free method is more robust than Gaussian distribution but the latter is more efficient) not just with CDA but with other astronomical data archives. The latter like vizieR or SDSS provides data sets which are less painful to explore with without astronomical software/package familiarity.

Computer scientists are well aware of UCI machine learning archive, with which they can validate their new methods with previous ones and empirically prove how superior their methods are. Statisticians are used to handle well trimmed data; otherwise we suggest strategies how to collect data for statistical inference. Although tons of data collecting and sampling protocols exist, most of them do not match with data formats, types, natures, and the way how data are collected from observing the sky via complexly structured instruments. Some archives might be extensively exclusive to the funded researchers and their beneficiaries. Some archives might be super hot potatoes with which no statistician wants to involve even though they are free of charges. I’d like to warn you overall not to expect the well tabulated simplicity of text book data sets found in exploratory data analysis and machine learning books.

Some one will raise another question why I do not speculate VOs (virtual observatories, click for slog postings) and Google Sky (click for slog postings), which I praised in the slog many times as good resources to explore the sky and to learn astronomy. Unfortunately, for the purpose of direct statistical applications, either VOs or Google sky may not be fancied as much as their names’ sake. It is very likely spending hours exploring these facilities and later you end up with one of archives or web interfaces that I mentioned above. It would be easier talking to your nearest astronomer who hopefully is aware of the importance of statistics and could offer you a statistically challenging data set without worries about how to process and clean raw data sets and how to build statistically suitable catalogs/databases. Every astronomer of survey projects builds his/her catalog and finds common factors/summary statistics of the catalog from the perspective of understanding/summarizing data, the primary goal of executing statistical analyses.

I believe some astronomers want to advertise their archives and show off how public friendly they are. Such advertising comments are very welcome because I intentionally left room for those instead of listing more archives I heard of without hands-on experience. My only wish is that more statisticians can use astronomical data from these archives so that the application section of their papers is filled with data from these archives. As if with sunspots, I wish that more astronomical data sets can be used to validate methodologies, algorithms, and eventually theories. I sincerely wish that this shall happen in a short time before I become adrift from astrostatistics and before I cannot preach about the benefits of astronomical data and their archives anymore to make ends meet.

There is no single well known data repository in astronomy like UCI machine learning archive. Nevertheless, I can assure you that the nature of astronomical data and catalogs bear various statistical problems and many of those problems have never been formulated properly towards various statistical inference problems. There are so many statistical challenges residing in them. Not enough statisticians bother to look these data because of the gigantic demands for statisticians from uncountably many data oriented scientific disciplines and the persistent shortage in supplies.

]]>
http://hea-www.harvard.edu/AstroStat/slog/2009/accessing-data/feed/ 3
When you register http://hea-www.harvard.edu/AstroStat/slog/2008/maintenance-issue/ http://hea-www.harvard.edu/AstroStat/slog/2008/maintenance-issue/#comments Thu, 09 Oct 2008 15:55:30 +0000 hlee http://hea-www.harvard.edu/AstroStat/slog/?p=1020 I bet there are various scams. One of them is automatic user registration. This blog requires a registration for contributing free of approval comments unless one does not put many web links. Recently, there were frequent anonymous user registrations. What I mean by anonymous is that I don’t see their names or part of identities (for example, someone uses initials of their names in their email accounts or uses email accounts from their affiliations). This slog is open to anyone who is interested in AstroStatistics, although not many are currently active. Upon your request, this can be changed very simply and you immediately start writing your ideas about AstroStatistics. However, I must restrict those scams from now on. Please, provide me a small information about you if you do not want to be eliminated after your registration. As I mentioned, the information does not require your full name, nor email account of academic institution. When you register, use your email account that you use daily bases, not the ones that look like results from phishing.

]]>
http://hea-www.harvard.edu/AstroStat/slog/2008/maintenance-issue/feed/ 0
I Like Eq http://hea-www.harvard.edu/AstroStat/slog/2008/i-like-eq/ http://hea-www.harvard.edu/AstroStat/slog/2008/i-like-eq/#comments Wed, 13 Aug 2008 16:59:54 +0000 vlk http://hea-www.harvard.edu/AstroStat/slog/?p=379 I grew up in an environment that glamourized mathematical equations. Equations adorned a text like jewelry, set there to dazzle, and often to outshine the text that they were to illuminate. Needless to say, anything I wrote was dense, opaque, and didn’t communicate what it set out to. It was not until I saw a Reference Frame essay by David Mermin on how to write equations (1989, Physics Today, 42, p9) that I realized that equations should be treated as part of the text. You should be able to read them. David Mermin set out 3 rules for writing out equations, which I’ve tried to follow diligently (if not always successfully) since then.

  1. Number or label all displayed equations (Fisher’s Rule):

    The most common violation of Fisher’s rule is the misguided practice of numbering only those displayed equations to which the text subsequently refers back. … it is necessary to state emphatically that Fisher’s rule is for the benefit not of the author, but the reader.
    For although you, dear author, may have no need to refer in your text to the equations you therefore left unnumbered, it is presumptuous to assume the same disposition in your readers. And although you may well have acquired the solipsistic habit of writing under the assumption that you will have no readers at all, you are wrong.

  2. When referring to an equation within the text, identify it by a phrase as well as a number (aka the Good Samaritan Rule):

    A Good Samaritan is compassionate and helpful to one in distress, and there is nothing more distressing than having to hunt your way back in a manuscript in search of Eq. (2.47), not because your subsequent progress requires you to inspect it in detail, but merely to find out what it is about so you may know the principles that go int othe construction of Eq. (7.38).

  3. Punctuate the equation (aka the Math is Prose Rule):

    The equations you display are embedded in your prose and constitute an inseparable part of it.

    Regardless … of how to parse the equation internally, certain things are clear to anyone who understands the equation and the prose in which it is embedded.

    We punctuate equations because they are a form of prose (they can, after all, be read aloud as a sequence of words) and are therefore subject to the same rules as any other prose. … punctuation makes them easier to read and often clarifies the discussion in which they occur. … viewing an equation not as a grammatically irrelevant blob, but as a part of the text … can only improve the fluency and grace of one’s expository mathematical prose.

]]>
http://hea-www.harvard.edu/AstroStat/slog/2008/i-like-eq/feed/ 2
Too many syllables? http://hea-www.harvard.edu/AstroStat/slog/2007/too-many-syllables/ http://hea-www.harvard.edu/AstroStat/slog/2007/too-many-syllables/#comments Fri, 07 Dec 2007 04:06:46 +0000 vlk http://hea-www.harvard.edu/AstroStat/slog/2007/too-many-syllables/ Stumbled across this “blog readability test“, and well, put the Slog to the Test. Apparently you need a college degree to read this stuff.  Who woulda thunk?

reading level

]]>
http://hea-www.harvard.edu/AstroStat/slog/2007/too-many-syllables/feed/ 0
Categories http://hea-www.harvard.edu/AstroStat/slog/2007/categories/ http://hea-www.harvard.edu/AstroStat/slog/2007/categories/#comments Fri, 25 May 2007 15:51:43 +0000 vlk http://hea-www.harvard.edu/AstroStat/slog/2007/categories/ It seems to be necessary to devise a good system of categories beforehand with wordpress, because plain tags are not (yet?) implemented. Categories can only be defined by the admin, so it is useful to have as comprehensive a list as possible. I suggest the following set:

(Edited slightly to reorganize)

Meta (about the site, the software, etc)
News
(conferences, deadlines, workshops, schools, etc.)
Nuggets
(research reports, new algorithms, etc.)
CHASC
(about CHASC)
Quotes
Jargon
Bad AstroStat

Astro (primarily Astronomy or Astrophysics oriented posts)
Astro > High-Energy
Astro > High-Energy > X-ray
Astro > High-Energy > gamma-ray
Astro > Optical
Astro > Physics
Astro > Stars
Astro > Galaxies
Astro > Objects

Stat (primarily Statistics oriented posts)
Stat > Bayesian
Stat > Frequentist
Stat > Fitting
Stat > Uncertainty
Stat > MC
Stat > MC > MCMC

Algorithms
Imaging
Spectral
Timing
Data Processing
Languages

Corrections and additions welcome!

]]>
http://hea-www.harvard.edu/AstroStat/slog/2007/categories/feed/ 5