Sometimes as I click through imagery from spacecraft and observatories, I think about what the world was like before we had an Internet to deliver this kind of information. Consider the early surveys of the heavens, exemplified by William Herschel sweeping the sky in the late 1700s. Herschel’s survey would find a new planet, create a basic map of the Milky Way, and note the location of the ‘cloudy things’ called nebulae, many of which turned out to be galaxies in their own right. His lists and annotations would grow into the New General Catalogue, which identifies thousands of objects by the now familiar NGC numbers.

The sky is all about statistics, as Herschel saw. When you’re dealing with objects whose lifespan is far longer than a human’s, you try to understand them by looking at enough examples to see the objects at every stage of their existence. Ann Finkbeiner offers this lovely Herschel quote in her new book A Grand and Bold Thing (Free Press, 2010):

“[The heavens] are now seen to resemble a luxuriant garden, [and]…is it not the same thing, whether we live successively to witness the germination, blooming, foliage, fecundity, fading, withering, and conception of a plant, or whether a vast number of specimens, selected from every stage through which the planet passes in the course of its existence be brought at once to our view?”

And then there’s Fred Hoyle, who said, “The Universe is so vast, and the lengths of time…are so long, that almost every conceivable type of astronomical process is still going on somewhere or other.” Finkbeiner’s book is all about the Sloan Digital Sky Survey, placing it firmly in the context of earlier astronomical work while bringing home with clarity and conviction how the new tools at our disposal, from adaptive optics to light-absorbing CCDs, have changed the game. Go back to the mid-20th Century and ponder the first optical survey done with a camera, the Palomar Observatory Sky Survey done with Caltech’s 48-inch telescope on Mount Palomar.

Surveys and Their Limitations

The Palomar survey, begun in 1949, produced some four thousand glass photographic plates, and if you were going to be a thorough astronomer in that era, you would need a set of photographic prints of the plates ($14,000) or glass copies of them ($25,000). Find something odd in a radio or X-ray observation and you would take your data to the Palomar plates to see what you were detecting. With the coordinates established, you could then go to a telescope and take a spectrum of whatever you had found. Needless to say, there was no ready access to telescopes or databases of their observations through a network available in the office.

But that wasn’t the only thing wrong with the plate method. For one thing, although their resolution was excellent, the plates used grains of silver halide that darkened when exposed to light. Too much light caused the plates to saturate, sharply reducing astronomers’ ability to measure the brightness of an object. And like all photographic plates, the Palomar plates differed from each other, exposed at different times and under different conditions. The advent of CCDs meant we could start measuring light without saturation and with immediate response. Moreover, the brightness of an object could now be measured to exceedingly high levels of accuracy.

Compared to all previous methods, the beauty of a digital sky survey like the SDSS is obvious. Data go from telescopes into computer storage, from which computers everywhere can access them. But doing a digital survey was a job of mind-numbing complexity (and I’ll spare you the political and academic dimension, which Finkbeiner covers in great detail and with a whimsical verve). Here she describes what the the Sloan Digital Sky Survey was attempting to do:

The Sloan itself was a new creature, effectively a real-time robotic astronomer. The software needed to observe and record, then locate, characterize, identify, and archive. It needed to be able to handle at least a trillion bytes of data — a terabyte, a unit astronomers had never before had any cause to use — coming off the telescope at a rate of 17 gigabytes, 17 billion bytes, every hour. It needed to command the automation of the telescope and the observations so that the only people on the mountain would be a staff of professional observers to oversee and troubleshoot, and everybody else could stay home and download galaxies. It needed to take the data coming off the instruments — the camera and the spectrographs — and reduce it, that is, turn data from the CCDs into images and spectra, standardized so that the stars and galaxies and quasars all looked as though they’d been taken on the same night under the same conditions.

What changes we’ve seen. In the late 1980s, Finkbeiner writes, there were about 4500 astronomers in the US and ten large telescopes, the majority of which were private. An astronomer would make observations at such an instrument and return with plates that would be analyzed in isolation. Astronomers without private telescopes could use the National Science Foundation’s two national observatories, coping with the fact that the instruments were far oversubscribed. Even then, scientists labored without well calibrated archives. Some early Sloan participants contemplated putting their data on CDs and selling them for $20,000 per set.

Image: 2.5m dedicated SDSS telescope at Apache Point Observatory. Credit: SDSS.

Taking the Data Online

Digital networking, of course, changed the equation. Today we can take a quick trip online to the SkyServer, downloading data with abandon and the kind of ease that will have future generations wondering how earlier astronomers ever functioned with so few major instruments and such inaccessible data. Putting terabytes of data into an online gateway, though, first meant collecting the information. It was James Gunn (Princeton) who coupled the idea of using state of the art CCDs with a camera and spectrograph fed by optical fibers to take spectra of hundreds of galaxies at once. His story animates Finkbeiner’s account. Gunn’s notion just grew and grew: Let the camera and spectrograph be fixed to a telescope that can drift scan one strip of sky after another and you wind up with images and spectra of galaxies galore.

Gunn’s idea would galvanize the astronomical community. Listen to Finkbeiner as she describes the reaction of Tim Heckman (Johns Hopkins) upon reading the Sloan proposal:

…the Sloan was going to change the way astronomers without private telescopes worked: Instead of taking a couple of years writing and rewriting proposals for three clouded-out nights on a public telescope, you could just take the interesting question that occurred to you and find the data you needed in the library of the universe. And the interesting questions would not be just about large-scale structure: for every question in optical astronomy that Heckman could think of, the Sloan archive would have data. And the amount of data on each question would be orders of magnitude larger than it had ever been before.

Amazing what you can do with a 2.5-meter mirror. Amazing, too, how institutions and individuals can alternately work together and frustrate each other to the point where projects like the SDSS are brought within an inch of being canceled. Finkbeiner has the whole story, and it’s sharply etched with the personalities of its protagonists. But what a result: The SkyServer holds everything in the surveyed sky, everything available through spectra, every image in five colors, every object from star to galaxy to quasar sorted as required and available for data manipulation.

How big has the SDSS become?

By the end of 2003, Science magazine’s Breakthrough of the Year was the new standard model of the universe revealed by comparing the cosmic microwave background as measured by NASA’s WMAP satellite with the large-scale structure as mapped by Sloan. By mid-2004, Sloanies had written 400 papers, and non-Sloanies using Sloan data another 125. In August, Scot Kleinman, an Apache Point observer, went to the Fourteenth European White Dwarf Workshop in Germany and reported that nearly 40 percent of the talks mentioned the Sloan. A non-Sloanie attending an American Astronomical Society meeting said he was astounded at the way the Sloan permeated all the talks…In 2001 and 2006, of all the optical observatories — the Hubble Space Telescope included — the Sloan was the most productive; in the intervening years, no one bothered to rank observatories…. As of October 2009, 2,656 papers were based on Sloan data and were cited in other papers 100,000 times. A non-Sloanie at the Space Telescope Science Institute said that Sloan hadn’t even been on his radar, and now it was astronomy’s eight-hundred-pound gorilla.

The impact of the SDSS is undeniable, and it’s no surprise that the public impulse behind it has spawned further innovations like Google Sky, WikiSky, and the Galaxy Zoo. Sloan II followed, then Sloan III as astronomy continued to move from a solitary scientist on a mountain to group collaboration and papers with not one or two but 150 co-authors. Today large surveys like Pan-STARRS and the LSST are in the works and like the SDSS, they will be online. Astronomy has opened up, becoming more accessible for practitioner and amateur alike. What a sea-change we’ve witnessed, and the Sloan Digital Sky Survey has been in the thick of it, described in this new book with panache by a writer who knows her science as well as her verbal craft.

The book is Finkbeiner, A Grand and Bold Thing: An Extraordinary New Map of the Universe Ushering in a New Era of Discovery. New York: Free Press, 2010.