Earlier this year, astronomers stumbled upon a fascinating finding: Thousands of black holes are likely to exist near the center of our galaxy.
The X-ray images that made this discovery were not from a modern state-of-the-art telescope. Still, they were taken recently ̵
No, the researchers discovered the black holes by scouring old, long-archived data.
Discoveries such as these will only pile up as the era of "big data" changes the way science is made. Astronomers are collecting an exponentially larger amount of data each day – so much so that it will take years to uncover any hidden signals hidden in the archives.
The evolution of astronomy
60 years ago, the typical astronomer worked largely alone or in a small team. They probably had access to a respectable large-scale ground-based optical telescope at their home institution.
Her observations were largely limited to optical wavelengths – more or less what the eye can see. This meant that they missed signals from a variety of astrophysical sources that can emit non-visible radiation from very low frequency radio to high energy gamma rays. Mostly, if you wanted to do astronomy, you had to be an academic or eccentric person with access to a good telescope.
Old data was stored in the form of photographic plates or published catalogs. But accessing archives from other observatories could be difficult – and virtually impossible for amateur astronomers.
Today there are observatories that cover the entire electromagnetic spectrum. These state-of-the-art observatories, which are no longer run by individual institutions, are usually launched by space agencies and are often joint efforts of many countries.
With the dawn of the digital age, almost all data is publicly available shortly after its release. This makes astronomy very democratic – who wants can re-analyze almost every record that makes the news. (You can also look at the Chandra data that has led to the discovery of thousands of black holes!)
These observatories produce an overwhelming amount of data. For example, the Hubble Space Telescope, which has been operating since 1990, has made over 1.3 million observations and transmitted about 20 GB of raw data per week, which is impressive for a telescope that was developed in the 1970s. The Atacama Large Millimeter Array in Chile now expects to bring 2 TB of data daily to its archives.
The archives of astronomical data are already impressively large. But things will explode soon.
Each generation of observatories is usually at least ten times more sensitive than the previous one, either because of improved technology or because the mission is simply larger. Depending on how long a new mission is running, it can detect hundreds of more astronomical sources than previous missions at that wavelength.
Compare, for example, the early EGRET gamma-ray observatory, which flew in the 1990s, with NASA Fermi's flagship mission, which will be 10 years old this year. EGRET only discovered about 190 gamma-ray sources in the sky. Fermi has seen over 5,000.
The Large Synoptic Survey Telescope, an optical telescope currently being built in Chile, will image the entire sky every few nights. It will be so sensitive that it will generate 10 million alerts per night via new or temporary sources, resulting in a catalog of over 15 petabytes after 10 years.
The square kilometer array, completed in 2020, will be the most sensitive telescope in the world and capable of detecting radar stations of alien civilizations up to 50 light-years away. In just one year of activity, it will generate more data than the entire Internet.
These ambitious projects will test the ability of scientists to handle data. Images must be processed automatically. This means that the data must be reduced to a manageable size or converted into a finished product. The new observatories expand computing power and require facilities that can handle hundreds of terabytes per day.
The resulting archives – all publicly searchable – contain 1 million times more information about what can be stored on your typical 1TB backup disk.
Unlock New Science
The flood of data will make astronomy a more collaborative and open science than ever before. Thanks to Internet archives, robust learning communities and new outreach initiatives, citizens can now participate in science. For example, with the Einstein @ Home computer program, each user can use the idle time of their computer to help find gravitational waves from colliding black holes.
It's an exciting time for scientists, too. Astronomers like me often study physical phenomena on time scales that go so far beyond typical human life that real-time observing simply will not happen. Events like a typical galaxy fusion – that's what it sounds like – can take hundreds of millions of years. All we can capture is a snapshot, like a still image from a video of a car accident.
However, there are some phenomena that occur on shorter time scales and only take a few decades, years or even seconds. So scientists in the new study have discovered these thousands of black holes. It is also, as they have recently noticed, that X-ray emission from the center of a nearby dwarf galaxy has diminished since its discovery in the 1990s. These new discoveries indicate that more can be found in archival data that spans decades.
In my own work, I use Hubble archives to make movies of "jets," high-speed plasma ejected in jets of black holes. I used over 400 raw 13-year images to make a movie about the jet in the nearby M87 galaxy. This film showed for the first time the rotational motion of the plasma, suggesting that the beam has a helical structure.
This kind of work was only possible because other observers happened to have taken pictures of the source for other purposes that I was interested in when I was in kindergarten. As astronomical images become larger, higher-resolution and more sensitive, this type of research becomes the norm.
Largest ever published catalog of high-energy gamma-ray sources in the galaxy