This data revolution has created two challenges for astronomers.
Firstly, as the capability of digital detector systems has advanced, the volume of digital data that astronomical facilities are producing has expanded greatly. The rate of growth of the volume of stored data far exceeds the rate of increase in the performance of computer systems or storage devices.
Secondly, astronomers have realised that many important insights into the deepest secrets in the Universe can come from combining information obtained at many wavelengths into a consistent and comprehensive physical picture. However, because the datasets from different parts of the spectrum come from different observatories using different instruments, the data are not easily combined. To unite data from different observatories, bridges must be built between digital archives to allow them to share data and 'interoperate' - an important and challenging task.
The human factor
These challenges are not only technological. Our brains are not equipped to, for instance, analyse simultaneously the millions and millions of images available. Astronomers must adapt and learn to deal with such diverse and extensive sets of data.
The 'digital sky' has the potential to become a vital tool with novel and fascinating capabilities that are essential for astronomers to make progress in their understanding of the Cosmos. But astronomers must be able to find the relevant information quickly and efficiently. Currently the data needed by a particular research program may well be stored in the archives already, but the tools and methods have not yet been developed to extract the relevant information from the flood of images available.
A new way of thinking, a new frame of mind and a new approach are needed.
The Astrophysical Virtual Observatory
The Astrophysical Virtual Observatory will allow astronomers to overcome the challenges and extract data from the digital sky, thus 'putting the Universe online'. Like a search engine helps us to find information on the Internet, astronomers need sophisticated 'search engines' as well as other tools to find and interpret the information.
"We're drowning in information and starving for knowledge" a Yale University librarian once said. Or to paraphrase a popular series on TV: "The information is out there, but you have to find it."
Using the latest in computer technology, data storage and analysis techniques, AVO will maximise the potential for new scientific insights from the stored data by making them available in a readily accessible and seamlessly unified form to professional researchers, amateur astronomers and students.
Users of AVO will have immense multi-wavelength vistas of the digital Universe at their fingertips and the potential to make breathtaking new discoveries. Virtual observatories signal a new era, where data collected by a multitude of sophisticated telescopes can be used globally and repeatedly to achieve substantial progress in the quest for knowledge.
The AVO project, funded by the European Commission, is a three-year study of the design and implementation of a virtual observatory for European astronomy. A virtual observatory is a collection of connected data archives and software tools that utilise the Internet to form a scientific research environment in which new multi-wavelength astronomical research programs can be conducted. In much the same way as a real observatory consists of telescopes, each with a collection of unique astronomical instruments, the virtual observatory consists of a collection of data centres each with unique collections of astronomical data, software systems and processing capabilities.
The programme will implement and test a prototype virtual observatory, focussing on the key areas of scientific requirements, interoperability and new technologies such as the GRID, needed to link powerful computers to the newly formed large data repositories.
The GRID and the future of the internet
The technical problems astronomers have to solve are similar to those being worked on by particle physicists, by biologists, and by commercial companies who want to search and fill customer databases across the world. The emerging idea is that of the GRID where computers collaborate across the internet. The World Wide Web made words and pictures available to anybody at the click of a mouse. The GRID will do the same for data, and for computer processing power. Anybody can have the power of a supercomputer sitting on their desktop. The Astrophysical Virtual Observatory, and GRID projects like the ASTROGRID project in the United Kingdom (funding 8 million Euro), are closely linked to these developments.