Community tools and frameworks for geoscientific analysis and visualization: NCL, PyNGL, and PyNIO
The NCAR Command Language (NCL) is a data analysis and visualization environment developed primarily at NCAR/CISL with deep collaboration with a number of core scientific groups. NCL enables scientists to easily and effectively read, analyze, and visualize their geoscientific data on platforms ranging from personal systems to high performance computers. PyNGL and PyNIO are software components that provide Python interfaces to NCL's file input/output and visualization capabilities, thereby enabling NCL's software frameworks to be exposed to a wider and more mainstream user base. The NCL and Python tools have been embraced broadly across an international Earth system sciences community spanning research, education, military, government, and commercial organizations. They are used by thousands of people in 74 countries.
Our efforts here support the NCAR strategic priorites of "Exploring atmospheric, Earth System, and solar processes, variability, and change" along with "Developing and providing advanced tools and services," and "Creating an Earth system knowledge environment."
A top priority for FY2007 was to add support for more scientific data formats. We accomplished this by releasing versions of NCL and PyNIO that:
- Read GRIB2, a second-generation data format standard developed by the World Meteorological Organization for distributing gridded data
- Read/write classic netCDF-4 files, enabling users to access significant data compression capabilities
- Provide better support for HDF-EOS swath and grid data
A second top priority was to collaborate with WRF scientists and software engineers to significantly upgrade the NCL post-processing capabilities for the rapidly growing WRF community. This effort resulted in several analysis and visualization functions being incorporated into NCL, specific to analyzing WRF-ARW data; the August 2007 release of NCL contained these new capabilities.
Finally, our Python efforts were ramped up in FY2007. We released two versions containing several new analysis and visualization functions, new examples, and support for a widely popular array processing package. We also continued to explore options for integrating exploratory and quantitative 3D capabilities via the Python framework.
Future plans for NCL and Python software development and support are largely based on continuous dialogs with the scientific community and on results conducted from a survey in FY2006. Goals for FY2008 include:
- Release NCL, PyNGL, and PyNIO under an open source license
- Research the scalability of NCL to address issues pertaining to large files, data aggregation, and 64-bit machine addressing
- Research the display model on which the visualiation tools are built and address future enhancements required
- Add full support for more data formats
- Integrate interfaces for generating vectors and streamlines on non-uniform grids to the NCL/PyNGL level
- Improve NCL's suite of interpolation routines by adding an interface to SCRIP
- Enhance the Python suite of analysis functions and continue to investigate the integration of 3D tools
- Continue providing high-quality consulting support and training services for the user community
The integration of NCL and its Python counterparts continues to be a core activity of our Earth System Grid Center for Enabling Technologies (ESG-CET) project. ESG will provide support for the integration of our tools as core client applications of ESG, and potentially as server-side capabilities as well.
Our development and deployment of community tools and frameworks for geoscientific analysis and visualization is supported by NSF Core funding.