GSL/PSL partner with AI startup to make observational data AI-ready

NOAA’s Physical Sciences Laboratory and Global Systems Laboratory and the artificial intelligence startup Brightband have entered into a two-year Cooperative Research And Development Agreement (CRADA) that will optimize a vast NOAA-managed archive of observational weather data for training artificial intelligence (AI)-based weather forecasting applications.

Under the CRADA, “Making NOAA Observation Data Artificial Intelligence-Ready,” NOAA will collaborate with Brightband to transform the NOAA NASA Joint Archive of observational data from satellites, weather balloons and surface stations into an open-source data a repository that will support a suite of geospatial foundation AI models. The joint archive is a collaboration between NOAA and NASA that has developed a homogenized repository of Earth System observations from 1970 to present.

“The homogenized archive represents a great opportunity for AI developers across the weather enterprise,” said NOAA scientist Sergey Frolov, who leads the Physical Sciences Laboratory team involved in the project. “While individual pieces of data are available elsewhere, this will be a one-stop shop that significantly lowers the bar for entry for everyone who would like to leverage AI to improve forecasting."

The engineering initiative will be led by Brightband co-founder and Head of Data and Weather, Daniel Rothenberg, who previously helped to build the Pangeo community and toolkit to handle very large-scale datasets. Brightband was formerly known as OpenEarthAI.

“As more groups work to use machine learning to improve data assimilation and incorporate observations into weather forecasts, having a single, comprehensive and easy-to-use dataset will accelerate research efforts,” said Rothenberg. “Our partnership with NOAA will ensure that the community has access to the best possible dataset to use for this work.”

NOAA scientifically curates one of the most authoritative and valuable collections of weather, climate, ocean and coastal observations and model data in the world. The National Centers for Environmental Information currently maintains over 60 petabytes of environmental data, which is expected to expand to 400 petabytes by 2030. A petabyte is a unit of data storage that represents 1 quadrillion bytes. A petabyte is one million times larger than a gigabyte. In comparison, the largest estimates of the size of training data used to train Chat GPT-4 are 1 petabyte.
NCEI, PSL and NASA developed the archive during a three-year project. Funding support was provided by NASA, NCEI and NOAA’s Weather Program Office.

NOAA is committed to open data, which, combined with leadership in AI technology, promises to help the U.S. become a global leader in AI applications for weather and climate modeling. The NOAA Open Data Dissemination (NODD) program, for example, provides open access to NOAA data in the cloud and supports U.S. advancements in weather and climate modeling. NOAA AI-ready data developed through the Brightband partnership can be made openly accessible through NODD.

Brightband aims to make NOAA’s observational data archive AI-ready by processing data from older, difficult-to-use formats into modern, analysis-ready and cloud-optimized formats. This transformation will enable rapid access to and processing of data in the cloud. The partners hope that this new dataset will be a foundation of data-driven weather forecasting tools.

Visit the NOAA Partnerships Page to learn more about NOAA partnership opportunities, view a list of NOAA’s active CRADAs, and read CRADA success stories.

Note: Any reference obtained from this website to a specific company, product, process, or service does not constitute or imply an endorsement by NOAA.
For more information about this initiative please visit: https://www.brightband.com/blog/partnership-with-n...

Media contact: Theo Stein, NOAA Communications, theo.stein@noaa.gov.

Story via NOAA Research