DESI gets to know its users: GSL performs the first comprehensive evaluation of powerful analysis tool

Researchers at the NOAA Global Systems Laboratory (GSL) have completed the first comprehensive evaluation of one of GSL’s flagship decision-support tools. The Dynamic Ensemble-based Scenarios for Impact-based Decision-Support Services (DESI) is a web-based application that allows meteorologists and other users to explore ensemble weather model data and extract probabilistic information about weather forecasts. Whether you need to know the probability of getting four inches of snow in the next six hours or the maximum potential wind gusts for next Saturday, DESI has the data.

While much effort has been put into the technical performance of DESI, until recently, the users themselves–their decision spaces, how they use weather information from DESI to develop and communicate forecasts, and how well DESI was meeting their weather data and decision-support needs–had not been rigorously studied. To address this, GSL launched a project in 2024 to evaluate user perceptions of DESI by methodically engaging forecasters who utilize it. In all, GSL researchers, using social and behavioral science principles, conducted 45 semi-structured interviews, primarily with National Weather Service (NWS) meteorologists and hydrologists. These interviews comprised over 40 hours of discussions about when, where, why, and how DESI and probabilistic forecast information are used by forecasters. 

Through this project, GSL was able to meaningfully document decision workflows of NWS meteorologists in extreme and non-extreme weather situations, informing a broader understanding of their operational contexts that can be applied and built upon in future research and evaluation studies. They identified new ways to provide DESI training to forecasters and examined how DESI influences some of the forecasters’ hardest decisions. The researchers learned how DESI informs meteorologists’ communication of the forecast to their partners and the public while also documenting future development ideas that are consistent with the vision and needs of our end users.

Among other findings, DESI increased the self-reported confidence of forecasters by providing an efficient and centralized platform that integrates seamlessly into forecaster workflows. This helps reduce their mental load and stress, resulting in more efficient inspection of weather variables and model insights. 

The probability of a weather variable meeting a specific threshold-–often called the probability of exceedance–was the most commonly used type of probabilistic information for data interrogation and partner communication among interviewed meteorologists when compared to use of percentiles and ranges. The cluster analysis function within DESI was also reported by a significant portion of participants as being useful for them in creating scenario-based messaging. Users emphasized the need for even greater development of DESI’s communication, graphic creation, and decision-support capabilities to enable them to continue meeting the growing demand for their partners’ decision-support needs. 

Overall, users expressed excitement and a strong sense of optimism for the future of DESI. One study participant summed up their feelings toward DESI, saying, “It is the most powerful up-and-coming tool I think that I’m using right now, and I absolutely love it.” 

Another echoed this tone, adding, “You’re moving into the future.” 

Findings from this evaluation are already directly informing new releases, ensuring DESI continues to evolve and meet the growing needs of its users, and in 2025, results and recommendations were delivered to the NWS to assist in the planning of DESI’s eventual transition to nationwide operations.