The 2016 AMS Annual Meeting. Big Data.

Every year, I look forward to coming to the American Meteorological Society Annual Meeting. Some of the motivation remains the same year to year. But there’s always something additional, something new, something that makes the blood sing.

This year? For me personally? It’s the encounter between the AMS community and Big Data. Both have been around a while. The AMS traces its history back almost a century. By that standard, Big Data is a relative newcomer, but you’ve been seeing the impact for years on your computer… for example, in those ads on your Facebook page tailored to you (in my case, Rockport shoes, Viking cruises, and my stepson’s bicycle-wheel business). The Big Data firms have tamed high-velocity, high-volume, high-variety, variable-veracity data sets and are making them sing.

Much of the music so far has been marketing-oriented (that Facebook page providing a vivid demonstration). But the possibilities are expanding rapidly, especially now that all agencies of government are making a serious effort to migrate their data to the cloud. Data on the Earth, the oceans, and the atmosphere are part of this trend.

Good news for the Earth and all its inhabitants! We face critical problems with respect to natural resources – water, food, and energy, for example – and all the more as we attempt to maintain resource levels while building resilience to natural extremes and minimizing environmental degradation. A tough challenge, and getting more complex and contentious as margins shrink and the stakes grow year by year. But tomorrow’s problems only look impossible if we try to solve them using yesterday’s tools[1]. Big data promises to be the most (positive) disruptive player in this arena since the development of observing instruments and platforms of high diagnostic power, and since the early application of computers to modeling elements of the Earth system of the last century.

A specific example comes to us from amazon web services. Some excerpts from one of their recent posts (refer to this original link to access the full set of hyperlinks in the text below):

“In October, we announced that the real-time feed and full historical archive of original resolution (Level II) NEXRAD data is freely available on Amazon Simple Storage Service (Amazon S3) for anyone to use. The Next Generation Weather Radar (NEXRAD) is a network of 160 high-resolution Doppler radar sites that enables severe storm prediction and is used by researchers and commercial enterprises to study and address the impact of weather across multiple sectors.

Early adopters have used the data to cut their product development time and ask new questions about weather-related phenomena. Today, we’re excited to share two new tools that make it even easier for you to analyze NEXRAD data and incorporate it into your workflows…

…WeatherPipe marshals the NEXRAD data into usable data structures and runs the job in Amazon Elastic MapReduce (EMR). The output is a NetCDF file that you can display in Unidata’s Integrated Data Viewer (IDV) and other visualization tools…

…One of the top requests from early users was for an easier way to incorporate the NEXRAD data into event-driven workflows. Today, we’re excited to announce that notifications are now available for both types of data.

We have set up public Amazon Simple Notification Service (SNS) topics for the “chunks” and archive data that create a notification for every new object added to the Amazon S3 buckets. To start, you can subscribe to these notifications using Amazon Simple Queue Service (SQS) and AWS Lambda. This means you can automatically add new real-time and near-real-time NEXRAD data into a queue or trigger event-based processing if the data meets certain criteria, such as geographic location…

…Visit our NEXRAD on AWS page for information on subscribing to these SNS topics and incorporating them into workflows. We’re excited to see what you do with this new capability!

These are just two of the early outcomes, from just one company, of the several CRADA’s NOAA has established with Big Data. A flood of such advances is coming[2], enabled by the simple switch from the old method of sending data to analytical tools to the new method of sending analytical tools to the data, and other opportunities Big Data provides – in days or weeks solving technical problems that have vexed Earth scientists for decades.

Which brings us to this year’s AMS Annual Meeting, and perhaps the next 3-4, running out to our 100^th birthday meeting in 2020, in Boston (where it all began). In 5-10 years, these approaches to resource-, hazard-, and environmental problems will have become old hat. A new generation of scientists and practitioners won’t remember any other way of doing business. But for now, Big Data firms don’t yet fully understand which of their many capabilities will be good for what purposes – what will be most useful, and how, and why? And for now Earth scientists don’t comprehend what new tools are now or soon will be at their disposal. AMS Annual Meetings can serve as a venue for these discoveries to take place. Think of the AMS Annual Meetings as a speed-dating arena where Big Data firms and professionals – especially early-career professionals – can connect.

We’ll all be seeing the first hint of this in New Orleans over the next seven days. A few examples: Big Data will be a major theme of Saturday’s NWS International Workshop. Ariel Gold and others from amazon web services will be at the student conference reception Saturday night. Commerce Secretary Penny Pritzker will be discussing this among other topics at her Monday noontime Town Hall in La Nouvelle C. Panelists will discuss the NOAA Big Data CRADAs on Tuesday afternoon from 1:30-3:00 in Room 342 of the Conference Center (for example, you can hear more about the NEXRAD work there).

Speaking of Secretary Pritzker, recall that Big Data’s capabilities emerge most powerfully when applied to diverse data sets. Imagine a new ability to merge NOAA data on hazard threats with Department of Commerce Census demographic data: the location of populations, especially populations of the vulnerable: the poor, the ethnic minorities, senior citizens and children. Envision being able to combine that information with data on the built environment from NIST, HUD, and other sources, and to anticipate special needs of those in harm’s way hours or days in advance. Think of what it will mean to put that assimilated information in the hands of emergency responders and social media. This is the best time in world history to be alive – except for tomorrow.

Expect to see this story line and other similar narratives unfolding at AMS in Seattle in 2017, in Austin in 2018, in Phoenix in 2019…

________________________________

[1] As explained in more detail in the 2014 book, Living on the Real World: How Thinking and Acting Like Meteorologists Will Help Save the Planet, American Meteorological Society

[2]one that will be a news story long after the current Mississippi flood here to greet AMS Meeting participants has been forgotten.