Big Data: Answering the Questions that Matter
15 years ago I graduated from Yale with a degree in the History of Art. To be precise, I focused on the history of architecture with my senior thesis examining the mathematics implicit in various forms of Indian temple architecture. This doesn’t seem to lend itself to my current avatar as the leader of a group within The Weather Company focused on Big Data applications of our proprietary weather data sets for Fortune 500 marketers.
However, what’s most important, in my mind, about the application of Big Data within large corporation is not the technology used or the language applied but applying analytical rigor around ensuring that the right question or series of questions are being answered.
Over the last year and a half, The Weather Company has allowed me and the other members of the WeatherFX group to explore the possible universe of questions that marketers could ask through the lens of a very particular data set—the weather.
Over its 30 year history, The Weather Company—the holding corporation for consumer brands such as The Weather Channel, weather.com and Weather Underground, and business-to-business brands such as WSI and Weather Central—has amassed a vast array of data from across the world. It includes variables such as temperature, pressure, humidity, dew point, cloud cover, pollen count and wind speed amongst others.
At first glance, it might not seem that this data is very useful to retailer, CPG manufacturers, insurance companies, pharmaceutical vendors and quick service restaurants, but in fact these are just some of the industry verticals that WeatherFX has created compelling products for, and the teams continues to build out services and technologies for verticals beyond this core group.
The Weather Company has been a large media house for most of its history and when we first went out to the marketing world to discuss ways we could bring our data to bear that would be compelling, we found a very interesting set of challenges awaiting us.
Polling the Clients
Most of our clients in the verticals we thought might be interesting were very clear that they already had access to weather data somewhere within their organization. Then we started asking some of the questions that seemed entirely foundational. Questions like—What is the interface between these weather inputs that you already have access to and your marketing plans? Do you know how weather actually impacts the sale or delivery of your products or services? Are there localized differences in the way people buy or do not buy your products?
More often than not, the response to these questions was a “no” or a “we’re not sure.” To be clear, these were not average marketers, these were some of the best in the business, but to date they had just not spent a lot of time focused on a variable that we were quickly beginning to realize was incredibly impactful to their businesses.
Building the Platform
So we set off to build a platform that we thought could both deliver insight and very practically enable marketers to build sophisticated, cross-channel message delivery programs. It consisted of three primary systems.
A back-end data engine that ingested and cleansed data of all stripes—weather and marketer business data being the key data types. We built this in close partnership with our larger IT group to ensure that the right weather data was being brought in from various data stores across the corporation.
“Some of the best in the business had just not spent a lot of time focused on a variable that we were quickly beginning to realize was incredibly impactful to their businesses”
The next piece was a business rules engine that enabled our platform to integrate rules that the marketer needed to create priority and taxonomy through their messaging options. This was a key component from the client perspective because it truly enabled our system to become an extension of their internal syntax.
Finally, and most easily given that many of my team hailed from an ad technology and media data management background, we built a media delivery system that plugged us into media servers across multiple channels including online display, mobile, social, cable TV, in-store display, out of home display, email and SMS outbound and CRM. In effect it allowed us to create a series of creative templates that would activate based on the output from the data engine and push a message or series of messages from the marketer out into the consumer world.
However, a platform alone is like a single hand trying to clap. In order to truly bring it to life, we needed to find clients that could validate the models we had created. We found those in plenty, in the retail, QSR and CPG spaces to start. By taking their day over day store or restaurant sales data and combining it with the highly localized weather data we had in abundance, we were able to start effectively predicting what consumers were going to buy before they actually bought. More importantly, it turns out that our predictions were actually both highly granular and very accurate.
To some extent, the high degree of granularity and accuracy came as a bit of a surprise. We certainly expected to be successful given the powerful explicit and implicit power of weather but the actual results far outstripped our wildest expectations. Once we knew we had a potentially nuclear-strength tool for marketers, we proceeded to ensure that the platform itself was as bomb-proof as we could make it while ensuring that the service levels around it was equally beyond reproach.
This resulted in a series of extremely deep relationships with marketers where we were actually able to link the utilization of weather triggers to actual product lift. In effect we were able to prove that our platform was able to actually sell more product or service.
To conclude, it appears that one needs luck, access, a great team and the ability to ask the right question, and Big Data can indeed actually answer questions that marketers need answered.
Big Data: Separating the Hype from Reality in Corporate Culture
Maintaining Maximum Relevancy for Buyers and Sellers
Building Levies to Manage Data Flood
Resolving Disassociated Processing of Real-Time and Historical Data in IoT
By Pete V. Sattler, VP-IT & CIO, International Flavors &...
By Benjamin Beberness, CIO, Snohomish County PUD
By Gary Watkins, CIO of IT Shared Services, KAR Auction...
By Tonya Jackson, VP Global Supply Chain, Lexmark
By Chad Lindbloom, CIO, C.H. Robinson
By Ryan Fay, CIO, ACI Specialty Benefits
By Kris Holla, VP& CSO, Nortek, Inc.
By Shawn Wiora, CIO & CISO, Creative Solutions In Healthcare
By Michael Alcock, Director-CIO Executive Programs &...
By Jeff Bauserman, VP-Information Systems & Technology,...
By Wes Wright, CTO, Sutter Health
By Peter Ambs, CIO, City of Albuquerque
By Mark Ziemianski, VP of Business Analytics, Children's...
By Jonathan Alboum, CIO, The United States Department of...
By Ryan Billings, MS, MBA, Executive Director, Digital...
By Christina Clark, Managing Principal, Cresa
By Evan Abrams, Associate, Steptoe & Johnson LLP
By Holly Baumgart, Vice President-Information Technology,...
By Melissa Douros, Director of Digital Product Management,...
By Andrew Palmer, SVP & Chief Information Officer, U.S....