
Four Phases of Operationalizing Big Data


George Demarest, director solutions marketing, MapR Technologies, George Demarest is a technology industry veteran with stints as a software engineer, technical marketer and marketing strategist for companies such as... More >>
IT organizations around the world are actively wrestling with the practical challenges of creating a big data program. The tantalizing combination of advanced analytics, a wide variety of interesting new data sets, an attractive cost model, and a proven scientific rigor put big data on pretty firm footing as an investment target for CIOs. But because of the rapid innovation cycle, the dizzying variety of new tools, and the relative newness of many of the key concepts (e.g., micro services, containerization, schemaless database operations, and data science), some IT organizations are finding it difficult to know where to begin—and perhaps most importantly—how to progress and avoid having their big data ambitions derailed.
This article provides IT leaders and enterprise architects some guidance as to what to plan for and what to expect as they progress through four phases of big data adoption.
• Phase 1. Experimentation — Understanding the capabilities of a big data platform
• Phase 2. Implementation — Developing first production use cases
• Phase 3. Expansion — Expanding to multiple use cases across the company
• Phase 4. Optimization — Optimize and integrate apps on the converged data platform
These phases represent discernable patterns that we have observed from hundreds of engagements with our production customers.
The goal is to be able to be predictive about most aspects of the business, and be able to respond and change operations in real time
Experimentation: Hadoop/Spark in Enterprise Data Architecture
In the first phase of the big data journey, companies are exploring how the Hadoop/Spark ecosystem works and how it can fit in with their existing enterprise data architecture. Those organizations that are trying to tackle the scale and richness of this new data with standard tools (RDBMS, enterprise middleware, premium storage arrays, and data warehouses) find that traditional IT economics don’t make sense beyond a certain threshold.
Initially, IT organizations are motivated by two factors. First is the IT preparedness. IT may or may not always completely drive or control big data adoption, but they do control their own readiness to implement big data when they are called upon. And the second factor is cost takeout. The combination of low cost commodity servers with cheap direct attached storage and free open source software provides an enticing reason to explore big data software.
IT-driven big data program will initiate cost takeout use cases like a Data Warehouse offload, where cool or cold data is stored in Hadoop. This can reduce costs in premium storage, server hardware, and database licenses. Other targets include mainframe offload, lowering dependence on premium storage arrays, and even replatforming RDBMS applications.
In terms of what actions can be taken, start by identifying a team or creating the beginning of a big data Center Of Excellence (COE) that would include distinguished staffers from IT Ops (for cluster admin and data management strategies), the BI/DW/Analytics team, and development. Engage in technology experimentation including Proof of Concepts (POCs), competitive bake-offs between vendors and approaches such as Hadoop/Spark vendors, NoSQL databases, and commercial vs. open source analytics.
Implementation: Creating a Lighthouse Win
In the implementation phase, companies are ready to get started in creating production use cases. They should be confident in their ability to build a solution on this platform. A key step is creating a professional big data discipline that moves beyond the proof of concept or value stage, and big data proponents must take steps to establish key platform components (like Spark, Hadoop, Elastic search, etc,) as permanent parts of their technology portfolio.
The most common production use cases in Phase 1 are an enterprise data lake or data hub that is used as a basis for a broad spectrum of analytics. Other common use cases include analytics use cases that move closer to the business. This could include security log analytics, fraud detection, and sales or marketing use cases such as creating a 360-degree view of the customer, reducing customer churn, and building a recommendation engine.
The implementation phase is all about creating a Lighthouse Win. The big data program team must drive a small number of use cases into production that deliver a demonstrable benefit to the business. These early use cases will set the tone for the entire program going forward. Looking ahead, we work on to expand use cases through an ideation process.
Expansion: Acceleration of Big Data Program
The expansion phase is where your big data program accelerates at a rapid rate. Success with early use cases demonstrates to others in the company about what can be accomplished. Implementing new applications will be a little easier since the platform is in place. However, if the use cases cross organizational boundaries, then multi-tenancy, security, and data governance will become a concern.
Now that your organization has a firm handle on the tools, data sources, formats, and algorithms, dozens if not hundreds of use cases become reasonable investment targets. With the expanded understanding of the big data ecosystem, the focus on use cases often turns to the customer’s vertical industry. For financial services, it may be risk management, fraud detection, and large scale self-service analytics. For healthcare, the focus may be on reducing readmission, population health management, and coordinating smart medical devices.
Organizations are now focused on industrializing their big data capability and making it relevant to all lines of business. That means establishing a dedicated IT or Dev staff, a dedicated cluster, and budgeting, staffing, training, and big data teams in central IT, BI/DW, enterprise development, and LOB leads.
Optimization: Reshape Business with the Benefits of Big Data
In this final phase, there is critical mass in the organization and all LOBs are seeing the benefits of big data. You should leverage this capability to improve business and operational processes, which will reshape the business and give you a competitive edge. The goal is to be able to be predictive about most aspects of the business, and be able to respond and change operations in real time across more than one line of business.
At this point, you may have created a Chief Data Officer (CDO) role and have hired data scientists and data engineers to look for problems that no one has solved yet. There is a deep investment and commitment to data science across all lines of business. Any actionable insights that are created have the potential to be automated. All key decision makers should have real-time visibility into the performance of their operations.
Admittedly, the number of organizations at this advanced level is relatively small. However, companies that are operating at this level are disrupting their industries and changing our notion of how responsive and adaptable a modern enterprise can be. Like any important discipline, it is important to observe the most advanced players to help you up your own game, and to successfully progress through the four phases of big data adoption.
ON THE DECK
Featured Vendors
Next Level Business Services (NLB): Applying Digital Transformation to Create Supply & Service Value Chains of the Future
Gerber Technology: Reshaping the Dynamics of the Fashion & Apparel and Flexible Materials Industries
FileFacets: A One-stop Solution for Locating and Identifying Data Across the Enterprise" title="Jennifer Nelson, VP, Sales & Marketing" style="float:left; margin-right:10px; margin-bottom:20px;" width="60px" height="50px">
FileFacets: A One-stop Solution for Locating and Identifying Data Across the Enterprise
Infoworks: Dynamic Data Warehousing on Hadoop that Automatically Ingests and Organizes Enterprise Data for All Use-cases
ThetaRay: Advanced Data Analytics Provide an Enhanced Security Layer to Combat Bank Fraud and Cybercrime
VentureSoft Global: Robust Big Data Solutions for Customer, Product Profitability and Operational Efficiency
Absolut-e Data Com BizStats – Leveraging Artificial Intelligence To Extract The True Potential Of Data
Relational Solutions, Inc.: Delivers Enterprise Demand Signal Repositories to the Consumer Goods Ind
Emagine International: Adaptive Contextual Marketing Platform for Personalized Customer Interactions
Cygnus Professionals: Translate Big Data into Actions: An Analytics Platform Transforming Enterprise
EDITOR'S PICK
Essential Technology Elements Necessary To Enable...
By Leni Kaufman, VP & CIO, Newport News Shipbuilding
Comparative Data Among Physician Peers
By George Evans, CIO, Singing River Health System
Monitoring Technologies Without Human Intervention
By John Kamin, EVP and CIO, Old National Bancorp
Unlocking the Value of Connected Cars
By Elliot Garbus, VP-IoT Solutions Group & GM-Automotive...
Digital Innovation Giving Rise to New Capabilities
By Gregory Morrison, SVP & CIO, Cox Enterprises
Staying Connected to Organizational Priorities is Vital...
By Alberto Ruocco, CIO, American Electric Power
Comprehensible Distribution of Training and Information...
By Sam Lamonica, CIO & VP Information Systems, Rosendin...
The Current Focus is On Comprehensive Solutions
By Sergey Cherkasov, CIO, PhosAgro
Big Data Analytics and Its Impact on the Supply Chain
By Pascal Becotte, MD-Global Supply Chain Practice for the...
Technology's Impact on Field Services
By Stephen Caulfield, Executive Director, Global Field...
Carmax, the Automobile Business with IT at the Core
By Shamim Mohammad, SVP & CIO, CarMax
The CIO's role in rethinking the scope of EPM for...
By Ronald Seymore, Managing Director, Enterprise Performance...
Driving Insurance Agent Productivity with Mobile and Big...
By Brad Bodell, SVP and CIO, CNO Financial Group, Inc.
Transformative Impact On The IT Landscape
By Jim Whitehurst, CEO, Red Hat
Get Ready for an IT Renaissance: Brought to You by Big...
By Clark Golestani, EVP and CIO, Merck
Four Initiatives Driving ECM Innovation
By Scott Craig, Vice President of Product Marketing, Lexmark...
Technology to Leverage and Enable
By Dave Kipe, SVP, Global Operations, Scholastic Inc.
By Meerah Rajavel, CIO, Forcepoint
AI is the New UI-AI + UX + DesignOps
By Amit Bahree, Executive, Global Technology and Innovation,...
Evolving Role of the CIO - Enabling Business Execution...
By Greg Tacchetti, CIO, State Auto Insurance
Read Also
The Under- And Overestimation Of Data.
Protecting IP Amidst The Pandemic
AI Will Improve Talent Management Practice But Change Management Is...
Keys To Reaching The Peak Of A Cyber Security Program Journey
Sailing through Uncertain COVID times
Paving the Way for Incentivized, Ethical Waste Collection
