Relational Database Systems have been powerful and useful instruments for almost every organization for many years. However, a problem arises when older technologies like Relational Database Management Systems are faced with the volumes of unstructured data that exist today. The ‘new’ industry has dubbed this concept “Big Data”—aggregates of data measured in terabytes / petabytes.
The term “Big Data” originated from within the open-source community, where there was an effort to develop analytic processes that would be faster and more scalable than traditional data warehousing. The goal was to extract value from the vast amounts of unstructured data produced daily by web users (e.g. Google searches).
Data enables us to understand customers and consumers. It also enables us to manage content and contact strategy. Data is a core component of integrated marketing, giving us the ability to speak with a single voice across all channels of business. Managing all of that information in a holistic manner that feeds customer engagement and experience is the challenge.
Even the most sophisticated and modern businesses are surprisingly ill-equipped to manage the most basic digital marketing standards and activities, let alone make the leap forward into the never-ending universe of Big Data techniques.
That effort requires a whole ecosystem of people, processes and technologies.
Big Data analytics is now more emergent and multifaceted, but less understood by the IT generalist. Development of Big Data analytic processes has been driven historically by the web. More recently, applications for Big Data analytics have seen rapid growth: all major vertical industry segments use some form of analytics. Big Data now represents a growth opportunity to vendors that’s worth all the hype.
John Webster (Evaluator Group) has written that Big Data analytics is an area of competency that is very dynamic and diverse. Trying to define it specifically is probably not helpful. But, identifying the traits that are common to the technologies currently identified with Big Data analytics is helpful:
- Traditional data warehousing processes are too slow and limited in scalability
- Converging data from multiple data sources, both structured and unstructured, is a source of opportunity
- Time-to-information is a critical variable in extracting value from data sources (e.g. mobile devices, RFID, the web, automated sensory technologies)
Traditional data warehousing is a large but relatively slow producer of information to business analytics users. It draws from limited data resources and depends on reiterative extract processes. Firms are now looking for quick access to information that is based on extracting data from multiple sources simultaneously. Big Data analytics may often be defined in relationship to the need to gather and interpret large data sets from multiple sources, and to do that in real-time. It is not dissimilar form Agile BI in this regard.
Webster also goes on to say that Big Data analytics represents a big opportunity for IT and consulting organizations that are exploring these concepts to extract value from the boom in social networking.
Types of organizations / industries standing to gain the most out of Big Data analytics include:
- Supply Chain, Logistics, and Manufacturing — With RFID sensors, handheld scanners, and on-board GPS vehicle and shipment tracking, logistics and manufacturing operations produce vast quantities of information offering significant insight into route optimization, cost savings and operational efficiency (as exemplified by Wal Mart)
- Online Services and Web Analytics — Internet companies invented Big Data specifically to handle processing information at Internet scale; Implementation of these analytical platforms is now viable for smaller online services companies to provide an edge over competitors for advertising, customer intelligence, capacity planning and more
- Financial Services — Financial markets generate immense quantities of stock market and banking transaction data that can help companies maximize trading opportunities or identify potentially fraudulent charges, amongst various other uses. New regulations also require detailed financial records to be maintained for longer periods
- Energy and Utilities — Smart instrumentation such as ‘smart grids’ and electronic sensors attached to machinery, oil pipelines and equipment generate streams of incoming data that must be stored and analyzed quickly to uncover and fix potential problems before they result in costly or disastrous failures
- Media and Telecommunications — Streaming media, smartphones, tablets, browsing behavior and text messages are captured at ever-increasing rates all over the world, representing a potential treasure trove of knowledge about user behavior and tastes
- Healthcare and Life Sciences — Electronic medical records systems are some of the most data-intensive systems in the world ; making sense of all this data to provide patient treatment options and analyze data for clinical studies can have a dramatic effect for both individual patients and public health management / policy
- Retail and Consumer Products — Retailers can analyze vast quantities of sales transaction data to unearth patterns in user behavior and monitor brand awareness and sentiment with social networking data
To apply this new technology effectively, it is important to understand when and how to integrate Big Data with the other components of the data warehouse environment. In a vast majority of cases, Big Data does not replace the data warehouse. Database management systems are therefore still a vital part of the overall solution architecture. The real question companies need to be asking is how will Big Data analytics be incorporated with existing BI/DW investments to vastly improve the bottom line?