As businesses continued to increase their involvement in the digital economy, the need for exponentially faster understanding of market changes, customer responses, and system performance has become abundantly clear in 2016. The survey that OpsClarity released today reported that 92% of business and technology professionals employed in both small and large technology, retail, healthcare, finance and public-sector organizations, found remarkable growth in the adoption of real-time, high-velocity data processing platforms to understand and manage the demands of the new marketplace.

Understanding Fast Data

Let’s first discuss how “Fast Data” is different from “Big Data”. Big Data is historical data that requires storage, curation and deep analytics within a tolerable elapsed time. The key term to focus here is “tolerable elapsed time”. Big Data provides offline analytics and the analytics jobs are processed in batch mode, once sufficient volumes of data have been collected and stored.

While historical analysis is important to identify patterns and aid in future decision-making, businesses are realizing that they can leverage multiple streams of real-time data to make real-time decisions and responses.  And this type of real-time analysis is built right into business applications. Applications can now handle massive volumes of data to address the growing needs of digital businesses that are trying to gain competitive leverage by extracting real-time insights from data. Today, the industry is undergoing a monumental shift in the way it develops applications to what is being called “Fast-data” or “Data-first” applications. Fast Data is about “new data” and processing it the instant it is created. Rather than being processed offline, fast data powers business critical applications, helping enterprises to create new business opportunities.

Organizations across industries are doubling down on fast data and stream processing technologies to gain a competitive edge and make intelligent decisions faster. Here are some use-cases:

  • Offer personalized and targeted offers in real-time to improve conversions
  • Detect fraud and prevent intrusions as soon as then occur
  • Process signals from millions of IoT devices to make time decisions
  • Drive efficiencies in ad-tech and high-frequency trading
  • Detecting failures in real-time and alerting sooner. Time-series data is now being leveraged to this effect.

According to the survey, 32 percent of the companies are acting on the need to closely manage customer-facing applications, while 29 percent reported the need to continually optimize key business processes. The remaining companies–39 percent–indicated both needs as driving their shift from historical analytics to streaming applications.

Challenges of adopting Fast Data

But even as these companies are shifting investment from batch processing to stream processing, the survey, sponsored by OpsClarity and conducted in May, found that adoption of advanced analytical capabilities is being confounded by the complexity of the new data frameworks involved.

The survey, which included developers (33%), data architects (26%), DevOps (19%), data scientists (13%), and a small number of sales professionals (3%), found that 92% of their companies plan to increase their investment in streaming data processing technologies.

The increased investment is planned despite serious concern among these professionals in their organizations’ ability to effectively exploit these important technologies. The majority of respondents (68%) said that insufficient experience and the complexity of the various data processing frameworks involved posed a major deterrent to adoption.

survey-challenges-600x462

We found that domain knowledge for the frameworks is already scarce, and that, as multiple data frameworks are stacked together, the complexity of monitoring these frameworks together severely increases.

Respondents also expressed significant concern over the lack of visibility into performance and reliability of the entire data pipeline, noting the tedious and time-consuming configuration needed, and limited dashboards for each framework, with no ability to look at common performance concerns across the entire data pipeline.

You can download the survey here.