6sense Case Study jonathan
6sense, a B2B predictive intelligence platform for sales and marketing, uses OpsClarity to monitor and troubleshoot their complex, distributed data processing pipeline. 6sense leverages OpsClarity to collect metrics on applications and infrastructure that process terabytes of data from sources encompassing Hadoop, HBase, Mesos, Presto and MySQL.
6sense is a B2B predictive intelligence platform for marketing and sales. Using its private network of billions of time-sensitive intent interactions, 6sense uncovers net-new prospects at every stage of the funnel and determines which existing prospects are in market to buy. 6sense predicts what products prospects will buy, how much they will buy, and when.
Single tool to monitor and troubleshoot across servers, services, AWS and custom applications
Automated health tracking and a rich set of auto-configured statistical monitors across apps and infra
Full-stack monitoring across data frameworks, custom applications and infrastructure
No tedious or manual processes required to configure OpsClarity
Auto-configured metric collection, health check and statistical monitors based on deep domain expertise for each component
Powerful correlated troubleshooting across application and data processing pipeline
“As 6sense has continued to witness breakthrough growth and momentum, our data processing demands have increased exponentially. OpsClarity provides us with the ability to monitor and troubleshoot our complex, distributed data processing pipeline with unmatched visibility. Since our business relies on the health of our data pipeline, OpsClarity has become an integral part of our operation.“
Challenge: Configuring and collecting vast amounts of metrics across their data-first applications to provide system-wide visibility
6sense built its application in the new “data-first” mode, creating a complex deployment of Hadoop, HBase, Presto, Mesos and MySQL. “As part of our data processing pipeline, we run hundreds of Hive and Presto queries everyday that go over terabytes of data stored in HDFS and S3 and must consistently meet our SLA demands,” said 6sense Co-Founder Premal Shah.
Given the complexity and distributed nature of the applications and infrastructure, 6sense had previously used a combination of open source and commercial tools to monitor their servers, AWS infrastructure and distributed data clusters. Even after spending considerable time configuring each of these tools, Premal’s team was still not able to get unified visibility and troubleshooting across this distributed infrastructure.
“Once we deployed OpsClarity agents across our infrastructure using an automated script, the next thing we knew it had discovered all the services, data frameworks, application code and infrastructure without any headache.“
Solution: Full-stack monitoring and troubleshooting across their data pipeline, applications and infrastructure
Having looked at the existing alternatives, 6sense decided to use OpsClarity. Within a day of deploying the OpsClarity agents, OpsClarity auto-discovered all the applications and frameworks and their inter-dependencies and configured metric collection from each component. The 6sense team did not have to manually configure any collection or specify which metrics to collect. Moreover, they could use the OpsClarity API to push custom metrics from their applications.
OpsClarity was able to automatically discover and configure their Hadoop, HBase, Presto and other data frameworks. It was essential for 6sense to understand the bottlenecks in their data pipeline and address them as quickly as possible given that this data infrastructure was increasingly more important for 6sense to deliver competitive differentiation. OpsClarity not only automatically discovered these components, but as the 6sense infrastructure scaled up or down, the topology view also updated in real-time. This was specifically important for Premal given the dynamic nature of their infrastructure.
OpsClarity’s intelligent monitoring solution provides unified visibility, proactive alerts and rapid troubleshooting for distributed, complex data intensive applications:
Unified visibility across data frameworks, applications and infrastructure – see everything in one powerful topology map-like view
Intelligent health tracking: View and track synthesized health for every data, service & infra component.
Rapid Troubleshooting: Guided focus to most likely affected service followed by analysis of automatically curated graphs and dashboards
Benefits: Automatically configure and collect vast amount of metrics from all the application and data frameworks along with correlated and rapid troubleshooting
Deploying OpsClarity agents across their infrastructure gave 6sense the critical visibility they needed across their distributed data frameworks, applications and AWS infrastructure. OpsClarity not only provided better visibility and faster troubleshooting, but did so without having Premal’s team spend months configuring the tools which they would have to do if they plan to stick to them. In contrast, the other monitoring alternatives 6sense considered would have required the team to spend hundreds of hours collecting the metrics and configuring the metrics and dashboards.
“As part of our data processing pipeline, we run hundreds of Hive and Presto queries everyday that go over terabytes of data stored in HDFS and S3 and must consistently meet our SLA demands. The query load is augmented by engineers and data scientists running ad-hoc analysis. In addition, we run a Mesos cluster with Chronos, Marathon and custom Mesos frameworks”, says Premal. “We looked at three other tools, but only OpsClarity provided visibility across our data framework rich application stack which included Hadoop, HBase, Presto, Mesos, MySQL and custom application code”
OpsClarity automatically configured the specific plugins required to collect detailed metrics from each of the data frameworks, schedulers, infrastructure and custom applications. OpsClarity automatically configured port availability checks, anomaly detection and statistical monitors that were specifically tailored for the each of components based on the deep domain knowledge that OpsClarity had developed. This shaved off months of configuration time typically required to identify the specific metrics to collect for each component, collect the metrics, plot them in graphs and then configure custom statistical monitors against them. “OpsClarity essentially did all of this in a day’s time,” says Premal.
The most important aspect of OpsClarity is its correlated troubleshooting across the data frameworks, application code and infrastructure. OpsClarity builds aggregated health models for all services that are overlaid on an always up-to-date application topology, enabling 6sense to get a pulse on the problem areas in their dynamic infrastructures at a glance. The event log captures relevant events from across the entire stack, such as code deployments, health status changes, data pipeline concerns, anomalies, metric violations, etc. The event log in combination with application topology and synthesized health status makes troubleshooting process faster.
Accelerate performance of your streaming applications.
Monitoring stream processing applications is hard, but it gets easier when you have a map.