Monitoring Kafka Consumer Group Lag with OpsClarity, Part 3

In this Kafka series we are taking a deep look at Apache Kafka and Consumer Lag monitoring. In Part 1 of this series we discussed the basics about Kafka Consumer Lag. In Part 2, we discussed how some open source alternatives monitor Consumer Lag and their limitations. In this post, we will discuss how OpsClarity monitors [...]

By | November 23rd, 2016|Data Processing Frameworks, Integrations|0 Comments

What have we been working on?

Our Product and Engineering teams here at OpsClarity have been busy for the last couple of months adding some new exciting capabilities as well as bringing about some major improvements with respect to usability, reliability and bug fixes. In this post we will share some of the exciting features we have added along with a [...]

Monitoring Kafka Consumer Lag – Part 2

With any new and fast moving technology stack such as Kafka, monitoring and operational tools are often a step behind or missing significant functionality. But we do have a couple of robust open source projects which are available and can be made to work in specific circumstances. One such tool is Burrow from LinkedIn, written [...]

By | November 18th, 2016|Analytics, Data Processing Frameworks|0 Comments

Leveraging AI and Machine Learning to tease out Seasonality patterns

The standard models, such as SMA, EWMA, etc., fail in the presence of trend or seasonality conditions. We have seen the effect of trend with metrics representing queue size. Many metrics representing important business concerns also exhibit strong seasonal behavior. For example, the number of active users on an e-commerce site shows both daily and [...]

Understanding Kafka Consumer Groups and Consumer Lag, Part 1

In our previous blog we talked about monitoring Kafka as a broker service, looking at ways to think about disk utilization and replication problems.  But the Kafka brokers sit in the middle of an ecosystem, with Kafka producers on one side writing data, and Kafka consumers on the other side reading data.  In this post, [...]

By | October 3rd, 2016|Data Processing Frameworks, Integrations|0 Comments

OpsClarity Adds Support for Native CoreOS

CoreOS is an open-source lightweight operating system based on the Linux kernel and designed for providing infrastructure to clustered deployments, while focusing on automation, ease of application deployment, security, reliability and scalability. As an operating system, CoreOS provides only the minimal functionality required for deploying applications inside software containers, together with built-in mechanisms for service [...]

By | September 28th, 2016|Data Processing Frameworks, Integrations, Uncategorized|0 Comments

Understanding, Operating and Monitoring Apache Kafka

Apache Kafka is an attractive service because it’s conceptually simple and powerful. It’s easy to understand writing messages to a log in one place, then reading messages from that log in another place. This simplicity not only allows for a nice separation of concerns, but also is relatively easier to understand than more complex alternatives. [...]

By | July 19th, 2016|Data Processing Frameworks, Integrations|0 Comments

Monitoring Apache Spark Streaming: Understanding Key Metrics

This post is part 2 of the Monitoring Apache Spark series. In part 1 - Monitoring Apache Spark : Why is it Challenging?, we discussed the basics of Spark architecture and why it is challenging to monitor a distributed and dynamic Spark cluster. In this post, we will dig deeper into the specific Apache Spark metrics [...]

Monitoring Apache Spark: Why is it Challenging?

Large-scale data processing has moved into a new age of sophistication and the greatest evidence is the increasing requirement for processing data in real time. The drivers for this move are many, but the basic impetus can be understood by simply recalling the classic Time-Value-of-Data concept and how the value of data is at its [...]

Riak to Elasticsearch: How we scaled our monitoring backend at OpsClarity

At OpsClarity we have developed an advanced monitoring and troubleshooting for modern stream processing applications. OpsClarity provides deep visibility of key concerns across a data pipeline and enables quick mean-time-to-resolution (MTTR) for the often hard-pressed operations and engineering teams. The OpsClarity platform itself is composed of several data processing pipelines and it ingests huge amount [...]

By | May 26th, 2016|Analytics, Data Processing Frameworks, Engineers|0 Comments