Monitoring Kafka Consumer Group Lag with OpsClarity, Part 3

In this Kafka series we are taking a deep look at Apache Kafka and Consumer Lag monitoring. In Part 1 of this series we discussed the basics about Kafka Consumer Lag. In Part 2, we discussed how some open source alternatives monitor Consumer Lag and their limitations. In this post, we will discuss how OpsClarity monitors [...]

By | November 23rd, 2016|Data Processing Frameworks, Integrations|0 Comments

Understanding Kafka Consumer Groups and Consumer Lag, Part 1

In our previous blog we talked about monitoring Kafka as a broker service, looking at ways to think about disk utilization and replication problems.  But the Kafka brokers sit in the middle of an ecosystem, with Kafka producers on one side writing data, and Kafka consumers on the other side reading data.  In this post, [...]

By | October 3rd, 2016|Data Processing Frameworks, Integrations|0 Comments

OpsClarity Adds Support for Native CoreOS

CoreOS is an open-source lightweight operating system based on the Linux kernel and designed for providing infrastructure to clustered deployments, while focusing on automation, ease of application deployment, security, reliability and scalability. As an operating system, CoreOS provides only the minimal functionality required for deploying applications inside software containers, together with built-in mechanisms for service [...]

By | September 28th, 2016|Data Processing Frameworks, Integrations, Uncategorized|0 Comments

Monitoring Docker, Part Two – The science of auto-discovery advances container monitoring

This is Part Two of our 2-part blog post on the challenges of monitoring containerized applications. In Part One we discussed basic requirements, considerations and components involved in a production-ready Docker setup. We touched upon the ephemeral nature of containers, the importance of collecting application and container metrics, and the ever increasing cost of configuration [...]

By | August 27th, 2016|Analytics, Integrations|0 Comments

Monitoring Docker Containers: Getting smart about tracking ephemeral infrastructure

This is part 1 of our Docker Monitoring blog series. In this part we will discuss the basics about Docker and container monitoring. In the next part, we will discuss how OpsClarity Intelligent Monitoring platform automatically tracks Docker containers and monitors the services inside them, even as they shift and the Docker images change, without [...]

Understanding, Operating and Monitoring Apache Kafka

Apache Kafka is an attractive service because it’s conceptually simple and powerful. It’s easy to understand writing messages to a log in one place, then reading messages from that log in another place. This simplicity not only allows for a nice separation of concerns, but also is relatively easier to understand than more complex alternatives. [...]

By | July 19th, 2016|Data Processing Frameworks, Integrations|0 Comments

Monitoring Apache Spark Streaming: Understanding Key Metrics

This post is part 2 of the Monitoring Apache Spark series. In part 1 - Monitoring Apache Spark : Why is it Challenging?, we discussed the basics of Spark architecture and why it is challenging to monitor a distributed and dynamic Spark cluster. In this post, we will dig deeper into the specific Apache Spark metrics [...]

Monitoring Apache Spark: Why is it Challenging?

Large-scale data processing has moved into a new age of sophistication and the greatest evidence is the increasing requirement for processing data in real time. The drivers for this move are many, but the basic impetus can be understood by simply recalling the classic Time-Value-of-Data concept and how the value of data is at its [...]

Building & Monitoring a Near Real-time Streaming Pipeline

(This is a guest post from Jayant Shekhar, CEO of, a company focussed at building and deploying machine learning & large scale applications.) Leveraging Kafka and Spark Streaming combined with a NoSQL database is a common pattern which is getting increasingly popular for Near Real-Time (NRT) analytics. NRT is empowering enterprises across various verticals such [...]

By | June 5th, 2016|Analytics, Integrations|0 Comments