Elasticsearch is arguably the most influential open source logging tool available today. It is widely supported by vendor platforms and other open source projects and has a huge community. The recent Elastic{ON} conference in San Francisco unveiled some of the cool new stuff the parent organization, Elastic, has been cooking up over the past year — and it was full of goodness for logging and more. Let’s take a closer look at Elastic{ON} 2018.
Elastic{ON} 2018: Unpacking X-Pack
First up, the biggest announcement at Elastic{ON} 2018 was the opening up of the source code for X-Pack, the package of premium features such as monitoring, debugging, application performance management, and more. Previously, these features were commercially available, but now that they’re open source, users can not only use them freely, but also contribute back to the source code. This move shows the confidence Elastic has in its product suite and its ability to innovate with new solutions around monitoring and search.
There are still parts of X-Pack that are commercially licensed such as machine learning, security, and advanced alerting. However, the bulk of the code is open source, and this is good news for Elasticsearch users.
Data: Roll it up for good
A new feature launched at Elastic{ON} 2018 that’s particularly relevant to logging is the data rollup feature. It aims to store historical log data at a fraction of the size and cost it would normally take. Typically, organizations store log data in high resolution for a few days or weeks at most, and beyond that they store archived log data for a few months very rarely going back to a few years. With the data rollup feature, you can store logs in a compressed manner going back many years. You can also choose the level of detail you’d like; if, for example, you choose to store logs at a daily interval, you can query the data even at a longer interval like weekly or monthly.
The best part is that you can query the historical data along with the recent data in the same query. The system automatically gives importance to the recent data which has more detail, but you can easily jump to any point in the past, as it is shown alongside the recent data. This gives you a better perspective when performing error monitoring and troubleshooting issues, especially ones that are very complex and span multiple years in the past. Think of data breaches and emergencies that can have their root cause many years in the past; going back to trace the evolution of the issue can be invaluable in determining how deeply the system is affected and in finding a path to recovery as well. Rather than obsessing over storage costs, why not forget about archived data and have confidence that it’s all stored and easily accessible by one of the fastest querying tools available today.
Beats for Docker and Kubernetes
Elastic also announced some new updates at Elastic{ON} to their Beats tool for monitoring Docker and Kubernetes. The key components of the Beats are Metricbeat and Filebeat, which handle monitoring metrics and logging, respectively. They are placed on the nodes that are running the Docker containers or Kubernetes pods and report on metrics and logs in real-time. They can send the data to Elasticsearch directly, or to Logstash for pre-processing before being sent to Elasticsearch for analysis.
The important new feature is the ability to automatically discover the parts of the system as they keep changing. Not only this, Beats listens for events and executes automated responses based on changes. This is a big deal considering automation is a key focus area for most Special Interest Groups (SIGs) of Kubernetes from builds to deployments to upgrades, and even to monitoring. Elastic is on the right track by focusing on automation. Container systems, especially Kubernetes, can be hard to manage because of the distributed nature of the system, and how dynamic it is. This makes monitoring essential to Kubernetes, and automated monitoring ever so valuable.
Beats also includes other functionality like Packetbeat for network monitoring and Heartbeat for uptime monitoring. Together, all these agents can give you deep visibility into your Docker and Kubernetes systems in production. To make it easier to analyze the data, Beats lets you add metadata to all your incoming metrics and logs. It does this using dedicated metadata processors. Whether it’s data about your cloud resources, your Docker containers, or your Kubernetes clusters, you can tag them and make them more manageable and easy to analyze.
APM with Elasticsearch and Kibana
Elasticsearch has been known so far for its prowess when it comes to logging. However, Elastic has bigger plans for the platform, and it sees metrics — particularly application performance monitoring (APM) metrics — as the next big area for growth. APM is now a mature market with tools like New Relic and AppDynamics dominating the segment, but Elastic believes it has the secret sauce to challenge the competition, even as a newcomer.
There are two particular strengths Elastic has with respect to APM. First, its Elasticsearch querying engine is on par with the best and can handle analysis of large quantities of data in real-time. In fact, log data is a lot more complex than metrics because it is more unstructured. If Elasticsearch can handle complex log data, it can eat metrics for breakfast.
Second, Kibana, despite being open source, is a very refined visualization tool and is key to the APM offering. APM is all about correlating various metrics and zeroing in on an issue, and visualization is key to this kind of troubleshooting. When assessing any APM tool, visualization is one of the key criteria, and Kibana is one thing Elastic has going for it in this sense.
Kibana allows for custom dashboards that are connected with each other, mature visualization options, features for sorting and filtering metrics, and it looks simply gorgeous. It’s so good looking that it was even featured on the TV series “Mr. Robot,” being used by Elliot the protagonist security professional and hacker. The technology consultant for the series made it to Elastic{ON} to talk about their use of Kibana in the show, which was one of the more entertaining parts of the conference.
There are numerous free tools available today to track the performance of web applications. For example, dotcom-tools lets you track various aspects of website performance using its suite of free tools, but APM goes beyond these metrics and looks to deliver a broader view of end-to-end application performance. So, while it helps to leverage free tools for specific metrics especially at the start, eventually, you want to use an APM tool that can integrate a wide range of metrics for end-to-end visibility.
Coming back to APM, it’s hard to tell how much adoption Elastic will gain in the entrenched APM market. At least its loyal customer base, which has seen its strengths with logging and full-text search, may have enough confidence to give it a go. Beyond this, a lot will depend on its marketing firepower, and how clearly Elastic can communicate the strengths of their offering versus the competition.
Machine learning and the future
Another interesting new feature is the ability to use machine learning to forecast future performance based on past performance. Until now, Elasticsearch machine learning was restricted to predicting current performance in real-time. Now, it goes a step further by showing projections for whether a certain metric is likely to increase or decrease over time. It makes this prediction based on past data.
For predictions to be accurate, Elasticsearch needs a historical record of data, preferably more than three weeks. If the past data is insufficient to make an educated guess, Elasticsearch will notify you of that. Granted, past behavior is not always an accurate way to predict future outcome, but it can be reliable for many scenarios. This is a nascent area ripe for innovation and Elastic is making an early start while being realistic about what to expect from the feature. This is available only in the X-Pack as a commercial offering and is worth exploring if you’re deeply invested in Elasticsearch and have explored most of the other features it offers.
In search of better site search
Another feature that is new to Elasticsearch, but is an interesting opportunity with a huge potential upside, is site search. The reason this feature is very timely is because the most popular solution thus far — Google Site Search — was sunsetted by Google on April 1. This leaves many organizations looking for alternatives, and Elastic is cashing in on this demand. Now, Elasticsearch is a full-text search engine, and the way it can analyze log data is proof of its ability to crunch not just numbers but text data just as well. And site search is just an extension of full-text search.
A site search tool is very dynamic in nature as the search results need to be fresh, being updated in real-time with the most recent search results. It needs to understand, interpret, and respond to user input as the user types in a query. It needs to be easy to implement on the website irrespective of what technologies or platforms are used to build the website. Finally, it needs to have a well-thought-out UI that gets users to the search result they’re looking for in split seconds. This is a tall task, but Elastic believes it has the answer in their site search product.
Giving users what they want
Elastic{ON} was chock-full of updates and makes you want to sit up and notice what the team is up to. The biggest update was the opening up of X-Pack. However, new products like APM and site search are just as big in terms of direction for the company. Finally, interesting features like data rollups, monitoring and logging for container systems, and machine learning are all not surprising but are right on the ball in terms of what today’s users want. As a logging aficionado, you’re bound to find all this exciting. There’s lots to play around with, so if you haven’t already, go get Elasticsearch.
