What is Elasticsearch?
Elasticsearch is a search engine created to solve big data issues. It’s been designed to be able to perform natural language search and analytics on a very wide range and volume of documents. It’s being used in a wide range of cases, from actual web-store product search (with features detecting similar items), to performance metrics analysis (number of Purchases or Downloads per day, CPU (central processing unit), usage across servers etc). All of this to provide excellent Business Intelligence.
Why is it so popular?
It is becoming popular because of it’s ease of use (for end users and developers). It’s built around the concept of scalability, which makes it easier to design a solution and increase the performance as needed by adding additional servers. It also comes with very reasonable default settings for it’s most common uses: natural text search (with relevance scoring), and efficient analysis queries. It has a very good open source ecosystem.
Why is it appealing from a tech perspective?
It’s technically interesting for a wide range of reasons:
- Real Time Processing capabilities: analysis and searches are performed on the scale of milliseconds suitable for web applications
- it’s easy to replace or add servers, and to continue working when losing a single server
- Generally good tooling for text analysis
Glossary of related tech solutions
Kibana: an User Interface (UI) and data discovery web application. It’s main focus is to allow for the visualisation of the data. (It is moving toward also becoming the UI for the management of a cluster in general).
Apache lucene: a free and open-source information retrieval software library, originally written in 100% pure Java.
Beats: allow for low level data collection and forwarding to ES. For instance, the capture of every http requests made one server served with surrounding information, or the passing of each new addition from a server log file to ES.
Logstash: an open source tool for collecting, parsing, and storing logs for future use. It allows for the handling and processing of different events.
XPack: a bundle of non-free plugins that can be added to each kibana, elasticsearch, beats and logstash instances.