The initial learning curve was very low: install Elasticsearch and Kibana and simply add some data using the Kibana dev console. Easy starter. I then used Apache Nifi to read a file and send it to the Elasticsearch server. Also very easy. I did not touch Logstash at that time.
With the little data I had - a few thousand records - I was learning how to use Kibana for creating visualizations and dashboards. And while doing that, I ran into two questions: how to handle dates and how to handle geo locations correctly? I went back and forth to read the documentation and search the Internet. It takes some time to understand how Elastic works with dates, times and how one can (or should) use them in KIbana. How does one works with timezones and time in general? Or geo locations.
So after some time I found out the following things:
- think thoroughly about the Elasticsearch schema (data types) and transform your data accordingly. Spend a lot of time for this. The better the schema, the better the analytics in Kibana.
- devide your data into slices, Analytics and queries can be done over multiple indexes and on the other side individual slices can be deleted easily. A typical and natural devision is e.g by date.
- don't create complex (or a lot of) visualizations or dashboards on incomplete indexes. Because when you delete the index - e.g. because you want to roll out a new version - then all the visualizations and dashboards are useless.
- If you like visual tools, use Apache Nifi to feed Elasticsearch. Getting started is easy and reading e.g. from files or databases is easily/quickly done.
- A major plus point in ELasticsearch is, that it inserts or updates the data based on a key. A no-brainer once you have a key defined for the data.
Apart from this, working with Kibana is real fun and it allows to create great dashboards.
Ok. This was the first intro from my side. I will publish more detailed projects here in the next weeks.
Carpe diem.