Before we continue, make sure you started your Elasticsearch and Kibana instances. Go to http://localhost:5601 in your Web browser.
We have created an index with 10000 data points in Elasticsearch. But meanwhile I have deleted this one and imported the complete geonames CSV file - around 12 million data points. I named the index "geonames".
To create an index pattern click on the "Management" link on the lower left-hand side of the browser window and then under the "Kibana" section, click on "Index Patters". If you have not defined any index pattern so far, you will see a page which allows you to define a first one:
Note: In Kibana date and time can be used to visualize data relatively to the moment when events happened. But in this data set we really only have the "modification date". This is not so interesting like say e.g. the event date and time of people clicking on web pages to order products. So for this showcase we will ignore date and time.
On the left side of the browser window click on "Visualize" and then click on the big blue "plus" button. The page displays the different visualization types available. Select "Coordinate Map" under the section "Maps". We will use this to visualize the latitude and longitude position of all points in the data set. You are now asked which index pattern to use - select "geonames", the pattern we just created.
You should now have an empty map displayed on the right side of the browser window.
To change this, under "Buckets", click on "Geo Coordinates" then "Geohash" and then select the field that carries the latitude and longitude positions of our data. In our case this is the field "position". It should look like this:
On the top right on the visualization there is a link labeled "Save". Click here to save the visualization by giving it an appropriate name.
Click on "Add a filter" and select e.g. the field "country_code.keyword" then "is" (equals) and then "DE" (for Germany).
You can also enter something like: feature_name.keyword : "water mill" to get all water mills displayed. In the search bar you can use "AND" and "OR" to combine search conditions according to your needs.
Make sure you save your visualization. I have saved it without any filters. We will now create another visualization - a bar chart. On the current visualization click on "Visualize" in the upper left hand corner - as shown below, to add a new one.
Now save this visualization as e.g. "Geonames - counts per feature_name".
Next, click on "Dashboard" on the left side of the browser window and then click on the blue button. You will get this:
You can now add a filter like we did before. E.g. add a filter to select only the data points in Spain (country_code.keyword is ES). Once the filter is saved the barchart and the geomap are updated. Again you can fit the data bounds or zoom in/out.
Or you could simply draw a rectangle (last button on the geomap, on the left side) to select an area and then an automatic filter for the selected area will be created and the barchart is updated accordingly.
- https://datamelt.weebly.com/blog/elasticsearch-a-practical-example-part-1
- https://datamelt.weebly.com/blog/elasticsearch-a-practical-example-part-2
So this is the first dashboard we create on top of the geonames data. There are many more visualizations available - go and try them out. You can combine them into one or multiple dashboards.
Hope you enjoyed this three-part blog about Logstash, Elasticsearch and Kibana. Make sure you come back frequently to read more articles about this amazing technology.
Carpe Diem