How to collect and visualize your logs with the ELK stack (Elasticsearch Logstash Kibana) | Scaleway


#1

How to collect and visualize your logs with the ELK stack (Elasticsearch Logstash Kibana)

This page shows you how to use the ELK stack InstantApp on your C1 server. ELK stack is an environment that lets you collect and visualize your logs with:

  • Elasticsearch for search and data analytics
  • Logstash for centralized logging, log enrichment and parsing
  • Kibana to visualize data

Requirements

  • You have an account and are logged into cloud.scaleway.com
  • You have configured your SSH Key

There are three steps to deploy the ELK stack InstantApp

  • Create and start a new C1 server using the ELK stack InstantApp
  • Collect syslogs data with Logstash
  • Visualize your data with Kibana

Step 1 - Create and start a new C1 server using the ELK stack InstantApp

First, we need to create a new server using the ELK stack InstantApp. Click the “Create Server” button in the control panel.

You land on the server creation page where you must input information and choose an image.

After inputting your server basic information, select the ELK stack image for your server. On the ImageHub tab, select ELK stack and click the “Create Server” button.

The server will be created with a ready to use install of elasticsearch, Kibana and logstash.

Step 2 - Collect Syslogs data with Logstash

In this tutorial we will see how to track syslogs data and visualize them from Kibana.

Let’s start by creating a new configuration file to collect system logs. Open a new file in /etc/logstash/conf.d/logstash-syslog.conf and fill it with the following:

input {
  file {
    path => [ "/var/log/*.log", "/var/log/messages", "/var/log/syslog" ]
    type => "syslog"
  }
}
 
output {
  elasticsearch { host => localhost }
  stdout { codec => rubydebug }
}

The configuration above tells Logstash to collect all files with .log extention in /var/log, /var/log/messages and /var/log/syslog.

Next, we will create a filter to prevent Elasticsearch to store logs in the message field and simplify the analysis.

filter {
  if [type] == "syslog" {
    grok {
      match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
      add_field => [ "received_at", "%{@timestamp}" ]
      add_field => [ "received_from", "%{host}" ]
    }
    syslog_pri { }
    date {
      match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
    }
  }
}

Restart logstash to apply our changes service logstash restart

Step 3 - Visualize your data with Kibana

System logs are now collected and stored in elasticsearch, you can visualize them with Kibana. Open a browser and go to http://<your_server_public_ip>. You are asked for a login and password. You can retrieve them on the message of the day (MOTD) when you connect your server.

Welcome on ELK stack on Scaleway' C1.
 * Kernel:           GNU/Linux 3.2.34-30 armv7l - Marvell (Proprietary)
                     - This kernel has the best performances on this hardware
                     - For mainline kernel with latest features and plenty of modules, use a 3.17 kernel instead
 * Distribution:     ELK stack (2015-06-09) on Ubuntu 14.10
 * Internal ip:      10.1.35.26
 * External ip:      212.47.241.133
 * Disk /dev/nbd0:   scw-app-elk-latest-2015-06-09_18:11 (l_ssd 50G)
 * Uptime:           09:50:11 up 17:31,  0 users,  load average: 3.23, 3.15, 3.08
Links
 * Documentation:    https://scaleway.com/docs/how-to-use-the-elk-stack-instant-apps/
 * Community:        https://community.scaleway.com
 * Image source:     https://github.com/scaleway/image-app-elk
To access Kibana, open http://xxx.yyy.zzz.www/.
Login with user kibana and password -> ieshahchuemohfohxooshieshieshiojiepiengeng <-
You can hide this message on the next connection by deleting the /etc/update-motd.d/70-elk file.

You land on Kibana homepage and are asked to configure an index pattern. Index patterns are used to identify the Elasticsearch index to run search and analytics against.

To create the first index, select @timestamp from the Time-field-name menu and click the Create button.

On the top navigation bar, click the Discover tab.

Here will be displayed all the log collected and an histogram representing the log activity.

It is your turn now! Start playing with Kibana, create graphics and filters on your logs :)

Conclusion

ELK stack lets you search and analyze your data with ease. From here you can go deeper and create a more complex configuration. For instance you can use logstash-forwarder which let you collect logs from remote servers and send them to Logstash.

If you have any suggestion or question on this documentation, please leave a comment.

Try this tutorial on your own C1 server TRY IT


This is a companion discussion topic for the original entry at https://www.scaleway.com/docs/how-to-use-the-elk-stack-instant-apps/

#2

No login details on MOTD.

Warning: Permanently added '163.172.160.150' (ECDSA) to the list of known hosts.
               _
 ___  ___ __ _| | _____      ____ _ _   _
/ __|/ __/ _` | |/ _ \ \ /\ / / _` | | | |
\__ \ (_| (_| | |  __/\ V  V / (_| | |_| |
|___/\___\__,_|_|\___| \_/\_/ \__,_|\__, |
                                    |___/

Welcome on ELK stack (GNU/Linux 4.4.6-std-2 x86_64 )

System information as of: Mon Jun 27 15:41:01 UTC 2016

System load:    0.07            Int IP Address: 10.2.167.15 
Memory usage:   0.9%            Pub IP Address: 163.172.160.150
Usage on /:     4%              Swap usage:     0.0%
Local Users:    0               Processes:      71
Image build:    2016-03-21      System uptime:  0 min
Disk nbd0:      l_ssd 50G

Documentation:  https://scaleway.com/docs/how-to-use-the-elk-stack-instant-apps/
Community:      https://community.scaleway.com
Image source:   https://github.com/scaleway-community/scaleway-elk

You can hide this message on the next connection by deleting the /etc/update-motd.d/70-elk file.

The programs included with the Ubuntu system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
applicable law.

root@scw-4623ba:~# 

#3

+1
Me too:
[SSH] Server Version OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.6
[SSH] Logged in (RSA key C:\Users\Ivan\Downloads\id_rsa)

               _
 ___  ___ __ _| | _____      ____ _ _   _
/ __|/ __/ _` | |/ _ \ \ /\ / / _` | | | |
\__ \ (_| (_| | |  __/\ V  V / (_| | |_| |
|___/\___\__,_|_|\___| \_/\_/ \__,_|\__, |
                                    |___/

Welcome on ELK stack (GNU/Linux 4.4.6-std-2 x86_64 )

System information as of: Wed Sep 28 15:54:48 UTC 2016

System load:    0.08            Int IP Address: 10.3.70.201
Memory usage:   0.4%            Pub IP Address: 163.172.163.162
Usage on /:     4%              Swap usage:     0.0%
Local Users:    0               Processes:      93
Image build:    2016-03-21      System uptime:  0 min
Disk nbd0:      l_ssd 50G
Disk nbd1:      l_ssd 150G

Documentation:  https://scaleway.com/docs/how-to-use-the-elk-stack-instant-apps/
Community:      https://community.scaleway.com
Image source:   https://github.com/scaleway-community/scaleway-elk

You can hide this message on the next connection by deleting the /etc/update-motd.d/70-elk file.

The programs included with the Ubuntu system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
applicable law.

root@scw-63c99f:~#

#4

You can get the password with the following command, the login for web is “kibana
cat /root/.kibana.passwd


#5

I can no longer find the ELK image in ImageHub on Scaleway - has it been removed or am I missing something?

i.e. Step 1 above: After inputting your server basic information, select the ELK stack image for your server. On the ImageHub tab, select ELK stack ... - seems to not show the |ELK| stack Image any more.

I’ve got an old ELK server on Scaleway from some time ago - and wanted to refresh it with a new clean instance - but can’t find it any more.

Any ideas?