Skip to navigationSkip to main contentSkip to footerScaleway Docs

Setting up the Elastic Stack (formerly ELK Stack)

ELK-stack
ELK
elastic-stack
elasticsearch
logstash
kibana

The Elastic Stack, formerly known as the ELK Stack, is a powerful suite of open-source tools designed for real-time data search, analysis, and visualization. It offers comprehensive capabilities for collecting, processing, and visualizing large volumes of data. Its components are:

  • Elasticsearch: A distributed, RESTful search and analytics engine based on the Lucene library.
  • Logstash: A flexible data collection, processing, and enrichment pipeline.
  • Kibana: A visualization and exploration tool for analyzing and visualizing data stored in Elasticsearch.
  • Beats: Lightweight data shippers for ingesting data into Elasticsearch or Logstash.

Before you start

To complete the actions presented below, you must have:

Install Elasticsearch

  1. Download and install the Elasticsearch signing key:

    curl -fsSL https://artifacts.elastic.co/GPG-KEY-elasticsearch | gpg --dearmor -o /usr/share/keyrings/elasticsearch-archive-keyring.gpg
  2. Add the Elasticsearch repository:

    echo "deb [signed-by=/usr/share/keyrings/elasticsearch-archive-keyring.gpg] https://artifacts.elastic.co/packages/8.x/apt stable main" | tee /etc/apt/sources.list.d/elastic-8.x.list
  3. Update the apt package repositories:

    apt update
  4. Install Elasticsearch:

    apt install elasticsearch
  5. Start and enable the Elasticsearch service:

    systemctl start elasticsearch
    systemctl enable elasticsearch
  6. Configure Elasticsearch for production: Modify the elasticsearch.yml file to optimize Elasticsearch for production use:

    nano /etc/elasticsearch/elasticsearch.yml

    Add the following:

    cluster.name: "my-cluster"
    node.name: "node-1"
    network.host: 0.0.0.0
    xpack.security.enabled: true
    xpack.security.transport.ssl.enabled: true
    xpack.security.http.ssl.enabled: true
    xpack.security.http.ssl.keystore.path: /etc/elasticsearch/certs/keystore.p12
    xpack.security.http.ssl.truststore.path: /etc/elasticsearch/certs/truststore.p12
    Note

    Make sure you have SSL certificates set up for secure communication.

Install and configure Logstash

  1. Install Logstash using the same repository added for Elasticsearch:

    apt install logstash
  2. Create and modify configuration files for Logstash: The configuration files for Logstash are typically located in /etc/logstash/conf.d/. You can create pipelines to manage your data processing.

  3. Start and enable the Logstash service:

    systemctl start logstash
    systemctl enable logstash

Install and configure Kibana

  1. Install Kibana:

    apt install kibana
  2. Start and enable the Kibana service:

    systemctl start kibana
    systemctl enable kibana
  3. Configure Kibana for remote access: By default, Kibana is accessible on http://localhost:5601. To make Kibana accessible remotely, edit the Kibana configuration file:

    nano /etc/kibana/kibana.yml

    Change the server host to:

    server.host: "0.0.0.0"
  4. Secure Kibana: Ensure Kibana uses SSL to encrypt communications by adding SSL certificates in the kibana.yml file:

    server.ssl.enabled: true
    server.ssl.certificate: /etc/kibana/certs/kibana.crt
    server.ssl.key: /etc/kibana/certs/kibana.key
    elasticsearch.ssl.certificate: /etc/kibana/certs/kibana.crt
    elasticsearch.ssl.key: /etc/kibana/certs/kibana.key

Install and configure Filebeat

  1. Install Filebeat:

    apt install filebeat
  2. Configure Filebeat to ship logs to Elasticsearch: Edit the Filebeat configuration file to point to your Elasticsearch instance:

    nano /etc/filebeat/filebeat.yml

    Set the output to Elasticsearch:

    output.elasticsearch:
      hosts: ["http://localhost:9200"]

    Alternatively, configure Filebeat to send logs to Logstash:

    output.logstash:
      hosts: ["localhost:5044"]
  3. Start and enable the Filebeat service:

    systemctl enable filebeat
    systemctl start filebeat

Secure the Elastic Stack

Securing your Elastic Stack is essential, especially if exposed to the internet. Following are some recommendations:

  • Enable built-in security features (as shown above in Elasticsearch and Kibana setup).

  • Use a firewall: You can use ufw or iptables to restrict access to only the necessary IPs:

    ufw allow from <your_ip> to any port 9200
    ufw allow from <your_ip> to any port 5601
  • Set up an HTTPS reverse proxy: You can secure Kibana by setting up an HTTPS reverse proxy with Nginx: Set up an HTTPS reverse proxy with Nginx.

  • Set up TLS/SSL for Elasticsearch and Kibana: Ensure communications are encrypted between components using SSL/TLS as shown above.

Test the installation

After completing the setup, you can verify if everything is working:

  • Elasticsearch: Run the following command to check Elasticsearch health:

    curl -X GET "localhost:9200/_cluster/health?pretty"
  • Kibana: Navigate to http://your_server_ip:5601 in your web browser.

  • Filebeat: Ensure logs are being shipped by checking the status:

    curl -X GET "localhost:5601/api/status"

Now, you should have a basic Elastic stack up and running! Adjust configurations as needed for your specific use case and further secure and optimize your setup for production use. Refer to the official Elastic documentation for the most accurate and up-to-date instructions and advanced configuration information.

Questions?

Visit our Help Center and find the answers to your most frequent questions.

Visit Help Center
No Results