Setting up the Elastic Stack (formerly ELK Stack)

• ELK-stack
• ELK
• elastic-stack
• elasticsearch
• logstash
• kibana

The Elastic Stack, formerly known as the ELK Stack, is a powerful suite of open-source tools designed for real-time data search, analysis, and visualization. It offers comprehensive capabilities for collecting, processing, and visualizing large volumes of data. Its components are:

• Elasticsearch: A distributed, RESTful search and analytics engine based on the Lucene library.
• Logstash: A flexible data collection, processing, and enrichment pipeline.
• Kibana: A visualization and exploration tool for analyzing and visualizing data stored in Elasticsearch.
• Beats: Lightweight data shippers for ingesting data into Elasticsearch or Logstash.

Before you startLink to this anchor

To complete the actions presented below, you must have:

• A Scaleway account logged into the console
• Owner status or IAM permissions allowing you to perform actions in the intended Organization
• An SSH key
• An Instance or an Elastic Metal server with at least 4 GB of RAM

Install ElasticsearchLink to this anchor

1. Download and install the Elasticsearch signing key:

   curl -fsSL https://artifacts.elastic.co/GPG-KEY-elasticsearch | gpg --dearmor -o /usr/share/keyrings/elasticsearch-archive-keyring.gpg
   

2. Add the Elasticsearch repository:

   echo "deb [signed-by=/usr/share/keyrings/elasticsearch-archive-keyring.gpg] https://artifacts.elastic.co/packages/8.x/apt stable main" | tee /etc/apt/sources.list.d/elastic-8.x.list
   

3. Update the apt package repositories:

   apt update
   

4. Install Elasticsearch:

   apt install elasticsearch
   

5. Start and enable the Elasticsearch service:

   systemctl start elasticsearch
   systemctl enable elasticsearch
   

6. Configure Elasticsearch for production: Modify the elasticsearch.yml file to optimize Elasticsearch for production use:

   nano /etc/elasticsearch/elasticsearch.yml
   

   Add the following:

   cluster.name: "my-cluster"
   node.name: "node-1"
   network.host: 0.0.0.0
   xpack.security.enabled: true
   xpack.security.transport.ssl.enabled: true
   xpack.security.http.ssl.enabled: true
   xpack.security.http.ssl.keystore.path: /etc/elasticsearch/certs/keystore.p12
   xpack.security.http.ssl.truststore.path: /etc/elasticsearch/certs/truststore.p12
   

   Note

   Make sure you have SSL certificates set up for secure communication.

Install and configure LogstashLink to this anchor

1. Install Logstash using the same repository added for Elasticsearch:

   apt install logstash
   

2. Create and modify configuration files for Logstash: The configuration files for Logstash are typically located in /etc/logstash/conf.d/. You can create pipelines to manage your data processing.

3. Start and enable the Logstash service:

   systemctl start logstash
   systemctl enable logstash
   

Install and configure KibanaLink to this anchor

1. Install Kibana:

   apt install kibana
   

2. Start and enable the Kibana service:

   systemctl start kibana
   systemctl enable kibana
   

3. Configure Kibana for remote access: By default, Kibana is accessible on http://localhost:5601. To make Kibana accessible remotely, edit the Kibana configuration file:

   nano /etc/kibana/kibana.yml
   

   Change the server host to:

   server.host: "0.0.0.0"
   

4. Secure Kibana: Ensure Kibana uses SSL to encrypt communications by adding SSL certificates in the kibana.yml file:

   server.ssl.enabled: true
   server.ssl.certificate: /etc/kibana/certs/kibana.crt
   server.ssl.key: /etc/kibana/certs/kibana.key
   elasticsearch.ssl.certificate: /etc/kibana/certs/kibana.crt
   elasticsearch.ssl.key: /etc/kibana/certs/kibana.key
   

Install and configure FilebeatLink to this anchor

1. Install Filebeat:

   apt install filebeat
   

2. Configure Filebeat to ship logs to Elasticsearch: Edit the Filebeat configuration file to point to your Elasticsearch instance:

   nano /etc/filebeat/filebeat.yml
   

   Set the output to Elasticsearch:

   output.elasticsearch:
     hosts: ["http://localhost:9200"]
   

   Alternatively, configure Filebeat to send logs to Logstash:

   output.logstash:
     hosts: ["localhost:5044"]
   

3. Start and enable the Filebeat service:

   systemctl enable filebeat
   systemctl start filebeat
   

Secure the Elastic StackLink to this anchor

Securing your Elastic Stack is essential, especially if exposed to the internet. Following are some recommendations:

• Enable built-in security features (as shown above in Elasticsearch and Kibana setup).

• Use a firewall: You can use ufw or iptables to restrict access to only the necessary IPs:

  ufw allow from <your_ip> to any port 9200
  ufw allow from <your_ip> to any port 5601
  

• Set up an HTTPS reverse proxy: You can secure Kibana by setting up an HTTPS reverse proxy with Nginx: Set up an HTTPS reverse proxy with Nginx.

• Set up TLS/SSL for Elasticsearch and Kibana: Ensure communications are encrypted between components using SSL/TLS as shown above.

Test the installationLink to this anchor

After completing the setup, you can verify if everything is working:

• Elasticsearch: Run the following command to check Elasticsearch health:

  curl -X GET "localhost:9200/_cluster/health?pretty"
  

• Kibana: Navigate to http://your_server_ip:5601 in your web browser.

• Filebeat: Ensure logs are being shipped by checking the status:

  curl -X GET "localhost:5601/api/status"
  

Now, you should have a basic Elastic stack up and running! Adjust configurations as needed for your specific use case and further secure and optimize your setup for production use. Refer to the official Elastic documentation for the most accurate and up-to-date instructions and advanced configuration information.