Grafana Loki: A Comprehensive Guide to Installation, Configuration and Integration with Reverse Proxy Server.

In my previous blog, I already explained about Grafana. So, here I am going to skip that part. if you miss that make sure you read that blog, before starting this.

Previous blog link: https://hashnode.com/post/clmxe1jgn000109iacjtl5wko

This guide is designed for Centos, RHEL, Fedora, and Amazon Linux 2. Here I am using Amazon Linux 2. Using three different servers for running four different services. Grafana, Loki, Promtail and Nginx Proxy server. Grafana and Nginx Proxy server is running on the same server.

what is Loki?

Loki is an open-source log aggregation system designed for collecting, storing, and querying large volumes of log data in a distributed and efficient manner. It is particularly well-suited for cloud-native and containerized environments where applications generate a substantial amount of log data.

Some key features and aspects of Loki:

• Log Aggregation

• Labels and Log Streams

• Efficient Storage

• Query Language

• Integration with Grafana

• Scalability

• Alerting

• Community and Ecosystem

How to Install Loki?

To install Loki, you typically use containerization technology like Docker or Kubernetes. But I'll provide instructions for installing Loki on a local machine. it's pretty much straightforward.

• Before you start, it's a good practice to update the package lists and upgrade the installed packages.

  yum update -y

• (Optional) Make a directory.

  mkdir loki

• (Optional) change the Directory.

  cd loki

• Now Download the Loki .zip files that correspond to your system.

  curl -O -L "https://github.com/grafana/loki/releases/download/v2.8.5/loki-linux-amd64.zip"

• Unzip the package contents into the same directory. This is where the two programs will run.

  unzip loki-linux-amd64.zip

• Download the yaml configuration file.

  wget https://raw.githubusercontent.com/grafana/loki/main/cmd/loki/loki-local-config.yaml

• you might be wondering, what's inside this file. let's check, using cat / vim command we can check it.

  vim loki-local-config.yaml

  Output:

  

• Enter the following command to start Loki:

  ./loki-linux-amd64 -config.file=loki-local-config.yaml

Loki runs and displays Loki logs in your command line and you can check it from your browser. http://your-IP-Address:3100/metrics

The next step will be running an agent to send logs to Loki. To do so with Promtail.

What Is Promtail?

Promtail is an open-source log agent and collector that is part of the Grafana Loki ecosystem. Grafana Loki is a log aggregation and log management system designed for cloud-native and containerized environments. Promtail is responsible for collecting log data from various sources and forwarding it to Loki for storage, indexing, and querying.

some key features and functions of Promtail:

• Log Collection

• Log Labeling

• Log Parsing

• Log Aggregation

• Reliability

• Integration with Loki

• Scalability

• Configuration

How to Install Promtail?

To install Loki, you typically use containerization technology like Docker or Kubernetes. But I'll provide instructions for installing Loki on a local machine.

• Before you start, it's a good practice to update the package lists and upgrade the installed packages:

  yum update -y

• (Optional) Make a directory.

  mkdir promtail

• (Optional) Change the Directory.

  cd promtail

• Now Download the Promtail .zip files that correspond to your system.

  wget https://github.com/grafana/loki/releases/download/v2.8.5/promtail-linux-amd64.zip

• Unzip the package contents into the same directory. This is where the two programs will run.

  unzip promtail-linux-amd64.zip

• Download the yaml configuration file.

  wget https://raw.githubusercontent.com/grafana/loki/main/clients/cmd/promtail/promtail-local-config.yaml

• let's check. using cat / vim command we can check it.

  vim promtail-local-config.yaml

  Output:

  

• Enter the following command to start Promtail:

  ./promtail-linux-amd64 -config.file=promtail-local-config.yaml

Promtail runs and displays promtail logs in your command line.

At this point our Grafana, Loki, and Promtail server are ready. Now we need to integrate a reverse proxy server. But the question is what is a reverse proxy? and why do we need a reverse proxy? Let's find out.

What is a Reverse Proxy Server?

A reverse proxy is a server or software component that sits between client devices (such as web browsers) and backend servers. Its primary function is to act as an intermediary between clients and servers, forwarding client requests to the appropriate backend server and then returning the server's response to the client. In essence, it reverses the traditional client-server communication pattern.

why do we need a reverse proxy?

A reverse proxy serves several important purposes in web application architectures and network setups. Here are some of the key reasons why we need a reverse proxy.

• Load Balancing

• Security

• SSL/TLS Termination

• Caching

• Content Compression

• Content Routing

• Web Acceleration

• Server Isolation

• Authentication and Authorization

• Logging and Monitoring

To set up a reverse proxy:

Popular reverse proxy software includes Nginx, Apache HTTP Server (when configured as a reverse proxy), HAProxy, and others. here we will use an nginx reverse proxy.

• Before you start, it's a good practice to update the package lists and upgrade the installed packages:

  sudo yum update -y

• If Nginx is not installed, you can install it using the following command:

  sudo yum install nginx -y

• After installation, start the Nginx service.

  sudo systemctl start nginx

• To ensure that Nginx starts automatically when your server reboots.

  sudo systemctl enable nginx

• Create a new configuration file in the /etc/nginx/conf.d/ directory. You can name the file whatever you like but make sure it has the .conf extension. Here's an example of a basic reverse proxy configuration for forwarding requests to another server (replace backend_server_ip and backend_server_port with your backend server's IP address and port).

  sudo vim /etc/nginx/conf.d/reverse-proxy.conf

• Add the following configuration to the file.

  • Save the file and exit the text editor.

    

  • Test Nginx Configuration: Before reloading Nginx, you should check the configuration for syntax errors.

    sudo nginx -t

    If there are no errors, you will see a message indicating that the configuration test is successful.

  • Reload Nginx: To apply the new configuration, reload Nginx.

    sudo systemctl reload nginx

Ensure that your server security group allows traffic on port 80 if it's not already configured.

If you're using a domain, make sure your domain's DNS settings point to your server's public IP address.

Your Nginx reverse proxy should now be set up and forwarding incoming requests to the backend server. Make sure to replace grafana.test.bd, grafana.test.bd, and port 3000 with your actual values.

Let's check if it's working or not.

• Go to http://grafana.test.bd/ and add a new data source.

• Add Loki data source.

• Input http://your-ip-address:3100/ for the HTTP URL field.

• Click Save & Test. You should a green pop-up saying that data sources are connected and labels found.

• Go to http://grafana.test.bd/explore again. Your Loki data source should show up now.

• Enter {job=”varlogs”} on the query field and run it by typing CTRL + Enter or the blue button on the right top corner. You should see some logs are shown.

We are done. We have successfully set up Grafana Loki for log management.

Final Tips:

Conflicting Port

The most common problem is conflicting ports. Grafana is running on port 3000, Loki is running on port 3100, and Promtail is running on 9080. Ensure these ports are not occupied by other services. If it’s occupied, you will need to change the configuration to an available port and restart the services.