Monitor your AWS EC2 instance with Grafana and Prometheus.

Connect a Linux and Windows EC2 instance with Grafana. By doing so, we'll be able to monitor various components of our servers and gain valuable insights into their performance. So let's dive right in and get started!

Step 1: Install Grafana and setup instances to monitor

To begin, we need to have Grafana installed on a separate EC2 instance or a server. Grafana is an open-source monitoring and visualization tool that provides a rich set of features for data analytics and monitoring of various systems and applications.

1. Install Grafana by following this link---> Install Grafana

2. Launch another EC2 instance (Linux or Windows) on AWS for monitoring its resources using Grafana.

I have created a Linux instance - Ubuntu 22.04 LTS for this demo.

Once you have Grafana up and running on your server, Let's move to the next step.

Step 2: Install and Configure Data Sources (Prometheus in this case)

Now Grafana is installed, we need to configure it to connect with our Linux and Windows EC2 instances. Grafana supports various data sources, including Prometheus, Graphite, Elasticsearch, and more. In our case, we'll focus on connecting with the EC2 instances directly.

Open your web browser and access the Grafana web interface by entering the IP address or domain name of your Grafana server.

Log in to Grafana using your credentials. The default username and password are usually "admin" and "admin" respectively.

Make sure to change the password after logging in.

Once logged in, go to the Configuration menu and select "Data Sources."

Click on "Add data source" to add a new data source.

Select the appropriate data source type for your EC2 instances. For Linux, you can choose "Prometheus" or "CloudWatch." For Windows, select "CloudWatch."

Now you need to install the Prometheus server as our data source--->

What is Node Exporter?

Node Exporter is a Prometheus exporter specifically designed to collect and expose system-level metrics from a target machine. It is a popular component of the Prometheus monitoring ecosystem. Node Exporter runs as a separate service on the target machine and provides a wide range of metrics related to CPU usage, memory utilization, disk activity, network statistics, system load, and more.

Node Exporter collects these metrics by interacting with various operating system interfaces, such as /proc and /sys on Linux systems. It exposes these metrics in a Prometheus-compatible format, allowing Prometheus to scrape and store them for monitoring and alerting purposes.

Let's install Prometheus on the monitoring server where our Grafana is running and the node exporter on both the monitoring server and the Targeted instance.

Install node_exporter--->

• First, we will download the Node Exporter on both machines :

wget https://github.com/prometheus/node_exporter/releases/download/v0.15.2/node_exporter-0.15.2.linux-amd64.tar.gz

• Extract the downloaded archive

tar -xf node_exporter-0.15.2.linux-amd64.tar.gz

• Move the node_exporter binary to /usr/local/bin:

sudo mv node_exporter-0.15.2.linux-amd64/node_exporter /usr/local/bin

• Remove the residual files with:

rm -r node_exporter-0.15.2.linux-amd64*

• Next, we need to create users and service files for node_exporter.

For security reasons, it is always recommended to run any services/daemons in separate accounts of their own. Thus, we are going to create an user account for node_exporter. We have used the -r flag to indicate it is a system account, and set the default shell to /bin/false using -s to prevent logins.

sudo useradd -rs /bin/false node_exporter

Then, we will create a systemd unit file so that node_exporter can be started at boot.

sudo nano /etc/systemd/system/node_exporter.service

[Unit]
Description=Node Exporter
After=network.target

[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter

[Install]
WantedBy=multi-user.target

Since we have created a new unit file, we must reload the systemd daemon, set the service to always run at boot and start it :

sudo systemctl daemon-reload
sudo systemctl enable node_exporter
sudo systemctl start node_exporter

Installing Prometheus--->

• Download and install Prometheus only on the Prometheus Server.

wget https://github.com/prometheus/prometheus/releases/download/v2.1.0/prometheus-2.1.0.linux-amd64.tar.gz

• Extract the Prometheus archive :

tar -xf prometheus-2.1.0.linux-amd64.tar.gz

• Move the binaries to /usr/local/bin:

sudo mv prometheus-2.1.0.linux-amd64/prometheus prometheus-2.1.0.linux-amd64/promtool /usr/local/bin

• Now, we need to create directories for configuration files and other Prometheus data.

sudo mkdir /etc/prometheus /var/lib/prometheus

• Then, we move the configuration files to the directory we made previously:

sudo mv prometheus-2.1.0.linux-amd64/consoles prometheus-2.1.0.linux-amd64/console_libraries /etc/prometheus

• Finally, we can delete the leftover files as we do not need them any more:

rm -r prometheus-2.1.0.linux-amd64*

Configuring Prometheus--->

After having installed Prometheus, we have to configure Prometheus to let it know about the HTTP endpoints it should monitor. Prometheus uses the YAML format for its configuration.

• We will use /etc/prometheus/prometheus.yml as our configuration file

global:
  scrape_interval: 10s

scrape_configs:
  - job_name: 'prometheus_metrics'
    scrape_interval: 5s
    static_configs:
      - targets: ['localhost:9090']
  - job_name: 'node_exporter_metrics'
    scrape_interval: 5s
    static_configs:
      - targets: ['localhost:9100','prometheus-target-1:9100','prometheus-target-2:9100']

In this file, we have defined a default scraping 10 seconds. We have also defined two sources of metrics, named prometheus_metrics and node_exporter_metrics. For both of them, we have set the scraping interval to 5 seconds, overriding the default. Then, we have specified the locations where these metrics will be available. Prometheus uses port 9090 and node_exporter uses port 9100 to provide their metrics.

• Finally, we will also change the ownership of files that Prometheus will use:

sudo useradd -rs /bin/false prometheus
sudo chown -R prometheus: /etc/prometheus /var/lib/prometheus

• Then, we will create a systemd unit file in /etc/systemd/system/prometheus.service with the following contents :

[Unit]
Description=Prometheus
After=network.target

[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus \
    --config.file /etc/prometheus/prometheus.yml \
    --storage.tsdb.path /var/lib/prometheus/ \
    --web.console.templates=/etc/prometheus/consoles \
    --web.console.libraries=/etc/prometheus/console_libraries

[Install]
WantedBy=multi-user.target

• Finally, we will reload systemd:

sudo systemctl daemon-reload
sudo systemctl enable prometheus
sudo systemctl start prometheus

• Prometheus provides a web UI for running basic queries located at http://<your_server_IP>:9090/.

  This is what it looks like in a web browser:

Configure the data source settings. Make sure to provide the necessary permissions to access the EC2 instances' metrics and data.

Test the data source connection to ensure everything is set up correctly.

Congratulations! You have successfully configured the data sources to connect with your EC2 instances. Now it's time to create dashboards and visualize the data.

Step 3: Create Dashboards

Dashboards in Grafana are where you can design and visualize your data in a way that suits your monitoring needs. You can create multiple dashboards and customize them according to your preferences. Let's create a basic dashboard to monitor our EC2 instances.

In the Grafana web interface, click on the "Create" button and select "Dashboard."

Choose the visualization type you want to use, such as graphs, tables, or charts.

Before going further please check the Prometheus configuration for targets:

Configure the metrics and data you want to monitor. You can select specific EC2 instances metrics, such as CPU usage, memory usage, disk utilization, and network traffic.

Customize the dashboard layout, add panels, and arrange them as needed.

Save the dashboard and give it a meaningful name.

Once you have created your dashboard, Grafana will start fetching data from your EC2 instances and display it in real time.

You can explore various visualization options and features provided by Grafana to enhance your monitoring experience.

You are done with it!

Connecting EC2 instances with Grafana opens up a world of possibilities for monitoring and analyzing your server's performance. By following the steps outlined in this blog post, you can establish a seamless connection between your Linux EC2 instances and Grafana. This connection allows you to gain valuable insights into the different components of your servers, enabling you to make informed decisions and take necessary actions to optimize their performance.

Remember, monitoring is a continuous process, and Grafana provides a powerful platform to track your server's health and performance over time. Keep exploring Grafana's features, experiment with different metrics, and refine your dashboards to meet your specific requirements.

Happy Learning and Happy Monitoring :)

Day 74 task is complete!

90DaysOfDevOps Tasks👇

github.com/Chaitannyaa/90DaysOfDevOps.git

Chaitannyaa Gaikwad | LinkedIn