Automated Monitoring & Alerts Setup using Ansible

Objective

The goal of this Task is to automate the deployment of a monitoring stack using Ansible. It will include:

• Deploying Prometheus for metric collection
• Deploying Grafana for visualization
• Automating the setup of alert rules and dashboards
• Ensuring high availability and configuring node exporters for system monitoring

Architecture Overview

Components Used

• Prometheus – Collects and stores metrics
• Grafana – Visualizes metrics from Prometheus
• Node Exporter – Collects system metrics (CPU, Memory, Disk, Network)
• Alertmanager – Handles alerts based on Prometheus rules
• Ansible – Automates deployment and configuration

Infrastructure

• OS: Ubuntu 22.04 (or CentOS 8)
• Instances: 3 Virtual Machines (Prometheus, Grafana, and Node Exporters)
• Deployment: Using Ansible Playbooks
• Storage: Persistent volumes for Prometheus and Grafana

Implementation Plan

Step 1: Setup Ansible Inventory

Define inventory for the servers:

[monitoring]
prometheus ansible_host=192.168.1.100 ansible_user=ubuntu
grafana ansible_host=192.168.1.101 ansible_user=ubuntu

[node_exporters]
node1 ansible_host=192.168.1.102 ansible_user=ubuntu
node2 ansible_host=192.168.1.103 ansible_user=ubuntu

Step 2: Install and Configure Prometheus

Ansible Playbook: Install Prometheus

- name: Install Prometheus
  hosts: monitoring
  become: yes
  tasks:
    - name: Download Prometheus
      get_url:
        url: "https://github.com/prometheus/prometheus/releases/latest/download/prometheus-2.41.0.linux-amd64.tar.gz"
        dest: "/tmp/prometheus.tar.gz"

    - name: Extract Prometheus
      ansible.builtin.unarchive:
        src: "/tmp/prometheus.tar.gz"
        dest: "/opt/"
        remote_src: yes

    - name: Create Prometheus User
      user:
        name: prometheus
        system: yes

    - name: Configure Prometheus Service
      copy:
        dest: "/etc/systemd/system/prometheus.service"
        content: |
          [Unit]
          Description=Prometheus
          After=network.target
          
          [Service]
          User=prometheus
          ExecStart=/opt/prometheus/prometheus --config.file=/etc/prometheus/prometheus.yml --storage.tsdb.path=/var/lib/prometheus
          Restart=always
          
          [Install]
          WantedBy=multi-user.target

    - name: Enable and Start Prometheus
      systemd:
        name: prometheus
        enabled: yes
        state: started

Step 3: Install and Configure Grafana

Ansible Playbook: Install Grafana

- name: Install Grafana
  hosts: grafana
  become: yes
  tasks:
    - name: Add Grafana Repository
      shell: |
        wget -q -O - https://packages.grafana.com/gpg.key | sudo apt-key add -
        echo "deb https://packages.grafana.com/oss/deb stable main" | sudo tee -a /etc/apt/sources.list.d/grafana.list

    - name: Install Grafana
      apt:
        name: grafana
        state: present
        update_cache: yes

    - name: Enable and Start Grafana
      systemd:
        name: grafana-server
        enabled: yes
        state: started

Step 4: Configure Prometheus to Scrape Node Exporters

Node Exporter Installation

- name: Install Node Exporter
  hosts: node_exporters
  become: yes
  tasks:
    - name: Download Node Exporter
      get_url:
        url: "https://github.com/prometheus/node_exporter/releases/latest/download/node_exporter-1.5.0.linux-amd64.tar.gz"
        dest: "/tmp/node_exporter.tar.gz"

    - name: Extract Node Exporter
      unarchive:
        src: "/tmp/node_exporter.tar.gz"
        dest: "/usr/local/bin/"
        remote_src: yes

    - name: Create Node Exporter Service
      copy:
        dest: "/etc/systemd/system/node_exporter.service"
        content: |
          [Unit]
          Description=Node Exporter
          After=network.target

          [Service]
          User=root
          ExecStart=/usr/local/bin/node_exporter
          Restart=always

          [Install]
          WantedBy=multi-user.target

    - name: Start and Enable Node Exporter
      systemd:
        name: node_exporter
        enabled: yes
        state: started

Step 5: Configure Prometheus Targets

Update prometheus.yml:

- name: Configure Prometheus Targets
  hosts: monitoring
  become: yes
  tasks:
    - name: Update Prometheus Config
      copy:
        dest: "/etc/prometheus/prometheus.yml"
        content: |
          global:
            scrape_interval: 15s

          scrape_configs:
            - job_name: "prometheus"
              static_configs:
                - targets: ["localhost:9090"]

            - job_name: "node_exporter"
              static_configs:
                - targets: ["192.168.1.102:9100", "192.168.1.103:9100"]
      notify: Restart Prometheus

  handlers:
    - name: Restart Prometheus
      systemd:
        name: prometheus
        state: restarted

Step 6: Setup Alert Rules and Notifications

Create alert_rules.yml:

- name: Configure Prometheus Alerts
  hosts: monitoring
  become: yes
  tasks:
    - name: Add Alert Rules
      copy:
        dest: "/etc/prometheus/alert_rules.yml"
        content: |
          groups:
            - name: instance_down
              rules:
                - alert: InstanceDown
                  expr: up == 0
                  for: 1m
                  labels:
                    severity: critical
                  annotations:
                    summary: "Instance {{ $labels.instance }} is down"
                    description: "{{ $labels.instance }} is unreachable for more than 1 minute"
      notify: Restart Prometheus

  handlers:
    - name: Restart Prometheus
      systemd:
        name: prometheus
        state: restarted

Step 7: Automate Grafana Dashboards

Use JSON dashboards to automate the setup.

- name: Upload Grafana Dashboards
  hosts: grafana
  become: yes
  tasks:
    - name: Copy Dashboard JSON
      copy:
        src: "dashboard.json"
        dest: "/var/lib/grafana/dashboards/dashboard.json"

    - name: Restart Grafana
      systemd:
        name: grafana-server
        state: restarted

Conclusion

• Ansible automates the installation, configuration, and alert setup for Prometheus and Grafana.
• Node Exporter collects metrics, which Prometheus scrapes.
• Grafana visualizes data and integrates with alerting.
• Alertmanager handles alert notifications.

This setup ensures an automated monitoring solution for infrastructure, ready to scale and integrate with existing CI/CD pipelines.