Realm Labs Proxmox LXC monitoring with Prometheus, PVE Exporter and the KITT Control Centre

Monitoring Proxmox LXC Containers Accurately with Prometheus and PVE Exporter

How I made the KITT Control Centre match Proxmox LXC CPU, RAM and disk figures by combining prometheus-pve-exporter with node_exporter, Prometheus and PromQL.

While adding live health data to my KITT-inspired Control Centre, I ran into a monitoring problem that is easy to miss with Proxmox LXC containers: the CPU, RAM and disk figures reported by node_exporter inside the container did not match the figures shown in Proxmox.

Both sets of numbers were valid, but they were measuring the system from different viewpoints. For the KITT dashboard I wanted the figures on the Android control panel to agree with the Proxmox summary for the container, so I added prometheus-pve-exporter alongside node_exporter and let each exporter do the job it is best suited to.

This guide documents the complete setup used in Realm Labs: Proxmox VE on the Shuttle DS81 host, a KITT-AI LXC, Prometheus, node_exporter, PVE Exporter, PromQL and the Android KITT Control Centre.

Related Realm Labs projects: KITT Smart Home Dashboard, RabbitMQ for KITT TTS, Home Lab Monitoring with Prometheus and Grafana, Proxmox VE: The Infrastructure Foundation, and the Realm Labs hardware page.

What I Was Trying to Monitor

KITT-AI runs as LXC 108 on my Proxmox host, QuanChi. The control-centre app already talks to Home Assistant, the KITT personality engine and the Piper/RabbitMQ TTS stack. For v1.03 I wanted a small 1980s-style health panel showing four simple figures:

  • CPU
  • RAM
  • Swap
  • Root disk usage

The first implementation used node_exporter inside KITT-AI. That worked immediately, but the values did not always resemble the Proxmox summary page. RAM and disk were the most obvious examples.

KITT Android Control Centre
          |
          v
      Prometheus
       /      \
      v        v
PVE Exporter  node_exporter
      |        |
      v        v
 Proxmox API  KITT-AI Linux
      |
      v
   LXC 108

Why node_exporter and Proxmox Can Show Different Values

node_exporter runs inside the LXC and reads the Linux view exposed to that container. Proxmox is looking at the guest from the host side using LXC/cgroup and storage accounting. Those are not necessarily identical definitions of “used CPU”, “used RAM” or “used disk”.

CPU can also differ because the sampling periods are different. A PromQL rate calculated over two minutes is not the same thing as a near-current value in the Proxmox UI. Memory is especially easy to misread because Linux MemAvailable calculations and Proxmox container memory accounting answer slightly different questions. Filesystem free/available space can similarly differ from the host-side rootfs figure.

The solution was not to declare one exporter wrong. It was to choose the measurement that matched the purpose of the display. For this control panel, I wanted the figures to represent what Proxmox thinks LXC 108 is using.

Hardware and Existing Infrastructure

The Proxmox host in this setup is QuanChi, a repurposed Shuttle DS81. It is exactly the sort of older hardware I like reusing in the lab: compact, reliable and still perfectly capable of running several infrastructure containers. Current hardware and equivalent product links are maintained on the Realm Labs Hardware page.

If you are building the virtualisation side from scratch, see Proxmox VE – Initial Installation & Setup Guide and Proxmox VE Guide: Deploying Linux, Windows 11, and LXC Containers. The wider monitoring stack is covered in Centralised Homelab Monitoring with Prometheus and Grafana.

1. Install node_exporter Inside the LXC

I kept node_exporter because it is still valuable for metrics from inside KITT-AI. On the Debian-based LXC:

apt update
apt install prometheus-node-exporter -y
systemctl enable --now prometheus-node-exporter
systemctl status prometheus-node-exporter

node_exporter listens on port 9100. The Prometheus job for KITT-AI is:

  - job_name: 'kitt_ai'
    static_configs:
      - targets: ['192.168.68.41:9100']

For this project I retained node_exporter specifically for the swap percentage because the PVE exporter metrics available in this setup did not expose a per-LXC swap metric.

2. Install prometheus-pve-exporter on the Proxmox Host

QuanChi already had Python 3.13. I installed pipx so the exporter could live in an isolated Python environment instead of modifying the system Python packages:

apt update
apt install -y pipx
pipx ensurepath
pipx install prometheus-pve-exporter

The executable was installed under root’s pipx directory. I deliberately use the absolute path in the systemd service so the service does not depend on an interactive shell’s PATH:

/root/.local/bin/pve_exporter

3. Create a Read-Only Proxmox Account

I did not want the exporter using the Proxmox root account. Instead I created a dedicated account and gave it the built-in PVEAuditor role:

pveum user add prometheus@pve --comment "Prometheus PVE Exporter"

pveum aclmod / \
  -user prometheus@pve \
  -role PVEAuditor

pveum user token add prometheus@pve exporter \
  --privsep 0

The token value is a secret. Store it in the exporter configuration and do not publish it, commit it to Git, or paste it into screenshots or documentation.

You can confirm the user and ACL with:

pveum user list | grep prometheus
pveum acl list | grep prometheus

4. Configure PVE Exporter

I created a protected configuration directory and file:

mkdir -p /etc/prometheus
chmod 700 /etc/prometheus
nano /etc/prometheus/pve.yml

The configuration is:

default:
  user: prometheus@pve
  token_name: exporter
  token_value: YOUR_TOKEN_VALUE
  verify_ssl: false

Then protect it:

chmod 600 /etc/prometheus/pve.yml

verify_ssl: false is appropriate to this particular internal lab setup using the local Proxmox certificate. If your Proxmox API is using a certificate trusted by the exporter host, enable certificate verification instead.

5. Test the Exporter Before Making It Permanent

I first started it manually on port 9221:

/root/.local/bin/pve_exporter \
  --config.file /etc/prometheus/pve.yml \
  --web.listen-address 0.0.0.0:9221

From another shell I queried LXC 108:

curl -s "http://127.0.0.1:9221/pve?target=127.0.0.1" \
  | grep 'id="lxc/108"' \
  | head

The useful metrics included:

pve_up{id="lxc/108"}
pve_disk_size_bytes{id="lxc/108"}
pve_disk_usage_bytes{id="lxc/108"}
pve_memory_size_bytes{id="lxc/108"}
pve_memory_usage_bytes{id="lxc/108"}
pve_cpu_usage_ratio{id="lxc/108"}
pve_cpu_usage_limit{id="lxc/108"}
pve_uptime_seconds{id="lxc/108"}

This was the important moment: calculating RAM and disk from those values produced essentially the same percentages as the Proxmox summary for KITT-AI.

6. Run PVE Exporter with systemd

Once the manual test worked, I created /etc/systemd/system/prometheus-pve-exporter.service:

[Unit]
Description=Prometheus Proxmox VE Exporter
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
ExecStart=/root/.local/bin/pve_exporter --config.file /etc/prometheus/pve.yml --web.listen-address 0.0.0.0:9221
Restart=always
RestartSec=5

[Install]
WantedBy=multi-user.target

Then enabled it:

systemctl daemon-reload
systemctl enable --now prometheus-pve-exporter
systemctl status prometheus-pve-exporter --no-pager

A final local check confirms the service still exposes the LXC:

curl -s "http://127.0.0.1:9221/pve?target=127.0.0.1" \
  | grep 'id="lxc/108"' \
  | head

7. Add the Proxmox Exporter to Prometheus

My Prometheus configuration already included Prometheus itself, RabbitMQ and KITT-AI. I added a separate proxmox_pve job rather than replacing the existing node_exporter job:

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['192.168.68.20:9090']

  - job_name: 'rabbitmq'
    static_configs:
      - targets: ['192.168.68.42:15692']

  - job_name: 'rabbitmq_detailed'
    metrics_path: /metrics/detailed
    params:
      family:
        - queue_coarse_metrics
        - queue_consumer_count
      vhost:
        - /kitt
    static_configs:
      - targets: ['192.168.68.42:15692']

  - job_name: 'kitt_ai'
    static_configs:
      - targets: ['192.168.68.41:9100']

  - job_name: 'proxmox_pve'
    metrics_path: /pve
    params:
      target:
        - '192.168.68.10'
    static_configs:
      - targets: ['192.168.68.10:9221']

After reloading Prometheus, this query confirms that LXC 108 is available:

pve_up{id="lxc/108"}

8. PromQL Used by the KITT Dashboard

The final control-centre display uses the Proxmox exporter for CPU, RAM and disk, and node_exporter for swap.

CPU

pve_cpu_usage_ratio{id="lxc/108",job="proxmox_pve"} * 100

RAM

100 * pve_memory_usage_bytes{id="lxc/108",job="proxmox_pve"}
    / pve_memory_size_bytes{id="lxc/108",job="proxmox_pve"}

Swap

100 * (
  1 -
  node_memory_SwapFree_bytes{job="kitt_ai"}
  / node_memory_SwapTotal_bytes{job="kitt_ai"}
)

Disk

100 * pve_disk_usage_bytes{id="lxc/108",job="proxmox_pve"}
    / pve_disk_size_bytes{id="lxc/108",job="proxmox_pve"}

During testing, the Proxmox-derived values were approximately 0.13% CPU, 3.33% RAM and 25.76% disk. More importantly, the RAM and disk figures now matched the Proxmox UI rather than the different guest-side calculation.

9. Feeding the Metrics into the Android KITT Control Centre

The Android app does not need direct access to Proxmox. It queries the existing Prometheus HTTP API. That keeps the Proxmox API token on the server and out of the APK.

Android Control Centre
        |
        | PromQL via HTTP
        v
Prometheus :9090
   |             |
   v             v
PVE Exporter   node_exporter
   |             |
   v             v
Proxmox API    KITT-AI LXC

The Kotlin client queries /api/v1/query, URL-encodes the PromQL and extracts the first returned value. The health object contains:

online
cpu
ram
swap
disk

The online state now comes from the Proxmox view as well:

pve_up{id="lxc/108",job="proxmox_pve"}

This is deliberate. If Proxmox says the KITT-AI container is down, the dashboard should treat the monitored system as offline even if some stale or unrelated metric remains available elsewhere.

10. The v1.03 Control-Centre Display

The final Android panel is deliberately small and period-inspired rather than a miniature Grafana dashboard. CPU, RAM, SWAP and DISK are displayed as percentage readouts with chunky segmented bars. The active segments and values use KITT red, while yellow labels provide contrast against the dark console.

The app also runs in immersive fullscreen mode on the wall/control tablet, removing the Android status/navigation chrome from the normal dashboard view. Elsewhere on the same screen, Home Assistant door sensors show the state and battery level of the front, back, bar and garage doors.

That gives the console three distinct information layers: Home Assistant for physical state, Prometheus for infrastructure health, and the KITT personality/TTS stack for interaction. The voice side of the project is covered in Building a Voice-Activated KITT Smart Home Dashboard, while the reliable TTS queue is documented in Adding RabbitMQ to My Homelab for Reliable KITT TTS Jobs.

Why I Kept Both Exporters

It would have been tempting to remove node_exporter after adding PVE Exporter, but that would throw away useful guest-level visibility.

I now think of the two sources like this:

PVE Exporter
  Proxmox's view of the guest
  - container state
  - CPU
  - allocated/used memory
  - root disk usage
  - uptime
  - host-side guest statistics

node_exporter
  Linux's view from inside the guest
  - swap
  - filesystems
  - network interfaces
  - Linux memory detail
  - load
  - OS-level metrics

Neither replaces the other. They answer different questions.

Security Notes

There are a few details I would not skip in a permanent setup:

  • Use a dedicated Proxmox account rather than root.
  • Give the exporter read-only PVEAuditor access.
  • Use an API token and protect the configuration file with restrictive permissions.
  • Never put the Proxmox token in the Android app.
  • Keep the exporter and Prometheus interfaces on trusted management networks or firewall them appropriately.
  • If you have a trusted TLS certificate on Proxmox, verify it rather than disabling certificate validation.

Troubleshooting

The exporter command is not found after pipx install

pipx may install the binary into /root/.local/bin without that path being active in the current shell. Check:

ls -l /root/.local/bin/

Using the full path in systemd avoids this problem entirely.

pve_exporter –version returns an error

The build used here did not implement a --version argument. Seeing its help/usage output still confirmed that the executable itself was working.

Prometheus returns no PVE data

Test the exporter directly before troubleshooting Prometheus:

curl -s "http://127.0.0.1:9221/pve?target=127.0.0.1" | head

Then confirm the Prometheus target is up and query pve_up{id="lxc/108"}.

node_exporter still does not match Proxmox

That is the central point of this setup: do not expect every guest-side Linux metric to match host-side Proxmox accounting exactly. Decide which viewpoint the dashboard is intended to represent.

Where This Fits in Realm Labs

This monitoring work sits at the intersection of several Realm Labs projects. Proxmox supplies the virtualisation layer, Prometheus and Grafana provide the monitoring backbone, Home Assistant provides device state and automation, and KITT brings those systems together in a custom control interface.

The physical host is documented on the Hardware Behind Realm Labs page, and the broader infrastructure is mapped on The Realm Labs Homelab.

Final Result

The useful lesson was not simply how to install another exporter. It was understanding that a virtualised system can have more than one valid view of resource usage.

For KITT Control Centre v1.03 I wanted the console to agree with Proxmox, so the final split is simple:

CPU   -> PVE Exporter
RAM   -> PVE Exporter
DISK  -> PVE Exporter
SWAP  -> node_exporter

Prometheus remains the single query point for the Android app, so the client does not need to know how each measurement was collected. That separation also leaves plenty of room to add RabbitMQ, TTS service health and other KITT subsystems later without turning the control-centre app into an infrastructure client.

For more of the build, see the KITT smart-home dashboard project, the RabbitMQ TTS integration, and the Realm Labs monitoring stack.