Grafana and Prometheus migration from Synology Docker to Proxmox LXC, preserving dashboards and historical monitoring data
Migrating the Grafana, Prometheus and SNMP monitoring stack from Synology Docker to Proxmox LXCs.

Moving Grafana and Prometheus from Synology Docker to Proxmox LXC — Without Losing History

A real homelab migration from Synology Docker to Proxmox LXCs: Grafana dashboards, Prometheus TSDB history, SNMP Exporter, Synology-hosted YAML sync, NFS gotchas, rollback and actual RAM savings.

Why I moved the monitoring stack

My Synology DS224+ (Seido) was hosting far more than files: Samba Active Directory, DNS, WordPress, Home Assistant, Plex and 18 Docker containers. Grafana, Prometheus and SNMP Exporter were good candidates to move to my Proxmox server (QuanChi), without moving databases or NAS-dependent services unnecessarily.

Before the move, Seido had 5.6 GiB usable RAM, 3.9 GiB used, only 994 MiB available and 2.4 GiB swap occupied. A point-in-time docker stats --no-stream showed Grafana at 671 MiB, Prometheus at 143 MiB and SNMP Exporter at 21 MiB. The goal wasn’t to make two percentage gauges match: it was to reduce actual memory pressure on the NAS.

The layout

Component Original Destination
Grafana Seido Docker QuanChi CT 109, 192.168.68.43:3000
Prometheus Seido Docker QuanChi CT 111, 192.168.68.45:9090
SNMP Exporter Seido Docker CT 111, localhost:9116
Master YAML Seido Sektor Still edited on Seido; validated local copies run in CT 111
Metrics and Grafana database Seido Docker bind mounts Local mSata storage on QuanChi

The IP addresses above are my homelab addresses. Substitute your own. CT IDs, mount paths, export rules and network gateway will differ from one installation to another.

1. Create the two Proxmox LXCs

I used the community Proxmox VE Helper Scripts for Grafana and Prometheus rather than Docker-in-a-VM. After installation, I set static addresses on Proxmox’s vmbr0 network and pointed DNS at Seido. Preserve existing NIC MAC addresses and VLAN settings when editing net0; the Proxmox Network tab is a safe way to do this.

pct list
pct config 109 | grep net0
pct config 111 | grep net0

For reference, Grafana was v13.2.2 in Docker and v13.2.3 in the new LXC. Prometheus moved from v3.14.0 to v3.15.0. Check compatibility rather than blindly moving data across major versions.

2. Check available storage before resizing anything

pvesm status
pct config 109 | grep rootfs
pct config 111 | grep rootfs
pct exec 109 -- df -h /
pct exec 111 -- df -h /var/lib/prometheus

This mattered: QuanChi’s local-lvm was nearly full (97.9%), and the mSata storage had only about 9.8 GiB free. Both monitoring LXCs were already on mSata with 4 GB root disks. Grafana used about 2.7 GB after the helper-script installation, mostly from /usr/share/grafana. I kept the allocations unchanged because the restored Grafana data would replace, rather than add to, the fresh database. Prometheus had 2.6 GB free—enough for the 859 MB dataset at migration time, but future growth needs monitoring.

3. Locate and back up Grafana

sudo docker inspect grafana prometheus snmp-exporter \
  --format '{{.Name}} {{json .Mounts}}'

My Grafana bind mount was /volume1/Sektor/nfs/docker-volumes/grafana; Prometheus’s data directory was nested within it at grafana/prometheus. That nesting is easy to miss. The Grafana SQLite file (grafana.db) was about 32 MB, while its non-Prometheus directory was about 485 MB.

I stopped the old Grafana container for a consistent SQLite backup, then archived the mounted directory while excluding the nested Prometheus database. Keep original containers and files for rollback.

sudo docker stop grafana
sudo tar -czpf /volume1/Sektor/grafana-migration.tar.gz \
  -C /volume1/Sektor/nfs/docker-volumes/grafana \
  --exclude='./prometheus' .
sudo tar -tzf /volume1/Sektor/grafana-migration.tar.gz | grep './grafana.db'

Also inspect the old Docker environment and grafana.ini for any explicitly configured encryption key, provisioned data sources or custom paths. Never publish keys or credentials. A database may load fine while encrypted data-source passwords fail if the key changes.

4. Restore Grafana into CT 109

An ordinary scp attempt failed with subsystem request failed on channel 0. Newer OpenSSH clients use SFTP by default; my Synology SSH configuration wasn’t serving that subsystem. Forcing legacy SCP fixed it:

scp -O [email protected]:/volume1/Sektor/grafana-migration.tar.gz /root/

After stopping the new Grafana service, I preserved its fresh data directory and restored the archive. Because CT 109 is unprivileged, tar initially complained about ownership mapping; extracting without source ownership avoids that error.

pct exec 109 -- systemctl stop grafana-server
pct exec 109 -- mv /var/lib/grafana /var/lib/grafana-fresh
pct exec 109 -- mkdir /var/lib/grafana
pct exec 109 -- tar --no-same-owner --no-same-permissions \
  -xzpf /dev/stdin -C /var/lib/grafana \
  < /root/grafana-migration.tar.gz
pct exec 109 -- chown -R grafana:grafana /var/lib/grafana
pct exec 109 -- chmod 750 /var/lib/grafana
pct exec 109 -- chmod 640 /var/lib/grafana/grafana.db

Plugin directories and binaries may also need sensible permissions; don’t indiscriminately remove executable bits from plugin binaries. Once encryption settings were checked, Grafana started successfully and the original users and dashboards were present. I later changed its Prometheus data-source URL to http://192.168.68.45:9090.

5. Keep configuration on Seido, but databases local

I wanted to continue editing YAML in the familiar \\Seido\Sektor share. Rather than placing live SQLite or Prometheus TSDB data over NFS, I mounted the Sektor NFS export on the Proxmox host and used local, synchronised copies inside the LXCs. That means monitoring can still start with the last known-good configuration even when Seido is temporarily unavailable.

I added QuanChi’s IP to Sektor’s NFS permissions in DSM, mounted the share for testing, then added a read-only systemd automount entry in /etc/fstab:

192.168.68.20:/volume1/Sektor /mnt/seido-sektor nfs4 ro,noauto,x-systemd.automount,_netdev,nofail,x-systemd.mount-timeout=30s 0 0

Editing fstab didn’t retroactively turn the original manual read/write mount into read-only. A remount attempt reported the NFS mount as busy, so I checked the mount state, unmounted the original connection safely and activated the automount.

cd /root
umount /mnt/seido-sektor
systemctl daemon-reload
systemctl start 'mnt-seido\x2dsektor.automount'
ls /mnt/seido-sektor/nfs/docker-volumes/grafana/
findmnt -no SOURCE,FSTYPE,OPTIONS /mnt/seido-sektor

Confirm ro appears among the options. Don’t force-unmount an NFS share that is actively in use.

6. Migrate the Prometheus scrape configuration

The master configuration was already at /volume1/Sektor/nfs/docker-volumes/grafana/prometheus/prometheus.yml. On the new server I changed the self-scrape from Seido’s IP to 192.168.68.45:9090 and the two SNMP relabeling destinations from Docker’s snmp-exporter:9116 hostname to 127.0.0.1:9116. I also removed obsolete Windows-exporter targets, one of which clashed with Prometheus’s new IP.

pct exec 111 -- cp /etc/prometheus/prometheus.yml /etc/prometheus/prometheus.yml.bak
pct push 111 /mnt/seido-sektor/nfs/docker-volumes/grafana/prometheus/prometheus.yml /etc/prometheus/prometheus.yml
pct exec 111 -- /usr/local/bin/promtool check config /etc/prometheus/prometheus.yml
pct exec 111 -- systemctl restart prometheus

The Prometheus helper script installed its executable at /usr/local/bin/prometheus, so running pct exec 111 -- prometheus --version without a full path initially failed, even though systemd showed Prometheus running normally.

7. Migrate the SNMP Exporter and troubleshoot an old broken module

The original SNMP Exporter (v0.30.1) had no Docker volume mounts. Its snmp.yml lived inside the image. I extracted it to Sektor for ongoing editing:

sudo mkdir -p /volume1/Sektor/nfs/docker-volumes/grafana/snmp
sudo docker cp snmp-exporter:/etc/snmp_exporter/snmp.yml \
  /volume1/Sektor/nfs/docker-volumes/grafana/snmp/snmp.yml

A useful discovery: the YAML contained the standard if_mib module but not my old hp_custom module. Tests against the original exporter returned HTTP 200 for if_mib and HTTP 400 for hp_custom. That job was already broken before migration. I removed the redundant custom scrape job pending a proper module definition rather than collecting duplicated switch statistics.

I installed SNMP Exporter 0.30.1 in CT 111, copied the snmp.yml locally and created a systemd service running the exporter on 127.0.0.1:9116. A test proved the new exporter could scrape the switch:

pct exec 111 -- curl -s -o /dev/null \
  -w 'SNMP HTTP %{http_code}\n' \
  'http://127.0.0.1:9116/snmp?target=192.168.68.2&module=if_mib'

Result: SNMP HTTP 200. SNMP configuration may contain community strings or credentials, so it belongs in a permission-restricted share, not in a publicly downloadable sample.

8. Restore Prometheus history safely

The historical data reached 859 MB by the final backup. I stopped the old Prometheus Docker container before archiving the TSDB blocks and WAL. This matters: copying a live TSDB with plain tar can produce an inconsistent database.

sudo docker stop prometheus
sudo tar -czpf /volume1/Sektor/prometheus-migration.tar.gz \
  -C /volume1/Sektor/nfs/docker-volumes/grafana/prometheus \
  --exclude='./prometheus.yml' \
  --exclude='./prometheus - Copy.yml' \
  --exclude='./@eaDir' .

The compressed archive was about 341 MB. With Sektor already mounted on QuanChi, no additional SCP transfer was necessary. I stopped the new service, preserved its small fresh dataset, and restored the historical archive without trying to preserve Synology’s unmappable numeric file owners.

pct exec 111 -- systemctl stop prometheus
pct exec 111 -- mv /var/lib/prometheus /var/lib/prometheus-fresh
pct exec 111 -- mkdir /var/lib/prometheus
pct exec 111 -- tar --no-same-owner --no-same-permissions \
  -xzpf /dev/stdin -C /var/lib/prometheus \
  < /mnt/seido-sektor/prometheus-migration.tar.gz
pct exec 111 -- systemctl start prometheus
pct exec 111 -- journalctl -u prometheus -n 30 --no-pager

Prometheus reported healthy TSDB blocks, completed WAL replay and announced that the server was ready for web requests. Historical graphs also returned in Grafana after the data-source URL was changed. Its current log showed the default 15-day retention, something to revisit only after monitoring disk growth.

9. Automatically sync YAML from Seido

To avoid manual copying, I added a monitoring-config-sync.service and monitoring-config-sync.timer on QuanChi. Every minute the script checks the Seido source files, pushes staged copies into CT 111, compares them with the running files and validates the Prometheus YAML with promtool before copying and reloading.

For SNMP, the script launches a temporary exporter on an unused port to check that its YAML loads, backs up the current file, installs the replacement and restarts the exporter. If the restart fails, it attempts to restore the previous file. This is a pragmatic home-lab solution rather than a full transactional deployment system: add stronger post-reload health checks, SNMP test assertions and alerting before relying on it for critical infrastructure.

The script itself was saved as /usr/local/sbin/sync-monitoring-config.sh on QuanChi (chmod 700). This is the version used in the migration; see the limitations above before adapting it for production infrastructure:

#!/bin/bash
set -euo pipefail
SOURCE="/mnt/seido-sektor/nfs/docker-volumes/grafana"
CT=111
log() { echo "$(date '+%F %T') $*"; }

if [[ ! -r "$SOURCE/prometheus/prometheus.yml" ||
      ! -r "$SOURCE/snmp/snmp.yml" ]]; then
  log "Seido unavailable - keeping existing configuration"
  exit 0
fi

pct push "$CT" "$SOURCE/prometheus/prometheus.yml" /tmp/prometheus-new.yml
if ! pct exec "$CT" -- cmp -s /tmp/prometheus-new.yml /etc/prometheus/prometheus.yml; then
  if pct exec "$CT" -- /usr/local/bin/promtool check config /tmp/prometheus-new.yml; then
    pct exec "$CT" -- cp /etc/prometheus/prometheus.yml /etc/prometheus/prometheus.yml.bak
    pct exec "$CT" -- install -m 644 /tmp/prometheus-new.yml /etc/prometheus/prometheus.yml
    pct exec "$CT" -- systemctl reload prometheus
    log "Prometheus configuration updated"
  else
    log "INVALID Prometheus YAML - update skipped"
  fi
fi

pct push "$CT" "$SOURCE/snmp/snmp.yml" /tmp/snmp-new.yml
if ! pct exec "$CT" -- cmp -s /tmp/snmp-new.yml /etc/snmp_exporter/snmp.yml; then
  set +e
  pct exec "$CT" -- timeout 5s /usr/local/bin/snmp_exporter \
    --config.file=/tmp/snmp-new.yml --web.listen-address=127.0.0.1:19116
  result=$?
  set -e
  if [[ "$result" -eq 124 ]]; then
    pct exec "$CT" -- cp /etc/snmp_exporter/snmp.yml /etc/snmp_exporter/snmp.yml.bak
    pct exec "$CT" -- install -m 600 /tmp/snmp-new.yml /etc/snmp_exporter/snmp.yml
    if pct exec "$CT" -- systemctl restart snmp-exporter &&
       pct exec "$CT" -- systemctl is-active --quiet snmp-exporter; then
      log "SNMP Exporter configuration updated"
    else
      pct exec "$CT" -- cp /etc/snmp_exporter/snmp.yml.bak /etc/snmp_exporter/snmp.yml
      pct exec "$CT" -- systemctl restart snmp-exporter
      log "SNMP restart failed - restored previous configuration"
      exit 1
    fi
  else
    log "SNMP configuration validation failed - update skipped"
  fi
fi

The timer was enabled with:

systemctl daemon-reload
systemctl enable --now monitoring-config-sync.timer
systemctl start monitoring-config-sync.service
journalctl -u monitoring-config-sync.service -n 30 --no-pager
systemctl list-timers --all | grep monitoring

For an end-to-end test, I changed scrape_interval from 15s to 20s on Seido. At 22:25:45, QuanChi logged Prometheus configuration updated. The service then completed successfully. After testing, the scrape interval should be returned to 15s in Seido’s master YAML if the change was only temporary.

10. Verify resource savings and retire the old stack

Seido metric Before After
RAM used 3.9 GiB 3.2 GiB
RAM available 994 MiB 1.7 GiB
Cache 1.4 GiB 2.1 GiB
Swap occupied 2.4 GiB 2.4 GiB

Afterwards, Seido was around 60% memory usage and QuanChi around 65%. The swap allocation stayed at 2.4 GiB, but that alone does not mean active memory pressure. Synology didn’t have vmstat available, so I measured the kernel counters over 10 seconds: pswpin rose by just 6 pages and pswpout didn’t increase at all. That is negligible swap activity during that sample.

grep -E '^pswpin |^pswpout ' /proc/vmstat
sleep 10
grep -E '^pswpin |^pswpout ' /proc/vmstat

After verifying Grafana dashboards, Prometheus targets and the automated YAML sync, I disabled automatic restart of the original Seido Docker containers while keeping their data and migration archives available for rollback:

sudo docker update --restart=no grafana prometheus snmp-exporter
sudo docker stop grafana prometheus snmp-exporter
sudo docker inspect grafana prometheus snmp-exporter \
  --format '{{.Name}} | Restart: {{.HostConfig.RestartPolicy.Name}} | Running: {{.State.Running}}'

If Portainer or Compose still manages the original stack, a redeployment may recreate those containers or restart policies. Retire the old stack definition only after the new service and backups are verified.

Lessons learned

  • Don’t compare RAM percentages alone. Check MemAvailable, swap activity and the actual container consumers.
  • Keep SQLite and Prometheus TSDB on local storage; centralise the editable YAML rather than the write-heavy databases.
  • Check NFS export permissions, active mount flags and LXC UID mappings before restoring.
  • Always validate monitoring targets. Some failures predate a migration, as the missing hp_custom module demonstrated.
  • Back up first, stop databases for consistent filesystem copies and keep the old containers as a rollback until verification is complete.
  • Automate routine YAML changes, but retain local last-known-good files so monitoring survives a NAS outage.

Result: The monitoring stack is now isolated in Proxmox LXCs, configuration remains easy to edit on Synology, historical metrics are intact and the NAS has recovered roughly 700 MiB of used RAM. That’s a worthwhile migration without moving services merely for the sake of it.