Monitoring a Mixed Fleet with Grafana and Prometheus

The Box, and Why One Small One Is Enough

The instinct with monitoring is to assume it needs a real server, because the enterprise version of this involves a cluster. It doesn’t. Prometheus is a single Go binary with a local time-series database, and the workload for a fleet this size is close to nothing.

This runs on a 4 GB, 2 vCPU VPS with NVMe storage, at $16/month on an ongoing plan. That’s more headroom than the job strictly needs, and it’s shared with other things. The reason to have NVMe rather than spinning disk is the only performance note worth making: Prometheus writes continuously and compacts on a schedule, so it likes fast storage more than it likes cores.

The question people actually want answered is how much disk the metrics eat, and that has a real formula rather than a shrug:

disk = retention_period * samples_per_second * bytes_per_sample
samples_per_second = (total_active_series) / scrape_interval

Prometheus compresses well, landing somewhere around 1 to 2 bytes per sample in practice. A node_exporter with the usual collectors publishes on the order of 1,000 active series; windows_exporter is in the same neighborhood. So for ten targets at a 15 second scrape interval:

10,000 series / 15s     = ~667 samples/sec
667 * 2 bytes           = ~1.3 KB/sec
                        = ~115 MB/day
                        = ~3.5 GB/month

At 90 days retention that’s roughly 10 GB. Treat that as an order-of-magnitude estimate, not a measurement, because active series count varies a lot with which collectors you enable, and the DCGM and UPS exporters in this fleet are far smaller contributors than the node and Windows exporters. The useful conclusion is the shape of the answer: a fleet like this costs single-digit gigabytes per month, not hundreds, and a small VPS is genuinely the right size. If you want the real number for your setup rather than this arithmetic, Prometheus reports it at /api/v1/status/tsdb.

Set retention deliberately rather than leaving the default. --storage.tsdb.retention.time accepts things like 90d, and --storage.tsdb.retention.size gives you a hard ceiling so a runaway cardinality problem fills a bounded amount of disk instead of all of it. Setting both is the belt-and-suspenders version, and the size limit is the one that saves you.

Pull, Not Push, and Why That Decides Everything

Prometheus is a pull-based system. It does not sit and wait for machines to report in; on a timer, it opens an HTTP connection to each target, requests /metrics, and gets back a block of plain text that it parses and stores. Every design consequence in this guide falls out of that one sentence.

An exporter, then, is not an agent in the usual sense. It’s a tiny HTTP server that translates something (kernel counters, Windows performance counters, an NVIDIA driver, a UPS daemon) into that text format and waits to be asked. It has no idea Prometheus exists, it never phones home, and if nothing scrapes it, it does nothing at all. That’s why adding a new machine to this fleet is two steps: run an exporter on it, then add three lines to a scrape config.

The uncomfortable consequence is the one everybody hits: the monitoring server must be able to reach every target, not the other way around. For a rack of servers on one subnet, that’s free. For a fleet that includes a laptop at home and a VPS in a datacenter, it is the entire problem, and it’s why so many people give up and run Grafana locally instead.

The wrong answers are worth naming so you can skip them. Port-forwarding each machine through a home router means exposing node_exporter to the internet, and node_exporter has no authentication whatsoever; anyone who finds it gets a detailed inventory of your machine. Pushing metrics out instead, via Pushgateway, is explicitly documented as the wrong tool for this: it’s for ephemeral batch jobs, it breaks the up/down semantics that make alerting work, and it turns into a single point of failure. A VPN back to the house works but is a heavier thing to run and keep running.

The right answer is a mesh network, and that’s the next step.

The Scrape Network

Tailscale solves the reachability problem cleanly enough that it stops being a topic. Every machine in the fleet joins the tailnet, each gets a stable 100.x.y.z address that works regardless of which network it’s physically on, and the VPS scrapes those addresses. The Windows laptop behind NAT is reachable at the same address whether it’s at home or on someone else’s wifi. No ports are forwarded, nothing is exposed to the public internet, and the connection is encrypted without you configuring TLS on ten exporters.

It’s also already the access path for the rest of this infrastructure, which matters more than it sounds. If Tailscale is how you reach these boxes for SSH anyway, monitoring rides along on a thing you already run, already trust, and already have a recovery story for. This site’s VPS Security Foundations guide covers that base setup.

The part worth doing carefully is what the exporters bind to. An exporter listening on 0.0.0.0 is reachable on every interface the machine has, including whatever coffee-shop network the laptop is on, and it’s unauthenticated by design. There are two defensible postures:

• Bind to the tailnet address only. Most exporters take a flag for it, such as --web.listen-address=100.x.y.z:9100. Now the listener does not exist on any other network. This is the tighter option.
• Bind everywhere, restrict with the firewall. Leave the default and add a rule allowing the port only from the tailnet CGNAT range, 100.64.0.0/10. Slightly looser, considerably more robust, for the reason immediately below.

That reason is a genuine trap, and it’s the same one this site’s hardening guide raises about binding sshd to a Tailscale address. A service bound to a 100.x address will fail to start if it starts before tailscaled has brought the interface up. The address does not exist yet, the bind fails, and the service exits. On reboot you don’t get an obvious error, you get a target that silently never comes back, and you find out days later when a dashboard has a hole in it. If you bind to the tailnet address, order the unit after Tailscale with a systemd drop-in (After=tailscaled.service, Wants=tailscaled.service) and add a restart policy so a lost race self-heals rather than requiring a visit.

For Windows machines, the firewall-rule approach is usually less friction than fighting the service configuration, and the equivalent rule scopes the inbound allow on the exporter’s port to the same 100.64.0.0/10 range.

The Ordinary Targets

Two of the five collection shapes are the conventional ones, and they’re conventional for a reason: they work, they’re well-maintained, and the community dashboards for them are genuinely good.

Linux, via node_exporter. This covers the three Raspberry Pis, the GPU box, and the VPS itself. It publishes CPU, memory, disk, filesystem, network, load, and temperature metrics, and on the Pis the thermal data is not decorative given how they behave under sustained load in a warm room. It runs on ARM without ceremony, so the Pis are not a special case. Two collectors are worth enabling beyond the defaults, --collector.systemd and --collector.processes, because the standard dashboard expects them and panels come up empty without them.

Windows, via windows_exporter. This covers the three laptop-servers. It’s the same idea translated to Windows performance counters, it installs as a proper Windows service from an MSI, and it defaults to port 9182 rather than 9100. Collector selection matters more here than on Linux, since the available set is large and enabling all of it produces a lot of series you will never look at. The defaults plus service and logical_disk cover most of what you actually want.

The scrape configuration is the same shape for both, and this is the entire “adding a machine” ritual:

scrape_configs:
  - job_name: node
    static_configs:
      - targets:
          - '100.x.y.z:9100'   # vps
          - '100.x.y.z:9100'   # rpi-1
          - '100.x.y.z:9100'   # rpi-2
          - '100.x.y.z:9100'   # rpi-3
          - '100.x.y.z:9100'   # gpu box

  - job_name: windows
    static_configs:
      - targets:
          - '100.x.y.z:9182'   # laptop-1
          - '100.x.y.z:9182'   # laptop-2
          - '100.x.y.z:9182'   # laptop-3

Keep the job names meaningful, because they become the variable you filter dashboards by later. Splitting Linux and Windows into separate jobs rather than one flat list is what lets a dashboard scope itself to one or the other without string-matching hostnames.

One note on the Pis specifically: if they’re running off SD cards, the card is the component most likely to fail in this fleet, and node_exporter gives you enough filesystem and I/O error signal to see it coming. That alone justifies monitoring them.

The GPU

The RTX 3080 box runs Debian, and it’s the clearest example of why “just install node_exporter everywhere” isn’t the whole answer. node_exporter reports that machine’s CPU, memory, and disk perfectly well, and knows nothing at all about the GPU. Utilization, VRAM, temperature, clocks, and power draw come from a completely separate source.

That source is NVIDIA DCGM Exporter, which sits on top of NVIDIA’s Data Center GPU Manager and publishes GPU telemetry on port 9400. It’s a genuine NVIDIA project rather than a community wrapper around nvidia-smi parsing, which is the main reason to prefer it: the metric set is stable and it doesn’t break when driver output formatting changes.

So the GPU box carries two exporters, node_exporter on 9100 and DCGM on 9400, and appears in two scrape jobs. That’s normal and worth internalizing as the general pattern: exporters are per-concern, not per-machine, and a host with three interesting subsystems runs three of them.

  - job_name: gpu
    static_configs:
      - targets:
          - '100.x.y.z:9400'   # gpu box, dcgm

DCGM is aimed at datacenter hardware, and a consumer card is not its primary target. In practice the metrics that matter for a workstation-class AI box (utilization, memory used, temperature, power) are present and correct. Some of the more exotic datacenter fields, profiling metrics and MIG partitioning among them, are either absent or meaningless on a 3080. If a panel in an imported dashboard is stubbornly empty, that’s usually why, and the fix is deleting the panel rather than debugging it.

For the dashboard, NVIDIA publishes an official one at ID 12239, and the JSON also lives in the exporter’s own repository.

Two UPS Units, Two Vendors, Two Daemons

This is the section that earns the guide, because it’s where the tidy story breaks down and you have to make a decision rather than follow a README.

There are two UPS units here. An APC Back-UPS ES 600 protects the Proxmox side, where it also does real work beyond reporting: it signals the host to shut down gracefully once an outage runs long enough, which is a separate concern documented elsewhere. A CyberPower CP1350 protects the AI box. Both are consumer-grade units connected over USB. Both need to end up in Grafana.

They get there by different routes, and the reason is not arbitrary. What a UPS exposes on Linux depends on what software speaks its dialect.

• The APC goes through apcupsd, a daemon written specifically for APC hardware. It speaks APC’s protocol natively, exposes a rich set of fields, and publishes them on a network information port, 3551 by default, which is what its exporter reads.
• The CyberPower goes through NUT, Network UPS Tools, the generic multi-vendor layer. It reaches the CP1350 through the standard USB HID power-device path with the usbhid-ups driver and serves data via upsd, normally on 3493. CyberPower does ship its own vendor daemon, PowerPanel, and NUT is the deliberate choice over it: one generic layer that will also handle the next UPS regardless of brand beats a second vendor-specific daemon.

Each daemon then needs a bridge to Prometheus, and here is the wrinkle. There is an apcupsd exporter that reads the apcupsd NIS port, and a NUT exporter that talks to upsd. They are different projects by different authors, and they do not agree on metric names. Battery charge percentage arrives under one name from one and a different name from the other. Runtime remaining might be seconds in one and minutes in the other.

Which leaves a decision:

Option A: two dashboards. Import or build a panel set per exporter and accept that the APC page and the CyberPower page look different. Zero effort, and completely fine if you check them rarely and separately.

Option B: normalize, and get one dashboard. Use Prometheus metric relabeling at scrape time, or recording rules after the fact, to land both vendors’ battery charge on one series name with a ups label distinguishing them. Then one panel set covers both units and adding a third UPS of any brand is a config entry rather than a new dashboard.

Option B is more work up front and clearly correct if you’re going to look at these together, which is the whole point of having one Grafana. It’s also the transferable idea in this guide: the moment you have two sources for the same concept, the choice is normalize once or special-case forever, and that generalizes well past UPS hardware.

A caution on units. Normalizing names without normalizing units produces a dashboard that is confidently wrong, which is worse than two ugly dashboards. If one exporter reports runtime in seconds and the other in minutes, the conversion belongs in the recording rule, and the rule should be named so the unit is unambiguous, ups_runtime_seconds rather than ups_runtime. Check the actual output of both exporters against the units in their documentation before writing the rule, because guessing here yields a “47 minutes remaining” panel that means 47 seconds.

Grafana, and What This Skips

With Prometheus scraping everything, Grafana is nearly an afterthought, which is the correct amount of attention to give it. Add Prometheus as a data source, point it at the local Prometheus, and start importing.

For dashboards, import first and build second. The community library is good and the maintained dashboards for the common exporters are better than what you would build in an evening:

• 1860, Node Exporter Full is the one everyone uses and deserves the reputation. It expects a job named node and the systemd and processes collectors enabled, which is why Step 4 turned them on.
• 12239, NVIDIA DCGM Exporter is NVIDIA’s official GPU dashboard.
• For windows_exporter there are several actively maintained options and which one is best genuinely moves, so search the dashboard library rather than trusting a fixed ID from any guide, including this one.
• For the UPS units, this is where the Step 6 decision cashes out. Two exporters means either two imported dashboards or one you assemble against normalized names.

Then adjust rather than admire. Imported dashboards ship with panels for hardware you don’t have and thresholds tuned for someone else’s environment. Delete the empty panels, fix the thresholds, and set the variables so the host dropdown actually lists your machines.

What this guide deliberately skips, so you know what you’re not getting:

• Alerting. Grafana and Alertmanager both do it, and a monitoring stack without alerts is a stack you have to remember to look at. This is the most defensible next thing to add, and the single highest-value alert is not a CPU threshold, it’s up == 0 on any target, because a dead exporter is how you discover a dead machine.
• Logs. Loki is the companion piece for log aggregation and it’s a different enough problem to deserve separate treatment rather than a paragraph here.
• Long-term storage. Thanos and Mimir exist for retention beyond what a single Prometheus should hold. For a fleet this size, local retention with a size cap is the right answer and these are the wrong tool.
• The Proxmox host as a target. It can join this fleet, but Proxmox is better served by pve-exporter, which reports guests, storage, and cluster state rather than just the host’s kernel counters. That belongs with this site’s Proxmox guide rather than bolted on here.
• High availability. One Prometheus, one Grafana, one box. If it dies you lose monitoring and, with the default setup, history. That’s an acceptable trade for a home fleet and would not be for anything load-bearing.

What You Spent

Nothing, more or less, and that’s the honest summary. Every component here is free and open source: Prometheus, Grafana OSS, node_exporter, windows_exporter, DCGM Exporter, apcupsd, and NUT all cost zero. Tailscale’s free tier covers up to 100 devices, which is an order of magnitude more than this fleet.

The only real line item is the box:

• VPS: $16/month for 4 GB, 2 vCPU, and NVMe storage, on an ongoing plan, and shared with other services rather than dedicated to this
• Everything else: $0

Two caveats on that number. It’s a snapshot from a specific plan at a specific time, and hosting prices moved sharply during 2026, so verify current pricing rather than budgeting off it. And it’s generous for the workload: the arithmetic in Step 1 puts a fleet this size in single-digit gigabytes per month and well under a gigabyte of working memory, so a smaller box would do it. If you already run a VPS for anything else, the marginal cost of adding this to it is genuinely zero, which is the strongest argument for doing it at all.

Against a hosted alternative, the comparison is less lopsided than it looks. Free tiers at the hosted monitoring vendors are real and workable at this scale, and they buy you alerting, retention, and an on-call story without running anything. What self-hosting buys is that the data stays on hardware you control, the retention is whatever your disk allows rather than what a free tier permits, and you can monitor a laptop in your house without routing its metrics through someone else’s cloud. For a fleet that is mostly personal hardware, that’s the right trade.