Lemonfiber
← The specification

Diagnostics

Rendered from spec at76d46f3 (2026-10-25) · its source · Edit this page

Status: Accepted · Audience: Operator · Area: C — Trust & correctness

Purpose

Answer “is this actually working?” with evidence rather than optimism.

Every serious failure mode in this ecosystem is silent. A VPN that failed open still reports Up. Imports that degraded from hardlink to copy still succeed. An indexer returning nothing looks identical to an indexer with nothing to return. The stack reports green while doing the wrong thing, and the operator finds out weeks later — from a full disk, a tracker ban, or a letter.

Diagnostics is the direct expression of P3: where a claim is checkable, check it.

Behaviour

Checks prove, they don’t infer

A check that reads configuration and concludes “hardlinks should work” is worthless. A check that creates a hardlink, stats it, and compares inode and link count has established a fact.

Weak Strong
Filesystem is APFS, so hardlinks work Created a link; inode matched; link count 2
Gluetun container is running Compared public IP inside qBittorrent and Gluetun; identical
Port 8989 is configured Bound it; it was free
API key is present Called the API; it answered with the service identity

Every result carries a remedy

A finding without a remedy is a dead end. Findings state what was observed, what it means, and what to do — per G4.

Severity is honest

Severity Meaning
pass Verified working
warn Working, but degraded or risky
fail Not working; something is broken
unverified Could not be checked. Not a pass.

unverified is the most important and the most often omitted. A killswitch that hasn’t been tested is not a working killswitch; reporting it green would be a lie of exactly the kind this feature exists to prevent.

Checks are independent

One check failing never prevents others from running. A stopped Docker daemon makes container checks impossible, but filesystem and configuration checks still run and still report.

Runnable in whole or in part

The full suite, a single check, or a category. Checks are re-runnable at any time and cheap enough to run often.

Non-disruptive by default

Some checks can only be performed by disturbing the system — proving a killswitch actually blocks traffic requires dropping the tunnel. Those are opt-in, never part of a default run, and state what they will disturb and for how long.

Check catalogue

Category Checks
Environment Docker present, daemon reachable, Compose version, platform detected, virtualisation backend
Storage Data root exists and is writable, single-filesystem, hardlink test, free space, projected exhaustion, permissions
Network Port availability, port conflicts, service reachability, LAN binding matches policy
VPN Tunnel up, egress match, forwarded port assigned, port matches client, killswitch (disruptive)
Credentials Each credential still valid, freshness
Services Health, crash loops, version skew, inter-service wiring intact, a series the media server holds episodes for and answers no seasons for
Providers Quota remaining, subscription validity, indexer responsiveness
Queue Stuck items, repeated import failures, orphaned downloads
Config Drift from lemonfiber-managed state, file permissions, manifest validity

Machine-readable output

Diagnostics emit structured output as well as human output, so results can be scripted, monitored, or attached to a support bundle.

A finding names its service as the operator knows it

A finding about a service carries the service’s identifier, which is what a command takes and what a script matches on, and beside it the name the stack gives that service in front of an operator. The identifier is a key, not a name: qbittorrent is not how anybody writes qBittorrent, and a surface cannot recover the one from the other, since capitalising an identifier guesses at a name the stack has already written down.

So the report carries both, and the words lemonfiber writes for a finding use the name. A title reads “Gluetun tunnel”, never “gluetun tunnel”. A command in a remedy keeps the identifier, because that is what the command takes.

A wrong check says when it began

A check found wrong carries its onset: when the stack first saw it wrong since it last saw it right. It is the stack’s fact, kept with the condition it already tracks for notifications (B5), so it survives a restart of lemonfiber and is the same on every surface that reports the check. A check that comes right and goes wrong again began again, and carries the later time.

It is what lets a surface say a problem is newer than the last one somebody saw (N27): a problem has no number and no order of its own, and a time it began is the one order the stack can vouch for.

A series the media server cannot open

A media server can hold a series’ episodes and still answer that the series has no seasons. A member then sees the title and nothing to play under it, and every service stays green. The doctor asks whatever fills media.serve how many seasons and episodes each series holds, over that capability’s contract and no other way. Each series that holds episodes and has no seasons is a warning of its own, named by its title, and its remedy is to refresh that series (C3). The check reads only. It is an operator’s finding, and nothing a member is shown names it.

States

Per check: pass, warn, fail, unverified, skipped (not applicable).

Overall: healthy (all pass), degraded (warnings only), broken (any failure), unknown (checks could not run).

Edge cases

Situation Behaviour
Docker daemon unreachable Run every check that doesn’t need it; mark the rest unverified, not fail.
Check times out unverified with the elapsed time. Never hang the suite.
Check would disturb the system Excluded from default runs; unverified with an explanation of how to run it.
Stack not running Static checks still run. Runtime checks are skipped, not failures.
Check needs a credential that’s absent skipped with the reason, not fail.
Two checks find the same root cause Report both, and indicate the shared cause so the operator fixes one thing rather than five symptoms.
Transient network failure Retry once before reporting; distinguish transient from persistent.
Check itself errors unexpectedly Report the check as errored. A bug in diagnostics must never present as a finding about the stack.
Disruptive check run while downloads are active Warn about the interruption and require confirmation.
Very slow storage Use generous timeouts for filesystem checks; a NAS is not a local SSD.
Operator runs diagnostics during setup Supported and encouraged — setup uses the same checks.
A plugin’s declared proof fails Report it as that plugin’s proof failing, naming the plugin. It is a finding about the plugin, not about the stack the plugin was added to.
A plugin’s proof is slow or never returns The same bounded timeout every check has, and the same unverified on expiry. A contributed check does not get to decide how long diagnostics take.
A plugin’s proof throws rather than answers A check error attributed to the plugin (C1-R8’s rule, with a name on it). An operator reading it should not be left wondering whether their stack is broken.
Nothing fills media.serve The series check reports skipped with the reason.
The media server cannot be asked The series check reports unverified. It is not a pass.
A series holds no episodes and no seasons Not a finding. An empty series is not a broken one.

Acceptance criteria

ID Requirement
C1-R1 Every check MUST establish its finding empirically, not infer it from configuration.
C1-R2 Every non-passing result MUST carry a remedy stating what to do.
C1-R3 unverified MUST be distinct from pass, and a check that could not run MUST NOT report as passing.
C1-R4 Checks MUST be independent; one failing MUST NOT prevent others running.
C1-R5 Checks that disturb the running system MUST be opt-in and MUST state what they disturb and for how long.
C1-R6 The suite, a category, or a single check MUST be individually runnable.
C1-R7 Every check MUST have a bounded timeout and MUST report unverified on expiry.
C1-R8 An error inside a check MUST be reported as a check error, never as a finding about the stack.
C1-R9 Diagnostics MUST emit machine-readable output alongside human-readable output.
C1-R10 Findings sharing a root cause MUST indicate that relationship.
C1-R11 Checks whose prerequisites are absent MUST report skipped with a reason, not fail.
C1-R12 A full non-disruptive run SHOULD complete within 30 seconds.
C1-R13 Setup MUST use the same checks as diagnostics, not a parallel implementation.
C1-R14 Filesystem check timeouts MUST accommodate network and external storage.
C1-R15 A proof an installed plugin declares MUST run as a check like any other, a run proof on the cadence ARCH-R287 sets, — under a bounded timeout, reporting the same verdicts — MUST be attributed to the plugin, and an error inside it MUST be reported as that plugin’s check error rather than as a finding about the stack.
C1-R16 A finding about a service MUST carry, beside the service’s identifier, the name the stack gives that service in front of an operator, and the title lemonfiber writes for a finding MUST name a service by that name, never by its identifier.
C1-R17 A check reported wrong MUST carry its onset, the time the stack first saw it wrong since it last saw it right; the onset MUST survive a restart of lemonfiber and MUST be the same on every surface that reports the check, and a check that comes right and goes wrong again MUST carry the later time.
C1-R18 The doctor MUST ask the filler of media.serve, over that capability’s contract alone, how many seasons and episodes each series holds, and MUST report each series holding episodes and no seasons as a warning of its own that names the series; where nothing fills media.serve it MUST report skipped, where the filler cannot be asked it MUST report unverified, and nothing a member is shown MUST name the finding.