Lemonfiber
← The specification

Uptime monitoring

Rendered from spec at76d46f3 (2026-10-25) · its source · Edit this page

Status: Accepted · Audience: Operator · Area: K — Observability

Purpose

Give the operator an independent, self-hosted uptime and heartbeat monitor for each critical service and each household-facing entry point — a second opinion distinct from the tool’s own health view (G7). When the stack’s own reporting is the thing that has failed, an outside watcher is the only observer that can still tell the truth.

Behaviour

It is a self-hosted, open-source second opinion

Uptime monitoring MUST use an open-source, self-hostable monitor, not a hosted uptime service. Its value is precisely that it is independent of lemonfiber’s own status path: if the tool’s telemetry channel is down, the monitor still reports, so an outage in the reporting is not mistaken for an outage in the stack — or worse, hidden by one.

A monitor per critical service and per entry point

Every critical service gets a live monitor, and so does every household-facing entry point — the paths a member actually uses to request and to watch. The question the monitor answers is not “is the container running?” but “does this respond to something the household would do?”

Heartbeats for scheduled work

Jobs that are supposed to run on a schedule — the periodic maintenance a healthy stack performs — are watched by heartbeat checks: the job is expected to check in each interval, and a missed check-in is itself an alarm. A cron job that silently stops is caught by its silence, not only by a downstream symptom appearing days later.

It is honest about how it can be configured

The chosen monitor may not expose a stable configuration API. The tool provisions it through whatever supported mechanism the monitor actually offers and states that limitation plainly rather than pretending an API exists or that configuration is fully declarative when it is not. Where provisioning cannot be fully file-driven, the tool says so instead of implying a guarantee it cannot keep.

It proves detection, not just presence

A configured monitor that has never seen a failure proves nothing. The tool asserts empirically:

  • Live coverage — each critical service and entry point has a monitor that is actually checking, not merely defined.
  • Detection works — a deliberately induced failure is detected by the monitor and surfaced, so the operator knows the alarm path fires before a real outage tests it.

Alerts route without duplicating notifications

When the monitor detects a failure it surfaces it, and its alert routing is reconciled with the stack’s own notifications (B5) so the operator is not paged twice for one event, nor left unsure which channel is authoritative.

On the companion

The argument this feature opens with is the one that survives the trip to a phone: when the stack’s own reporting is the thing that has failed, an outside watcher is the only observer that can still tell the truth (N16).

So N16-R1 forbids the app reconciling this watcher and the stack’s own health view into a single verdict. Where they disagree, the disagreement is the information, and a surface that resolved it would discard exactly the signal this exists to provide — silently, because one verdict looks authoritative in a way two do not.

N16-R2 and N16-R3 are about the absence. A monitor that has stopped reporting and a service that has stopped both arrive as nothing, and they are opposite facts: one is a thing to go and fix, the other read as all clear is a stack nobody is watching. A heartbeat that has not arrived says how long it has been missing and what it was covering, because late and never are different answers.

It is honest about how it can be configured becomes N16-R4: where a check could not be made to observe what it should, that is shown instead of its result. A green tick from a monitor watching the wrong thing is worse than no monitor at all, and from a phone there is no way to tell which one is on screen.

States

State Meaning
unconfigured No uptime monitor provisioned; offers to wire monitors for services and entry points
watching Monitors live and checking; all covered targets currently up
alarm One or more monitored targets down; the failure has been surfaced
heartbeat-missed A scheduled job failed to check in within its interval
monitor-down The monitor itself is unreachable; coverage is currently blind and this is stated
expected-down A target is intentionally stopped (a stopped form); suppressed, not alarmed

Edge cases

Situation Behaviour
The monitor itself goes down State that coverage is blind rather than implying all-clear; an absent watcher MUST NOT read as everything healthy.
False positive on a slow link Require a check to fail consistently before alarming, so a single slow response on a congested line does not page the operator.
A service intentionally down Treat an intentionally stopped service as expected-down and suppress the alarm; a deliberate stop is not an outage.
Alert routing overlaps B5 notifications Reconcile routing with the stack’s notifications (B5) so one event does not produce two alerts, and the authoritative channel is clear.
Monitor lacks a stable configuration API Provision through a supported mechanism and state the limitation, rather than presenting configuration as declarative when it is not.
Heartbeat job legitimately skipped Distinguish a job that ran and reported nothing to do from a job that never checked in; only genuine silence is an alarm.
Entry point reachable internally but not from the household path Monitor the path a member actually uses; internal reachability MUST NOT be reported as household reachability.
Monitor and metrics stack disagree Report both honestly rather than hiding the disagreement; an independent second opinion is worthless if silently overridden.
Flapping target Report the flap as flapping rather than as a stream of separate up/down events, so the operator sees instability, not noise.
Notification channel for alerts is itself down Surface that alerts could not be delivered; an undeliverable alarm MUST NOT be recorded as delivered.

Acceptance criteria

ID Requirement
K2-R1 Uptime monitoring MUST use an open-source, self-hostable monitor, and MUST NOT depend on a hosted uptime service.
K2-R2 The monitor MUST be independent of lemonfiber’s own health path, so a failure in the tool’s reporting does not blind or falsify uptime coverage.
K2-R3 Every critical service MUST have a live monitor, and every household-facing entry point MUST have one.
K2-R4 Jobs expected to run on a schedule MUST be watched by heartbeat checks, and a missed check-in MUST itself raise an alarm.
K2-R5 The tool MUST assert that each critical service and entry point has a monitor that is actively checking, not merely defined.
K2-R6 The tool MUST assert that a deliberately induced failure is detected by the monitor and surfaced.
K2-R7 Where the monitor lacks a stable configuration API, the tool MUST provision it through a supported mechanism and MUST state the limitation rather than implying an API exists.
K2-R8 An unreachable monitor MUST be reported as blind coverage and MUST NOT be rendered as all-clear.
K2-R9 A monitored target MUST fail consistently before alarming, so a single slow response does not raise a false positive.
K2-R10 An intentionally stopped service MUST be treated as expected-down and MUST NOT raise an outage alarm.
K2-R11 Alert routing MUST be reconciled with the stack’s notifications (B5) so a single event does not produce duplicate alerts.
K2-R12 An entry point MUST be monitored over the path the household actually uses, and internal reachability MUST NOT be reported as household reachability.
K2-R13 An alert that could not be delivered MUST be surfaced as undelivered rather than recorded as delivered.
K2-R14 Provisioning monitors and running the detection proof MUST each be reachable non-interactively.