Distinguish "the peer answered" from "nothing answered", widen attribution to
every check that leaves the instance, make the maintenance exit explicit, and
fix the recovery poll cadence.
Providers can now say outright whether the peer answered, via peerResponded on
MonitorStatusResponse. Previously the producer inferred it from an HTTP status
code, which two providers do not report:
- GrpcProvider: NOT_SERVING shares NETWORK_ERROR with a connection failure, and
gRPC codes 0-16 are not HTTP statuses, so PERMISSION_DENIED, UNIMPLEMENTED
and the rest were all read as transport failures. Only DEADLINE_EXCEEDED and
UNAVAILABLE, or a failure carrying no gRPC code, now count as unreachable.
- DNSProvider: every resolver error became NETWORK_ERROR, so an authoritative
NXDOMAIN, SERVFAIL or REFUSED was read as unreachable. The c-ares codes where
the server replied are now separated from timeout and connection refused.
Left unset, the HTTP-status heuristic still applies, so other providers are
unchanged.
Attribution now follows whether the check leaves the instance, not the monitor
type. A hardware check is an ordinary outbound HTTP request to the Capture
agent, and a Docker daemon can be reached over TCP, so both are attributable;
only a Docker daemon on a unix socket is local and exempt.
Leaving a maintenance window is now explicit in the producer, alongside the
entry it already owned. It was a side effect of evaluating a check, which the
degraded-egress short-circuit skips, so a monitor whose window ended during an
outage kept reporting "maintenance" for the length of it.
The recovery poll interval is fixed at 5 seconds and no longer configurable.
While degraded, checks are flagged from stored state without probing, so the
interval is the window in which a recovered instance still discards its checks;
the only effect of raising it was a longer blind window. egressPollIntervalSeconds
is removed from the settings, the validator and the OpenAPI spec.
🔧
- The egress job runs through WorkerHelper.getEgressRecoveryJob(), like
the cleanup jobs, instead of a direct service call from the queue
- Toggling egressCheckEnabled resets the egress state and removes any
pending recovery job
- Probes resolve and pass the proxy, so http targets honour the global
proxy as monitors do
- PingProvider keeps IPv6 addresses intact instead of truncating at the
first colon
- Egress targets are parsed by one function shared by the validator and
the probe; malformed addresses such as unbalanced brackets are rejected
- Notification id pattern drops the /i flag, which produced an invalid
OpenAPI pattern
- EgressState findSingleton reads before it writes, so a read no longer
bumps updatedAt
- "disabled" is a distinct assessment result and is cached like any other
- Concurrent failures share an in-flight assessment for as long as it
runs, then reuse the settled result for 5 s
- The recovery job row is only removed if nobody rescheduled it since
this run claimed it, so a concurrent new episode keeps its job
- Deleting a notification removes it from egressNotifications
🔧
Adds the EgressService that probes a configurable set of reliability targets
(ping, TCP or HTTP) when asked, persists a single instance-wide egress state
document, and runs a recovery poll loop while egress is degraded. Adds the four
egress settings fields with validation. The worker constructs and initialises
the service but nothing calls assessAfterFailure yet, so behaviour is unchanged.
🔧
Craig Lawson