Address the inline review on the egress check pipeline

Distinguish "the peer answered" from "nothing answered", widen attribution to
every check that leaves the instance, make the maintenance exit explicit, and
fix the recovery poll cadence.

Providers can now say outright whether the peer answered, via peerResponded on
MonitorStatusResponse. Previously the producer inferred it from an HTTP status
code, which two providers do not report:

- GrpcProvider: NOT_SERVING shares NETWORK_ERROR with a connection failure, and
  gRPC codes 0-16 are not HTTP statuses, so PERMISSION_DENIED, UNIMPLEMENTED
  and the rest were all read as transport failures. Only DEADLINE_EXCEEDED and
  UNAVAILABLE, or a failure carrying no gRPC code, now count as unreachable.
- DNSProvider: every resolver error became NETWORK_ERROR, so an authoritative
  NXDOMAIN, SERVFAIL or REFUSED was read as unreachable. The c-ares codes where
  the server replied are now separated from timeout and connection refused.

Left unset, the HTTP-status heuristic still applies, so other providers are
unchanged.

Attribution now follows whether the check leaves the instance, not the monitor
type. A hardware check is an ordinary outbound HTTP request to the Capture
agent, and a Docker daemon can be reached over TCP, so both are attributable;
only a Docker daemon on a unix socket is local and exempt.

Leaving a maintenance window is now explicit in the producer, alongside the
entry it already owned. It was a side effect of evaluating a check, which the
degraded-egress short-circuit skips, so a monitor whose window ended during an
outage kept reporting "maintenance" for the length of it.

The recovery poll interval is fixed at 5 seconds and no longer configurable.
While degraded, checks are flagged from stored state without probing, so the
interval is the window in which a recovered instance still discards its checks;
the only effect of raising it was a longer blind window. egressPollIntervalSeconds
is removed from the settings, the validator and the OpenAPI spec.

🔧
This commit is contained in:
Craig Lawson committed 2026-09-17 22:06:34 +01:00
1 parent d865103fb5
commit 3dc529733a
17 files changed
+265 -80

No files matched your search

+13
View File
@@ -1,5 +1,6 @@
import { DNSStatusPayload, MonitorStatusResponse } from "@/types/network.js";
import { IStatusProvider } from "@/service/network/IStatusProvider.js";
import dns from "dns";
import type { Resolver } from "dns/promises";
import { Monitor, MonitorType } from "@/domain/monitors/monitor.type.js";
import { AppError } from "@/utils/AppError.js";
@@ -8,6 +9,16 @@ import { NETWORK_ERROR } from "@/types/network.js";
const SERVICE_NAME = "DNSProvider";
// c-ares codes where the DNS server replied: the name does not exist, holds no such record, or the server
// declined or answered badly. Anything else (timeout, connection refused, a local resolver fault) means we
// never got an answer, so the failure could be our own loss of egress rather than the target's.
const ANSWERED_DNS_CODES: ReadonlySet<string> = new Set([dns.NODATA, dns.FORMERR, dns.SERVFAIL, dns.NOTFOUND, dns.NOTIMP, dns.REFUSED, dns.BADRESP]);
const dnsServerAnswered = (error: unknown): boolean => {
const code = (error as NodeJS.ErrnoException | undefined)?.code;
return typeof code === "string" && ANSWERED_DNS_CODES.has(code);
};
export class DNSProvider implements IStatusProvider<DNSStatusPayload> {
readonly type = "dns";
constructor(private createResolver: () => Resolver) {}
@@ -63,6 +74,7 @@ export class DNSProvider implements IStatusProvider<DNSStatusPayload> {
type: monitor.type,
status: false,
code: NETWORK_ERROR,
peerResponded: dnsServerAnswered(error),
message: error instanceof Error ? error.message : String(error),
responseTime,
payload: {
@@ -81,6 +93,7 @@ export class DNSProvider implements IStatusProvider<DNSStatusPayload> {
type: monitor.type,
status: true,
code: 200,
peerResponded: true,
message: "Success",
responseTime,
payload: {
@@ -25,6 +25,17 @@ export class GrpcProvider implements IStatusProvider<GrpcStatusPayload> {
return type === "grpc";
}
// gRPC codes that mean the call never reached a server. Everything else in 0-16 is the server's own answer,
// including NOT_SERVING, which shares NETWORK_ERROR with a connection failure and so cannot be read from `code`.
// A call that fails with no gRPC code at all never got as far as the protocol.
private peerAnswered(code: number | undefined): boolean {
const UNREACHABLE_GRPC_CODES: ReadonlySet<number> = new Set([
4, // DEADLINE_EXCEEDED
14, // UNAVAILABLE
]);
return typeof code === "number" && !UNREACHABLE_GRPC_CODES.has(code);
}
private getGrpcStatusName(code: number): string {
const statusNames: Record<number, string> = {
0: "OK",
@@ -148,6 +159,7 @@ export class GrpcProvider implements IStatusProvider<GrpcStatusPayload> {
type: type,
status: false,
code: grpcError.grpcCode ?? 5000,
peerResponded: this.peerAnswered(grpcError.grpcCode),
message: grpcError.message ?? "gRPC health check failed",
responseTime,
payload: payload ?? null,
@@ -163,6 +175,8 @@ export class GrpcProvider implements IStatusProvider<GrpcStatusPayload> {
type: type,
status: isServing,
code: isServing ? 200 : 5000,
// The server replied, whatever it said, so this failure is never the instance's own egress.
peerResponded: true,
message: isServing ? `gRPC service healthy (${grpcPayload.servingStatus})` : `gRPC service unhealthy (${grpcPayload.servingStatus})`,
responseTime,
payload: grpcPayload,