An egress spell masked a service on the surviving network going down: with
`peerResponded` set only by the DNS and gRPC providers, every other failure fell
back to the HTTP-status heuristic, which answers "nothing replied" for anything
that is not speaking HTTP. A refused connection on the local network was read as
the instance's own loss of egress, so the check was recorded as degraded and the
monitor stayed up.
`peerAnsweredFromError` holds the errno codes that require the target's own stack
to have replied — a refusal, a reset, a TLS handshake that got as far as a
certificate. Http, Port, WebSocket and Docker use it; Http also treats any status
code as an answer, and Docker any status from the daemon. Hardware and PageSpeed
go through Http and inherit it.
`true` is the claim that costs something, since it blames the target, so it is
only set on evidence. Two cases stay conservative and are commented as such: a
ping that draws no echo reply, and a gamedig query that fails, neither of which
can be told apart from the instance losing its egress. Nor can a proxied HTTP
request that fails with no response, where the refusal may be the proxy's.
🔧
Distinguish "the peer answered" from "nothing answered", widen attribution to
every check that leaves the instance, make the maintenance exit explicit, and
fix the recovery poll cadence.
Providers can now say outright whether the peer answered, via peerResponded on
MonitorStatusResponse. Previously the producer inferred it from an HTTP status
code, which two providers do not report:
- GrpcProvider: NOT_SERVING shares NETWORK_ERROR with a connection failure, and
gRPC codes 0-16 are not HTTP statuses, so PERMISSION_DENIED, UNIMPLEMENTED
and the rest were all read as transport failures. Only DEADLINE_EXCEEDED and
UNAVAILABLE, or a failure carrying no gRPC code, now count as unreachable.
- DNSProvider: every resolver error became NETWORK_ERROR, so an authoritative
NXDOMAIN, SERVFAIL or REFUSED was read as unreachable. The c-ares codes where
the server replied are now separated from timeout and connection refused.
Left unset, the HTTP-status heuristic still applies, so other providers are
unchanged.
Attribution now follows whether the check leaves the instance, not the monitor
type. A hardware check is an ordinary outbound HTTP request to the Capture
agent, and a Docker daemon can be reached over TCP, so both are attributable;
only a Docker daemon on a unix socket is local and exempt.
Leaving a maintenance window is now explicit in the producer, alongside the
entry it already owned. It was a side effect of evaluating a check, which the
degraded-egress short-circuit skips, so a monitor whose window ended during an
outage kept reporting "maintenance" for the length of it.
The recovery poll interval is fixed at 5 seconds and no longer configurable.
While degraded, checks are flagged from stored state without probing, so the
interval is the window in which a recovered instance still discards its checks;
the only effect of raising it was a longer blind window. egressPollIntervalSeconds
is removed from the settings, the validator and the OpenAPI spec.
🔧
- Schedule the recovery job before recording the degraded transition,
so a crash between the two leaves a stray row (removed on the job's
first run) rather than a degraded state nothing is polling
- Count any HTTP response, including 4xx/5xx, as reachable for http
targets; only a transport failure counts against egress
- Cascade a deleted notification out of egressNotifications with an
atomic $pull in the settings repository, after the team-scoped delete
- Handle two processes racing the first egress state insert
- Address global job rows by one key (deleteGlobalJob on _id) and
restore the deleteByIdAndType signature
- Inject net into PingProvider like PortProvider
- Integration tests for MongoEgressStateRepository and the global-row
delete methods on the jobs repository
🔧
- The egress job runs through WorkerHelper.getEgressRecoveryJob(), like
the cleanup jobs, instead of a direct service call from the queue
- Toggling egressCheckEnabled resets the egress state and removes any
pending recovery job
- Probes resolve and pass the proxy, so http targets honour the global
proxy as monitors do
- PingProvider keeps IPv6 addresses intact instead of truncating at the
first colon
- Egress targets are parsed by one function shared by the validator and
the probe; malformed addresses such as unbalanced brackets are rejected
- Notification id pattern drops the /i flag, which produced an invalid
OpenAPI pattern
- EgressState findSingleton reads before it writes, so a read no longer
bumps updatedAt
- "disabled" is a distinct assessment result and is cached like any other
- Concurrent failures share an in-flight assessment for as long as it
runs, then reuse the settled result for 5 s
- The recovery job row is only removed if nobody rescheduled it since
this run claimed it, so a concurrent new episode keeps its job
- Deleting a notification removes it from egressNotifications
🔧
Matrix notifications failed for anyone who entered a room alias. The send
endpoint only accepts a room ID, and an unresolved alias broke the request
outright: the leading "#" opens a URI fragment, so the PUT landed on
/_matrix/client/v3/rooms/ instead of the send endpoint. Element shows users
the alias, so this is the value most people paste in.
- Resolve "#alias:server" through GET /_matrix/client/v3/directory/room/{alias}
before sending, per the client-server spec
- Percent-encode the room ID in the send path; ":" was previously sent raw
- Strip trailing slashes from the homeserver URL to avoid "//_matrix/..."
Alias resolution failures return false and log, matching how the provider
already handles send failures.
Closes#3478
sendEmail caught transport failures internally and resolved to false or
undefined, so every caller's .catch() was dead code and send failures were
silently suppressed. Password recovery in particular reported "email sent
successfully" while the user received nothing.
sendEmail now throws an AppError carrying the underlying cause, and returns
Promise<string> instead of Promise<string | false | undefined>.
Call sites updated for the new contract:
- UserService.registerUser: the existing .catch() now actually fires
- UserService.requestRecovery: the failure reaches the client instead of a
misleading success response
- InviteService and the test-email endpoint: keep their user-facing messages
and attach the SMTP cause
- EmailProvider: catches and returns false, because NotificationsService fans
providers out through Promise.all and a rejection there would abort the
other channels' alerts
Closes#3757
ntfy servers that require auth rejected every Checkmate alert because the
provider posted without an Authorization header, so alerts to protected
topics failed silently.
Adds ntfyAuthType ("none" | "token" | "basic"):
- token: Authorization: Bearer <accessToken>
- basic: Authorization: Basic base64(ntfyUsername:accessToken)
Both types store their secret in the existing accessToken field rather than
introducing a new secret, so credential handling stays in one place.
Incomplete credentials fail the send instead of falling back to an anonymous
post, and validation rejects partial auth config up front so a channel can't
be saved looking authenticated while posting anonymously.
Backend only; frontend follows once this lands.
Closes#3717
Build title_link from the monitor type (hardware -> infrastructure,
pagespeed -> pagespeed, uptime types -> uptime) and omit it when
clientHost is not configured, instead of always linking to the
infrastructure page.
When customUpCodes mark an HTTPError as up, parse the response body, apply advancedMatcher.validate, and set status to statusUp && matchResult.ok. Return parsed payload and extracted values to match the success path.
Alex Holliday
Craig Lawson
fabiangigler
signalgridco
egeoztass
Marcos Paulo Diniz
harsh-aghara