This adds the ability for users to click on any day block in the historical uptime chart to see the specific incidents that occurred on that day.
Closes#3972
The window was carried across a spell, so the results either side of it ended up
adjacent: a monitor at [t, t, t, f, f] when egress dropped resumed at
[t, t, f, f, f] on its first real failure afterwards, crossing the threshold on
results taken before the outage.
Cleared in the producer beside the maintenance gate, which already clears the
window on entering a window, and guarded on its length so it is one write per
spell rather than one per degraded check. The monitor still goes down once the
rebuilt window holds enough real failures.
The two integration tests that asserted the old behaviour are rewritten: one
expected the window preserved through a spell, the other expected failures
either side of a spell to combine.
🔧
An egress spell masked a service on the surviving network going down: with
`peerResponded` set only by the DNS and gRPC providers, every other failure fell
back to the HTTP-status heuristic, which answers "nothing replied" for anything
that is not speaking HTTP. A refused connection on the local network was read as
the instance's own loss of egress, so the check was recorded as degraded and the
monitor stayed up.
`peerAnsweredFromError` holds the errno codes that require the target's own stack
to have replied — a refusal, a reset, a TLS handshake that got as far as a
certificate. Http, Port, WebSocket and Docker use it; Http also treats any status
code as an answer, and Docker any status from the daemon. Hardware and PageSpeed
go through Http and inherit it.
`true` is the claim that costs something, since it blames the target, so it is
only set on evidence. Two cases stay conservative and are commented as such: a
ping that draws no echo reply, and a gamedig query that fails, neither of which
can be told apart from the instance losing its egress. Nor can a proxied HTTP
request that fails with no response, where the refusal may be the proxy's.
🔧
Fixes#3970 by pointing protoPath to the correct relative directory (../protos) and adding a step in the package.json build script to copy the protos directory to the final dist output.
findUnevaluatedByMonitorId now takes the job's pendingChecks rather than a
timestamp, so the test builds those entries from the ids createChecks returns.
🔧
Those aggregations were reverted when docker and hardware could not carry the
flag. Both types are now egress attributable, so their checks can be degraded
and their totals, bucket counts and latest-check reads must exclude them like
every other aggregate.
🔧
Distinguish "the peer answered" from "nothing answered", widen attribution to
every check that leaves the instance, make the maintenance exit explicit, and
fix the recovery poll cadence.
Providers can now say outright whether the peer answered, via peerResponded on
MonitorStatusResponse. Previously the producer inferred it from an HTTP status
code, which two providers do not report:
- GrpcProvider: NOT_SERVING shares NETWORK_ERROR with a connection failure, and
gRPC codes 0-16 are not HTTP statuses, so PERMISSION_DENIED, UNIMPLEMENTED
and the rest were all read as transport failures. Only DEADLINE_EXCEEDED and
UNAVAILABLE, or a failure carrying no gRPC code, now count as unreachable.
- DNSProvider: every resolver error became NETWORK_ERROR, so an authoritative
NXDOMAIN, SERVFAIL or REFUSED was read as unreachable. The c-ares codes where
the server replied are now separated from timeout and connection refused.
Left unset, the HTTP-status heuristic still applies, so other providers are
unchanged.
Attribution now follows whether the check leaves the instance, not the monitor
type. A hardware check is an ordinary outbound HTTP request to the Capture
agent, and a Docker daemon can be reached over TCP, so both are attributable;
only a Docker daemon on a unix socket is local and exempt.
Leaving a maintenance window is now explicit in the producer, alongside the
entry it already owned. It was a side effect of evaluating a check, which the
degraded-egress short-circuit skips, so a monitor whose window ended during an
outage kept reporting "maintenance" for the length of it.
The recovery poll interval is fixed at 5 seconds and no longer configurable.
While degraded, checks are flagged from stored state without probing, so the
interval is the window in which a recovered instance still discards its checks;
the only effect of raising it was a longer blind window. egressPollIntervalSeconds
is removed from the settings, the validator and the OpenAPI spec.
🔧
A degraded episode now reads as a gap in monitoring rather than as downtime
or as a dip in response time. The exclusion is applied once per query, in the
top-level $match, so no pipeline can count a degraded check by omission.
- check.query.ts: EXCLUDE_DEGRADED_EGRESS_MATCH is the only helper;
IS_NOT_DEGRADED_EGRESS_EXPR and the $cond accumulators it fed are gone.
- findUptimeDateRangeChecks: excludes at the match, so the response-time
series, the up/down series and the averages drop degraded checks too.
- findPageSpeedDateRangeChecks: excludes as well; pagespeed is egress
attributable and was previously missed.
- getDailyStatusBuckets: excludes at the match; the group shapes and the
totalChecks > 0 guard revert to their original form.
- check.docker.aggregation.ts, check.hardware.aggregations.ts: reverted.
Neither type is egress attributable, so their checks never carry the flag
and the filters were dead code. A test now pins that decision.
- Tests: the gap behaviour, a window that was entirely degraded, pagespeed,
the team listing and the resolve filter, and an incident that stays open
when egress drops during it.
🔧
CheckProducer consults egress only for a transport failure: the check
failed, no HTTP status came back (isHttpStatusCode from PR 1), and the
monitor type is one whose failure can be a loss of egress
(EgressAttributableTypes; hardware and docker are excluded). A 4xx/5xx
or a content mismatch is evaluated as usual while the instance is
degraded. A rejected assessment is logged and treated as unknown so the
check is still buffered.
EXCLUDE_DEGRADED_EGRESS_MATCH moves out of the model into
check.query.ts, with an expression form for $group accumulators.
Daily status buckets and docker stats buckets exclude degraded checks
from their up/total counts while keeping them in the response-time
average, matching the uptime facet. toEntity maps egressStatus like its
siblings.
Adds heartbeatJobDegradedEgress integration tests, extends the
repository exclusion tests, and makes the harness egress stub settable.
🔧
Runs the summary, daily bucket, date-range, hardware and docker aggregations
against a live mongod with a mix of degraded and ordinary checks, and covers
the egressStatus round trip through createChecks and findUnevaluatedByMonitorId.
🔧
CheckProducer asks EgressService to assess the instance's own connectivity
after a failing check and stamps the result on the check as egressStatus.
CheckEvaluator short-circuits degraded checks so no incident opens and no
spurious recovery fires later. Uptime and down-count aggregations exclude
degraded checks; the response-time series keeps them.
🔧
Alex Holliday
nikhil2004
fabiangigler
Craig Lawson