Clear the status window when a degraded spell starts
The window was carried across a spell, so the results either side of it ended up
adjacent: a monitor at [t, t, t, f, f] when egress dropped resumed at
[t, t, f, f, f] on its first real failure afterwards, crossing the threshold on
results taken before the outage.
Cleared in the producer beside the maintenance gate, which already clears the
window on entering a window, and guarded on its length so it is one write per
spell rather than one per degraded check. The monitor still goes down once the
rebuilt window holds enough real failures.
The two integration tests that asserted the old behaviour are rewritten: one
expected the window preserved through a spell, the other expected failures
either side of a spell to combine.
🔧
This commit is contained in:
1 parent
b55560e2f1
commit
1b3b8e5312
3 files changed
+63
-7
No files matched your search
@@ -133,10 +133,19 @@ export class CheckProducer implements ICheckProducer {
|
||||
if (egressStatus !== null) {
|
||||
check.egressStatus = egressStatus;
|
||||
}
|
||||
|
||||
// Step 2c: Add to buffer
|
||||
this.bufferService.addToBuffer(check);
|
||||
|
||||
// Step 2d: Handle docker logs
|
||||
// Step 2d: As with the maintenance gate above, the window is cleared at the point the monitor stops
|
||||
// being evaluated, so the results either side of a spell never end up adjacent and cross the threshold
|
||||
// on stale data. The length guard keeps this to one write per spell rather than one per degraded check,
|
||||
// and it runs after the check is buffered so that a failed write cannot cost us the check.
|
||||
if (egressStatus === "degraded" && monitor.statusWindow?.length) {
|
||||
await this.monitorsRepository.updateById(monitor.id, monitor.teamId, { statusWindow: [] });
|
||||
}
|
||||
|
||||
// Step 2e: Handle docker logs
|
||||
if (status.type === "docker") {
|
||||
const dockerLogs = await this.dockerLogsService.buildDockerLogs(status as MonitorStatusResponse<DockerStatusPayload>);
|
||||
for (const dockerLog of dockerLogs) {
|
||||
|
||||
Reference in new issue
Block a user