Commit Graph
14 Commits
Author SHA1 Message Date
Craig Lawson b68367b04f Harden the egress recovery job after a second review pass (#3937)
- Schedule the recovery job before recording the degraded transition,
  so a crash between the two leaves a stray row (removed on the job's
  first run) rather than a degraded state nothing is polling
- Count any HTTP response, including 4xx/5xx, as reachable for http
  targets; only a transport failure counts against egress
- Cascade a deleted notification out of egressNotifications with an
  atomic $pull in the settings repository, after the team-scoped delete
- Handle two processes racing the first egress state insert
- Address global job rows by one key (deleteGlobalJob on _id) and
  restore the deleteByIdAndType signature
- Inject net into PingProvider like PortProvider
- Integration tests for MongoEgressStateRepository and the global-row
  delete methods on the jobs repository

🔧
2026-09-15 08:59:13 +01:00
Craig Lawson 08f104e283 Address review on the egress recovery job (#3937)
- The egress job runs through WorkerHelper.getEgressRecoveryJob(), like
  the cleanup jobs, instead of a direct service call from the queue
- Toggling egressCheckEnabled resets the egress state and removes any
  pending recovery job
- Probes resolve and pass the proxy, so http targets honour the global
  proxy as monitors do
- PingProvider keeps IPv6 addresses intact instead of truncating at the
  first colon
- Egress targets are parsed by one function shared by the validator and
  the probe; malformed addresses such as unbalanced brackets are rejected
- Notification id pattern drops the /i flag, which produced an invalid
  OpenAPI pattern
- EgressState findSingleton reads before it writes, so a read no longer
  bumps updatedAt
- "disabled" is a distinct assessment result and is cached like any other
- Concurrent failures share an in-flight assessment for as long as it
  runs, then reuse the settled result for 5 s
- The recovery job row is only removed if nobody rescheduled it since
  this run claimed it, so a concurrent new episode keeps its job
- Deleting a notification removes it from egressNotifications

🔧
2026-09-15 08:42:09 +01:00
Craig Lawson 46eb0b4063 Replace the egress recovery timer with a scheduled job (#3937)
A failed check still calls assessAfterFailure, which now shares one
in-flight assessment between concurrent failures (same cache shape as
ProxyResolver). When egress degrades it upserts a global "egress" job;
DBQueueWorker runs checkRecovery for that type at the configured
interval, and the job deletes its own row once any target answers.
The unref'd timer, startRecoveryPolling, stop and init are gone: a
degraded episode survives a restart through the jobs collection.
deleteByIdAndType accepts a null refId for global rows.

🔧
2026-09-14 20:41:58 +01:00
Craig Lawson ec01a5dfe4 Add egress state and probe service (#3937)
Adds the EgressService that probes a configurable set of reliability targets
(ping, TCP or HTTP) when asked, persists a single instance-wide egress state
document, and runs a recovery poll loop while egress is degraded. Adds the four
egress settings fields with validation. The worker constructs and initialises
the service but nothing calls assessAfterFailure yet, so behaviour is unchanged.

🔧
2026-09-14 10:34:12 +01:00
Alex Holliday 9ad20daa51 bump minor version, fix unused import 2026-09-12 14:56:07 -07:00
Alex Holliday b14b2142e3 add secrets rotation service and tests 2026-09-11 10:09:30 -07:00
Alex Holliday 1035e397e0 inject encryptionService into DockerProvider 2026-09-10 11:57:43 -07:00
Alex Holliday 6587ddf389 wire up services/repositories 2026-09-03 11:32:48 -07:00
Alex Holliday 1152a70afb wire up proxy resolver 2026-08-21 11:41:26 -07:00
Alex Holliday 936d9cd9a0 update call site 2026-07-21 10:26:02 -07:00
Alex Holliday c9f0254aed start health server, update shutdown, update env config 2026-06-25 10:36:53 -07:00
Alex Holliday 5fab2d0a30 inject buffer service, implement drain 2026-06-24 12:17:02 -07:00
Alex Holliday 71d3e789ff split services to api, worker, and shared 2026-06-24 10:30:21 -07:00
Alex Holliday f59a7c3113 extract shared services 2026-06-23 12:26:42 -07:00