Compare commits

...
Author SHA1 Message Date
Alex Holliday 011b93ecc1 update tests 2026-09-16 11:40:42 -07:00
Alex Holliday 20ce11497e revert cleanup job to uncached settings 2026-09-16 11:40:14 -07:00
Alex Holliday 50d0721822 cache DB settings, used cached db settings in ProxyResolver, PageSpeedProvider, and EgressService 2026-09-16 11:37:29 -07:00
Alexander Holliday c66872c7f4 Merge pull request #3966 from bluewave-labs/chore/remove-easy-panel-screenshot
chore: remove easypanel screenshot
2026-09-15 20:13:42 -07:00
Alex Holliday b1be38368c remove screenshot 2026-09-15 20:11:10 -07:00
Alexander Holliday 278f652ea9 Merge pull request #3944 from bluewave-labs/feat/refined-dashboard
feat: create dashboard of updates
2026-09-15 13:40:17 -07:00
Malena Aguiar d3e12722b7 add code adjustments 2026-09-15 12:42:57 -06:00
Malena Aguiar 8c9ff41074 fix: resolve conflicts 2026-09-15 11:40:29 -06:00
Malena Aguiar 71a1194373 fix: add code adjustments 2026-09-15 11:29:08 -06:00
Alexander Holliday 580a025353 Merge pull request #3962 from bluewave-labs/fix/reset-job-state-on-success
fix: reset job state on success
2026-09-15 10:18:28 -07:00
Alexander Holliday 33994cb3df Merge pull request #3961 from Ahson-Shaikh/add-easypanel-deploy-docs
Add Easypanel to 1-click installation options
2026-09-15 10:18:08 -07:00
Alex Holliday 44edf320a0 reset job state on success 2026-09-15 10:13:20 -07:00
Alexander Holliday fea2ef4dbe Merge pull request #3959 from web-engineer/feat/egress-1-check-service
Egress self-check (1/4): egress state and probe service (#3937)
2026-09-15 09:51:01 -07:00
Ahson Shaikh 360b4b8035 Add Easypanel deployment screenshot 2026-09-15 19:11:38 +05:00
Ahson Shaikh 3857a71e62 Add Easypanel to 1-click installation options 2026-09-15 16:58:35 +05:00
Craig Lawson b68367b04f Harden the egress recovery job after a second review pass (#3937)
- Schedule the recovery job before recording the degraded transition,
  so a crash between the two leaves a stray row (removed on the job's
  first run) rather than a degraded state nothing is polling
- Count any HTTP response, including 4xx/5xx, as reachable for http
  targets; only a transport failure counts against egress
- Cascade a deleted notification out of egressNotifications with an
  atomic $pull in the settings repository, after the team-scoped delete
- Handle two processes racing the first egress state insert
- Address global job rows by one key (deleteGlobalJob on _id) and
  restore the deleteByIdAndType signature
- Inject net into PingProvider like PortProvider
- Integration tests for MongoEgressStateRepository and the global-row
  delete methods on the jobs repository

🔧
2026-09-15 08:59:13 +01:00
Craig Lawson 08f104e283 Address review on the egress recovery job (#3937)
- The egress job runs through WorkerHelper.getEgressRecoveryJob(), like
  the cleanup jobs, instead of a direct service call from the queue
- Toggling egressCheckEnabled resets the egress state and removes any
  pending recovery job
- Probes resolve and pass the proxy, so http targets honour the global
  proxy as monitors do
- PingProvider keeps IPv6 addresses intact instead of truncating at the
  first colon
- Egress targets are parsed by one function shared by the validator and
  the probe; malformed addresses such as unbalanced brackets are rejected
- Notification id pattern drops the /i flag, which produced an invalid
  OpenAPI pattern
- EgressState findSingleton reads before it writes, so a read no longer
  bumps updatedAt
- "disabled" is a distinct assessment result and is cached like any other
- Concurrent failures share an in-flight assessment for as long as it
  runs, then reuse the settled result for 5 s
- The recovery job row is only removed if nobody rescheduled it since
  this run claimed it, so a concurrent new episode keeps its job
- Deleting a notification removes it from egressNotifications

🔧
2026-09-15 08:42:09 +01:00
Craig Lawson 46eb0b4063 Replace the egress recovery timer with a scheduled job (#3937)
A failed check still calls assessAfterFailure, which now shares one
in-flight assessment between concurrent failures (same cache shape as
ProxyResolver). When egress degrades it upserts a global "egress" job;
DBQueueWorker runs checkRecovery for that type at the configured
interval, and the job deletes its own row once any target answers.
The unref'd timer, startRecoveryPolling, stop and init are gone: a
degraded episode survives a restart through the jobs collection.
deleteByIdAndType accepts a null refId for global rows.

🔧
2026-09-14 20:41:58 +01:00
Alexander Holliday 84539f76cf Merge pull request #3913 from fabiangigler/feat/localized-monitor-times
Localize formatDuration output
2026-09-14 09:49:15 -07:00
Alex Holliday 308672d558 pass locale to formatting 2026-09-14 09:45:24 -07:00
Alex Holliday 0f474d82d6 use exact unit for duration 2026-09-14 09:45:09 -07:00
Alex Holliday 650cfe00ce Merge branch 'develop' into feat/localized-monitor-times 2026-09-14 09:33:47 -07:00
Alexander Holliday 69eb9957c6 Merge pull request #3881 from fabiangigler/feat/custom-monitor-intervals
Add custom monitor interval options
2026-09-14 09:29:15 -07:00
Craig Lawson ec01a5dfe4 Add egress state and probe service (#3937)
Adds the EgressService that probes a configurable set of reliability targets
(ping, TCP or HTTP) when asked, persists a single instance-wide egress state
document, and runs a recovery poll loop while egress is degraded. Adds the four
egress settings fields with validation. The worker constructs and initialises
the service but nothing calls assessAfterFailure yet, so behaviour is unchanged.

🔧
2026-09-14 10:34:12 +01:00
fabiangigler 5da472da14 Prevent clearing monitor intervals 2026-09-13 20:28:27 +02:00
fabiangigler 4bb366f5ab Add custom monitor interval options 2026-09-13 20:28:27 +02:00
Malena Aguiar cc4b8b38f5 feat: leave only ui changes 2026-09-12 16:17:16 -06:00
Alexander Holliday f41e0f0619 Merge pull request #3949 from kah-ja/fix/register-invite-token-validation
validate invite token type on registration
2026-09-12 15:02:48 -07:00
Alexander Holliday 3f10b6e567 Merge pull request #3952 from bluewave-labs/chore/release-prep
chore: bump minor version, fix unused import
2026-09-12 14:57:45 -07:00
Alex Holliday 9ad20daa51 bump minor version, fix unused import 2026-09-12 14:56:07 -07:00
Malena Aguiar 86b59da909 feat: clean code 2026-09-12 15:53:21 -06:00
Alexander Holliday 6a9068a151 Merge pull request #3951 from bluewave-labs/fix/svg-optimization
fix: reoptimize svg
2026-09-12 14:52:01 -07:00
Alex Holliday b31ffa5318 reoptimize svg 2026-09-12 14:46:05 -07:00
Malena Aguiar 71dd998390 feat: simplify code and functionalities 2026-09-12 15:41:03 -06:00
Alexander Holliday 102a9f1484 Merge pull request #3950 from bluewave-labs/chore/readme
chore: update readme
2026-09-12 14:17:01 -07:00
Alex Holliday 581ec9382b docker monitor instructions 2026-09-12 14:16:14 -07:00
Alex Holliday cebe9350f7 update readme 2026-09-12 14:07:31 -07:00
Jan Kahmen 406518e0a5 validate invite token type on registration 2026-09-12 12:08:33 +00:00
Alexander Holliday c458d5fb65 Merge pull request #3948 from bluewave-labs/feat/client-side-docker-tls
feat: client side docker tls
2026-09-11 12:10:28 -07:00
Alex Holliday ba0d685c6d commetns 2026-09-11 12:07:45 -07:00
Alex Holliday 912f1f960d add missing timeout 2026-09-11 11:50:26 -07:00
Alex Holliday 1de0edf7e3 Merge branch 'develop' into feat/client-side-docker-tls 2026-09-11 11:37:11 -07:00
Alex Holliday 8c9cb4be77 update create monitor form 2026-09-11 11:23:11 -07:00
Alex Holliday 727b904ca9 add docker TLS to monitor type 2026-09-11 11:21:39 -07:00
Alex Holliday 5c0279a6a7 add dockerTLS to form 2026-09-11 11:21:13 -07:00
Alex Holliday 9d72d20dd2 docker TLS validation 2026-09-11 11:20:47 -07:00
Alexander Holliday 62d0630702 Merge pull request #3947 from bluewave-labs/feat/validate-and-encrypt-docker-tls-key
feat: add validation for docker TLS fields, encrypt TLS key on create/edit monitor
2026-09-11 11:13:52 -07:00
Alex Holliday 2260b4a4d2 add validation for docker TLS fields, encrypt TLS key on create/edit monitor 2026-09-11 11:10:37 -07:00
Alexander Holliday cfb50afe42 Merge pull request #3946 from bluewave-labs/feat/secret-rotation
feat: secret rotation
2026-09-11 10:13:58 -07:00
Alex Holliday b14b2142e3 add secrets rotation service and tests 2026-09-11 10:09:30 -07:00
Malena Aguiar 896d73f12e feat: create dashboard of updates 2026-09-10 18:00:12 -06:00
Alex Holliday c64c4cc5ff update tests 2026-09-10 12:26:25 -07:00
Alex Holliday 7071de294f add docker TLS fields to toEntity 2026-09-10 12:26:12 -07:00
Alex Holliday 5eb16fdbdb add validation, helpers, tests 2026-09-10 12:23:20 -07:00
Alex Holliday 131fd1a6b0 update tests 2026-09-10 11:58:26 -07:00
Alex Holliday f9bc02ed1c update msgs 2026-09-10 11:58:12 -07:00
Alex Holliday 1035e397e0 inject encryptionService into DockerProvider 2026-09-10 11:57:43 -07:00
Alex Holliday 9c788561a7 update docker provider to add a TLS path 2026-09-10 11:49:53 -07:00
Alex Holliday 449e96d842 add dockerTlsKey to context for docker type monitors 2026-09-10 11:30:26 -07:00
Alex Holliday c7626c669a Add docker tls key to check context 2026-09-10 11:29:41 -07:00
Alex Holliday ede5bc4ecd add docker TLS fields to type, model, access methods to repo 2026-09-10 11:15:20 -07:00
Alexander Holliday 8bb60bfd03 Merge pull request #3938 from bluewave-labs/feat/encryption-module
feat: encryption module
2026-09-10 10:43:02 -07:00
Alex Holliday f009aa17d9 Add encryption service and tests. Update README 2026-09-10 10:31:41 -07:00
Alexander Holliday ef7ac8bef8 Merge pull request #3936 from bluewave-labs/feat/docker-chart-tooltips
feat: add docker chart tooltips
2026-09-09 10:26:15 -07:00
Alex Holliday b75f895a1f add docker chart tooltips 2026-09-09 10:22:02 -07:00
Alexander Holliday 53f5b61e8e Merge pull request #3935 from bluewave-labs/fix/404-illustration-credit
fix(client): credit Annie Hsu on the 404 illustration
2026-09-09 09:37:30 -07:00
Alex Holliday 64a2c69540 convert styles 2026-09-09 09:35:20 -07:00
Alex Holliday c4ec14db5d add link to SVG 2026-09-09 09:28:40 -07:00
gorkem-bwl 4560820649 fix(client): balance the 404 credit colour across both themes
The signature was #9a9a9a, picked against the dark ground. The app
switches the whole tree between lightTheme and darkTheme at App.tsx:25
and the 404 page sits inside that, so it also renders on the light
background - where #9a9a9a manages only 2.64:1 and reads as washed out.

#787878 is the balance point: 4.13:1 on the dark ground (#151518) and
4.14:1 on the light one (#F6F8F7), chosen by maximising the worse of
the two rather than tuning for either. Checked by rendering on both.
2026-09-08 19:31:22 -04:00
gorkem-bwl 3051bc7edc fix(client): credit Annie Hsu on the 404 illustration
The sushi artwork on the 404 page was designed by Annie Hsu and carried
no attribution. Adds her name in the bottom-right corner, in a script
face at a size that reads as a signature rather than a label.

The glyphs are embedded as vector outlines rather than a `<text>`
element. The asset is loaded through an `<img>` tag in
Pages/NotFound/index.tsx, and an img-loaded SVG cannot fetch external
resources, so a web font would silently fall back to whatever cursive
face the viewer happens to have - different on every platform. Outlines
render identically everywhere and stay sharp at any size, which a raster
export would not.

Adds ~21KB to the asset.
2026-09-08 19:23:44 -04:00
Alexander Holliday 22232489a8 Merge pull request #3934 from bluewave-labs/fix/admin-gate-docker-logs
fix: admin gate docker logs
2026-09-08 15:59:17 -07:00
Alex Holliday 04277a6eb4 add admin check to fetch request 2026-09-08 15:55:17 -07:00
Alex Holliday a8dc867e78 add admin gate to API route 2026-09-08 15:52:43 -07:00
Alex Holliday 533783ba3e gate client side log page behind admin check 2026-09-08 15:51:17 -07:00
Alexander Holliday fd7b3e5f20 Merge pull request #3933 from bluewave-labs/feat/docker-state-palette
feat: docker state palette
2026-09-08 15:36:31 -07:00
Alex Holliday ae4b851620 pass palette to stat box for docker state 2026-09-08 15:33:50 -07:00
Alex Holliday e83f97a05a add function to translate docker state to palette keys 2026-09-08 15:33:31 -07:00
Alexander Holliday 070682af74 Merge pull request #3932 from bluewave-labs/fix/test-all-admin-guard
fix: add admin guard to test all button, fix tests
2026-09-08 11:01:21 -07:00
Alex Holliday 8609e4bca0 add admin guard to test all button, fix tests 2026-09-08 10:46:36 -07:00
Alexander Holliday 2661d7a384 Merge pull request #3922 from alishanahmad907/fix/rbac-auth
fix(api): enforce RBAC role checks on maintenance windows, notifications, and check deletion
2026-09-08 10:43:01 -07:00
Alexander Holliday 5b9d554f46 Merge pull request #3910 from bluewave-labs/fix/checks-table-copy-message
fix(client): don't navigate away when selecting text in a table row
2026-09-08 10:31:56 -07:00
Alex Holliday 7fb8bab4be purge narrative comment 2026-09-08 10:21:44 -07:00
Alexander Holliday 75d29124b6 Merge pull request #3909 from signalgridco/feat/signalgrid-notifications
feat: add Signalgrid notification provider
2026-09-08 10:14:19 -07:00
Alexander Holliday 1f0ea6cedc Merge pull request #3914 from eegrok/fix/maintenance-window-timezone-copy
fix Start Time description to accurately reflect that the time zone is local not UTC
2026-09-08 10:07:10 -07:00
Alexander Holliday 07a38b9c2c Merge pull request #3925 from egeoztass/fix/create-monitor-return-document
fix: return the created monitor from createMonitor
2026-09-08 10:03:34 -07:00
egeoztassandClaude Fable 5.1 67806c94b1 fix: return the created monitor from createMonitor
The controller writes the service's return value into the response, but
createMonitor was typed Promise<void> and returned nothing, so every
successful POST /monitors answered data: null and callers had no way to
read the new monitor's id.

Fixes #3917

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D7eViZ2Np5ZpWUYNoRhmAt
2026-09-07 12:31:07 +02:00
Ali Shan Ahmad 2eeb088b61 [FIX]: rbac by adding the missing isAllowed(["admin", "superadmin"]) authorization middleware to mutating API endpoints for maintenance windows, notifications, and monitor check history. 2026-09-06 08:35:43 +00:00
gorkem-bwl 9c796edbde refactor(client): drop the copy button, keep the selection fix
The selection guard alone closes #3907: drag-selecting a check's error
message no longer navigates away, so the text can be copied. The copy
button was a second, larger answer to the same problem, and it brought a
reusable component, an HTTP clipboard fallback, four translation keys and
a timer to maintain.

Reverting it leaves the fix at ~29 lines in one file, and keeps the
behaviour change to the thing that was actually broken. CopyButton is
worth having for monitor URLs, tokens and incident ids, but it should
land on its own merits rather than inside a bug fix.

Claude-Session: https://claude.ai/code/session_01AbYLvwpEaZFXktHVcqrFRd
2026-09-03 00:24:07 -04:00
gorkem-bwl 84a1b994a1 fix(client): address review feedback on the copy-message change
Four issues from code review:

- The selection guard read the document-wide selection, so a leftover
  selection anywhere on the page could swallow the first click on any
  row. It now checks that the selection is inside the row that was
  clicked, with the event threaded through to both call sites.

- CopyButton's reset timer was never cleared, so it fired against an
  unmounted component when a row went away on pagination, and a rapid
  second click could not cancel the pending reset. Held in a ref and
  cleared on unmount and on re-copy.

- navigator.clipboard only exists on secure origins, and the README
  documents reaching Checkmate over plain HTTP on a LAN address. There
  the API is undefined, so every click showed "could not copy" with no
  way forward. Falls back to a hidden textarea.

- The two message cells were byte-identical, so they are now one shared
  CopyableText component.

Claude-Session: https://claude.ai/code/session_01AbYLvwpEaZFXktHVcqrFRd
2026-09-03 00:24:07 -04:00
gorkem-bwl db4689207a fix(client): make check error messages selectable and copyable
Closes #3907.

Selecting the text of a failed check's error message navigated away
before the selection could be copied. The mouseup that ends a drag-select
lands on the table row, which fires the row's onClick.

Two changes:

- DataTable ignores a row click when the user has an active text
  selection, so drag-selecting text inside any row no longer navigates.
  This applies to all eleven tables that pass onRowClick, not just checks.
- The message column gets an explicit copy button, which is the more
  discoverable of the two fixes the reporter suggested. Added as a shared
  CopyButton in design-elements so other columns can reuse it.

Note the row click on the uptime details checks table navigates to
/checks/<check id>, but that route's parameter is :monitorId. A check id
never matches a monitor, so the destination page renders empty - which is
the "all this information disappears" in the report. Left alone here to
keep this change reviewable; it needs its own decision about what that
row should link to.

Claude-Session: https://claude.ai/code/session_01AbYLvwpEaZFXktHVcqrFRd
2026-09-03 00:24:07 -04:00
eegrok af465766c8 fix Start Time description to accurately reflect that the time zone is local time, not UTC 2026-09-03 09:13:32 +12:00
fabiangigler c9b99eabb4 Localize monitor timing details 2026-09-02 20:15:39 +02:00
signalgridco c13f26a4c3 feat: add Signalgrid notification provider 2026-09-01 23:17:11 +02:00
127 changed files with 5457 additions and 1933 deletions

No files matched your search

+123 -8
View File
@@ -92,7 +92,7 @@ JWT_SECRET="$(openssl rand -hex 32)" docker compose up -d
Then open http://localhost:52345. If the app is reached at another origin (domain or LAN IP), set `CLIENT_HOST` accordingly. To build the image yourself, run `docker build -f docker/Dockerfile -t checkmate .` from a checkout. For TLS, put any reverse proxy (Caddy, Traefik, nginx) in front of port 52345.
There are also 1-click installation options like [Repocloud](https://repocloud.io/details/Checkmate),
[Pikapods](https://www.pikapods.com/), [Coolify](https://coolify.io/), [Elestio](https://elest.io/open-source/checkmate), [K8s](../../charts/helm/checkmate/INSTALLATION.md), [Sive Host](https://sive.host) or [Cloudzy](https://cloudzy.com/marketplace/checkmate).
[Pikapods](https://www.pikapods.com/), [Coolify](https://coolify.io/), [Elestio](https://elest.io/open-source/checkmate), [Easypanel](https://easypanel.io/templates/checkmate), [K8s](./charts/helm/checkmate/INSTALLATION.md), [Sive Host](https://sive.host) or [Cloudzy](https://cloudzy.com/marketplace/checkmate). Note that the Helm chart has not yet been migrated to the all-in-one image: it still deploys the legacy `checkmate-client`, `checkmate-backend` and `checkmate-mongo` images, pinned at v3.8.1.
### Configuration
@@ -104,7 +104,15 @@ The image is configured entirely through environment variables on the server con
| `DB_CONNECTION_STRING` | Yes | MongoDB connection string, e.g. `mongodb://mongodb:27017/uptime_db` |
| `JWT_SECRET` | Yes | Secret used to sign auth tokens; generate one with `openssl rand -hex 32` |
| `CLIENT_HOST` | Yes | The URL users reach the app at, e.g. `https://checkmate.example.com`; used for CORS and for links in notifications and emails |
| `ENCRYPTION_KEY` | No | Encrypts stored Docker TLS client keys at rest; generate one with `openssl rand -base64 32`. Comma-separated list: the first key encrypts, every key decrypts. Must be identical on the API and every worker. To rotate without downtime, deploy `OLD_KEY,NEW_KEY` everywhere, then `NEW_KEY,OLD_KEY` everywhere, wait for the worker to re-encrypt every row, then drop `OLD_KEY`. |
| `PORT` | No | Port the API and web client are served on (default `52345`) |
| `HEALTH_PORT` | No | Port for the `/livez`, `/readyz` and `/metrics` endpoints, served by any process that runs the job worker (default `52346`) |
| `NODE_ENV` | No | `development`, `production` or `test` (default `development`). `development` disables the general API rate limiter; set `production` on real deployments |
| `LOG_LEVEL` | No | Server log level: `error`, `warn`, `info`, or `debug` (default `debug`) |
| `TOKEN_TTL` | No | Lifetime of issued auth tokens, e.g. `12h` or `7d` (default `99d`) |
| `QUEUE_MODE` | No | `primary` (default) runs the API, the web client and the job scheduler; `worker` runs a job-processing worker only, with no API |
| `QUEUE_PRIMARY_PROCESSES` | No | `true` (default) or `false`. Whether a `primary` node also processes monitoring jobs itself; set `false` when dedicated `worker` nodes handle all checks. Ignored in `worker` mode |
| `STATUS_PAGE_THEMES_ENABLED` | No | `true` (default) or `false`. When `false`, status pages ignore theme settings and always render the default theme |
The web client needs no configuration by default: it calls the API on the same origin it was served from (`/api/v1`). For setups where the defaults don't apply — for example, the API is reached through a different origin than the page — the server renders overrides into the client at runtime via these optional variables:
@@ -118,12 +126,119 @@ The web client needs no configuration by default: it calls the API on the same o
See full installation instructions in the [Checkmate documentation portal](https://checkmate.so/docs).
Alternatively, you can also use [Coolify](https://coolify.io/), [Elestio](https://elest.io/open-source/checkmate), [K8s](./charts/helm/checkmate/INSTALLATION.md), [Sive Host](https://sive.host) (South Africa), [Cloudzy](https://cloudzy.com/marketplace/checkmate) or [Pikapods](https://www.pikapods.com/) to quickly spin off a Checkmate instance. If you would like to monitor your server infrastructure, you'll need [Capture agent](https://github.com/bluewave-labs/capture). Capture repository also contains the installation instructions.
Alternatively, you can also use [Coolify](https://coolify.io/), [Elestio](https://elest.io/open-source/checkmate), [K8s](./charts/helm/checkmate/INSTALLATION.md) (legacy images, pinned at v3.8.1), [Sive Host](https://sive.host) (South Africa), [Cloudzy](https://cloudzy.com/marketplace/checkmate) or [Pikapods](https://www.pikapods.com/) to quickly spin off a Checkmate instance. If you would like to monitor your server infrastructure, you'll need [Capture agent](https://github.com/bluewave-labs/capture). Capture repository also contains the installation instructions.
### Using a Custom CA
If you need to monitor internal HTTPS endpoints with certificates from private Certificate Authorities (like Smallstep), see our [Custom CA Trust Guide](./docs/custom-ca-trust.md) for Docker configuration options.
### Docker monitors
A Docker monitor connects to a Docker daemon and reports on every container it runs. The daemon's ping response decides whether the monitor is up or down and its latency is the response time. Each check also records every container's state, health, CPU and memory usage, restart count, published ports and mounts. Enabling **Collect container logs** additionally stores the latest 200 log lines per container on every check; logs are kept for 7 days.
The **Docker host** field accepts two forms:
| Host | Example | Notes |
|---|---|---|
| Local socket | `unix:///var/run/docker.sock` | Also accepts a bare absolute path such as `/var/run/docker.sock`. Use this for the daemon on the same machine Checkmate runs on. |
| Remote daemon | `tcp://docker.example.com:2376` | Always uses mutual TLS; the port defaults to `2376`. Unencrypted daemons on `2375` are not supported. |
**Monitoring the local socket.** The reference Compose file does not mount the socket, so add it and grant the container the host's `docker` group. The image runs as an unprivileged user and cannot read the socket otherwise. Find the group id with `stat -c %g /var/run/docker.sock`, then:
```yaml
services:
checkmate:
volumes:
- /var/run/docker.sock:/var/run/docker.sock:ro
group_add:
- "989" # the gid printed by stat
```
**Monitoring a remote daemon.** Set the host to `tcp://host:port` and fill in the **TLS credentials** section of the monitor form with the same PEM files you would pass to `docker --tlsverify`:
- **CA certificate**: the CA that signed the daemon's server certificate. Required unless **Ignore TLS/SSL errors** is on, which skips verifying the daemon's identity.
- **Client certificate**: the certificate the daemon uses to authenticate Checkmate.
- **Client key**: the matching private key, unencrypted (no passphrase). Checkmate checks that it matches the certificate, encrypts it with `ENCRYPTION_KEY`, and never shows it again. Leave the field blank when editing to keep the stored key.
TLS Docker monitors require `ENCRYPTION_KEY` to be set on the server (see [Configuration](#configuration)). Saving one without it fails with an error telling you so. If the key is ever removed or rotated incorrectly, affected checks fail with a decryption error until it is restored.
If the daemon's certificate is signed by a private CA that you also want the rest of Checkmate to trust, see the [Custom CA Trust Guide](./docs/custom-ca-trust.md); for Docker monitors alone, the **CA certificate** field is enough.
### Securing the Docker daemon with TLS
The Docker daemon does not enable TLS by default. Follow these steps on the Docker host to generate a CA, a server certificate, and a client certificate for Checkmate. This is the procedure from [Docker's own guide](https://docs.docker.com/engine/security/protect-access/), condensed. Replace `docker.example.com` and `203.0.113.10` with your daemon's DNS name and IP.
**1. Create a CA.** The CA key gets a passphrase; keep it offline once the certificates are issued.
```bash
openssl genrsa -aes256 -out ca-key.pem 4096
openssl req -new -x509 -days 365 -key ca-key.pem -sha256 -subj "/CN=docker-ca" -out ca.pem
```
**2. Create the server certificate.** The subject alternative names must cover every name or address Checkmate will use to reach the daemon.
```bash
openssl genrsa -out server-key.pem 4096
openssl req -subj "/CN=docker.example.com" -sha256 -new -key server-key.pem -out server.csr
cat > server-ext.cnf <<EOF
subjectAltName = DNS:docker.example.com,IP:203.0.113.10
extendedKeyUsage = serverAuth
EOF
openssl x509 -req -days 365 -sha256 -in server.csr -CA ca.pem -CAkey ca-key.pem -CAcreateserial \
-out server-cert.pem -extfile server-ext.cnf
```
**3. Create the client certificate for Checkmate.** Do not add a passphrase to this key; Checkmate cannot use encrypted private keys.
```bash
openssl genrsa -out key.pem 4096
openssl req -subj "/CN=checkmate" -new -key key.pem -out client.csr
echo "extendedKeyUsage = clientAuth" > client-ext.cnf
openssl x509 -req -days 365 -sha256 -in client.csr -CA ca.pem -CAkey ca-key.pem -CAcreateserial \
-out cert.pem -extfile client-ext.cnf
rm client.csr server.csr server-ext.cnf client-ext.cnf
chmod 0400 ca-key.pem key.pem server-key.pem
chmod 0444 ca.pem server-cert.pem cert.pem
```
**4. Point the daemon at the certificates.** Move `ca.pem`, `server-cert.pem` and `server-key.pem` to `/etc/docker/certs/` and configure `/etc/docker/daemon.json`:
```json
{
"hosts": ["unix:///var/run/docker.sock", "tcp://0.0.0.0:2376"],
"tls": true,
"tlsverify": true,
"tlscacert": "/etc/docker/certs/ca.pem",
"tlscert": "/etc/docker/certs/server-cert.pem",
"tlskey": "/etc/docker/certs/server-key.pem"
}
```
On distributions where the systemd unit already passes `-H fd://` (Debian, Ubuntu and derivatives), the daemon refuses to start with `hosts` set in both places. Remove the flag from the unit with an override, then restart:
```bash
sudo systemctl edit docker.service
```
```ini
[Service]
ExecStart=
ExecStart=/usr/bin/dockerd
```
```bash
sudo systemctl daemon-reload && sudo systemctl restart docker
```
Open port `2376` on the host firewall only to the machine running Checkmate.
**5. Verify from the Checkmate host**, then paste `ca.pem`, `cert.pem` and `key.pem` into the monitor form:
```bash
docker --tlsverify --tlscacert=ca.pem --tlscert=cert.pem --tlskey=key.pem \
-H=docker.example.com:2376 version
```
For more documentation, see the [docs directory](./docs/).
<a id="performance"></a>
@@ -133,7 +248,7 @@ Thanks to extensive optimizations, Checkmate operates with an exceptionally smal
![image](https://github.com/user-attachments/assets/37e04a75-d83a-488f-b25c-025511b492c9)
You can see the memory footprint of MongoDB and Redis on the same server (398Mb and 15Mb) for the same amount of servers:
You can see the memory footprint of MongoDB on the same server (398Mb) for the same amount of servers:
![image](https://github.com/user-attachments/assets/3b469e85-e675-4040-a162-3f24c1afc751)
@@ -151,16 +266,16 @@ Feel free to ask questions or share your ideas - we'd love to hear from you!
## Features
- Completely open source, deployable on your servers or home devices (e.g Raspberry Pi 4 or 5)
- Several monitoring options: Uptime, Docker, Ping, SSL, Port, Game server
- Several monitoring options: HTTP (with SSL certificate expiry), Ping, Port, DNS, Docker, gRPC, WebSocket, Game server
- Page speed monitoring
- Infrastructure monitoring (memory, disk usage, CPU performance, network etc) - requires [Capture](https://github.com/bluewave-labs/capture) agent
- Selective disk monitoring with mountpoint selection
- Incidents at a glance
- Status pages with 4 beautiful themes
- E-mail, Webhooks, Discord, Slack, PagerDuty, Matrix, Rocket.Chat, Microsoft Teams, Telegram, Pushover, Twilio (SMS) notifications
- Status pages with 5 beautiful themes
- E-mail, Webhooks, Discord, Slack, PagerDuty, Matrix, Rocket.Chat, Microsoft Teams, Telegram, Pushover, ntfy, SignalGrid, Twilio (SMS) notifications
- Scheduled maintenance
- JSON query monitoring
- Multi-language support for Arabic, Chinese (Simplified), Chinese (Traditional, Taiwan), Czech, English, Finnish, French, German, Japanese, Portuguese (Brazil), Russian, Spanish, Thai, Turkish, Ukrainian, and Vietnamese
- Multi-language support for Arabic, Catalan, Chinese (Simplified), Chinese (Traditional, Taiwan), Czech, English, Finnish, French, German, Italian, Japanese, Polish, Portuguese (Brazil), Russian, Spanish, Thai, Turkish, Ukrainian, and Vietnamese
## Monitor Lifecycle
@@ -213,7 +328,7 @@ Feel free to ask questions or share your ideas - we'd love to hear from you!
We are [Alex](http://github.com/ajhollid) (team lead), [Gorkem](http://github.com/gorkem-bwl/), [Aryaman](https://github.com/Br0wnHammer), [Malena](https://github.com/malenacaroline) and [Mert](https://github.com/mertssmnoglu) helping individuals and businesses monitor their infra and servers.
We pride ourselves on building strong connections with contributors at every level. Despite being a young project, Checkmate has already earned almost 11K+ stars and attracted 150+ contributors from around the globe.
We pride ourselves on building strong connections with contributors at every level. Despite being a young project, Checkmate has already earned almost 11K stars and attracted 150+ contributors from around the globe.
Our repo is starred by employees from **Google, Microsoft, Intel, Cisco, Tencent, Electronic Arts, ByteDance, JP Morgan Chase, Deloitte, Accenture, Foxconn, Broadcom, China Telecom, Barclays, Capgemini, Wipro, Cloudflare, Dassault Systèmes and NEC**, so don’t hold back — jump in, contribute and learn with us!
+2 -2
View File
@@ -1,12 +1,12 @@
{
"name": "client",
"version": "3.11.0",
"version": "3.12.0",
"lockfileVersion": 3,
"requires": true,
"packages": {
"": {
"name": "client",
"version": "3.11.0",
"version": "3.12.0",
"dependencies": {
"@csstools/css-tokenizer": "^3.0.4",
"@emotion/react": "^11.13.3",
+1 -1
View File
@@ -1,7 +1,7 @@
{
"name": "client",
"private": true,
"version": "3.11.0",
"version": "3.12.0",
"type": "module",
"scripts": {
"lockfile": "docker run --rm -v \"$PWD\":/w -w /w $(grep -m1 -o 'node:[^ ]*' ../docker/Dockerfile) npm install --package-lock-only",
@@ -8,8 +8,11 @@ import { typographyLevels } from "@/Utils/Theme/Palette";
import useMediaQuery from "@mui/material/useMediaQuery";
import { LAYOUT } from "@/Utils/Theme/constants";
import { formatDuration } from "@/Utils/TimeUtils";
import { useTranslation } from "react-i18next";
export const MonitorStatus = ({ monitor }: { monitor: Monitor }) => {
const theme = useTheme();
const { t, i18n } = useTranslation();
const isSmall = useMediaQuery(theme.breakpoints.down("md"));
if (!monitor) {
@@ -52,7 +55,12 @@ export const MonitorStatus = ({ monitor }: { monitor: Monitor }) => {
<>
<Dot />
<Typography>
Checking every {formatDuration(monitor?.interval, true)}
{t("pages.common.monitors.statBoxes.checkingEvery", {
interval: formatDuration(monitor.interval, {
long: true,
locale: i18n.resolvedLanguage,
}),
})}
</Typography>
</>
)}
@@ -81,6 +81,21 @@ export function DataTable<
};
const isSmall = useMediaQuery(theme.breakpoints.down("md"));
const hasTextSelectionWithin = (element: Element) => {
const selection = window.getSelection();
if (!selection || selection.isCollapsed || !selection.toString().trim()) {
return false;
}
return (
selection.anchorNode !== null &&
element.contains(
selection.anchorNode.nodeType === Node.TEXT_NODE
? selection.anchorNode.parentNode
: selection.anchorNode
)
);
};
const isInteractive = Boolean(onRowClick) || expandableRows;
if (data.length === 0 || headers.length === 0) {
@@ -102,7 +117,10 @@ export function DataTable<
keys.push(key);
return (
<Stack
onClick={() => (onRowClick ? onRowClick(row) : null)}
onClick={(e) => {
if (hasTextSelectionWithin(e.currentTarget)) return;
if (onRowClick) onRowClick(row);
}}
spacing={theme.spacing(LAYOUT.XS)}
sx={{
borderStyle: "solid",
@@ -256,7 +274,8 @@ export function DataTable<
}),
...(getRowSx?.(row) as object),
}}
onClick={() => {
onClick={(e) => {
if (hasTextSelectionWithin(e.currentTarget)) return;
if (expandableRows) handleExpand(row);
else if (onRowClick) onRowClick(row);
}}
@@ -5,9 +5,10 @@ import {
type FieldValues,
} from "react-hook-form";
import { Autocomplete } from "@/Components/inputs";
import type { AutocompleteProps } from "@mui/material/Autocomplete";
export interface AutocompleteOption {
id: string;
id: string | number;
name: string;
}
@@ -16,6 +17,9 @@ interface FormAutocompleteProps<T extends FieldValues, O extends AutocompleteOpt
options: O[];
fieldLabel?: string;
placeholder?: string;
filterOptions?: AutocompleteProps<O, false, false, false>["filterOptions"];
getOptionDisabled?: AutocompleteProps<O, false, false, false>["getOptionDisabled"];
disableClearable?: boolean;
}
export const FormAutocompleteField = <
@@ -26,6 +30,9 @@ export const FormAutocompleteField = <
options,
fieldLabel,
placeholder,
filterOptions,
getOptionDisabled,
disableClearable,
}: FormAutocompleteProps<T, O>) => {
const { control } = useFormContext<T>();
return (
@@ -39,10 +46,14 @@ export const FormAutocompleteField = <
isOptionEqualToValue={(option, value) => option.id === value.id}
value={options.find((o) => o.id === field.value) ?? null}
onChange={(_: unknown, newValue: O | null) => {
if (newValue === null && disableClearable) return;
field.onChange(newValue?.id ?? "");
}}
fieldLabel={fieldLabel}
placeholder={placeholder}
filterOptions={filterOptions}
getOptionDisabled={getOptionDisabled}
disableClearable={disableClearable}
error={!!fieldState.error}
helperText={fieldState.error?.message ?? ""}
/>
@@ -60,20 +60,22 @@ export const HeaderMonitorControls = ({
direction={{ xs: "column", md: "row" }}
gap={theme.spacing(2)}
>
<Button
variant="contained"
color="secondary"
loading={isPosting}
startIcon={<Icon icon={Mail} />}
onClick={async () => {
await post(`/notifications/test/all`, { monitorId: monitor.id });
}}
sx={{
whiteSpace: "nowrap",
}}
>
{t("common.buttons.testNotifications")}
</Button>
{isAdmin && (
<Button
variant="contained"
color="secondary"
loading={isPosting}
startIcon={<Icon icon={Mail} />}
onClick={async () => {
await post(`/notifications/test/all`, { monitorId: monitor.id });
}}
sx={{
whiteSpace: "nowrap",
}}
>
{t("common.buttons.testNotifications")}
</Button>
)}
<Button
variant="contained"
color="secondary"
@@ -21,7 +21,7 @@ export const MonitorStatBoxes = ({
domainExpiry,
}: MonitorStatBoxesProps) => {
const theme = useTheme();
const { t } = useTranslation();
const { t, i18n } = useTranslation();
if (!monitorStats || !monitor) {
return null;
}
@@ -29,13 +29,16 @@ export const MonitorStatBoxes = ({
const timeOfLastFailure = monitorStats?.timeOfLastFailure || 0;
const timeSinceLastFailure = timeOfLastFailure > 0 ? Date.now() - timeOfLastFailure : 0;
// Determine time since last check
const timeOfLastCheck = monitorStats?.lastCheckTimestamp || 0;
const timeSinceLastCheck = Date.now() - timeOfLastCheck || 0;
const streakTime = formatDuration(timeSinceLastFailure);
const lastCheckTime = formatDuration(timeSinceLastCheck);
const lastCheckTime = timeOfLastCheck
? formatDuration(Date.now() - timeOfLastCheck, {
long: true,
relative: true,
locale: i18n.resolvedLanguage,
})
: t("common.labels.na");
const isActive =
monitor?.status === "up" ||
monitor?.status === "paused" ||
@@ -1,15 +1,58 @@
import { BaseChart } from "@/Components/design-elements";
import { Area, AreaChart, CartesianGrid, ResponsiveContainer } from "recharts";
import { BaseChart, BaseBox } from "@/Components/design-elements";
import { Area, AreaChart, CartesianGrid, ResponsiveContainer, Tooltip } from "recharts";
import Typography from "@mui/material/Typography";
// Types
import type { DockerContainerStats } from "@/Types/Monitor";
import type { TooltipProps } from "recharts";
import type { NameType, ValueType } from "recharts/types/component/DefaultTooltipContent";
import type { Theme } from "@mui/material/styles";
import type { RootState } from "@/Types/state";
// Hooks
import { useTheme } from "@mui/material";
import { Fragment, useId } from "react";
import { useTranslation } from "react-i18next";
import { useSelector } from "react-redux";
// Utils
import { createGradient } from "@/Components/monitors/charts/ChartUtils";
import { tooltipDateFormatLookup, formatDateWithTz } from "@/Utils/TimeUtils";
import type { DateRange } from "@/Types/Query";
type DockerHistogramToolTipProps = TooltipProps<ValueType, NameType> & {
dataKey: string;
range: string;
theme: Theme;
uiTimezone: string;
valueFormatter: (value: number) => string;
labelKey: string;
};
const DockerHistogramToolTip = ({
dataKey,
active,
payload,
range,
theme,
uiTimezone,
valueFormatter,
labelKey,
}: DockerHistogramToolTipProps) => {
const { t } = useTranslation();
if (!payload) return null;
if (!active) return null;
const format = tooltipDateFormatLookup(range);
const ts = payload[0].payload?._id ?? null;
const dataPoint = payload[0].payload?.[dataKey] ?? null;
return (
<BaseBox sx={{ py: theme.spacing(2), px: theme.spacing(4) }}>
<Typography>{formatDateWithTz(ts, format, uiTimezone)}</Typography>
<Typography>{t(labelKey, { value: valueFormatter(dataPoint) })}</Typography>
</BaseBox>
);
};
interface HistogramDockerContainerProps {
title: string;
rightTitle: string;
@@ -17,6 +60,9 @@ interface HistogramDockerContainerProps {
dataKey: string;
strokeColor: string;
gradientStartColor: string;
range: DateRange;
formatter: (value: number) => string;
labelKey: string;
}
export const HistogramDockerContainer = ({
@@ -26,11 +72,14 @@ export const HistogramDockerContainer = ({
dataKey,
strokeColor,
gradientStartColor,
range,
formatter,
labelKey,
}: HistogramDockerContainerProps) => {
const uniqueId = useId();
const theme = useTheme();
const gradientId = `gradient-${uniqueId}`;
const uiTimezone = useSelector((state: RootState) => state.ui.timezone);
return (
<BaseChart
icon="null"
@@ -56,6 +105,19 @@ export const HistogramDockerContainer = ({
endColor: "transparent",
direction: "vertical",
})}
<Tooltip
content={(props) => (
<DockerHistogramToolTip
{...props}
range={range}
theme={theme}
uiTimezone={uiTimezone}
dataKey={dataKey}
valueFormatter={formatter}
labelKey={labelKey}
/>
)}
/>
<Area
dataKey={dataKey}
type="monotone"
+6
View File
@@ -20,12 +20,18 @@ import {
Tag,
Waypoints,
Container,
// LayoutDashboard,
} from "lucide-react";
export const getMenu = (t: Function) => {
return [
{
group: t("components.sidebar.menu.groups.monitoring"),
items: [
// {
// name: t("components.sidebar.menu.dashboard"),
// path: "dashboard",
// icon: <Icon icon={LayoutDashboard} />,
// },
{
name: t("components.sidebar.menu.uptime"),
path: "uptime",
+6
View File
@@ -77,6 +77,12 @@ export const getMonitorDefaults = (
...base,
type: "docker",
url: data?.url || "",
ignoreTlsErrors: data?.ignoreTlsErrors || false,
dockerTlsCa: data?.dockerTlsCa || "",
dockerTlsCert: data?.dockerTlsCert || "",
// The key is write-only; the API never returns it.
dockerTlsKey: "",
dockerTlsKeySet: data?.dockerTlsKeySet ?? false,
dockerLogsEnabled: data?.dockerLogsEnabled ?? false,
};
break;
+130 -12
View File
@@ -38,6 +38,7 @@ import {
PageSpeedStrategies,
DnsRecordTypes,
MonitorIntervalOptions,
MIN_MONITOR_INTERVAL_MS,
DefaultMonitorMatchMethod,
MonitorMatchMethods,
GeoCheckIntervalOptions,
@@ -50,6 +51,7 @@ import type { AppSettingsResponse } from "@/Types/Settings";
import {
stepFieldsFor,
monitorStepCount,
isDockerTlsUrl,
type MonitorFormData,
} from "@/Validation/monitor";
import { FormNumberField } from "@/Components/inputs/forms/FormNumberField";
@@ -57,15 +59,62 @@ import { FormTextField } from "@/Components/inputs/forms/FormTextField";
import { FormRadioGroup } from "@/Components/inputs/forms/FormRadioGroupField";
import { FormMultiSelectField } from "@/Components/inputs/forms/FormMultiSelectField";
import { FormSelectField } from "@/Components/inputs/forms/FormSelectField";
import {
FormAutocompleteField,
type AutocompleteOption,
} from "@/Components/inputs/forms/FormAutoCompleteField";
import { FormSliderField } from "@/Components/inputs/forms/FormSliderField";
import { FormSwitchField } from "@/Components/inputs/forms/FormSwitchField";
import type { ProxyResponse } from "@/Types/Proxy";
import { formatDuration } from "@/Utils/TimeUtils";
const httpMethodOptions = HttpMethods.map((method) => ({
value: method,
label: method,
}));
const PEM_FIELD_ROWS = 6;
// Hides the pasted key while the field is not focused. Ignored by browsers
// without -webkit-text-security, which then show the key as plain text.
const maskedInputSx = {
"&:not(.Mui-focused) textarea": { WebkitTextSecurity: "disc" },
};
interface IntervalOption extends AutocompleteOption {
id: number;
}
const INTERVAL_UNIT_MULTIPLIERS = [1000, 60000, 3600000, 86400000] as const;
const toIntervalOption = (
intervalMs: number,
locale: string | undefined
): IntervalOption => ({
id: intervalMs,
name: formatDuration(intervalMs, { long: true, locale }),
});
const filterIntervalOptions = (
options: IntervalOption[],
inputValue: string,
locale: string | undefined
): IntervalOption[] => {
const input = inputValue.trim();
if (input === "" || options.some((option) => option.name === input)) {
return options;
}
const amount = Number(input);
if (!Number.isFinite(amount) || amount <= 0) {
return [];
}
return INTERVAL_UNIT_MULTIPLIERS.map((multiplier) => amount * multiplier)
.filter(Number.isSafeInteger)
.map((ms) => toIntervalOption(ms, locale));
};
interface GeneralSettingsConfig {
urlLabel: string;
urlPlaceholder: string;
@@ -243,7 +292,7 @@ const getGeneralSettingsConfig = (
const CreateMonitorPage = () => {
const theme = useTheme();
const { t } = useTranslation();
const { t, i18n } = useTranslation();
const { monitorId } = useParams();
const location = useLocation();
const navigate = useNavigate();
@@ -300,10 +349,12 @@ const CreateMonitorPage = () => {
const showStep = (step: number) => isEditMode || currentStep === step;
const watchedType = watch("type") as MonitorType;
const watchedUrl = watch("url") as string;
const watchedMethod = watch("method") as HttpMethod | undefined;
const watchedProxyMode = watch("proxyMode") as ProxyMode | undefined;
const watchedUseAdvancedMatching = watch("useAdvancedMatching") as boolean;
const watchGeoCheckEnabled = watch("geoCheckEnabled") as boolean;
const watchedInterval = watch("interval");
// Steps without an advanced section drop it, so the last step is the form's.
const totalSteps = monitorStepCount(watchedType);
@@ -334,6 +385,8 @@ const CreateMonitorPage = () => {
[watchedType, t]
);
const pemInputSx = { fontFamily: theme.typography.fontFamilyMonospace };
const { post, loading: isCreating } = usePost<MonitorFormData, Monitor>();
const { patch, loading: isUpdating } = usePatch<MonitorFormData, Monitor>();
const isSubmitting = isCreating || isUpdating;
@@ -459,16 +512,18 @@ const CreateMonitorPage = () => {
[t]
);
const intervalOptions = useMemo(
() =>
MonitorIntervalOptions.map((option) => ({
value: option.value,
label: t(
`pages.createMonitor.form.frequency.option.frequency.value.${option.labelKey}`
),
})),
[t]
);
const intervalOptions = useMemo(() => {
const options = MonitorIntervalOptions.map(({ value }) =>
toIntervalOption(value, i18n.resolvedLanguage)
);
if (
typeof watchedInterval === "number" &&
!options.some((option) => option.id === watchedInterval)
) {
options.push(toIntervalOption(watchedInterval, i18n.resolvedLanguage));
}
return options;
}, [watchedInterval, i18n.resolvedLanguage]);
const matchMethodOptions = useMemo(
() =>
@@ -689,17 +744,80 @@ const CreateMonitorPage = () => {
/>
)}
{showStep(0) && watchedType === "docker" && isDockerTlsUrl(watchedUrl) && (
<ConfigBox
title={t("pages.createMonitor.form.dockerTls.title")}
subtitle={t("pages.createMonitor.form.dockerTls.description")}
rightContent={
<Stack spacing={theme.spacing(LAYOUT.MD)}>
<FormTextField
name="dockerTlsCa"
multiline
rows={PEM_FIELD_ROWS}
fieldLabel={t("pages.createMonitor.form.dockerTls.option.ca.label")}
placeholder={t(
"pages.createMonitor.form.dockerTls.option.ca.placeholder"
)}
slotProps={{ input: { sx: pemInputSx } }}
/>
<FormTextField
name="dockerTlsCert"
multiline
rows={PEM_FIELD_ROWS}
fieldLabel={t("pages.createMonitor.form.dockerTls.option.cert.label")}
placeholder={t(
"pages.createMonitor.form.dockerTls.option.cert.placeholder"
)}
slotProps={{ input: { sx: pemInputSx } }}
/>
<FormTextField
name="dockerTlsKey"
multiline
rows={PEM_FIELD_ROWS}
fieldLabel={t("pages.createMonitor.form.dockerTls.option.key.label")}
placeholder={t(
"pages.createMonitor.form.dockerTls.option.key.placeholder"
)}
helperText={
existingMonitor?.dockerTlsKeySet
? t("pages.createMonitor.form.dockerTls.option.key.stored")
: undefined
}
autoComplete="off"
slotProps={{ input: { sx: { ...pemInputSx, ...maskedInputSx } } }}
/>
<FormSwitchField
name="ignoreTlsErrors"
label={t("pages.createMonitor.form.dockerTls.option.ignoreTls.label")}
/>
<Typography
component="span"
color={theme.palette.text.secondary}
sx={{ opacity: 0.8 }}
>
{t("pages.createMonitor.form.dockerTls.option.ignoreTls.description")}
</Typography>
</Stack>
}
/>
)}
{showStep(1) && (
<ConfigBox
title={t("pages.createMonitor.form.frequency.title")}
subtitle={t("pages.createMonitor.form.frequency.description")}
rightContent={
<FormSelectField
<FormAutocompleteField
name="interval"
fieldLabel={t(
"pages.createMonitor.form.frequency.option.frequency.label"
)}
options={intervalOptions}
filterOptions={(options, { inputValue }) =>
filterIntervalOptions(options, inputValue, i18n.resolvedLanguage)
}
getOptionDisabled={(option) => option.id < MIN_MONITOR_INTERVAL_MS}
disableClearable
/>
}
/>
@@ -0,0 +1,54 @@
import Box from "@mui/material/Box";
import Stack from "@mui/material/Stack";
import Typography from "@mui/material/Typography";
import { useTheme } from "@mui/material/styles";
import { LAYOUT } from "@/Utils/Theme/constants";
import { typographyLevels } from "@/Utils/Theme/Palette";
interface DashboardCardProps {
title: string;
topRight?: React.ReactNode;
children: React.ReactNode;
}
export const DashboardCard = ({ title, topRight, children }: DashboardCardProps) => {
const theme = useTheme();
return (
<Box
bgcolor={theme.palette.background.paper}
border={1}
borderColor={theme.palette.divider}
borderRadius={`${theme.shape.borderRadius}px`}
padding={theme.spacing(LAYOUT.MD)}
display="flex"
flexDirection="column"
gap={theme.spacing(LAYOUT.SM)}
height="100%"
boxSizing="border-box"
>
<Stack
direction="row"
alignItems="center"
justifyContent="space-between"
>
<Typography
fontSize={typographyLevels.m}
fontWeight={500}
color={theme.palette.text.primary}
>
{title}
</Typography>
{topRight && (
<Typography
fontSize={typographyLevels.s}
color={theme.palette.text.secondary}
>
{topRight}
</Typography>
)}
</Stack>
{children}
</Box>
);
};
@@ -0,0 +1,28 @@
import { LAYOUT } from "@/Utils/Theme/constants";
import Box from "@mui/material/Box";
import { useTheme } from "@mui/material/styles";
interface ProgressBarProps {
fillRatio: number;
}
export const ProgressBar = ({ fillRatio }: ProgressBarProps) => {
const theme = useTheme();
const clamped = Math.min(1, Math.max(0, fillRatio));
return (
<Box
height={theme.spacing(LAYOUT.XS)}
borderRadius={theme.shape.borderRadius}
bgcolor={theme.palette.action.disabledBackground}
overflow="hidden"
>
<Box
height="100%"
width={`${clamped * 100}%`}
borderRadius={theme.shape.borderRadius}
bgcolor={theme.palette.primary.main}
/>
</Box>
);
};
@@ -0,0 +1,42 @@
import Stack from "@mui/material/Stack";
import Typography from "@mui/material/Typography";
import { useTheme } from "@mui/material/styles";
import { typographyLevels } from "@/Utils/Theme/Palette";
import { SPACING } from "@/Utils/Theme/constants";
import { ProgressBar } from "@/Pages/Dashboard/components/ProgressBar";
interface BarRowProps {
label: string;
value: string;
fillRatio: number;
valueColor?: string;
}
export const BarRow = ({ label, value, fillRatio, valueColor }: BarRowProps) => {
const theme = useTheme();
return (
<Stack gap={theme.spacing(SPACING.LG)}>
<Stack
direction="row"
justifyContent="space-between"
alignItems="center"
>
<Typography
fontSize={typographyLevels.m}
color={theme.palette.text.primary}
>
{label}
</Typography>
<Typography
fontSize={typographyLevels.m}
color={valueColor ?? theme.palette.primary.main}
fontWeight={500}
>
{value}
</Typography>
</Stack>
<ProgressBar fillRatio={fillRatio} />
</Stack>
);
};
@@ -0,0 +1,49 @@
import Stack from "@mui/material/Stack";
import Typography from "@mui/material/Typography";
import { useTheme } from "@mui/material/styles";
import { DashboardCard } from "@/Pages/Dashboard/components/DashboardCard";
import { ProgressBar } from "@/Pages/Dashboard/components/ProgressBar";
import { typographyLevels } from "@/Utils/Theme/Palette";
import { SPACING } from "@/Utils/Theme/constants";
import { useTranslation } from "react-i18next";
import type { MonitorsSummary } from "@/Types/Monitor";
interface MonitorStatusCardProps {
summary: MonitorsSummary;
}
export const MonitorStatusCard = ({ summary }: MonitorStatusCardProps) => {
const theme = useTheme();
const { t } = useTranslation();
const up = summary.upMonitors;
const total = summary.totalMonitors;
const ratio = total > 0 ? up / total : 0;
return (
<DashboardCard title={t("pages.dashboard.cards.monitorStatus")}>
<Stack gap={theme.spacing(SPACING.SM)}>
<Stack
direction="row"
alignItems="baseline"
gap={theme.spacing(SPACING.XS)}
>
<Typography
fontSize={typographyLevels.xxl}
fontWeight={500}
color={theme.palette.text.primary}
>
{up}
</Typography>
<Typography
fontSize={typographyLevels.m}
color={theme.palette.text.secondary}
>
{t("pages.dashboard.monitorStatus.ofUp", { total })}
</Typography>
</Stack>
<ProgressBar fillRatio={ratio} />
</Stack>
</DashboardCard>
);
};
@@ -0,0 +1,35 @@
import Stack from "@mui/material/Stack";
import { DashboardCard } from "@/Pages/Dashboard/components/DashboardCard";
import { BarRow } from "@/Pages/Dashboard/components/cards/BarRow";
import { useTranslation } from "react-i18next";
import { useTheme } from "@mui/material/styles";
import type { DashboardTypeCount } from "@/Pages/Dashboard/types";
import { getMonitorTypeLabel } from "@/Types/Monitor";
import { LAYOUT } from "@/Utils/Theme/constants";
interface MonitorsByTypeCardProps {
monitorsByType: DashboardTypeCount[];
}
export const MonitorsByTypeCard = ({ monitorsByType }: MonitorsByTypeCardProps) => {
const { t } = useTranslation();
const theme = useTheme();
const maxCount = monitorsByType[0]?.count ?? 1;
return (
<DashboardCard title={t("pages.dashboard.cards.monitorsByType")}>
<Stack gap={theme.spacing(LAYOUT.SM)}>
{monitorsByType.map(({ type, count }) => (
<BarRow
key={type}
label={getMonitorTypeLabel(type, t)}
value={String(count)}
fillRatio={count / maxCount}
valueColor={theme.palette.text.primary}
/>
))}
</Stack>
</DashboardCard>
);
};
+45
View File
@@ -0,0 +1,45 @@
import Box from "@mui/material/Box";
import { useTheme } from "@mui/material/styles";
import { BasePage } from "@/Components/design-elements/BasePage";
import { LAYOUT } from "@/Utils/Theme/constants";
import { MonitorStatusCard } from "@/Pages/Dashboard/components/cards/MonitorStatusCard";
import { MonitorsByTypeCard } from "@/Pages/Dashboard/components/cards/MonitorsByTypeCard";
import type { MonitorsSummary } from "@/Types/Monitor";
import type { DashboardTypeCount } from "@/Pages/Dashboard/types";
const MOCK_SUMMARY: MonitorsSummary = {
totalMonitors: 12,
upMonitors: 10,
downMonitors: 1,
pausedMonitors: 1,
initializingMonitors: 0,
maintenanceMonitors: 0,
breachedMonitors: 0,
};
const MOCK_MONITORS_BY_TYPE: DashboardTypeCount[] = [
{ type: "http", count: 7 },
{ type: "ping", count: 3 },
{ type: "port", count: 2 },
];
const Dashboard = () => {
const theme = useTheme();
return (
<BasePage headerKey="dashboard">
<Box
display="grid"
gridTemplateColumns={{ xs: "1fr", md: "1fr 1fr" }}
gap={theme.spacing(LAYOUT.MD)}
>
<MonitorStatusCard summary={MOCK_SUMMARY} />
<MonitorsByTypeCard monitorsByType={MOCK_MONITORS_BY_TYPE} />
</Box>
</BasePage>
);
};
export default Dashboard;
+6
View File
@@ -0,0 +1,6 @@
import type { MonitorType } from "@/Types/Monitor";
export interface DashboardTypeCount {
type: MonitorType;
count: number;
}
Whitespace-only changes.
@@ -10,6 +10,7 @@ import type { DockerContainerInfo } from "@/Types/Check";
// Utils
import { formatDuration } from "@/Utils/TimeUtils";
import { getDockerPalette } from "@/Utils/MonitorUtils";
export const DockerContainerStatusBoxes = ({
container,
@@ -24,6 +25,7 @@ export const DockerContainerStatusBoxes = ({
const { state, restartCount, health } = container;
const startedAt = container.startedAt ?? "0";
const runningFor = Date.now() - new Date(startedAt).getTime();
const palette = getDockerPalette(state);
return (
<Stack
direction="row"
@@ -33,6 +35,7 @@ export const DockerContainerStatusBoxes = ({
<StatBox
title={t("common.labels.state")}
subtitle={state}
palette={palette}
/>
<StatBox
title={t("common.labels.uptime")}
@@ -23,6 +23,7 @@ import {
import { useTheme } from "@mui/material";
import { useTranslation } from "react-i18next";
import { useToast } from "@/Hooks/UseToast";
import { useIsAdmin } from "@/Hooks/useIsAdmin";
// Utils
import { get } from "@/Utils/ApiClient";
@@ -50,6 +51,7 @@ export const TabLogs = ({ monitorId, containerName, enabled }: TabLogsProps) =>
const theme = useTheme();
const { t } = useTranslation();
const { toastError } = useToast();
const isAdmin = useIsAdmin();
// logs is newest-first, matching the API and what flattenDockerLogs expects.
const [logs, setLogs] = useState<DockerLog[]>([]);
@@ -59,7 +61,7 @@ export const TabLogs = ({ monitorId, containerName, enabled }: TabLogsProps) =>
const [lastUpdatedAt, setLastUpdatedAt] = useState<number | null>(null);
const logsUrl =
enabled && monitorId && containerName
isAdmin && enabled && monitorId && containerName
? `/monitors/docker/details/${encodeURIComponent(monitorId)}/containers/${encodeURIComponent(containerName)}/logs`
: null;
@@ -168,6 +170,15 @@ export const TabLogs = ({ monitorId, containerName, enabled }: TabLogsProps) =>
}
}, [logsUrl, oldestCursor, loadingOlder, t, toastError]);
if (!isAdmin) {
return (
<EmptyState
title={t("pages.docker.container.logs.forbidden.title")}
description={t("pages.docker.container.logs.forbidden.description")}
/>
);
}
if (!enabled) {
return (
<EmptyState
@@ -17,6 +17,7 @@ import { formatPercentage } from "@/Utils/FormatUtils";
import prettyBytes from "pretty-bytes";
import { BaseBox } from "@/Components/design-elements";
import { dedupeDockerPorts, getDockerMountLabel } from "@/Utils/MonitorUtils";
import type { DateRange } from "@/Types/Query";
const getChartConfigs = (theme: Theme, t: TFunction, stats: DockerContainerStats) => [
{
@@ -24,20 +25,29 @@ const getChartConfigs = (theme: Theme, t: TFunction, stats: DockerContainerStats
key: "avgCpuPct",
color: theme.palette.success.main,
rightTitle: formatPercentage(stats?.latest?.container.cpuPct || 0),
formatter: formatPercentage,
labelKey: "common.charts.histogram.avgCpuUsage",
},
{
title: t("common.charts.labels.memoryUsage"),
key: "avgMemoryUsedBytes",
color: theme.palette.primary.main,
rightTitle: prettyBytes(stats?.latest?.container.memoryUsedBytes || 0),
formatter: prettyBytes,
labelKey: "common.charts.histogram.avgMemoryUsage",
},
];
export const TabOverview = ({ stats }: { stats: DockerContainerStats }) => {
export const TabOverview = ({
stats,
range,
}: {
stats: DockerContainerStats;
range: DateRange;
}) => {
const theme = useTheme();
const { t } = useTranslation();
const isSmall = useMediaQuery(theme.breakpoints.down("md"));
const { ports, mounts } = stats?.latest?.container ?? {};
const dedupedPorts = dedupeDockerPorts(ports ?? []);
return (
@@ -58,6 +68,9 @@ export const TabOverview = ({ stats }: { stats: DockerContainerStats }) => {
dataKey={config.key}
strokeColor={config.color}
gradientStartColor={config.color}
range={range}
formatter={config.formatter}
labelKey={config.labelKey}
/>
</Grid>
);
@@ -21,7 +21,7 @@ const TAB_MAP = {
logs: 1,
} as const;
const DockerDetailsPage = ({ open = "logs" }: DockerContainerDetailsProps) => {
const DockerDetailsPage = ({ open = "overview" }: DockerContainerDetailsProps) => {
const [activeTab, setActiveTab] = useState<number>(TAB_MAP[open]);
const [dateRange, setDateRange] = useState<DateRange>("recent");
const { monitorId, containerName } = useParams<{
@@ -69,7 +69,12 @@ const DockerDetailsPage = ({ open = "logs" }: DockerContainerDetailsProps) => {
<Tab label={t("pages.docker.container.tabs.overview")} />
<Tab label={t("pages.docker.container.tabs.logs")} />
</Tabs>
{activeTab === 0 && <TabOverview stats={stats} />}
{activeTab === 0 && (
<TabOverview
stats={stats}
range={dateRange}
/>
)}
{activeTab === 1 && (
<TabLogs
monitorId={monitorId}
+1 -1
View File
@@ -8,7 +8,7 @@ import { useTranslation } from "react-i18next";
const LogsPage = () => {
const { t } = useTranslation();
const [activeTab, setActiveTab] = useState<number>(1);
const [activeTab, setActiveTab] = useState<number>(0);
return (
<BasePage headerKey="logs">
<Tabs
+6 -5
View File
@@ -1,4 +1,4 @@
import NotFoundSvg from "@/assets/Images/sushi_404.svg";
import NotFoundSvg from "@/assets/Images/sushi_404.svg?react";
import { Button } from "@/Components/inputs";
import Box from "@mui/material/Box";
import Stack from "@mui/material/Stack";
@@ -36,10 +36,11 @@ const NotFoundPage = ({ title, desc }: NotFoundProps) => {
alignItems="center"
>
<Box
component="img"
src={NotFoundSvg}
alt="404"
maxHeight={"25rem"}
component={NotFoundSvg}
role="img"
aria-label="404"
width="100%"
maxHeight="25rem"
/>
<Typography
component="h1"
@@ -126,6 +126,7 @@ const NotificationsCreatePage = () => {
{watchedType !== "matrix" &&
watchedType !== "telegram" &&
watchedType !== "pushover" &&
watchedType !== "signalgrid" &&
watchedType !== "twilio" &&
watchedType !== "ntfy" && (
<ConfigBox
@@ -201,6 +202,31 @@ const NotificationsCreatePage = () => {
}
/>
)}
{watchedType === "signalgrid" && (
<ConfigBox
title={t("pages.notifications.form.signalgrid.title")}
subtitle={t("pages.notifications.form.signalgrid.description")}
rightContent={
<Stack spacing={theme.spacing(8)}>
<FormTextField
name="accessToken"
fieldLabel={t("pages.notifications.form.signalgrid.optionClientKey")}
placeholder={t(
"pages.notifications.form.signalgrid.placeholderClientKey"
)}
/>
<FormTextField
name="address"
fieldLabel={t("pages.notifications.form.signalgrid.optionChannel")}
placeholder={t(
"pages.notifications.form.signalgrid.placeholderChannel"
)}
/>
</Stack>
}
/>
)}
{watchedType === "twilio" && (
<ConfigBox
title={t("pages.notifications.form.twilio.title")}
+7
View File
@@ -8,6 +8,9 @@ import AuthRegister from "@/Pages/Auth/Register";
import AuthForgotPassword from "@/Pages/Auth/Recovery";
import AuthSetNewPassword from "@/Pages/Auth/SetNewPassword";
// Dashboard
import Dashboard from "@/Pages/Dashboard";
// Uptime
import Uptime from "@/Pages/Uptime/Monitors";
import UptimeDetails from "@/Pages/Uptime/Details";
@@ -92,6 +95,10 @@ const Routes = () => {
path="/"
element={<Navigate to="/uptime" />}
/>
<Route
path="dashboard"
element={<Dashboard />}
/>
<Route
path="/uptime"
element={<Uptime />}
+12
View File
@@ -60,6 +60,15 @@ export const monitorTypeLabelKey: Record<SelectableMonitorType, string> &
pagespeed: "optionPagespeed",
};
export const getMonitorTypeLabel = (
type: MonitorType,
t: (key: string) => string
): string => {
const monitorLabel = monitorTypeLabelKey[type];
return monitorLabel ? t(`pages.common.monitors.monitorTypes.${monitorLabel}`) : type;
};
export const MIN_MONITOR_INTERVAL_MS = 10000;
export const MonitorIntervalOptions = [
{ value: 15000, labelKey: "fifteenSeconds" },
{ value: 30000, labelKey: "thirtySeconds" },
@@ -160,6 +169,9 @@ export interface Monitor {
geoCheckLocations?: GeoContinent[];
geoCheckInterval?: number;
dockerLogsEnabled?: boolean;
dockerTlsCa?: string;
dockerTlsCert?: string;
dockerTlsKeySet?: boolean;
dnsServer?: string;
dnsRecordType?: DnsRecordType;
recentChecks: CheckSnapshot[];
+1
View File
@@ -9,6 +9,7 @@ export const NotificationChannels = [
"teams",
"telegram",
"pushover",
"signalgrid",
"twilio",
"ntfy",
] as const;
+8
View File
@@ -40,6 +40,14 @@ export const getStatusPalette = (status: MonitorStatus): PaletteKey => {
return "warning";
};
export const getDockerPalette = (dockerState: DockerContainerState): PaletteKey => {
if (dockerState === "created") return "success";
if (dockerState === "running") return "success";
if (dockerState === "dead") return "error";
if (dockerState === "exited") return "error";
return "warning";
};
export const getValuePalette = (value: ValueType): PaletteKey => {
const paletteMap: Record<ValueType, PaletteKey> = {
positive: "success",
+46 -3
View File
@@ -104,9 +104,52 @@ export const formatMs = (ms: number, hasSpace: boolean = true): string => {
return formatted;
};
export const formatDuration = (ms: number, verbose: boolean = false, hasSpace = true) => {
if (verbose) {
return prettyMilliseconds(ms, { verbose: true });
type DurationUnit = "day" | "hour" | "minute" | "second";
interface FormatDurationOptions {
long?: boolean;
relative?: boolean;
hasSpace?: boolean;
locale?: string;
}
const DURATION_UNITS: Array<{ unit: DurationUnit; milliseconds: number }> = [
{ unit: "day", milliseconds: MS_PER_DAY },
{ unit: "hour", milliseconds: MS_PER_HOUR },
{ unit: "minute", milliseconds: MS_PER_MINUTE },
{ unit: "second", milliseconds: MS_PER_SECOND },
];
const FALLBACK_DURATION_UNIT = DURATION_UNITS[DURATION_UNITS.length - 1];
const getDurationUnit = (ms: number) =>
DURATION_UNITS.find(({ milliseconds }) => Math.abs(ms) >= milliseconds) ??
FALLBACK_DURATION_UNIT;
const getExactDurationUnit = (ms: number) =>
DURATION_UNITS.find(({ milliseconds }) => ms !== 0 && ms % milliseconds === 0) ??
FALLBACK_DURATION_UNIT;
export const formatDuration = (
ms: number,
{ long = false, relative = false, hasSpace = true, locale }: FormatDurationOptions = {}
): string => {
if (relative) {
const { unit, milliseconds } = getDurationUnit(ms);
const value = Math.sign(ms) * Math.floor(Math.abs(ms) / milliseconds);
return new Intl.RelativeTimeFormat(locale, {
numeric: "always",
style: long ? "long" : "narrow",
}).format(-value, unit);
}
if (long) {
const { unit, milliseconds } = getExactDurationUnit(ms);
return new Intl.NumberFormat(locale, {
style: "unit",
unit,
unitDisplay: "long",
}).format(ms / milliseconds);
}
const formatted = prettyMilliseconds(ms, {
+51 -4
View File
@@ -6,6 +6,7 @@ import {
HttpMethods,
PageSpeedStrategies,
ProxyModes,
MIN_MONITOR_INTERVAL_MS,
type MonitorType,
} from "@/Types/Monitor";
import { ALL_HTTP_STATUS_CODES } from "@/Utils/statusCode";
@@ -27,7 +28,8 @@ const baseSchema = z.object({
description: z.string().optional(),
interval: z
.number()
.min(15000, "Interval must be at least 15 seconds")
.int("Interval must be a whole number of milliseconds")
.min(MIN_MONITOR_INTERVAL_MS, "Interval must be at least 10 seconds")
.register(monitorStepRegistry, { step: 1 }),
notifications: z.array(z.string()).register(monitorStepRegistry, { step: 1 }),
tags: z.array(z.string()).register(monitorStepRegistry, { step: 1 }),
@@ -108,10 +110,30 @@ const portSchema = baseSchema.extend({
.max(65535, "Port must be at most 65535"),
});
// Docker monitor schema
// Docker monitor schema. The url forms mirror DOCKER_TLS_URL and
// DOCKER_SOCKET_URL in server/src/utils/dockerHost.ts.
const dockerTlsUrlRegex = /^(tcp|https):\/\/([^/\s:]+)(?::(\d+))?\/?$/;
const dockerSocketUrlRegex = /^(?:unix:\/\/\/\S+|\/\S+)$/;
export const isDockerTlsUrl = (url: string | undefined): boolean =>
dockerTlsUrlRegex.test(url?.trim() ?? "");
const dockerSchema = baseSchema.extend({
type: z.literal("docker"),
url: z.string().min(1, "Docker host URL is required"),
url: z
.string()
.min(1, "Docker host URL is required")
.refine(
(url) => dockerSocketUrlRegex.test(url.trim()) || isDockerTlsUrl(url),
"Docker host must be unix:///path, an absolute socket path, or tcp://host[:port]"
),
ignoreTlsErrors: z.boolean(),
dockerTlsCa: z.string(),
dockerTlsCert: z.string(),
// blank on edit keeps the stored key.
dockerTlsKey: z.string(),
// Read only mirror of the server flag, stripped on submit
dockerTlsKeySet: z.boolean(),
dockerLogsEnabled: z.boolean().register(monitorStepRegistry, { step: 1 }),
});
@@ -217,7 +239,8 @@ const monitorSchemaUnion = z.discriminatedUnion("type", [
dnsSchema,
]);
// Mirrors the server's refineProxySelection (monitorValidation.ts)
// Mirrors the server's refineProxySelection and refineDockerTls
// (monitorValidation.ts). PEM contents are only parsed on the server.
export const monitorSchema = monitorSchemaUnion.superRefine((data, ctx) => {
if (data.type === "http" && data.proxyMode === "custom" && !data.proxyId) {
ctx.addIssue({
@@ -226,6 +249,30 @@ export const monitorSchema = monitorSchemaUnion.superRefine((data, ctx) => {
message: "A proxy must be selected when proxy mode is custom",
});
}
if (data.type === "docker" && isDockerTlsUrl(data.url)) {
if (!data.dockerTlsCert.trim()) {
ctx.addIssue({
code: "custom",
path: ["dockerTlsCert"],
message: "Client certificate is required for a TLS Docker host",
});
}
if (!data.dockerTlsKeySet && !data.dockerTlsKey.trim()) {
ctx.addIssue({
code: "custom",
path: ["dockerTlsKey"],
message: "Client key is required for a TLS Docker host",
});
}
if (!data.ignoreTlsErrors && !data.dockerTlsCa.trim()) {
ctx.addIssue({
code: "custom",
path: ["dockerTlsCa"],
message: "CA certificate is required unless TLS errors are ignored",
});
}
}
});
export type MonitorFormData = z.infer<typeof monitorSchema>;
+7
View File
@@ -70,6 +70,12 @@ const pushoverSchema = baseSchema.extend({
accessToken: z.string().min(1, "App token is required"),
});
const signalgridSchema = baseSchema.extend({
type: z.literal("signalgrid"),
address: z.string().min(1, "Channel is required"),
accessToken: z.string().min(1, "Client key is required"),
});
const twilioSchema = baseSchema.extend({
type: z.literal("twilio"),
accountSid: z.string().min(1, "Account SID is required"),
@@ -95,6 +101,7 @@ export const notificationSchema = z.discriminatedUnion("type", [
teamsSchema,
telegramSchema,
pushoverSchema,
signalgridSchema,
twilioSchema,
ntfySchema,
]);
File diff suppressed because it is too large. Load diff

Before

Width:  |  Height:  |  Size: 536 KiB

After

Width:  |  Height:  |  Size: 286 KiB

+1
View File
@@ -448,6 +448,7 @@
},
"statBoxes": {
"activeFor": "Aktiv seit",
"checkingEvery": "Check alle {{interval}}",
"certificateExpiry": "Zertifikatablauf",
"lastCheck": "Letzter Check",
"lastResponseTime": "Letzte Antwortzeit",
+57 -4
View File
@@ -80,7 +80,9 @@
"avg": "Avg: {{value}}",
"max": "Max: {{value}}",
"responseTime": "Response time: {{value}}",
"total": "Total: {{value}}"
"total": "Total: {{value}}",
"avgCpuUsage": "Average CPU usage: {{value}}",
"avgMemoryUsage": "Average memory usage: {{value}}"
},
"phases": {
"dns": "DNS",
@@ -224,7 +226,8 @@
"alerting": "Alerting",
"reporting": "Reporting",
"configuration": "Configuration"
}
},
"dashboard": "Dashboard"
},
"bottomMenu": {
"support": "Support",
@@ -545,6 +548,7 @@
},
"statBoxes": {
"activeFor": "Active for",
"checkingEvery": "Checking every {{interval}}",
"certificateExpiry": "Certificate expiry",
"domainExpiry": "Domain expiry",
"lastCheck": "Last check",
@@ -716,7 +720,7 @@
"http": "Enter the URL or IP to monitor (e.g., https://example.com/ or 192.168.1.100) and add a clear display name that appears on the dashboard. Set an optional proxy server to connect through.",
"ping": "Enter the IP address or hostname to ping (e.g., 192.168.1.100 or example.com) and add a clear display name that appears on the dashboard.",
"port": "Enter the URL or IP of the server, the port number and a clear display name that appears on the dashboard.",
"docker": "Enter the Docker socket to monitor, as unix:///var/run/docker.sock or an absolute path such as /var/run/docker.sock. Every container on the host is monitored.",
"docker": "Enter the Docker socket to monitor, as unix:///var/run/docker.sock or an absolute path such as /var/run/docker.sock, or a remote daemon as tcp://host:2376 to connect with TLS client certificates. Every container on the host is monitored.",
"game": "Enter the IP address or hostname and the port number to ping (e.g., 192.168.1.100 or example.com) and choose game type. Some games require enabling server queries in their config file — check your game server's documentation. See the <gamedigLink>supported games list</gamedigLink> for protocol details.",
"pagespeed": "Track page load performance, Core Web Vitals, and optimization scores for your website.",
"grpc": "Enter the hostname and port of the gRPC server, optionally specify a service name for the health check, and add a display name.",
@@ -832,6 +836,29 @@
"description": "This will permanently delete all monitoring history, statistics, and check data for your team. This action cannot be undone.",
"confirm": "Yes, clear all stats"
}
},
"dockerTls": {
"title": "TLS credentials",
"description": "Connect to a remote Docker daemon over mutual TLS. Paste the PEM files you would pass to docker --tlsverify. The client key is encrypted before it is stored and is never shown again.",
"option": {
"ca": {
"label": "CA certificate",
"placeholder": "-----BEGIN CERTIFICATE-----"
},
"cert": {
"label": "Client certificate",
"placeholder": "-----BEGIN CERTIFICATE-----"
},
"key": {
"label": "Client key",
"placeholder": "-----BEGIN PRIVATE KEY-----",
"stored": "A key is stored. Leave blank to keep it."
},
"ignoreTls": {
"label": "Ignore TLS/SSL errors",
"description": "Skips verifying the daemon's certificate, so the CA certificate becomes optional. The client certificate and key are still required because the daemon verifies them."
}
}
}
}
},
@@ -985,6 +1012,10 @@
"title": "Log collection is off for this host",
"description": "Turn on container logs in the monitor settings to start collecting.",
"action": "Configure monitor"
},
"forbidden": {
"title": "You don't have access to container logs",
"description": "Only administrators can view logs. Ask an administrator for access if you need to see them."
}
}
},
@@ -1247,7 +1278,7 @@
},
"startTime": {
"title": "Start time",
"description": "Set the start time and duration for your maintenance window. All values are in UTC",
"description": "Set the start time and duration for your maintenance window. All values are in your local time zone.",
"option": {
"duration": {
"label": "Duration"
@@ -1329,6 +1360,7 @@
"teams": "Teams",
"telegram": "Telegram",
"pushover": "Pushover",
"signalgrid": "Signalgrid",
"twilio": "Twilio",
"ntfy": "ntfy"
}
@@ -1349,6 +1381,14 @@
"optionUserKey": "User key",
"placeholderUserKey": "uQiRzpo4DXghDmr9QzzfQu27cmVRsG"
},
"signalgrid": {
"title": "Signalgrid configuration",
"description": "Configure Signalgrid to receive push notifications on your devices.",
"optionClientKey": "Client key",
"placeholderClientKey": "Your Signalgrid client key",
"optionChannel": "Channel",
"placeholderChannel": "Your Signalgrid channel"
},
"twilio": {
"title": "Twilio SMS configuration",
"description": "Configure Twilio to send SMS notifications to a phone number.",
@@ -1869,6 +1909,19 @@
"title": "Uptime monitors",
"description": "Watch HTTP endpoints, pings, containers, and ports — and get alerted the moment something goes down."
}
},
"dashboard": {
"monitorStatus": {
"ofUp": "of {{total}} up"
},
"cards": {
"monitorStatus": "Monitor status",
"monitorsByType": "Monitors by type"
},
"header": {
"title": "Dashboard",
"description": "See every monitor type in one place and spot what needs attention."
}
}
}
}
+1
View File
@@ -9,6 +9,7 @@ services:
- DB_CONNECTION_STRING=mongodb://mongodb:27017/uptime_db
- CLIENT_HOST=${CLIENT_HOST:-http://localhost:52345}
- JWT_SECRET=${JWT_SECRET:?set JWT_SECRET in your environment}
- ENCRYPTION_KEY=${ENCRYPTION_KEY:-}
stop_grace_period: 60s
healthcheck:
test:
+11
View File
@@ -27,6 +27,17 @@ CLIENT_HOST=http://localhost:5173
# ----------------------
ORIGIN=localhost
# Encryption
# ------------------
# Encrypts stored Docker TLS client keys. Not needed for socket-only Docker monitors.
# Generate a key: openssl rand -base64 32
# Key list: comma-separated; the first key encrypts, every key decrypts.
# Split deployments: set the same value on the API and every worker.
# Rotate a key: 1. deploy OLD_KEY,NEW_KEY everywhere so every process can read both
# 2. deploy NEW_KEY,OLD_KEY everywhere so new writes use the new key
# 3. wait for the worker to re-encrypt every row, then drop OLD_KEY
# ENCRYPTION_KEY=<output of openssl rand -base64 32>
# Feature Flags
# -------------
# Set to "false" to disable the selectable theme feature on status pages.
+409 -21
View File
@@ -422,9 +422,6 @@
"secret": {
"type": "string"
},
"sshPrivateKey": {
"type": "string"
},
"cpuAlertThreshold": {
"type": "number"
},
@@ -466,6 +463,18 @@
"geoCheckInterval": {
"type": "number"
},
"dockerLogsEnabled": {
"type": "boolean"
},
"dockerTlsCa": {
"type": "string"
},
"dockerTlsCert": {
"type": "string"
},
"dockerTlsKeySet": {
"type": "boolean"
},
"dnsServer": {
"type": "string"
},
@@ -513,6 +522,7 @@
"geoCheckEnabled",
"geoCheckLocations",
"geoCheckInterval",
"dockerLogsEnabled",
"teamId",
"userId",
"createdAt",
@@ -640,9 +650,6 @@
"secret": {
"type": "string"
},
"sshPrivateKey": {
"type": "string"
},
"cpuAlertThreshold": {
"type": "number"
},
@@ -684,6 +691,18 @@
"geoCheckInterval": {
"type": "number"
},
"dockerLogsEnabled": {
"type": "boolean"
},
"dockerTlsCa": {
"type": "string"
},
"dockerTlsCert": {
"type": "string"
},
"dockerTlsKeySet": {
"type": "boolean"
},
"dnsServer": {
"type": "string"
},
@@ -733,6 +752,7 @@
"geoCheckEnabled",
"geoCheckLocations",
"geoCheckInterval",
"dockerLogsEnabled",
"teamId",
"userId",
"createdAt",
@@ -929,6 +949,9 @@
{
"$ref": "#/components/schemas/PushoverNotification"
},
{
"$ref": "#/components/schemas/SignalgridNotification"
},
{
"$ref": "#/components/schemas/TwilioNotification"
},
@@ -949,6 +972,7 @@
"teams": "#/components/schemas/TeamsNotification",
"telegram": "#/components/schemas/TelegramNotification",
"pushover": "#/components/schemas/PushoverNotification",
"signalgrid": "#/components/schemas/SignalgridNotification",
"twilio": "#/components/schemas/TwilioNotification",
"ntfy": "#/components/schemas/NtfyNotification"
}
@@ -1380,6 +1404,34 @@
"accessToken": "a1234567890abcdef"
}
},
"SignalgridNotification": {
"type": "object",
"properties": {
"notificationName": {
"type": "string",
"minLength": 1
},
"type": {
"type": "string",
"enum": ["signalgrid"]
},
"address": {
"type": "string",
"minLength": 1
},
"accessToken": {
"type": "string",
"minLength": 1
}
},
"required": ["notificationName", "type", "address", "accessToken"],
"example": {
"notificationName": "Signalgrid",
"type": "signalgrid",
"address": "your-channel",
"accessToken": "your-client-key"
}
},
"TwilioNotification": {
"type": "object",
"properties": {
@@ -7090,9 +7142,6 @@
"secret": {
"type": "string"
},
"sshPrivateKey": {
"type": "string"
},
"cpuAlertThreshold": {
"type": "number"
},
@@ -7134,6 +7183,18 @@
"geoCheckInterval": {
"type": "number"
},
"dockerLogsEnabled": {
"type": "boolean"
},
"dockerTlsCa": {
"type": "string"
},
"dockerTlsCert": {
"type": "string"
},
"dockerTlsKeySet": {
"type": "boolean"
},
"dnsServer": {
"type": "string"
},
@@ -7183,6 +7244,7 @@
"geoCheckEnabled",
"geoCheckLocations",
"geoCheckInterval",
"dockerLogsEnabled",
"teamId",
"userId",
"createdAt",
@@ -7655,9 +7717,6 @@
"secret": {
"type": "string"
},
"sshPrivateKey": {
"type": "string"
},
"cpuAlertThreshold": {
"type": "number"
},
@@ -7699,6 +7758,18 @@
"geoCheckInterval": {
"type": "number"
},
"dockerLogsEnabled": {
"type": "boolean"
},
"dockerTlsCa": {
"type": "string"
},
"dockerTlsCert": {
"type": "string"
},
"dockerTlsKeySet": {
"type": "boolean"
},
"dnsServer": {
"type": "string"
},
@@ -7748,6 +7819,7 @@
"geoCheckEnabled",
"geoCheckLocations",
"geoCheckInterval",
"dockerLogsEnabled",
"teamId",
"userId",
"createdAt",
@@ -7984,6 +8056,231 @@
}
}
},
"/monitors/docker/details/{monitorId}/containers/{containerName}/logs": {
"get": {
"tags": ["monitors"],
"summary": "Get stored log lines for one container on a Docker host monitor, newest first",
"security": [
{
"bearerAuth": []
}
],
"parameters": [
{
"schema": {
"type": "string",
"minLength": 1
},
"required": true,
"name": "monitorId",
"in": "path"
},
{
"schema": {
"type": "string",
"minLength": 1
},
"required": true,
"name": "containerName",
"in": "path"
},
{
"schema": {
"type": "string",
"format": "date-time"
},
"required": false,
"name": "before",
"in": "query"
},
{
"schema": {
"type": "string",
"format": "date-time"
},
"required": false,
"name": "after",
"in": "query"
},
{
"schema": {
"type": "integer",
"minimum": 1,
"maximum": 50,
"default": 20
},
"required": false,
"name": "limit",
"in": "query"
}
],
"responses": {
"200": {
"description": "OK",
"content": {
"application/json": {
"schema": {
"type": "object",
"properties": {
"success": {
"type": "boolean",
"enum": [true]
},
"msg": {
"type": "string"
},
"data": {
"type": "object",
"properties": {
"logs": {
"type": "array",
"items": {
"type": "object",
"properties": {
"id": {
"type": "string"
},
"metadata": {
"type": "object",
"properties": {
"monitorId": {
"type": "string"
},
"teamId": {
"type": "string"
},
"containerId": {
"type": "string"
},
"containerName": {
"type": "string"
}
},
"required": ["monitorId", "teamId", "containerId", "containerName"]
},
"lines": {
"type": "array",
"items": {
"type": "object",
"properties": {
"ts": {
"type": "string"
},
"stream": {
"type": "string",
"enum": ["stdout", "stderr"]
},
"text": {
"type": "string"
}
},
"required": ["ts", "stream", "text"]
}
},
"gap": {
"type": "boolean"
},
"checkedAt": {
"type": "string"
},
"expiry": {
"type": "string"
},
"createdAt": {
"type": "string"
},
"updatedAt": {
"type": "string"
}
},
"required": ["id", "metadata", "lines", "gap", "checkedAt", "expiry", "createdAt", "updatedAt"]
}
},
"nextCursor": {
"type": "string",
"nullable": true
}
},
"required": ["logs", "nextCursor"]
}
},
"required": ["success", "msg", "data"]
}
}
}
},
"401": {
"description": "Unauthorized",
"content": {
"application/json": {
"schema": {
"type": "object",
"properties": {
"success": {
"type": "boolean",
"enum": [false]
},
"msg": {
"type": "string"
},
"data": {
"nullable": true
}
},
"required": ["success", "msg"]
}
}
}
},
"403": {
"description": "Forbidden",
"content": {
"application/json": {
"schema": {
"type": "object",
"properties": {
"success": {
"type": "boolean",
"enum": [false]
},
"msg": {
"type": "string"
},
"data": {
"nullable": true
}
},
"required": ["success", "msg"]
}
}
}
},
"500": {
"description": "Internal server error",
"content": {
"application/json": {
"schema": {
"type": "object",
"properties": {
"success": {
"type": "boolean",
"enum": [false]
},
"msg": {
"type": "string"
},
"data": {
"nullable": true
}
},
"required": ["success", "msg"]
}
}
}
}
}
}
},
"/monitors/pagespeed/details/{monitorId}": {
"get": {
"tags": ["monitors"],
@@ -8908,9 +9205,6 @@
"secret": {
"type": "string"
},
"sshPrivateKey": {
"type": "string"
},
"jsonPath": {
"anyOf": [
{
@@ -9011,6 +9305,42 @@
"type": "number",
"minimum": 300000
},
"dockerLogsEnabled": {
"type": "boolean"
},
"dockerTlsCa": {
"anyOf": [
{
"type": "string"
},
{
"type": "string",
"enum": [""]
}
]
},
"dockerTlsCert": {
"anyOf": [
{
"type": "string"
},
{
"type": "string",
"enum": [""]
}
]
},
"dockerTlsKey": {
"anyOf": [
{
"type": "string"
},
{
"type": "string",
"enum": [""]
}
]
},
"dnsServer": {
"type": "string",
"minLength": 1
@@ -9604,9 +9934,6 @@
"secret": {
"type": "string"
},
"sshPrivateKey": {
"type": "string"
},
"cpuAlertThreshold": {
"type": "number",
"default": 100
@@ -9702,6 +10029,10 @@
"minimum": 300000,
"default": 300000
},
"dockerLogsEnabled": {
"type": "boolean",
"default": false
},
"dnsServer": {
"type": "string",
"minLength": 1
@@ -10137,9 +10468,6 @@
"secret": {
"type": "string"
},
"sshPrivateKey": {
"type": "string"
},
"ignoreTlsErrors": {
"type": "boolean"
},
@@ -10269,6 +10597,42 @@
"type": "number",
"minimum": 300000
},
"dockerLogsEnabled": {
"type": "boolean"
},
"dockerTlsCa": {
"anyOf": [
{
"type": "string"
},
{
"type": "string",
"enum": [""]
}
]
},
"dockerTlsCert": {
"anyOf": [
{
"type": "string"
},
{
"type": "string",
"enum": [""]
}
]
},
"dockerTlsKey": {
"anyOf": [
{
"type": "string"
},
{
"type": "string",
"enum": [""]
}
]
},
"dnsServer": {
"type": "string",
"minLength": 1
@@ -12930,6 +13294,30 @@
"globalProxyId": {
"type": "string",
"nullable": true
},
"egressCheckEnabled": {
"type": "boolean"
},
"egressCheckTargets": {
"type": "array",
"items": {
"type": "string",
"minLength": 1,
"maxLength": 253
},
"maxItems": 10
},
"egressPollIntervalSeconds": {
"type": "integer",
"minimum": 5,
"maximum": 600
},
"egressNotifications": {
"type": "array",
"items": {
"type": "string",
"pattern": "^[0-9a-fA-F]{24}$"
}
}
}
}
+4
View File
@@ -65,6 +65,10 @@ const notificationVariantMeta: Record<string, { component: string; example: Reco
component: "PushoverNotification",
example: { notificationName: "Pushover personal", type: "pushover", address: "u1234567890abcdef", accessToken: "a1234567890abcdef" },
},
signalgrid: {
component: "SignalgridNotification",
example: { notificationName: "Signalgrid", type: "signalgrid", address: "your-channel", accessToken: "your-client-key" },
},
twilio: {
component: "TwilioNotification",
example: {
+2 -1
View File
@@ -5,6 +5,7 @@ import { requireTeamId, requireUserEmail, requireUserId, requireUserRoles } from
import {
registrationBodyValidation,
registerInviteTokenValidation,
loginValidation,
recoveryValidation,
recoveryTokenBodyValidation,
@@ -48,7 +49,7 @@ class AuthController implements IAuthController {
registerUser = catchAsync(async (req: Request, res: Response) => {
const newUser = req.body.user;
const newUserToken = req.body.token;
const newUserToken = registerInviteTokenValidation.parse(req.body.token);
if (newUser?.email) {
const newUserEmail = requireUserEmail(newUser.email);
newUser.email = newUserEmail.toLowerCase();
@@ -6,6 +6,7 @@ import { AppError } from "@/utils/AppError.js";
import { ISettingsService } from "@/domain/app-settings/app-settings.service.js";
import { IEmailService } from "@/service/emailService.js";
import { IProxiesService } from "@/domain/proxies/proxy.service.js";
import { IEgressStateService } from "@/domain/egress/egress-state.service.js";
import { Settings } from "@/domain/app-settings/app-settings.type.js";
export interface ISettingsController {
@@ -18,10 +19,17 @@ class SettingsController implements ISettingsController {
private settingsService: ISettingsService;
private emailService: IEmailService;
private proxiesService: IProxiesService;
constructor(settingsService: ISettingsService, emailService: IEmailService, proxiesService: IProxiesService) {
private egressStateService: IEgressStateService;
constructor(
settingsService: ISettingsService,
emailService: IEmailService,
proxiesService: IProxiesService,
egressStateService: IEgressStateService
) {
this.settingsService = settingsService;
this.emailService = emailService;
this.proxiesService = proxiesService;
this.egressStateService = egressStateService;
}
buildAppSettings = async (dbSettings: Settings) => {
@@ -70,7 +78,14 @@ class SettingsController implements ISettingsController {
}
}
const previousSettings = await this.settingsService.getDBSettings();
const updatedSettings = await this.settingsService.updateDbSettings(validatedBody);
// Switching the egress check on or off starts from a clean state: no degraded episode, no pending recovery job
if (validatedBody.egressCheckEnabled !== undefined && validatedBody.egressCheckEnabled !== previousSettings.egressCheckEnabled) {
await this.egressStateService.reset();
}
const returnSettings = await this.buildAppSettings(updatedSettings);
return res.status(200).json({
success: true,
+1 -1
View File
@@ -9,6 +9,6 @@ export const createCheckRoutes = (checkController: ICheckController): Router =>
router.get("/team", checkController.getChecksByTeam);
router.delete("/team", isAllowed(["admin", "superadmin"]), checkController.deleteChecksByTeamId);
router.get("/:monitorId", checkController.getChecksByMonitor);
router.delete("/:monitorId", checkController.deleteChecks);
router.delete("/:monitorId", isAllowed(["admin", "superadmin"]), checkController.deleteChecks);
return router;
};
@@ -1,15 +1,16 @@
import { IMaintenanceWindowController } from "@/api/controllers/maintenanceWindowController.js";
import { isAllowed } from "@/api/middleware/isAllowed.js";
import { Router } from "express";
export const createMaintenanceWindowRoutes = (maintenanceWindowController: IMaintenanceWindowController): Router => {
const router = Router();
router.post("/", maintenanceWindowController.createMaintenanceWindows);
router.post("/", isAllowed(["admin", "superadmin"]), maintenanceWindowController.createMaintenanceWindows);
router.get("/team/", maintenanceWindowController.getMaintenanceWindowsByTeamId);
router.get("/monitor/:monitorId", maintenanceWindowController.getMaintenanceWindowsByMonitorId);
router.get("/:id", maintenanceWindowController.getMaintenanceWindowById);
router.patch("/:id", maintenanceWindowController.editMaintenanceWindow);
router.delete("/:id", maintenanceWindowController.deleteMaintenanceWindow);
router.patch("/:id", isAllowed(["admin", "superadmin"]), maintenanceWindowController.editMaintenanceWindow);
router.delete("/:id", isAllowed(["admin", "superadmin"]), maintenanceWindowController.deleteMaintenanceWindow);
return router;
};
+5 -1
View File
@@ -21,7 +21,11 @@ export const createMonitorRoutes = (monitorController: IMonitorController): Rout
// Docker routes
router.get("/docker/details/:monitorId", monitorController.getDockerDetailsById);
router.get("/docker/details/:monitorId/containers/:containerName", monitorController.getDockerContainerByName);
router.get("/docker/details/:monitorId/containers/:containerName/logs", monitorController.getDockerContainerLogs);
router.get(
"/docker/details/:monitorId/containers/:containerName/logs",
isAllowed(["admin", "superadmin"]),
monitorController.getDockerContainerLogs
);
// Geo checks routes
router.get("/:monitorId/geo-checks", monitorController.getGeoChecksByMonitorId);
+6 -5
View File
@@ -1,14 +1,15 @@
import { INotificationController } from "@/api/controllers/notificationController.js";
import { isAllowed } from "@/api/middleware/isAllowed.js";
import { Router } from "express";
export const createNotificationRoutes = (notificationController: INotificationController): Router => {
const router = Router();
router.post("/", notificationController.createNotification);
router.post("/test/all", notificationController.testAllNotifications);
router.post("/test", notificationController.testNotification);
router.post("/", isAllowed(["admin", "superadmin"]), notificationController.createNotification);
router.post("/test/all", isAllowed(["admin", "superadmin"]), notificationController.testAllNotifications);
router.post("/test", isAllowed(["admin", "superadmin"]), notificationController.testNotification);
router.get("/team", notificationController.getNotificationsByTeamId);
router.get("/:id", notificationController.getNotificationById);
router.delete("/:id", notificationController.deleteNotification);
router.patch("/:id", notificationController.editNotification);
router.delete("/:id", isAllowed(["admin", "superadmin"]), notificationController.deleteNotification);
router.patch("/:id", isAllowed(["admin", "superadmin"]), notificationController.editNotification);
return router;
};
@@ -23,6 +23,8 @@ export const registrationBodyValidation = z.object({
inviteToken: z.string().optional().default(""),
});
export const registerInviteTokenValidation = z.string().optional().default("");
export const recoveryValidation = z.object({
email: z.email("Must be a valid email address"),
});
+61 -7
View File
@@ -15,6 +15,9 @@ import {
import { DateRanges, SortOrders } from "@/types/query.js";
import { DockerContainerStates, DockerHealthStatuses, DockerLogStreams, DockerPortProtocols } from "@/domain/docker/docker.type.js";
import { DOCKER_LOG_PAGE_DEFAULT, DOCKER_LOG_PAGE_MAX } from "@/domain/docker/docker-log.type.js";
import { isDockerSocketUrl, isDockerTlsUrl } from "@/utils/dockerHost.js";
import { X509Certificate } from "node:crypto";
import { keyMatchesCertificate, parseCertificates, parsePrivateKey } from "@/utils/pem.js";
const httpStatusCode = z.number().refine((code) => HttpStatusCodeSet.has(code), { message: "Must be a valid HTTP status code" });
@@ -120,19 +123,60 @@ const refineProxySelection = (body: { proxyMode?: string; proxyId?: string }, ct
}
};
const dockerUrlRegex = /^(?:unix:\/\/\/\S+|\/\S+)$/;
const refineDockerUrl = (data: { type?: string; url?: string }, ctx: z.RefinementCtx) => {
if (data.type !== "docker" || data.url === undefined) return;
if (!dockerUrlRegex.test(data.url)) {
if (!isDockerSocketUrl(data.url) && !isDockerTlsUrl(data.url)) {
ctx.addIssue({
code: "custom",
path: ["url"],
message: "Docker host must be unix:///path or an absolute socket path",
message: "Docker host must be unix:///path, an absolute socket path, or tcp://host[:port]",
});
}
};
type DockerTlsFields = {
type?: string;
url?: string;
ignoreTlsErrors?: boolean;
dockerTlsCa?: string;
dockerTlsCert?: string;
dockerTlsKey?: string;
};
const refineDockerTls = (mode: "create" | "edit") => (data: DockerTlsFields, ctx: z.RefinementCtx) => {
if (data.type !== "docker" || !isDockerTlsUrl(data.url)) return;
const issue = (path: string, message: string) => ctx.addIssue({ code: "custom", path: [path], message });
if (!data.dockerTlsCert) issue("dockerTlsCert", "TLS certificate is required for a TLS Docker host");
if (mode === "create" && !data.dockerTlsKey) issue("dockerTlsKey", "TLS key is required for a TLS Docker host");
if (!data.ignoreTlsErrors && !data.dockerTlsCa) issue("dockerTlsCa", "CA certificate is required unless TLS errors are ignored");
let certificate: X509Certificate | undefined;
try {
if (data.dockerTlsCa) parseCertificates(data.dockerTlsCa);
} catch (error: unknown) {
issue("dockerTlsCa", error instanceof Error ? error.message : "Docker TLS CA error");
}
try {
if (data.dockerTlsCert) [certificate] = parseCertificates(data.dockerTlsCert);
} catch (error) {
issue("dockerTlsCert", error instanceof Error ? error.message : "Docker TLS cert error");
}
try {
if (data.dockerTlsKey) {
const key = parsePrivateKey(data.dockerTlsKey);
if (certificate && !keyMatchesCertificate(key, certificate)) issue("dockerTlsKey", "Key does not match certificate");
}
} catch (error: unknown) {
issue("dockerTlsKey", error instanceof Error ? error.message : "Docker TLS key error");
}
};
export const createMonitorBodyValidation = z
.object({
_id: z.string().optional(),
@@ -170,6 +214,9 @@ export const createMonitorBodyValidation = z
geoCheckLocations: z.array(z.enum(GeoContinents)).optional(),
geoCheckInterval: z.number().min(300000).optional(),
dockerLogsEnabled: z.boolean().optional(),
dockerTlsCa: z.union([z.string(), z.literal("")]).optional(),
dockerTlsCert: z.union([z.string(), z.literal("")]).optional(),
dockerTlsKey: z.union([z.string(), z.literal("")]).optional(),
dnsServer: dnsServerValidation.optional(),
dnsRecordType: z.enum(DnsRecordTypes).optional(),
})
@@ -178,7 +225,8 @@ export const createMonitorBodyValidation = z
.superRefine(refineHeadMatching)
.superRefine(refineRegexPattern)
.superRefine(refineProxySelection)
.superRefine(refineDockerUrl);
.superRefine(refineDockerUrl)
.superRefine(refineDockerTls("create"));
export const editMonitorBodyValidation = z
.object({
@@ -215,6 +263,9 @@ export const editMonitorBodyValidation = z
geoCheckLocations: z.array(z.enum(GeoContinents)).optional(),
geoCheckInterval: z.number().min(300000).optional(),
dockerLogsEnabled: z.boolean().optional(),
dockerTlsCa: z.union([z.string(), z.literal("")]).optional(),
dockerTlsCert: z.union([z.string(), z.literal("")]).optional(),
dockerTlsKey: z.union([z.string(), z.literal("")]).optional(),
dnsServer: dnsServerValidation.optional(),
dnsRecordType: z.enum(DnsRecordTypes).optional(),
})
@@ -223,7 +274,8 @@ export const editMonitorBodyValidation = z
.superRefine(refineHeadMatching)
.superRefine(refineRegexPattern)
.superRefine(refineProxySelection)
.superRefine(refineDockerUrl);
.superRefine(refineDockerUrl)
.superRefine(refineDockerTls("edit"));
export const pauseMonitorParamValidation = z.object({
monitorId: z.string().min(1, "Monitor ID is required"),
@@ -291,7 +343,6 @@ const importedMonitorSchema = z
geoCheckLocations: z.array(z.enum(GeoContinents)).default([]),
geoCheckInterval: z.number().min(300000).default(300000),
dockerLogsEnabled: z.boolean().default(false),
dnsServer: dnsServerValidation.optional(),
dnsRecordType: z.enum(DnsRecordTypes).optional(),
createdAt: z.string().optional(),
@@ -381,6 +432,9 @@ export const monitorResponseSchema = z
geoCheckLocations: z.array(z.enum(GeoContinents)),
geoCheckInterval: z.number(),
dockerLogsEnabled: z.boolean(),
dockerTlsCa: z.string().optional(),
dockerTlsCert: z.string().optional(),
dockerTlsKeySet: z.boolean().optional(),
dnsServer: z.string().optional(),
dnsRecordType: z.enum(DnsRecordTypes).optional(),
teamId: z.string(),
@@ -123,6 +123,13 @@ export const createNotificationBodyValidation = z.discriminatedUnion("type", [
address: z.string().min(1, "User key is required"),
accessToken: z.string().min(1, "App token is required"),
}),
// Signalgrid notification
z.object({
notificationName: z.string().min(1, "Notification name is required"),
type: z.literal("signalgrid"),
address: z.string().min(1, "Channel is required"),
accessToken: z.string().min(1, "Client key is required"),
}),
// Twilio SMS notification
z.object({
notificationName: z.string().min(1, "Notification name is required"),
@@ -1,6 +1,17 @@
import { CHECK_TTL_SENTINEL } from "@/domain/checks/check.type.js";
import { MAX_EGRESS_POLL_INTERVAL_SECONDS, MAX_EGRESS_TARGETS, MIN_EGRESS_POLL_INTERVAL_SECONDS } from "@/domain/egress/egress.type.js";
import { parseEgressTarget } from "@/utils/egressTarget.js";
import { z } from "zod";
// A reliability target is a bare host/IP (ICMP ping), host:port (TCP connect) or an http(s) URL.
// Parsed with the same function the probe uses, so nothing the validator accepts can fail to probe.
const egressTargetValidation = z
.string()
.trim()
.min(1)
.max(253)
.refine((target) => parseEgressTarget(target) !== null, "Enter a hostname or IP address, host:port, or http(s) URL");
//****************************************
// Settings Validations
//****************************************
@@ -42,6 +53,11 @@ export const updateAppSettingsBodyValidation = z
.optional(),
globalProxyEnabled: z.boolean().optional(),
globalProxyId: z.string().nullable().optional(),
egressCheckEnabled: z.boolean().optional(),
egressCheckTargets: z.array(egressTargetValidation).max(MAX_EGRESS_TARGETS).optional(), // empty list falls back to DEFAULT_EGRESS_TARGETS
egressPollIntervalSeconds: z.number().int().min(MIN_EGRESS_POLL_INTERVAL_SECONDS).max(MAX_EGRESS_POLL_INTERVAL_SECONDS).optional(),
egressNotifications: z.array(z.string().regex(/^[0-9a-fA-F]{24}$/, "Invalid notification id")).optional(),
})
.strip()
.superRefine((body, ctx) => {
+6 -1
View File
@@ -36,7 +36,12 @@ export const initializeControllers = (apiServices: ApiServices): InitializedCont
return {
authController: new AuthController(apiServices.userService),
monitorController: new MonitorController(apiServices.monitorService, apiServices.notificationsService),
settingsController: new SettingsController(apiServices.settingsService, apiServices.emailService, apiServices.proxiesService),
settingsController: new SettingsController(
apiServices.settingsService,
apiServices.emailService,
apiServices.proxiesService,
apiServices.egressStateService
),
checkController: new CheckController(apiServices.checkService),
geoCheckController: new GeoCheckController(apiServices.geoChecksService),
inviteController: new InviteController(apiServices.inviteService),
+36
View File
@@ -3,6 +3,39 @@ import { DbTypes, LogLevels, QueueModes } from "@/domain/app-settings/app-settin
import { booleanCoercion } from "@/api/validation/shared.js";
import { ILogger } from "@/utils/logger.js";
// Padded standard base64 of exactly 32 bytes, as produced by `openssl rand -base64 32`
const ENCRYPTION_KEY_ENTRY = /^[A-Za-z0-9+/]{43}=$/;
export const encryptionKeyList = z
.string()
.optional()
.transform((raw, ctx) => {
if (raw === undefined) return [];
const entries = raw
.split(",")
.map((entry, index) => ({ entry: entry.trim(), slot: index + 1 }))
.filter(({ entry }) => entry.length > 0);
const seen = new Set<string>();
const keys: Buffer[] = [];
entries.forEach(({ entry, slot }) => {
if (!ENCRYPTION_KEY_ENTRY.test(entry)) {
ctx.addIssue({
code: "custom",
message: `ENCRYPTION_KEY entry ${slot} must be 32 bytes as padded standard base64 (openssl rand -base64 32)`,
});
return;
}
const decoded = Buffer.from(entry, "base64");
const fingerprint = decoded.toString("hex");
if (seen.has(fingerprint)) {
ctx.addIssue({ code: "custom", message: `ENCRYPTION_KEY entry ${slot} duplicates an earlier entry` });
}
seen.add(fingerprint);
keys.push(decoded);
});
return keys;
});
const envSchema = z.object({
// Server Configuration
PORT: z.string().default("52345"),
@@ -36,6 +69,9 @@ const envSchema = z.object({
// Feature flags
STATUS_PAGE_THEMES_ENABLED: booleanCoercion.default(true),
// Encryption
ENCRYPTION_KEY: encryptionKeyList,
});
export type ValidatedEnv = z.infer<typeof envSchema>;
+6
View File
@@ -7,6 +7,7 @@ import { ITagsService, TagsService } from "@/domain/tags/tag.service.js";
import { IUserService, UserService } from "@/domain/users/user.service.js";
import { IJobScheduler } from "@/worker/worker.interface.js";
import { ProxiesService, IProxiesService } from "@/domain/proxies/proxy.service.js";
import { EgressStateService, IEgressStateService } from "@/domain/egress/egress-state.service.js";
import { SharedServices } from "@/config/services.shared.js";
// Third-party
@@ -24,6 +25,7 @@ export interface ApiServices extends SharedServices {
tagsService: ITagsService;
diagnosticService: IDiagnosticService;
proxiesService: IProxiesService;
egressStateService: IEgressStateService;
}
export const buildApi = (shared: SharedServices, jobScheduler: IJobScheduler): ApiServices => {
@@ -48,6 +50,7 @@ export const buildApi = (shared: SharedServices, jobScheduler: IJobScheduler): A
maintenanceWindowsRepository,
jobsRepository,
proxiesRepository,
egressStateRepository,
} = shared;
const userService = new UserService({
@@ -80,6 +83,7 @@ export const buildApi = (shared: SharedServices, jobScheduler: IJobScheduler): A
monitorStatsRepository,
statusPagesRepository,
incidentsRepository,
encryptionService: shared.encryptionService,
});
const maintenanceWindowService = new MaintenanceWindowService({
@@ -99,6 +103,7 @@ export const buildApi = (shared: SharedServices, jobScheduler: IJobScheduler): A
const tagsService = new TagsService(tagsRepository, monitorsRepository);
const diagnosticService = new DiagnosticService(db);
const proxiesService = new ProxiesService(proxiesRepository, monitorsRepository, settingsService);
const egressStateService = new EgressStateService(egressStateRepository, jobsRepository);
return {
...shared,
worker: jobScheduler,
@@ -110,5 +115,6 @@ export const buildApi = (shared: SharedServices, jobScheduler: IJobScheduler): A
tagsService,
diagnosticService,
proxiesService,
egressStateService,
};
};
+18 -2
View File
@@ -1,6 +1,6 @@
import { Mongoose } from "mongoose";
import { hostname } from "node:os";
import { randomUUID } from "node:crypto";
import * as nodeCrypto from "node:crypto";
import fs from "fs";
import path from "path";
import nodemailer from "nodemailer";
@@ -20,6 +20,7 @@ import { INotificationsService, NotificationsService } from "@/domain/notificati
import { IEmailService, EmailService } from "@/service/emailService.js";
import { GlobalPingService } from "@/service/globalPingService.js";
import { ILogger } from "@/utils/logger.js";
import { IEncryptionService, EncryptionService } from "@/service/encryption/encryptionService.js";
// Notification providers
import type { NotificationProviderRegistry } from "@/domain/notifications/notification.service.js";
@@ -33,6 +34,7 @@ import { MatrixProvider } from "@/domain/notifications/providers/matrix.js";
import { TeamsProvider } from "@/domain/notifications/providers/teams.js";
import { TelegramProvider } from "@/domain/notifications/providers/telegram.js";
import { PushoverProvider } from "@/domain/notifications/providers/pushover.js";
import { SignalgridProvider } from "@/domain/notifications/providers/signalgrid.js";
import { TwilioProvider } from "@/domain/notifications/providers/twilio.js";
import { NtfyProvider } from "@/domain/notifications/providers/ntfy.js";
@@ -54,6 +56,7 @@ import { ITagsRepository } from "@/domain/tags/tag.repository.interface.js";
import { ITeamsRepository } from "@/domain/teams/team.repository.interface.js";
import { IUsersRepository } from "@/domain/users/user.repository.interface.js";
import { IProxiesRepository } from "@/domain/proxies/proxy.repository.interface.js";
import { IEgressStateRepository } from "@/domain/egress/egress-state.repository.interface.js";
// Mongo repository implementations
import MongoSettingsRepository from "@/domain/app-settings/app-settings.repository.mongo.js";
@@ -74,6 +77,7 @@ import MongoTagsRepository from "@/domain/tags/tag.repository.mongo.js";
import MongoTeamsRepository from "@/domain/teams/team.repository.model.js";
import MongoUsersRepository from "@/domain/users/user.repository.mongo.js";
import MongoProxiesRepository from "@/domain/proxies/proxy.repository.mongo.js";
import MongoEgressStateRepository from "@/domain/egress/egress-state.repository.mongo.js";
import { IDockerLogsRepository } from "@/domain/docker/docker-log.repository.interface.js";
// Shared infrastructure + business services that both the API and the worker process construct.
@@ -82,6 +86,7 @@ export interface SharedServices {
db: IDb<Mongoose>;
settingsService: ISettingsService;
emailService: IEmailService;
encryptionService: IEncryptionService;
notificationMessageBuilder: INotificationMessageBuilder;
incidentService: IIncidentService;
checkService: ICheckService;
@@ -111,19 +116,25 @@ export interface SharedServices {
teamsRepository: ITeamsRepository;
maintenanceWindowsRepository: IMaintenanceWindowsRepository;
proxiesRepository: IProxiesRepository;
egressStateRepository: IEgressStateRepository;
}
export const buildShared = async ({
logger,
envSettings,
settingsService,
encryptionKeys,
}: {
logger: ILogger;
envSettings: EnvConfig;
settingsService: ISettingsService;
encryptionKeys: Buffer[];
}): Promise<SharedServices> => {
// Shared worker ID
const workerId = `${hostname()}:${process.pid}:${randomUUID()}`;
const workerId = `${hostname()}:${process.pid}:${nodeCrypto.randomUUID()}`;
// Create the encryption service
const encryptionService = new EncryptionService(encryptionKeys, logger, nodeCrypto);
// Create DB
let db: IDb<Mongoose> | null = null;
@@ -149,6 +160,7 @@ export const buildShared = async ({
const teamsRepository = new MongoTeamsRepository();
const maintenanceWindowsRepository = new MongoMaintenanceWindowsRepository();
const proxiesRepository = new MongoProxiesRepository();
const egressStateRepository = new MongoEgressStateRepository();
// Inject settings repository into settings service (now that DB is connected)
(settingsService as SettingsService).setRepository(settingsRepository);
@@ -178,6 +190,7 @@ export const buildShared = async ({
const teamsProvider = new TeamsProvider(logger);
const telegramProvider = new TelegramProvider(logger);
const pushoverProvider = new PushoverProvider(logger);
const signalgridProvider = new SignalgridProvider(logger);
const twilioProvider = new TwilioProvider(logger);
const ntfyProvider = new NtfyProvider(logger);
@@ -192,6 +205,7 @@ export const buildShared = async ({
teams: teamsProvider,
telegram: telegramProvider,
pushover: pushoverProvider,
signalgrid: signalgridProvider,
twilio: twilioProvider,
ntfy: ntfyProvider,
};
@@ -210,6 +224,7 @@ export const buildShared = async ({
db,
settingsService,
emailService,
encryptionService,
notificationMessageBuilder,
incidentService,
checkService,
@@ -235,6 +250,7 @@ export const buildShared = async ({
teamsRepository,
maintenanceWindowsRepository,
proxiesRepository,
egressStateRepository,
};
return sharedServices;
};
+30 -4
View File
@@ -16,6 +16,7 @@ import { SharedServices } from "@/config/services.shared.js";
import { INetworkService, NetworkService } from "@/service/networkService.js";
import { IBufferService, BufferService } from "@/service/bufferService.js";
import { IStatusService, StatusService } from "@/service/statusService.js";
import { SecretsRotationService } from "@/service/encryption/secretsRotationService.js";
import { IQueueWorker } from "@/worker/worker.interface.js";
import { MonitorStatusPolicy } from "@/worker/worker.monitor-status-policy.js";
import { WorkerHelper } from "@/worker/worker.helper.js";
@@ -27,6 +28,7 @@ import { IncidentReactor } from "@/worker/reactors/reactor.incident.js";
import { ReactorDispatcher } from "@/worker/reactors/reactor.dispatcher.js";
import { DBQueueWorker } from "@/worker/worker.db-queue.js";
import { ProxyResolver } from "@/service/network/ProxyResolver.js";
import { IEgressService, EgressService } from "@/domain/egress/egress.service.js";
// Network providers
import { PingProvider } from "@/service/network/PingProvider.js";
import { HttpProvider } from "@/service/network/HttpProvider.js";
@@ -39,11 +41,13 @@ import { GameProvider } from "@/service/network/GameProvider.js";
import { GrpcProvider } from "@/service/network/GrpcProvider.js";
import { WebSocketProvider } from "@/service/network/WebSocketProvider.js";
import { DNSProvider } from "@/service/network/DNSProvider.js";
import { AppError } from "@/utils/AppError.js";
export interface WorkerServices {
worker: IQueueWorker;
networkService: INetworkService;
bufferService: IBufferService;
statusService: IStatusService;
egressService: IEgressService;
}
export const buildWorker = async (shared: SharedServices, envSettings: EnvConfig): Promise<WorkerServices> => {
@@ -59,6 +63,7 @@ export const buildWorker = async (shared: SharedServices, envSettings: EnvConfig
jobsRepository,
queueWorkersRepository,
monitorsRepository,
encryptionService,
checksRepository,
geoChecksRepository,
dockerLogsRepository,
@@ -67,16 +72,17 @@ export const buildWorker = async (shared: SharedServices, envSettings: EnvConfig
teamsRepository,
maintenanceWindowsRepository,
proxiesRepository,
egressStateRepository,
} = shared;
// ***********************
// Network providers
// ***********************
const pingProvider = new PingProvider(ping);
const pingProvider = new PingProvider(ping, net);
const httpProvider = new HttpProvider(got, new AdvancedMatcher(jmespath));
const pageSpeedProvider = new PageSpeedProvider(httpProvider, settingsService, logger);
const hardwareProvider = new HardwareProvider(httpProvider);
const dockerProvider = new DockerProvider(logger, Docker);
const dockerProvider = new DockerProvider(logger, Docker, encryptionService);
const portProvider = new PortProvider(net);
const gameProvider = new GameProvider(logger, GameDig);
const grpcProvider = new GrpcProvider(grpc, protoLoader);
@@ -100,6 +106,7 @@ export const buildWorker = async (shared: SharedServices, envSettings: EnvConfig
const bufferService = new BufferService(logger, checkService, geoChecksService, dockerLogsService, settingsService, jobsRepository);
const statusService = new StatusService(logger, monitorsRepository, monitorStatsRepository);
const monitorStatusPolicy = new MonitorStatusPolicy();
const egressService = new EgressService(settingsService, egressStateRepository, jobsRepository, networkService, proxyResolver, logger);
// ***********************
// Reactors and dispatcher
@@ -142,7 +149,8 @@ export const buildWorker = async (shared: SharedServices, envSettings: EnvConfig
checksRepository,
incidentsRepository,
geoChecksRepository,
dockerLogsRepository
dockerLogsRepository,
egressService
);
const worker = await DBQueueWorker.create({
@@ -164,5 +172,23 @@ export const buildWorker = async (shared: SharedServices, envSettings: EnvConfig
workerId,
});
return { worker, networkService, bufferService, statusService };
const secretsRotationService = new SecretsRotationService(monitorsRepository, encryptionService, logger);
try {
const result = await secretsRotationService.run();
logger.debug({
message: "Secrets rotation run successfully",
service: "WorkerServices",
method: "buildWorker",
details: { ...result },
});
} catch (error: unknown) {
logger.warn({
message: `Could not rotate Docker TLS secrets: ${error instanceof Error ? error.message : String(error)}`,
service: "WorkerServices",
method: "buildWorker",
details: { ...(error instanceof AppError ? error.details : {}) },
});
}
return { worker, networkService, bufferService, statusService, egressService };
};
@@ -6,6 +6,7 @@ export interface ISettingsRepository {
findSingleton(): Promise<Settings | null>;
// update
update(settings: SettingsUpdate): Promise<Settings>;
removeEgressNotification(notificationId: string): Promise<void>; // $pull, so it cannot race a concurrent settings update
// delete
deleteLegacy: () => Promise<boolean>;
}
@@ -1,5 +1,6 @@
import { Schema, model, type Types } from "mongoose";
import type { Settings, SettingsThresholds } from "@/domain/app-settings/app-settings.type.js";
import { DEFAULT_EGRESS_POLL_INTERVAL_SECONDS, DEFAULT_EGRESS_TARGETS } from "@/domain/egress/egress.type.js";
interface AppSettingsDocument extends Omit<Settings, "id" | "createdAt" | "updatedAt"> {
_id: Types.ObjectId;
@@ -42,6 +43,10 @@ const AppSettingsSchema = new Schema<AppSettingsDocument>(
globalThresholds: { type: thresholdsSchema },
globalProxyEnabled: { type: Boolean, default: false },
globalProxyId: { type: String, default: null },
egressCheckEnabled: { type: Boolean, default: false },
egressCheckTargets: { type: [String], default: () => [...DEFAULT_EGRESS_TARGETS] },
egressPollIntervalSeconds: { type: Number, default: DEFAULT_EGRESS_POLL_INTERVAL_SECONDS },
egressNotifications: { type: [String], default: () => [] }, // Notification ids, stored as strings like globalProxyId
},
{ timestamps: true }
);
@@ -3,6 +3,7 @@ import { ISettingsRepository } from "@/domain/app-settings/app-settings-reposito
import type { Settings, SettingsUpdate } from "@/domain/app-settings/app-settings.type.js";
import { AppSettingsModel, type AppSettingsDocument } from "@/domain/app-settings/app-settings.model.js";
import { toStringId, toDateString } from "@/utils/mongoMappers.js";
import { DEFAULT_EGRESS_POLL_INTERVAL_SECONDS, DEFAULT_EGRESS_TARGETS } from "@/domain/egress/egress.type.js";
class MongoSettingsRepository implements ISettingsRepository {
private toEntity = (doc: AppSettingsDocument): Settings => {
@@ -30,6 +31,10 @@ class MongoSettingsRepository implements ISettingsRepository {
globalThresholds: doc.globalThresholds ?? undefined,
globalProxyEnabled: doc.globalProxyEnabled ?? false,
globalProxyId: doc.globalProxyId ?? undefined,
egressCheckEnabled: doc.egressCheckEnabled ?? false,
egressCheckTargets: doc.egressCheckTargets ?? [...DEFAULT_EGRESS_TARGETS],
egressPollIntervalSeconds: doc.egressPollIntervalSeconds ?? DEFAULT_EGRESS_POLL_INTERVAL_SECONDS,
egressNotifications: doc.egressNotifications ?? [],
createdAt: toDateString(doc.createdAt),
updatedAt: toDateString(doc.updatedAt),
};
@@ -79,6 +84,10 @@ class MongoSettingsRepository implements ISettingsRepository {
return this.toEntity(updatedSettings);
};
removeEgressNotification = async (notificationId: string) => {
await AppSettingsModel.updateMany({ egressNotifications: notificationId }, { $pull: { egressNotifications: notificationId } });
};
deleteLegacy = async () => {
const res = await AppSettingsModel.deleteMany({ version: { $exists: false } });
return res.deletedCount > 0;
@@ -23,15 +23,27 @@ export interface ISettingsService {
getSettings(): EnvConfig;
areStatusPageThemesEnabled(): boolean;
getDBSettings(): Promise<Settings>;
getCachedDBSettings(): Promise<Settings>;
updateDbSettings(newSettings: SettingsUpdate): Promise<Settings>;
removeEgressNotification(notificationId: string): Promise<void>;
}
const DB_SETTINGS_CACHE_TTL_MS = 60_000;
type CachedSettings = {
value: Promise<Settings>;
expiresAt: number;
};
export class SettingsService implements ISettingsService {
static SERVICE_NAME = SERVICE_NAME;
private settings: EnvConfig;
private settingsRepository: ISettingsRepository | null = null;
private dbSettingsCache: CachedSettings | null = null;
constructor(env: ValidatedEnv) {
constructor(
env: ValidatedEnv,
private dbSettingsTTLMs: number = DB_SETTINGS_CACHE_TTL_MS
) {
this.settings = {
jwtSecret: env.JWT_SECRET,
jwtTTL: env.TOKEN_TTL as StringValue,
@@ -75,7 +87,14 @@ export class SettingsService implements ISettingsService {
}
updateDbSettings = async (newSettings: SettingsUpdate) => {
return await this.getRepository().update(newSettings);
const updated = await this.getRepository().update(newSettings);
this.dbSettingsCache = null;
return updated;
};
removeEgressNotification = async (notificationId: string) => {
await this.getRepository().removeEgressNotification(notificationId);
this.dbSettingsCache = null;
};
getDBSettings = async () => {
@@ -95,4 +114,17 @@ export class SettingsService implements ISettingsService {
return settings;
};
getCachedDBSettings = async () => {
const hit = this.dbSettingsCache;
if (hit && hit.expiresAt > Date.now()) {
return hit.value;
}
const entry: CachedSettings = { value: this.getDBSettings(), expiresAt: Date.now() + this.dbSettingsTTLMs };
this.dbSettingsCache = entry;
entry.value.catch(() => {
if (this.dbSettingsCache === entry) this.dbSettingsCache = null;
});
return entry.value;
};
}
@@ -51,6 +51,11 @@ export interface Settings {
globalThresholds?: SettingsThresholds;
globalProxyEnabled: boolean;
globalProxyId?: string | null;
// Egress self-check (see domain/egress)
egressCheckEnabled: boolean;
egressCheckTargets: string[];
egressPollIntervalSeconds: number; // recovery poll cadence while degraded, not a general schedule
egressNotifications: string[]; // Notification ids to receive the "egress recovered" alert
createdAt: string;
updatedAt: string;
}
@@ -0,0 +1,40 @@
import { Schema, model, type Types } from "mongoose";
import { EgressStatuses, type EgressProbeResult, type EgressState } from "@/domain/egress/egress.type.js";
interface EgressStateDocument extends Omit<EgressState, "id" | "degradedSince" | "lastRecoveredAt" | "lastProbeAt" | "createdAt" | "updatedAt"> {
_id: Types.ObjectId;
singleton: boolean;
degradedSince: Date | null;
lastRecoveredAt: Date | null;
lastProbeAt: Date | null;
createdAt: Date;
updatedAt: Date;
}
const probeResultSchema = new Schema<EgressProbeResult>(
{
target: { type: String, required: true },
reachable: { type: Boolean, required: true },
responseTime: { type: Number, default: 0 },
message: { type: String },
},
{ _id: false }
);
const EgressStateSchema = new Schema<EgressStateDocument>(
{
singleton: { type: Boolean, required: true, unique: true, default: true },
status: { type: String, enum: EgressStatuses, required: true, default: "ok" },
degradedSince: { type: Date, default: null },
lastRecoveredAt: { type: Date, default: null },
lastProbeAt: { type: Date, default: null },
lastProbeResults: { type: [probeResultSchema], default: () => [] },
},
{ timestamps: true }
);
const EgressStateModel = model<EgressStateDocument>("EgressState", EgressStateSchema);
export type { EgressStateDocument };
export { EgressStateModel };
export default EgressStateModel;
@@ -0,0 +1,13 @@
import type { EgressProbeResult, EgressState } from "@/domain/egress/egress.type.js";
export interface IEgressStateRepository {
// read
findSingleton(): Promise<EgressState>; // creates the "ok" singleton on first call, a plain read after that
// update
recordProbe(results: EgressProbeResult[], now: Date): Promise<EgressState>;
reset(): Promise<EgressState>; // back to "ok" with no episode; used when the feature is toggled
// Atomic transitions. Each returns the updated state when this caller performed the transition,
// or null when the state was already on the target side (another worker got there first).
markDegraded(results: EgressProbeResult[], now: Date): Promise<EgressState | null>;
markRecovered(results: EgressProbeResult[], now: Date): Promise<EgressState | null>;
}
@@ -0,0 +1,88 @@
import { IEgressStateRepository } from "@/domain/egress/egress-state.repository.interface.js";
import type { EgressProbeResult, EgressState } from "@/domain/egress/egress.type.js";
import { EgressStateModel, type EgressStateDocument } from "@/domain/egress/egress-state.model.js";
import { toStringId, toDateString } from "@/utils/mongoMappers.js";
const SERVICE_NAME = "EgressStateRepository";
class MongoEgressStateRepository implements IEgressStateRepository {
static SERVICE_NAME = SERVICE_NAME;
private toEntity = (doc: EgressStateDocument): EgressState => {
return {
id: toStringId(doc._id),
status: doc.status,
degradedSince: doc.degradedSince ? toDateString(doc.degradedSince) : null,
lastRecoveredAt: doc.lastRecoveredAt ? toDateString(doc.lastRecoveredAt) : null,
lastProbeAt: doc.lastProbeAt ? toDateString(doc.lastProbeAt) : null,
lastProbeResults: (doc.lastProbeResults ?? []).map((result) => ({
target: result.target,
reachable: result.reachable,
responseTime: result.responseTime ?? 0,
...(result.message !== undefined && { message: result.message }),
})),
createdAt: toDateString(doc.createdAt),
updatedAt: toDateString(doc.updatedAt),
};
};
findSingleton = async () => {
// Read first: with timestamps on, every findOneAndUpdate would $set updatedAt, so a read would write.
const existing = await EgressStateModel.findOne({ singleton: true }).lean<EgressStateDocument>();
if (existing) return this.toEntity(existing);
try {
const doc = await EgressStateModel.findOneAndUpdate(
{ singleton: true },
{ $setOnInsert: { singleton: true, status: "ok" } },
{ upsert: true, new: true, setDefaultsOnInsert: true }
).lean<EgressStateDocument>();
return this.toEntity(doc);
} catch (error: unknown) {
// Two processes can race the very first insert; the unique index rejects the loser, who reads the winner's row.
if ((error as { code?: number }).code === 11000) {
const doc = await EgressStateModel.findOne({ singleton: true }).lean<EgressStateDocument>();
if (doc) return this.toEntity(doc);
}
throw error;
}
};
reset = async () => {
const doc = await EgressStateModel.findOneAndUpdate(
{ singleton: true },
{ $set: { status: "ok", degradedSince: null, lastProbeAt: null, lastProbeResults: [] }, $setOnInsert: { singleton: true } },
{ upsert: true, new: true, setDefaultsOnInsert: true }
).lean<EgressStateDocument>();
return this.toEntity(doc);
};
recordProbe = async (results: EgressProbeResult[], now: Date) => {
const doc = await EgressStateModel.findOneAndUpdate(
{ singleton: true },
{ $set: { lastProbeAt: now, lastProbeResults: results }, $setOnInsert: { singleton: true, status: "ok" } },
{ upsert: true, new: true, setDefaultsOnInsert: true }
).lean<EgressStateDocument>();
return this.toEntity(doc);
};
markDegraded = async (results: EgressProbeResult[], now: Date) => {
// Only the caller that flips ok -> degraded sees a document back; everyone else sees null.
const doc = await EgressStateModel.findOneAndUpdate(
{ singleton: true, status: "ok" },
{ $set: { status: "degraded", degradedSince: now, lastProbeAt: now, lastProbeResults: results } },
{ new: true }
).lean<EgressStateDocument>();
return doc ? this.toEntity(doc) : null;
};
markRecovered = async (results: EgressProbeResult[], now: Date) => {
const doc = await EgressStateModel.findOneAndUpdate(
{ singleton: true, status: "degraded" },
{ $set: { status: "ok", lastRecoveredAt: now, lastProbeAt: now, lastProbeResults: results } },
{ new: true }
).lean<EgressStateDocument>();
return doc ? this.toEntity(doc) : null;
};
}
export default MongoEgressStateRepository;
@@ -0,0 +1,31 @@
import { IEgressStateRepository } from "@/domain/egress/egress-state.repository.interface.js";
import { IJobsRepository } from "@/domain/jobs/job.repository.interface.js";
import type { EgressState } from "@/domain/egress/egress.type.js";
export interface IEgressStateService {
getState(): Promise<EgressState>;
reset(): Promise<EgressState>;
}
const SERVICE_NAME = "EgressStateService";
// API-process view of the instance-global egress state. Probe-driven transitions are owned by the
// worker (egress.service.ts); this exists so the controller never touches the repositories.
export class EgressStateService implements IEgressStateService {
static SERVICE_NAME = SERVICE_NAME;
constructor(
private egressStateRepository: IEgressStateRepository,
private jobsRepository: IJobsRepository
) {}
getState = async (): Promise<EgressState> => {
return this.egressStateRepository.findSingleton();
};
// Called when the feature is switched on or off: drop any pending recovery job and start from "ok".
reset = async (): Promise<EgressState> => {
await this.jobsRepository.deleteGlobalJob("egress");
return this.egressStateRepository.reset();
};
}
+283
View File
@@ -0,0 +1,283 @@
import type { Monitor } from "@/domain/monitors/monitor.type.js";
import type { ISettingsService } from "@/domain/app-settings/app-settings.service.js";
import type { IEgressStateRepository } from "@/domain/egress/egress-state.repository.interface.js";
import type { IJobsRepository } from "@/domain/jobs/job.repository.interface.js";
import { jobId, type Job, type JobSeed } from "@/domain/jobs/job.type.js";
import type { INetworkService } from "@/service/networkService.js";
import type { IProxyResolver } from "@/service/network/ProxyResolver.js";
import type { ILogger } from "@/utils/logger.js";
import {
DEFAULT_EGRESS_POLL_INTERVAL_SECONDS,
DEFAULT_EGRESS_TARGETS,
type EgressAssessment,
type EgressProbeResult,
type EgressStatus,
} from "@/domain/egress/egress.type.js";
import { timeRequest } from "@/service/network/utils.js";
import { parseEgressTarget, type EgressTarget } from "@/utils/egressTarget.js";
const SERVICE_NAME = "EgressService";
const PROBE_TIMEOUT_MS = 5000;
// How long a settled assessment answers for later failures. An assessment still in flight is shared
// for as long as it takes, so this only bounds how stale a reused answer can be.
const ASSESSMENT_REUSE_MS = 5000;
// Global job row (refId null) that re-probes while degraded. Inserted when egress degrades, removed on recovery.
const RECOVERY_JOB_TYPE = "egress" as const;
const RECOVERY_JOB_ID = jobId(RECOVERY_JOB_TYPE, null);
// HttpProvider reports transport failures with NETWORK_ERROR (outside the HTTP range) and everything else with the real status code.
const isHttpStatusCode = (code: unknown): boolean => typeof code === "number" && code >= 100 && code <= 599;
// Synthetic identity stamped on the probe monitors so provider responses are recognisable in logs.
const PROBE_MONITOR_ID = "egress-probe";
const PROBE_TEAM_ID = "system";
export interface IEgressService {
probeTargets(targets: string[]): Promise<EgressProbeResult[]>;
// Called by the check producer when a check fails. Null when the feature is off or the assessment itself failed.
assessAfterFailure(): Promise<EgressStatus | null>;
// Handler for the "egress" job: re-probes while degraded and removes the job once egress is back.
checkRecovery(job: Job): Promise<void>;
}
type CachedAssessment = {
value: Promise<EgressAssessment | null>;
expiresAt: number | null; // null while the assessment is still in flight
};
export class EgressService implements IEgressService {
static SERVICE_NAME = SERVICE_NAME;
private assessment: CachedAssessment | null = null;
constructor(
private settingsService: ISettingsService,
private egressStateRepository: IEgressStateRepository,
private jobsRepository: IJobsRepository,
private networkService: INetworkService,
private proxyResolver: IProxyResolver,
private logger: ILogger
) {}
// ****************************
// Probing
// ****************************
// Reuses the existing ping/port/http providers by dressing each target up as a minimal monitor.
// proxyMode "inherit" lets the resolver apply the global proxy to http probes, as it does for monitors.
private toProbeMonitor = (target: EgressTarget, raw: string): Monitor => {
const base = {
id: PROBE_MONITOR_ID,
teamId: PROBE_TEAM_ID,
name: `Egress probe: ${raw}`,
method: "GET",
proxyMode: "inherit",
useAdvancedMatching: false,
ignoreTlsErrors: false,
customUpCodes: [],
};
switch (target.kind) {
case "http":
return { ...base, type: "http", url: target.url } as unknown as Monitor;
case "port":
return { ...base, type: "port", url: target.host, port: target.port } as unknown as Monitor;
case "ping":
return { ...base, type: "ping", url: target.host } as unknown as Monitor;
}
};
private probeTarget = async (raw: string): Promise<EgressProbeResult> => {
const target = parseEgressTarget(raw);
if (!target) {
// Validation rejects these; a stored value can still be stale, and it must not count as reachable.
return { target: raw, reachable: false, responseTime: 0, message: "Invalid egress target" };
}
const monitor = this.toProbeMonitor(target, raw);
let timer: NodeJS.Timeout | undefined;
const { response, responseTime, error } = await timeRequest(async () => {
const proxyUrl = await this.proxyResolver.resolve(monitor);
return Promise.race([
this.networkService.requestStatus(monitor, { proxyUrl }),
new Promise<never>((_, reject) => {
timer = setTimeout(() => reject(new Error(`Egress probe timed out after ${PROBE_TIMEOUT_MS}ms`)), PROBE_TIMEOUT_MS);
}),
]).finally(() => clearTimeout(timer));
});
if (error || !response) {
return {
target: raw,
reachable: false,
responseTime,
message: error instanceof Error ? error.message : error ? String(error) : "No response",
};
}
return {
target: raw,
// Any HTTP response, including 4xx/5xx, proves the instance can reach the target; only a transport
// failure counts against egress. Ping and port providers have no such distinction.
reachable: response.status === true || (target.kind === "http" && isHttpStatusCode(response.code)),
responseTime: response.responseTime ?? responseTime,
message: response.message,
};
};
probeTargets = async (targets: string[]): Promise<EgressProbeResult[]> => {
return await Promise.all(targets.map((target) => this.probeTarget(target)));
};
private resolveTargets = (configured: string[] | undefined): string[] => {
const targets = (configured ?? []).map((target) => target.trim()).filter((target) => target.length > 0);
// An empty list would make "all unreachable" vacuously true, so fall back to the defaults.
return targets.length > 0 ? targets : [...DEFAULT_EGRESS_TARGETS];
};
private toIntervalMs = (seconds: number | undefined): number => {
const valid = typeof seconds === "number" && Number.isFinite(seconds) && seconds > 0;
return (valid ? seconds : DEFAULT_EGRESS_POLL_INTERVAL_SECONDS) * 1000;
};
// ****************************
// Event-triggered assessment
// ****************************
assessAfterFailure = async (): Promise<EgressStatus | null> => {
const assessment = await this.sharedAssessment();
return assessment === "disabled" ? null : assessment;
};
// When egress is down every monitor fails at once. Concurrent failures share the in-flight assessment
// however long it takes, and a settled one answers for ASSESSMENT_REUSE_MS. Same shape as ProxyResolver's cache.
private sharedAssessment = (): Promise<EgressAssessment | null> => {
const cached = this.assessment;
if (cached && (cached.expiresAt === null || cached.expiresAt > Date.now())) {
return cached.value;
}
const entry: CachedAssessment = { value: this.assess(), expiresAt: null };
this.assessment = entry;
void entry.value.then((result) => {
if (this.assessment !== entry) return;
if (result === null) {
this.assessment = null; // an internal failure is not cached, so the next failing check retries
} else {
entry.expiresAt = Date.now() + ASSESSMENT_REUSE_MS;
}
});
return entry.value;
};
private assess = async (): Promise<EgressAssessment | null> => {
try {
const settings = await this.settingsService.getCachedDBSettings();
if (!settings.egressCheckEnabled) {
return "disabled";
}
const state = await this.egressStateRepository.findSingleton();
if (state.status === "degraded") {
// Recovery is detected by the scheduled job. Make sure the row exists without touching its schedule.
await this.scheduleRecoveryCheck(settings.egressPollIntervalSeconds, false);
return "degraded";
}
const results = await this.probeTargets(this.resolveTargets(settings.egressCheckTargets));
const now = new Date();
if (results.some((result) => result.reachable)) {
await this.egressStateRepository.recordProbe(results, now);
return "ok";
}
// Schedule the recovery job before recording the transition: a crash between the two then leaves a stray row
// while ok, which the job's first run removes, rather than a degraded state that nothing is polling.
await this.scheduleRecoveryCheck(settings.egressPollIntervalSeconds, true);
const degraded = await this.egressStateRepository.markDegraded(results, now);
this.logger.warn({
message: degraded
? "Instance egress degraded: every reliability target is unreachable. Monitor failures will not be counted until it recovers"
: "Instance egress degraded (transition already recorded by another worker)",
service: SERVICE_NAME,
method: "assessAfterFailure",
details: { results },
});
return "degraded";
} catch (error: unknown) {
// The egress check must never break check production; treat an internal failure as "unknown" and carry on.
this.logger.error({
message: error instanceof Error ? error.message : String(error),
service: SERVICE_NAME,
method: "assessAfterFailure",
stack: error instanceof Error ? error.stack : undefined,
});
return null;
}
};
// ****************************
// Recovery job
// ****************************
// reschedule=true sets nextScheduledAt (start of an episode); false only inserts the row if it is missing,
// so re-arming while degraded never pushes a pending run back.
private scheduleRecoveryCheck = async (pollIntervalSeconds: number | undefined, reschedule: boolean) => {
const intervalMs = this.toIntervalMs(pollIntervalSeconds);
const seed: JobSeed = {
id: RECOVERY_JOB_ID,
type: RECOVERY_JOB_TYPE,
refId: null,
isActive: true,
nextScheduledAt: Date.now() + intervalMs,
intervalMs,
};
if (reschedule) {
await this.jobsRepository.upsertCleanupJob(seed); // global-row upsert that also sets the schedule
} else {
await this.jobsRepository.upsertJob(seed);
}
};
// Removes the row only if nobody has rescheduled it since this run claimed it. Another worker may have seen
// egress fail again in the meantime and started a new episode; its schedule must survive.
private releaseRecoveryJob = async (job: Job) => {
await this.jobsRepository.deleteGlobalJobIfUnchanged(RECOVERY_JOB_TYPE, job.nextScheduledAt);
};
// Runs on the queue at the configured interval while degraded. Errors propagate so the queue records
// the failure and retries with its usual backoff.
checkRecovery = async (job: Job): Promise<void> => {
const settings = await this.settingsService.getCachedDBSettings();
const state = await this.egressStateRepository.findSingleton();
if (!settings.egressCheckEnabled || state.status !== "degraded") {
// Feature switched off, recovery recorded by another worker, or state reset: nothing left to poll for.
await this.releaseRecoveryJob(job);
return;
}
const results = await this.probeTargets(this.resolveTargets(settings.egressCheckTargets));
const now = new Date();
if (!results.some((result) => result.reachable)) {
await this.egressStateRepository.recordProbe(results, now);
return;
}
const recovered = await this.egressStateRepository.markRecovered(results, now);
await this.releaseRecoveryJob(job);
if (!recovered) {
// Another process performed the transition.
return;
}
this.logger.info({
message: `Instance egress recovered (degraded since ${recovered.degradedSince ?? "unknown"})`,
service: SERVICE_NAME,
method: "checkRecovery",
details: { results },
});
};
}
+35
View File
@@ -0,0 +1,35 @@
// Egress self-check: detects loss of the instance's own outbound connectivity so that
// a burst of simultaneous monitor failures is not reported as a burst of real outages.
export const EgressStatuses = ["ok", "degraded"] as const;
export type EgressStatus = (typeof EgressStatuses)[number];
// Outcome of one assessment: a status, or "disabled" when the feature is switched off (distinct from an internal failure).
export type EgressAssessment = EgressStatus | "disabled";
// Sensible defaults for a fresh install. Bare hosts are ICMP-pinged, host:port is a TCP connect,
// http(s):// URLs are fetched. Any one target being reachable means egress is fine.
export const DEFAULT_EGRESS_TARGETS = ["1.1.1.1", "8.8.8.8"] as const;
export const DEFAULT_EGRESS_POLL_INTERVAL_SECONDS = 30;
export const MIN_EGRESS_POLL_INTERVAL_SECONDS = 5;
export const MAX_EGRESS_POLL_INTERVAL_SECONDS = 600;
export const MAX_EGRESS_TARGETS = 10;
// Instance-global singleton persisted in MongoDB so that every worker process and the API
// process share one view of the state (the queue worker may run separately from the API).
export interface EgressState {
id: string;
status: EgressStatus;
degradedSince: string | null; // ISO timestamp of the current or most recent degraded entry
lastRecoveredAt: string | null; // ISO timestamp of the most recent recovery
lastProbeAt: string | null; // ISO timestamp of the most recent reliability probe
lastProbeResults: EgressProbeResult[];
createdAt: string;
updatedAt: string;
}
export interface EgressProbeResult {
target: string;
reachable: boolean;
responseTime: number; // ms
message?: string;
}
@@ -61,6 +61,11 @@ export interface IJobsRepository {
// Delete a single job of type
deleteByIdAndType(refId: string, type: JobType): Promise<boolean>;
// Global rows (refId null) are addressed by their canonical _id, see jobId
deleteGlobalJob(type: JobType): Promise<boolean>;
// Delete a global row only if nobody has rescheduled it since it was claimed (nextScheduledAt unchanged)
deleteGlobalJobIfUnchanged(type: JobType, nextScheduledAt: number): Promise<boolean>;
// Delete jobs that reference monitors that no longer exist
deleteByMonitorIdsNotIn(monitorIds: string[]): Promise<number>;
@@ -107,6 +107,8 @@ class MongoJobsRepository implements IJobsRepository {
lockedBy: null, // Remove lock
lockedUntil: null,
lastFinishedAt: now,
failCount: 0,
lastFailReason: null,
},
$inc: {
runCount: 1, // Increment run count
@@ -150,6 +152,8 @@ class MongoJobsRepository implements IJobsRepository {
lockedBy: null, // Remove lock
lockedUntil: null,
lastFinishedAt: now,
failCount: 0,
lastFailReason: null,
},
$inc: {
runCount: 1, // Increment run count
@@ -278,6 +282,16 @@ class MongoJobsRepository implements IJobsRepository {
return res.deletedCount > 0;
};
deleteGlobalJob = async (type: JobType) => {
const res = await JobModel.deleteOne({ _id: jobId(type, null) });
return res.deletedCount > 0;
};
deleteGlobalJobIfUnchanged = async (type: JobType, nextScheduledAt: number) => {
const res = await JobModel.deleteOne({ _id: jobId(type, null), nextScheduledAt });
return res.deletedCount > 0;
};
deleteByMonitorIdsNotIn = async (monitorIds: string[]): Promise<number> => {
// remove jobs who'se refId is not null (cleanup jobs) and not in the list of monitor IDs (orphaned monitor jobs)
const res = await JobModel.deleteMany({ refId: { $ne: null, $nin: monitorIds } });
+2 -2
View File
@@ -1,10 +1,10 @@
export const JOB_TYPES = ["check", "geo-check", "evaluate", "cleanup-orphaned", "cleanup-retention"] as const;
export const JOB_TYPES = ["check", "geo-check", "evaluate", "cleanup-orphaned", "cleanup-retention", "egress"] as const;
export type JobType = (typeof JOB_TYPES)[number];
export const LOCK_MS = 60_000;
export const BACKOFF_MS = 5_000;
export const PARKED = Number.MAX_SAFE_INTEGER;
// Canonical _id scheme: monitor-bound rows are `${type}:${refId}`; global rows (refId null) are just the type.
// Canonical _id scheme: monitor-bound rows are `${type}:${refId}`; global rows (refId null: cleanup, egress) are just the type.
export const jobId = (type: JobType, refId: string | null): string => (refId === null ? type : `${type}:${refId}`);
export type Job = {
@@ -307,6 +307,20 @@ const MonitorSchema = new Schema<MonitorDocument>(
type: Boolean,
default: false,
},
dockerTlsCa: {
type: String,
},
dockerTlsCert: {
type: String,
},
dockerTlsKey: {
type: String,
select: false,
},
dockerTlsKeySet: {
type: Boolean,
default: false,
},
dnsServer: {
type: String,
},
@@ -55,6 +55,11 @@ export interface IMonitorsRepository {
findMonitorCountByTeamIdAndType(teamId: string, config: TeamQueryConfig): Promise<number>;
findMonitorCountByProxyId(proxyId: string): Promise<number>;
// Docker TLS
findDockerTlsKeyById(monitorId: string): Promise<string | null>;
findAllDockerTlsKeys(): Promise<{ id: string; dockerTlsKey: string }[]>;
updateDockerTlsKey(monitorId: string, dockerTlsKey: string): Promise<void>; // Needed for worker to rotate keys, no teamId/userId
// other
findMonitorsSummaryByTeamId(teamId: string, config?: SummaryConfig): Promise<MonitorsSummary>;
removeNotificationFromMonitors(notificationId: string): Promise<void>;
@@ -298,6 +298,20 @@ class MongoMonitorsRepository implements IMonitorsRepository {
return { monitors: this.mapDocuments(monitors), deletedCount };
};
findDockerTlsKeyById = async (monitorId: string): Promise<string | null> => {
const doc = await MonitorModel.findOne({ _id: monitorId }, { dockerTlsKey: 1 }).lean();
return doc?.dockerTlsKey ?? null;
};
findAllDockerTlsKeys = async (): Promise<{ id: string; dockerTlsKey: string }[]> => {
const docs = await MonitorModel.find({ dockerTlsKeySet: true }, { dockerTlsKey: 1 }).lean();
return docs.flatMap((doc) => (doc.dockerTlsKey ? [{ id: doc._id.toString(), dockerTlsKey: doc.dockerTlsKey }] : []));
};
updateDockerTlsKey = async (monitorId: string, dockerTlsKey: string): Promise<void> => {
await MonitorModel.updateOne({ _id: monitorId }, { $set: { dockerTlsKey } });
};
findMonitorsSummaryByTeamId = async (teamId: string, config: SummaryConfig): Promise<MonitorsSummary> => {
const match = this.queryBuilder(config, teamId);
const pipeline = [
@@ -454,6 +468,9 @@ class MongoMonitorsRepository implements IMonitorsRepository {
geoCheckLocations: doc.geoCheckLocations ?? [],
geoCheckInterval: doc.geoCheckInterval ?? 300000,
dockerLogsEnabled: doc.dockerLogsEnabled ?? false,
dockerTlsCa: doc.dockerTlsCa ?? undefined,
dockerTlsCert: doc.dockerTlsCert ?? undefined,
dockerTlsKeySet: doc.dockerTlsKeySet ?? false,
dnsServer: doc.dnsServer ?? undefined,
dnsRecordType: doc.dnsRecordType ?? undefined,
createdAt: toDateString(doc.createdAt),
+57 -2
View File
@@ -33,6 +33,8 @@ import { ILogger } from "@/utils/logger.js";
import { IJobScheduler } from "@/worker/worker.interface.js";
import { DateRange } from "@/types/query.js";
import { IDockerLogsRepository } from "@/domain/docker/docker-log.repository.interface.js";
import { IEncryptionService } from "@/service/encryption/encryptionService.js";
import { isDockerTlsUrl } from "@/utils/dockerHost.js";
const SERVICE_NAME = "MonitorService";
@@ -60,7 +62,7 @@ const computeRestartsInRange = (aggregate: DockerContainerStatsBucket[]): number
export interface IMonitorService {
// create
createMonitor(teamId: string, userId: string, body: Partial<Monitor>): Promise<void>;
createMonitor(teamId: string, userId: string, body: Partial<Monitor>): Promise<Monitor>;
createMonitors(monitors: Array<Monitor>): Promise<Monitor[] | null>;
addDemoMonitors(args: { userId: string; teamId: string }): Promise<Monitor[]>;
@@ -144,6 +146,7 @@ export class MonitorService implements IMonitorService {
private monitorStatsRepository: IMonitorStatsRepository;
private statusPagesRepository: IStatusPagesRepository;
private incidentsRepository: IIncidentsRepository;
private encryptionService: IEncryptionService;
constructor({
scheduler,
@@ -156,6 +159,7 @@ export class MonitorService implements IMonitorService {
monitorStatsRepository,
statusPagesRepository,
incidentsRepository,
encryptionService,
}: {
scheduler: IJobScheduler;
logger: ILogger;
@@ -167,6 +171,7 @@ export class MonitorService implements IMonitorService {
monitorStatsRepository: IMonitorStatsRepository;
statusPagesRepository: IStatusPagesRepository;
incidentsRepository: IIncidentsRepository;
encryptionService: IEncryptionService;
}) {
this.scheduler = scheduler;
this.logger = logger;
@@ -178,19 +183,25 @@ export class MonitorService implements IMonitorService {
this.monitorStatsRepository = monitorStatsRepository;
this.statusPagesRepository = statusPagesRepository;
this.incidentsRepository = incidentsRepository;
this.encryptionService = encryptionService;
}
createMonitor = async (teamId: string, userId: string, body: Monitor): Promise<void> => {
createMonitor = async (teamId: string, userId: string, body: Monitor): Promise<Monitor> => {
// proxyId is only needed in custom mode
if (body.proxyMode !== "custom") {
delete body.proxyId;
}
// Encrypt docker TLS keys
this.encryptDockerTls(body, "create");
const monitor = await this.monitorsRepository.create(body, teamId, userId);
if (!monitor) {
throw new AppError({ message: "Failed to create monitor", status: 500, service: SERVICE_NAME, method: "createMonitor" });
}
this.scheduler.addJob(monitor.id, monitor);
return monitor;
};
createMonitors = async (monitors: Array<Monitor>): Promise<Monitor[] | null> => {
@@ -536,6 +547,20 @@ export class MonitorService implements IMonitorService {
if (unsetProxyId) {
delete body.proxyId;
}
// Handle Docker TLS
this.encryptDockerTls(body, "edit");
if (body.type === "docker" && isDockerTlsUrl(body.url) && body.dockerTlsKey === undefined) {
const stored = await this.monitorsRepository.findById(monitorId, teamId);
if (!stored.dockerTlsKeySet) {
throw new AppError({
message: "Key is required for a TLS Docker host",
status: 422,
service: SERVICE_NAME,
method: "editMonitor",
});
}
}
const editedMonitor = await this.monitorsRepository.updateById(monitorId, teamId, body, { unsetProxyId });
await this.scheduler.updateJob(editedMonitor);
return editedMonitor;
@@ -723,4 +748,34 @@ export class MonitorService implements IMonitorService {
return { imported: createdMonitors!.length, errors };
};
private encryptDockerTls = (body: Partial<Monitor>, mode: "create" | "edit") => {
if (body.type !== "docker" && mode === "create") return;
if (body.url !== undefined && !isDockerTlsUrl(body.url)) {
body.dockerTlsCa = undefined;
body.dockerTlsCert = undefined;
body.dockerTlsKey = undefined;
body.dockerTlsKeySet = false;
return;
}
if (!body.dockerTlsKey) {
delete body.dockerTlsKey; // empty field means keeps stored key
return;
}
if (!this.encryptionService.isConfigured()) {
throw new AppError({
message:
'Docker TLS credentials require ENCRYPTION_KEY to be set on the server. Generate one with "openssl rand -base64 32", set the same value for both the API and worker, and restart.',
status: 422,
service: SERVICE_NAME,
method: "encryptDockerTls",
});
}
body.dockerTlsKey = this.encryptionService.encrypt(body.dockerTlsKey);
body.dockerTlsKeySet = true;
};
}
@@ -109,6 +109,11 @@ export interface Monitor {
geoCheckLocations?: GeoContinent[];
geoCheckInterval?: number;
dockerLogsEnabled?: boolean;
dockerTlsCa?: string;
dockerTlsCert?: string;
// EncryptionService ciphertext. Normally excluded
dockerTlsKey?: string;
dockerTlsKeySet?: boolean;
dnsServer?: string;
dnsRecordType?: DnsRecordType;
recentChecks: CheckSnapshot[];
@@ -37,6 +37,7 @@ const NotificationSchema = new Schema<NotificationDocument>(
"teams",
"telegram",
"pushover",
"signalgrid",
"twilio",
"ntfy",
] as NotificationChannel[],
@@ -179,6 +179,9 @@ export class NotificationsService implements INotificationsService {
deleteById = async (id: string, teamId: string): Promise<Notification> => {
await this.monitorsRepository.removeNotificationFromMonitors(id);
const deleted = await this.notificationsRepository.deleteById(id, teamId);
// App settings hold notification ids for the egress recovered alert; drop the deleted one so it never dangles.
// After the team-scoped delete, so a request for another team's id (404 above) cannot touch instance settings.
await this.settingsService.removeEgressNotification(id);
return deleted;
};
}
@@ -9,6 +9,7 @@ export const NotificationChannels = [
"teams",
"telegram",
"pushover",
"signalgrid",
"twilio",
"ntfy",
] as const;
@@ -0,0 +1,123 @@
const SERVICE_NAME = "SignalgridProvider";
import type { Notification, NotificationMessage, NotificationSeverity } from "@/domain/notifications/notification.type.js";
import { NotificationProvider } from "@/domain/notifications/providers/INotificationProvider.js";
import { getTestMessage } from "@/domain/notifications/providers/utils.js";
import got from "got";
export class SignalgridProvider extends NotificationProvider {
async sendTestAlert(notification: Partial<Notification>): Promise<boolean> {
if (!notification.address || !notification.accessToken) {
return false;
}
try {
await got.post("https://api.signalgrid.co/v1/push", {
form: {
client_key: notification.accessToken,
channel: notification.address,
title: "Checkmate Test Notification",
body: getTestMessage(),
type: "INFO",
critical: false,
},
...this.gotRequestOptions(),
});
return true;
} catch (error) {
const errMsg = error instanceof Error ? error.message : "unknown error";
const errStack = error instanceof Error ? error.stack : undefined;
this.logger.warn({
message: "Signalgrid test alert failed",
service: SERVICE_NAME,
method: "sendTestAlert",
stack: errStack,
details: { error: errMsg },
});
return false;
}
}
async sendMessage(notification: Notification, message: NotificationMessage): Promise<boolean> {
if (!notification.address || !notification.accessToken) {
return false;
}
try {
await got.post("https://api.signalgrid.co/v1/push", {
form: {
client_key: notification.accessToken,
channel: notification.address,
title: message.content.title,
body: this.buildSignalgridText(message),
type: this.mapSeverity(message.severity),
critical: false,
},
...this.gotRequestOptions(),
});
this.logger.info({
message: "Signalgrid notification sent",
service: SERVICE_NAME,
method: "sendMessage",
});
return true;
} catch (error) {
const errMsg = error instanceof Error ? error.message : "unknown error";
const errStack = error instanceof Error ? error.stack : undefined;
this.logger.warn({
message: "Signalgrid alert failed",
service: SERVICE_NAME,
method: "sendMessage",
stack: errStack,
details: { error: errMsg },
});
return false;
}
}
private mapSeverity(severity: NotificationSeverity): "CRIT" | "WARN" | "INFO" | "SUCCESS" {
switch (severity) {
case "critical":
return "CRIT";
case "warning":
return "WARN";
case "success":
return "SUCCESS";
default:
return "INFO";
}
}
private buildSignalgridText(message: NotificationMessage): string {
const lines: string[] = [];
lines.push(message.content.summary);
lines.push("");
lines.push("Monitor Details:");
lines.push(`• Name: ${message.monitor.name}`);
lines.push(`• URL: ${message.monitor.url}`);
lines.push(`• Type: ${message.monitor.type}`);
lines.push(`• Status: ${message.monitor.status}`);
if (message.content.details && message.content.details.length > 0) {
lines.push("");
lines.push("Additional Information:");
message.content.details.forEach((detail) => lines.push(`• ${detail}`));
}
if (message.content.thresholds && message.content.thresholds.length > 0) {
lines.push("");
lines.push("Threshold Breaches:");
message.content.thresholds.forEach((breach) => {
lines.push(`• ${breach.metric.toUpperCase()}: ${breach.formattedValue} (threshold: ${breach.threshold}${breach.unit})`);
});
}
if (message.content.incident) {
lines.push("");
lines.push(`View Incident: ${message.clientHost}/incidents/${message.monitor.id}`);
}
return lines.join("\n");
}
}
+1 -1
View File
@@ -45,7 +45,7 @@ const startApp = async () => {
// ***********************
// Build shared services
// ***********************
const shared = await buildShared({ logger, envSettings, settingsService });
const shared = await buildShared({ logger, envSettings, settingsService, encryptionKeys: env.ENCRYPTION_KEY });
// ***********************
// Worker node path, don't need API
@@ -0,0 +1,176 @@
import { ILogger } from "@/utils/logger.js";
import type * as nodeCrypto from "node:crypto";
import { AppError } from "@/utils/AppError.js";
const SERVICE_NAME = "EncryptionService";
const ENCRYPTION_KEY_BYTES = 32;
// Encryption consts
const FORMAT_VERSION = "v1";
const ALGORITHM = "aes-256-gcm";
const IV_BYTES = 12;
const TAG_BYTES = 16;
const KEY_ID_BYTES = 4;
const SEGMENT_COUNT = 5;
type V1Segments = [string, string, string, string, string];
const isV1Segments = (segments: string[]): segments is V1Segments => segments.length === SEGMENT_COUNT;
const BASE64URL = /^[A-Za-z0-9_-]*$/;
type KeyEntry = { id: string; key: Buffer };
export type EncryptionCryptoLib = Pick<typeof nodeCrypto, "createCipheriv" | "createDecipheriv" | "createHash" | "randomBytes">;
export interface IEncryptionService {
isConfigured(): boolean;
currentKeyId(): string;
keyIdOf(ciphertext: string): string;
needsReencryption(ciphertext: string): boolean;
encrypt(plaintext: string): string;
decrypt(ciphertext: string): string;
}
export class EncryptionService implements IEncryptionService {
static SERVICE_NAME = SERVICE_NAME;
private readonly entries: KeyEntry[];
constructor(
keys: Buffer[],
private logger: ILogger,
private cryptoLib: EncryptionCryptoLib
) {
this.entries = keys.map((key, index) => this.toEntry(key, index));
this.entries.forEach((entry, index) => {
if (this.entries.findIndex((other) => other.id === entry.id) !== index) {
throw this.fail("constructor", "ENCRYPTION_KEY key ids collide; regenerate one key", { index });
}
});
const current = this.entries[0];
this.logger.info({
message: current
? `Encryption configured with ${this.entries.length} key(s); current key id ${current.id}`
: "ENCRYPTION_KEY not set; encrypted fields are unavailable",
service: SERVICE_NAME,
method: "constructor",
});
}
isConfigured = () => {
return this.entries.length > 0;
};
currentKeyId = () => {
return this.requireCurrentKey("currentKeyId").id;
};
keyIdOf = (ciphertext: string) => {
return this.parse(ciphertext, "keyIdOf").keyId;
};
// Check if the ciphertext was encrypted with the current key
needsReencryption = (ciphertext: string) => {
const currentKey = this.requireCurrentKey("needsReencryption");
return this.parse(ciphertext, "needsReencryption").keyId !== currentKey.id;
};
encrypt = (plaintext: string) => {
const currentKey = this.requireCurrentKey("encrypt"); // Get the current key for encryption
const iv = this.cryptoLib.randomBytes(IV_BYTES); // Get IV_BYTES worth of random bytes for nonce
const cipher = this.cryptoLib.createCipheriv(ALGORITHM, currentKey.key, iv, { authTagLength: TAG_BYTES }); // Create the cipher
cipher.setAAD(this.aad(currentKey.id)); // set signing key as additional data so it is protected by the verification tag
const data = Buffer.concat([cipher.update(plaintext, "utf8"), cipher.final()]); // Encode plaintext as utf8 and run through cipher
const tag = cipher.getAuthTag(); // Get verification tag
return [FORMAT_VERSION, currentKey.id, iv.toString("base64url"), tag.toString("base64url"), data.toString("base64url")].join(".");
};
decrypt = (ciphertext: string) => {
this.requireCurrentKey("decrypt"); // Config check
const { keyId, iv, tag, data } = this.parse(ciphertext, "decrypt");
const key = this.entries.find((entry) => entry.id === keyId)?.key;
if (!key) {
throw this.fail("decrypt", "No encryption key matches ciphertext", { keyId });
}
const decipher = this.cryptoLib.createDecipheriv(ALGORITHM, key, iv, { authTagLength: TAG_BYTES });
decipher.setAAD(this.aad(keyId)); // reconstruct the additional data, must be done BEFORE auth tag
decipher.setAuthTag(tag); // Give the decipher the tag to verify
try {
return Buffer.concat([decipher.update(data), decipher.final()]).toString("utf8");
} catch {
throw this.fail("decrypt", "Ciphertext failed authentication");
}
};
private fail = (method: string, message: string, details?: Record<string, unknown>): AppError => {
return new AppError({
message,
status: 500,
service: SERVICE_NAME,
method,
details,
});
};
// Create an ID:Key pair for each key.
private toEntry = (key: Buffer, keyIdx: number): KeyEntry => {
if (key.length !== ENCRYPTION_KEY_BYTES) throw this.fail("constructor", "ENCRYPTION_KEY is not 32 bytes", { keyIdx });
const id = this.cryptoLib
.createHash("sha256") // Create a sha256 hasher
.update(key) // Feed it the key
.digest() // Digest a hash
.subarray(0, KEY_ID_BYTES) // Keep first KEY_ID_BYTES, enough to avoid collision
.toString("base64url");
return { id, key };
};
private requireCurrentKey = (method: string): KeyEntry => {
const current = this.entries[0];
if (!current) {
throw this.fail(method, "ENCRYPTION_KEY is not configured");
}
return current;
};
// v1 Ciphertext stored as a string with five segments:
// <version>.<keyId>.<iv>.<tag>.<encryptedData>
// iv and tag belong to aes-256-gcm standard
private parse = (ciphertext: string, method: string) => {
const segments = ciphertext.split(".");
const version = segments[0];
if (version !== FORMAT_VERSION) {
throw this.fail(method, "Unsupported ciphertext version", { version });
}
// **************************
// v1 parser
// **************************
if (!isV1Segments(segments)) {
throw this.fail(method, "Ciphertext is malformed");
}
const [, keyId, ivText, tagText, dataText] = segments; // Skip first segment, we already have version
if (![keyId, ivText, tagText, dataText].every((segment) => BASE64URL.test(segment))) {
throw this.fail(method, "Ciphertext is malformed");
}
const iv = Buffer.from(ivText, "base64url");
const tag = Buffer.from(tagText, "base64url");
if (keyId.length === 0 || iv.length !== IV_BYTES || tag.length !== TAG_BYTES) {
throw this.fail(method, "Ciphertext is malformed");
}
return { keyId, iv, tag, data: Buffer.from(dataText, "base64url") };
// **************************
// End v1 parser
// **************************
};
private aad = (keyId: string) => {
return Buffer.from(`${FORMAT_VERSION}.${keyId}`, "utf8");
};
}
@@ -0,0 +1,60 @@
import { IMonitorsRepository } from "@/domain/monitors/monitor.repository.interface.js";
import { IEncryptionService } from "@/service/encryption/encryptionService.js";
import { AppError } from "@/utils/AppError.js";
import { ILogger } from "@/utils/logger.js";
const SERVICE_NAME = "SecretsRotationService";
export interface ISecretsRotationService {
run(): Promise<{ scanned: number; reencrypted: number; skipped: number }>;
}
export class SecretsRotationService implements ISecretsRotationService {
static SERVICE_NAME = SERVICE_NAME;
constructor(
private monitorsRepository: IMonitorsRepository,
private encryptionService: IEncryptionService,
private logger: ILogger
) {}
run = async () => {
const rows = await this.monitorsRepository.findAllDockerTlsKeys();
const result = { scanned: rows.length, reencrypted: 0, skipped: 0 };
if (rows.length === 0) return result;
if (!this.encryptionService.isConfigured()) {
this.logger.warn({
message: `${rows.length} monitor(s) have a stored Docker TLS key but ENCRYPTION_KEY is not set; their checks will fail until it is`,
service: SERVICE_NAME,
method: "run",
});
result.skipped = rows.length;
return result;
}
for (const monitor of rows) {
const { id, dockerTlsKey } = monitor;
try {
if (!this.encryptionService.needsReencryption(dockerTlsKey)) continue;
const plainText = this.encryptionService.decrypt(dockerTlsKey);
await this.monitorsRepository.updateDockerTlsKey(id, this.encryptionService.encrypt(plainText));
result.reencrypted += 1;
} catch (error: unknown) {
result.skipped += 1;
this.logger.warn({
message: `Could not reencrypt Docker TLS key for monitor ${id}: ${error instanceof Error ? error.message : String(error)}`,
service: SERVICE_NAME,
method: "run",
details: { monitorId: id, ...(error instanceof AppError ? error.details : {}) },
});
}
}
this.logger.info({
message: `Docker TLS key rotation: ${result.reencrypted} re-encrypted, ${result.skipped} skipped, ${result.scanned} scanned`,
service: SERVICE_NAME,
method: "run",
});
return result;
};
}
+81 -21
View File
@@ -1,5 +1,7 @@
import https from "node:https";
import { IStatusProvider } from "@/service/network/IStatusProvider.js";
import { DockerStatusPayload, MonitorStatusResponse } from "@/types/network.js";
import { CheckContext, DockerStatusPayload, MonitorStatusResponse } from "@/types/network.js";
import {
DOCKER_LOG_TAIL_LINES,
@@ -23,13 +25,21 @@ import { AppError } from "@/utils/AppError.js";
import Dockerode from "dockerode";
import { timeRequest } from "@/service/network/utils.js";
import { NETWORK_ERROR } from "@/types/network.js";
import { IEncryptionService } from "@/service/encryption/encryptionService.js";
import { DOCKER_TLS_URL, isDockerTlsUrl } from "@/utils/dockerHost.js";
import { splitCertificateBundle } from "@/utils/pem.js";
type DockerodeType = typeof Dockerode;
type DockerOptions = Dockerode.DockerOptions & { agent?: https.Agent; connectionTimeout?: number };
type TlsCredentials = { ok: true; key: string } | { ok: false; message: string };
const SERVICE_NAME = "DockerProvider";
const TIMEOUT_MS = 10000;
const STATS_CONCURRENCY = 5;
const DOCKER_LOG_MAX_LINE_BYTES = 4096;
const DOCKER_LOG_TRUNCATION_MARKER = " …[truncated]";
const DOCKER_LOG_TS_REGEX = /^(\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2})(?:\.(\d{1,9}))?Z /;
const DOCKER_TLS_DEFAULT_PORT = 2376;
export interface DockerError extends Error {
statusCode?: number;
@@ -42,28 +52,73 @@ export class DockerProvider implements IStatusProvider<DockerStatusPayload> {
constructor(
private logger: ILogger,
private DockerLib: DockerodeType
private DockerLib: DockerodeType,
private encryptionService: IEncryptionService
) {}
supports(type: MonitorType): boolean {
return type === "docker";
}
private resolveTlsCredentials = (monitor: Monitor, ctx?: CheckContext): TlsCredentials | undefined => {
if (!isDockerTlsUrl(monitor.url)) return undefined;
if (!ctx?.dockerTlsKey) return { ok: false, message: "Docker TLS key is missing" };
try {
return { ok: true, key: this.encryptionService.decrypt(ctx.dockerTlsKey) };
} catch (error) {
if (error instanceof AppError) {
const keyId = typeof error.details?.keyId === "string" ? ` (key id ${error.details.keyId})` : "";
return { ok: false, message: `${error.message}${keyId}` };
}
return { ok: false, message: "Docker TLS key could not be decrypted" };
}
};
private isDockerError(error: unknown): error is DockerError {
return error instanceof Error && ("statusCode" in error || "reason" in error || "json" in error);
}
private toDockerOptions = (monitor: Monitor): Dockerode.DockerOptions => {
private splitCaBundle = (pem: string | undefined): string[] | undefined => {
const blocks = pem ? splitCertificateBundle(pem) : [];
return blocks.length > 0 ? blocks : undefined;
};
private invalidUrl = (message: string) => new AppError({ message, status: 422, service: SERVICE_NAME, method: "toDockerOptions" });
private toDockerOptions = (monitor: Monitor, credentials?: TlsCredentials): DockerOptions => {
const url = monitor.url?.trim();
if (!url) throw new AppError({ message: "Docker host URL is required", status: 422, service: SERVICE_NAME, method: "toDockerOptions" });
if (!url) throw this.invalidUrl("Docker host URL is required");
// Unix socket
if (url.startsWith("unix://")) {
const socketPath = url.slice("unix://".length);
if (!socketPath.startsWith("/"))
throw new AppError({ message: `Invalid Docker host URL: ${url}`, status: 422, service: SERVICE_NAME, method: "toDockerOptions" });
return { socketPath };
if (!socketPath.startsWith("/")) throw this.invalidUrl(`Invalid Docker host URL: ${url}`);
return { socketPath, timeout: TIMEOUT_MS, connectionTimeout: TIMEOUT_MS };
}
if (url.startsWith("/")) return { socketPath: url };
throw new AppError({ message: `Invalid Docker host URL: ${url}`, status: 422, service: SERVICE_NAME, method: "toDockerOptions" });
if (url.startsWith("/")) return { socketPath: url, timeout: TIMEOUT_MS, connectionTimeout: TIMEOUT_MS };
// Docker TLS
const match = DOCKER_TLS_URL.exec(url);
if (!match) throw this.invalidUrl(`Invalid Docker host URL: ${url}`);
if (!credentials?.ok) throw this.invalidUrl(credentials?.message ?? "Docker TLS key is missing");
const [, , host, port] = match;
const ca = monitor.ignoreTlsErrors ? undefined : this.splitCaBundle(monitor.dockerTlsCa);
const cert = monitor.dockerTlsCert;
const key = credentials.key;
return {
host,
port: port ? Number(port) : DOCKER_TLS_DEFAULT_PORT,
protocol: "https",
ca,
cert,
key,
agent: new https.Agent({ ca, cert, key, rejectUnauthorized: !monitor.ignoreTlsErrors }),
socketPath: undefined,
username: undefined,
sshOptions: undefined,
timeout: TIMEOUT_MS,
connectionTimeout: TIMEOUT_MS,
};
};
private async mapWithConcurrency<T, R>(items: T[], limit: number, fn: (item: T) => Promise<R>): Promise<R[]> {
@@ -260,9 +315,23 @@ export class DockerProvider implements IStatusProvider<DockerStatusPayload> {
return rawLogs.filter((log) => log !== null);
};
handle = async (monitor: Monitor): Promise<MonitorStatusResponse<DockerStatusPayload>> => {
private failed = (monitor: Monitor, code: number, message: string, responseTime: number): MonitorStatusResponse<DockerStatusPayload> => ({
monitorId: monitor.id,
teamId: monitor.teamId,
type: monitor.type,
status: false,
code,
message,
responseTime,
payload: null,
});
handle = async (monitor: Monitor, ctx?: CheckContext): Promise<MonitorStatusResponse<DockerStatusPayload>> => {
try {
const docker = new this.DockerLib(this.toDockerOptions(monitor));
const credentials = this.resolveTlsCredentials(monitor, ctx);
if (credentials && !credentials.ok) return this.failed(monitor, NETWORK_ERROR, credentials.message, 0);
const docker = new this.DockerLib(this.toDockerOptions(monitor, credentials));
// Host reachability is the monitor's status; ping latency is the responseTime
const { responseTime, error } = await timeRequest(() => docker.ping());
if (error) {
@@ -274,16 +343,7 @@ export class DockerProvider implements IStatusProvider<DockerStatusPayload> {
} else if (error instanceof Error) {
message = error.message;
}
return {
monitorId: monitor.id,
teamId: monitor.teamId,
type: monitor.type,
status: false,
code,
message,
responseTime,
payload: null,
};
return this.failed(monitor, code, message, responseTime);
}
const summaries = await docker.listContainers({ all: true });
@@ -22,7 +22,7 @@ export class PageSpeedProvider implements IStatusProvider<PageSpeedStatusPayload
const { url, strategy } = monitor;
try {
if (!url) throw new Error("URL is required for PageSpeed monitor");
const dbSettings = await this.settingsService.getDBSettings();
const dbSettings = await this.settingsService.getCachedDBSettings();
const apiKey = dbSettings?.pagespeedApiKey;
const resolvedStrategy = strategy ?? DefaultPageSpeedStrategy;
let pageSpeedUrl = `https://pagespeedonline.googleapis.com/pagespeedonline/v5/runPagespeed?url=${encodeURIComponent(
+13 -5
View File
@@ -4,25 +4,33 @@ import { MonitorType, Monitor } from "@/domain/monitors/monitor.type.js";
import { MonitorStatusResponse } from "@/types/network.js";
import { AppError } from "@/utils/AppError.js";
import ping from "ping";
import * as net from "net";
import { timeRequest } from "@/service/network/utils.js";
const SERVICE_NAME = "PingProvider";
type Ping = typeof ping;
type NetType = typeof net;
export class PingProvider implements IStatusProvider<PingStatusPayload> {
readonly type = "ping";
constructor(private ping: Ping) {}
constructor(
private ping: Ping,
private net: NetType
) {}
supports(type: MonitorType): boolean {
return type === "ping";
}
private sanitizeHost(url: string): string {
return url
.replace(/^https?:\/\//, "")
.replace(/\/.*$/, "")
.replace(/:.*/, "");
const host = url.replace(/^https?:\/\//, "").replace(/\/.*$/, "");
// "[2001:db8::1]" or "[2001:db8::1]:53": the address is inside the brackets
const bracketed = /^\[([^\]]+)\](?::\d+)?$/.exec(host);
if (bracketed) return bracketed[1]!;
// A bare IPv6 address contains colons that are not a port separator
if (this.net.isIPv6(host)) return host;
return host.replace(/:.*/, "");
}
async handle(monitor: Monitor): Promise<MonitorStatusResponse<PingStatusPayload>> {
+1 -1
View File
@@ -73,7 +73,7 @@ export class ProxyResolver implements IProxyResolver {
};
private resolveInherit = async (monitor: Monitor): Promise<string | undefined> => {
const settings = await this.getCached("settings", () => this.settingsService.getDBSettings());
const settings = await this.settingsService.getCachedDBSettings();
if (!settings.globalProxyEnabled) {
return undefined;
}
+1
View File
@@ -162,4 +162,5 @@ export type MonitorStatusResponseOverrides<T> = Partial<Omit<MonitorStatusRespon
export type CheckContext = {
proxyUrl?: string;
dockerTlsKey?: string;
};
+5
View File
@@ -0,0 +1,5 @@
export const DOCKER_TLS_URL = /^(tcp|https):\/\/([^/\s:]+)(?::(\d+))?\/?$/;
export const DOCKER_SOCKET_URL = /^(?:unix:\/\/\/\S+|\/\S+)$/;
export const isDockerTlsUrl = (url: string | undefined): boolean => DOCKER_TLS_URL.test(url?.trim() ?? "");
export const isDockerSocketUrl = (url: string | undefined): boolean => DOCKER_SOCKET_URL.test(url?.trim() ?? "");
+48
View File
@@ -0,0 +1,48 @@
import { z } from "zod";
import { dnsHostnameRegex } from "@/api/validation/shared.js";
// A reliability target for the egress self-check, in one of three shapes. Parsed here so the
// settings validator and the probe agree on exactly what is accepted.
export type EgressTarget =
| { kind: "http"; url: string } // http(s):// URL, fetched
| { kind: "port"; host: string; port: number } // host:port or [ipv6]:port, TCP connect
| { kind: "ping"; host: string }; // bare hostname or IP address, ICMP
// "[ipv6]:port" or "host:port" where host has no colons, so a bare IPv6 address is never read as host:port.
const HOST_PORT_PATTERN = /^(?:\[([^\]]+)\]|([^:/\s[\]]+)):(\d{1,5})$/;
const BRACKETED_PATTERN = /^\[([^\]]+)\]$/;
const isIpv6 = (value: string): boolean => z.ipv6().safeParse(value).success;
const isHostOrIpv4 = (value: string): boolean => value === "localhost" || z.ipv4().safeParse(value).success || dnsHostnameRegex.test(value);
const parseHttp = (target: string): EgressTarget | null => {
try {
const { protocol, hostname } = new URL(target);
return (protocol === "http:" || protocol === "https:") && hostname.length > 0 ? { kind: "http", url: target } : null;
} catch {
return null;
}
};
export const parseEgressTarget = (raw: string): EgressTarget | null => {
const target = raw.trim();
if (target.length === 0) return null;
if (/^https?:\/\//i.test(target)) return parseHttp(target);
const hostPort = HOST_PORT_PATTERN.exec(target);
if (hostPort) {
const port = Number(hostPort[3]);
if (port < 1 || port > 65535) return null;
const bracketedHost = hostPort[1];
if (bracketedHost !== undefined) return isIpv6(bracketedHost) ? { kind: "port", host: bracketedHost, port } : null;
const host = hostPort[2]!;
return isHostOrIpv4(host) ? { kind: "port", host, port } : null;
}
const bracketed = BRACKETED_PATTERN.exec(target);
if (bracketed) return isIpv6(bracketed[1]!) ? { kind: "ping", host: bracketed[1]! } : null;
if (isIpv6(target) || isHostOrIpv4(target)) return { kind: "ping", host: target };
return null;
};
+28
View File
@@ -0,0 +1,28 @@
import { createPrivateKey, KeyObject, X509Certificate } from "node:crypto";
const PEM_CERT_BLOCK = /-----BEGIN CERTIFICATE-----[\s\S]*?-----END CERTIFICATE-----/g;
export const splitCertificateBundle = (pem: string): string[] => pem.match(PEM_CERT_BLOCK) ?? [];
export const parseCertificates = (pem: string): X509Certificate[] => {
const blocks = splitCertificateBundle(pem);
if (blocks.length === 0) throw new Error("No certificate found; expected one or more PEM CERTIFICATE blocks");
try {
return blocks.map((block) => new X509Certificate(block));
} catch {
throw new Error("Certificate is not valid PEM");
}
};
export const parsePrivateKey = (pem: string): KeyObject => {
if (/^-----BEGIN ENCRYPTED PRIVATE KEY-----/m.test(pem) || /^Proc-Type: 4,ENCRYPTED/m.test(pem)) {
throw new Error("Encrypted private keys are not supported; remove the passphrase first");
}
try {
return createPrivateKey(pem);
} catch {
throw new Error("Private key is not valid PEM");
}
};
export const keyMatchesCertificate = (key: KeyObject, certificate: X509Certificate): boolean => certificate.checkPrivateKey(key);
+9 -1
View File
@@ -36,6 +36,12 @@ export class CheckProducer implements ICheckProducer {
return windows.some((w) => isWindowActive(w, now));
}
private async resolveDockerTlsKey(monitor: Monitor): Promise<string | undefined> {
if (monitor.type !== "docker" || !monitor.dockerTlsKeySet || !monitor.id) return undefined;
const dockerTlsKey = await this.monitorsRepository.findDockerTlsKeyById(monitor.id);
return dockerTlsKey ?? undefined;
}
produce = async (monitor: Monitor) => {
if (!monitor.id) {
throw new AppError({ message: "No monitor id", service: SERVICE_NAME, method: "produce" });
@@ -61,7 +67,9 @@ export class CheckProducer implements ICheckProducer {
// Step 1b: Acquire status
const proxyUrl = await this.proxyResolver.resolve(monitor);
const status = await this.networkService.requestStatus(monitor, { proxyUrl });
const dockerTlsKey = await this.resolveDockerTlsKey(monitor);
const status = await this.networkService.requestStatus(monitor, { proxyUrl, dockerTlsKey });
if (!status) {
throw new Error("No network response");
}
+5
View File
@@ -28,6 +28,7 @@ const CONCURRENCY: Record<JobType, number> = {
evaluate: 20,
"cleanup-orphaned": 1,
"cleanup-retention": 1,
egress: 1,
};
export interface DBQueueWorkerDependencies {
@@ -61,6 +62,7 @@ export class DBQueueWorker extends JobScheduler implements IQueueWorker {
evaluate: 0,
"cleanup-orphaned": 0,
"cleanup-retention": 0,
egress: 0,
};
private checksRepository: IChecksRepository;
@@ -175,6 +177,9 @@ export class DBQueueWorker extends JobScheduler implements IQueueWorker {
case "cleanup-retention":
await this.helper.getCleanupRetentionJob()(); // Get job and execute
break;
case "egress":
await this.helper.getEgressRecoveryJob()(job); // Re-probe while degraded; removes its own row on recovery
break;
}
if (job.intervalMs === null) {
+15 -1
View File
@@ -11,10 +11,13 @@ import { IJobsRepository } from "@/domain/jobs/job.repository.interface.js";
import { ITeamsRepository } from "@/domain/teams/team.repository.interface.js";
import { ILogger } from "@/utils/logger.js";
import { IDockerLogsRepository } from "@/domain/docker/docker-log.repository.interface.js";
import { IEgressService } from "@/domain/egress/egress.service.js";
import type { Job } from "@/domain/jobs/job.type.js";
export interface IWorkerHelper {
getCleanupOrphanedJob(): () => Promise<void>;
getCleanupRetentionJob(): () => Promise<void>;
getEgressRecoveryJob(): (job: Job) => Promise<void>;
}
export interface MonitorActionDecision {
@@ -45,6 +48,7 @@ export class WorkerHelper implements IWorkerHelper {
private incidentsRepository: IIncidentsRepository;
private geoChecksRepository: IGeoChecksRepository;
private dockerLogsRepository: IDockerLogsRepository;
private egressService: IEgressService;
constructor(
logger: ILogger,
@@ -57,7 +61,8 @@ export class WorkerHelper implements IWorkerHelper {
checksRepository: IChecksRepository,
incidentsRepository: IIncidentsRepository,
geoChecksRepository: IGeoChecksRepository,
dockerLogsRepository: IDockerLogsRepository
dockerLogsRepository: IDockerLogsRepository,
egressService: IEgressService
) {
this.logger = logger;
this.checkService = checkService;
@@ -70,8 +75,17 @@ export class WorkerHelper implements IWorkerHelper {
this.incidentsRepository = incidentsRepository;
this.geoChecksRepository = geoChecksRepository;
this.dockerLogsRepository = dockerLogsRepository;
this.egressService = egressService;
}
// Runs while instance egress is degraded; the service removes the row once egress is back.
// Errors propagate so the queue records the failure and retries with its usual backoff.
getEgressRecoveryJob = () => {
return async (job: Job) => {
await this.egressService.checkRecovery(job);
};
};
getCleanupOrphanedJob = () => {
return async () => {
try {
@@ -41,6 +41,7 @@ export class JobScheduler implements IJobScheduler {
evaluate: POLL_MS,
"cleanup-orphaned": POLL_MS,
"cleanup-retention": POLL_MS,
egress: POLL_MS,
};
// Helpers, builds job seeds. immediate=true runs on the next tick; otherwise jitter spreads the herd.
protected toCheckJob = (monitor: MonitorScheduleFields, now: number, immediate = false): JobSeed => ({
+10
View File
@@ -0,0 +1,10 @@
Test fixtures for Docker TLS validation. Self-signed, 100-year, not trusted anywhere.
Regenerate with:
openssl req -x509 -newkey ec -pkeyopt ec_paramgen_curve:prime256v1 -nodes -days 36500 -subj "/CN=Checkmate Test CA" -keyout ca-key.pem -out ca.pem
openssl req -newkey ec -pkeyopt ec_paramgen_curve:prime256v1 -nodes -subj "/CN=checkmate-test-client" -keyout client-key.pem -out client.csr
openssl x509 -req -in client.csr -CA ca.pem -CAkey ca-key.pem -CAcreateserial -days 36500 -out client-cert.pem
openssl ecparam -name prime256v1 -genkey -noout -out other-key.pem
openssl pkcs8 -topk8 -in client-key.pem -out encrypted-key.pem -passout pass:fixture
rm client.csr ca.srl ca-key.pem
+11
View File
@@ -0,0 +1,11 @@
-----BEGIN CERTIFICATE-----
MIIBkDCCATWgAwIBAgIUL3SODzLIekF+Tc3iHhHr/7uxvdIwCgYIKoZIzj0EAwIw
HDEaMBgGA1UEAwwRQ2hlY2ttYXRlIFRlc3QgQ0EwIBcNMjYwOTEwMTkxNDUyWhgP
MjEyNjA4MTcxOTE0NTJaMBwxGjAYBgNVBAMMEUNoZWNrbWF0ZSBUZXN0IENBMFkw
EwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEXRJ0Y0jc+pKwXbuioDmbaHOqr5n3oU9x
Y16vYajhUiFeWohEqGRG8lHA9S82KPNmpjthLWUykHvrTsKosBb6IKNTMFEwHQYD
VR0OBBYEFNfGlRFC2MFZrCkoF4UxktbZUve1MB8GA1UdIwQYMBaAFNfGlRFC2MFZ
rCkoF4UxktbZUve1MA8GA1UdEwEB/wQFMAMBAf8wCgYIKoZIzj0EAwIDSQAwRgIh
APOnYaaKk5ED6NV/AkdpNXOEqlWcXwIQzh5Ycj3cjby+AiEApuAGTnk8k4dtVNmB
aVV5bRmq7f3BjhxwZorP+KiIrdI=
-----END CERTIFICATE-----
+11
View File
@@ -0,0 +1,11 @@
-----BEGIN CERTIFICATE-----
MIIBgjCCASigAwIBAgIUBx7r8tVOUFNX8c3YYyBxXTNltuUwCgYIKoZIzj0EAwIw
HDEaMBgGA1UEAwwRQ2hlY2ttYXRlIFRlc3QgQ0EwIBcNMjYwOTEwMTkxNDUyWhgP
MjEyNjA4MTcxOTE0NTJaMCAxHjAcBgNVBAMMFWNoZWNrbWF0ZS10ZXN0LWNsaWVu
dDBZMBMGByqGSM49AgEGCCqGSM49AwEHA0IABKeUUNe9kE57nK7wTfnnvmqgzVj1
ngNPajQYIAh9SAR+pVLAoTWLinWq5YarW1LFAzxyRKl6p2NdKtyLpVYV5BKjQjBA
MB0GA1UdDgQWBBReuZBD6K0kWfgGDkt05zzM24wPejAfBgNVHSMEGDAWgBTXxpUR
QtjBWawpKBeFMZLW2VL3tTAKBggqhkjOPQQDAgNIADBFAiAvfzha4dEVcJlKjH45
yOkbFXiS/+tU+uTvn+7uutn6QAIhAI2qCXVvxX+i3Jk1xTS7R0mzJNXZXqabjODi
cisS1VJG
-----END CERTIFICATE-----
+5
View File
@@ -0,0 +1,5 @@
-----BEGIN PRIVATE KEY-----
MIGHAgEAMBMGByqGSM49AgEGCCqGSM49AwEHBG0wawIBAQQgIHAuDMyYVMOi5PpA
yUPjuNjvy/QOI9dnjQZ+AKhuQO+hRANCAASnlFDXvZBOe5yu8E35575qoM1Y9Z4D
T2o0GCAIfUgEfqVSwKE1i4p1quWGq1tSxQM8ckSpeqdjXSrci6VWFeQS
-----END PRIVATE KEY-----
+8
View File
@@ -0,0 +1,8 @@
-----BEGIN ENCRYPTED PRIVATE KEY-----
MIH0MF8GCSqGSIb3DQEFDTBSMDEGCSqGSIb3DQEFDDAkBBDak7FU8VAgafGR1iEl
eFHbAgIIADAMBggqhkiG9w0CCQUAMB0GCWCGSAFlAwQBKgQQCUTF7P/8uj9a5kLU
3nBJdQSBkNpsmjC1FczojCBOn0g+2PlmMGI6MfR3vFvmPRAAF7SHLzQC1lXWhTU0
3v5yCCdjlC5gTGINKf8RhZQjKym+lIpf+16EAPFLzFbRJPvt/eKq65YiixvQP2PS
vysOG71YZJt1kSHrCOu5EPcCLvD2Kjyt48c/LJssCB81TOdqP1jV9iH29SCv4bnx
KtjXulGg3w==
-----END ENCRYPTED PRIVATE KEY-----
+5
View File
@@ -0,0 +1,5 @@
-----BEGIN EC PRIVATE KEY-----
MHcCAQEEIGnUNFbPMxfa3m7xo9J3Lu9qiss2ZxWpqp91B6X9rQbfoAoGCCqGSM49
AwEHoUQDQgAEGtbNBALMRIEROLtiEdNda4r/X0Hq1hcY/0V20gW3hEHhKRKkSilH
0LyFWtlZ5uILZh6zXCGWlOPm0PFO9rN/hQ==
-----END EC PRIVATE KEY-----
Loaded 100 of 127 files, more files were not shown because too many files have changed in this diff. Show more