Compare commits
71
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
011b93ecc1 | ||
|
|
20ce11497e | ||
|
|
50d0721822 | ||
|
|
c66872c7f4 | ||
|
|
b1be38368c | ||
|
|
278f652ea9 | ||
|
|
d3e12722b7 | ||
|
|
8c9ff41074 | ||
|
|
71a1194373 | ||
|
|
580a025353 | ||
|
|
33994cb3df | ||
|
|
44edf320a0 | ||
|
|
fea2ef4dbe | ||
|
|
360b4b8035 | ||
|
|
3857a71e62 | ||
|
|
b68367b04f | ||
|
|
08f104e283 | ||
|
|
46eb0b4063 | ||
|
|
84539f76cf | ||
|
|
308672d558 | ||
|
|
0f474d82d6 | ||
|
|
650cfe00ce | ||
|
|
69eb9957c6 | ||
|
|
ec01a5dfe4 | ||
|
|
5da472da14 | ||
|
|
4bb366f5ab | ||
|
|
cc4b8b38f5 | ||
|
|
f41e0f0619 | ||
|
|
3f10b6e567 | ||
|
|
9ad20daa51 | ||
|
|
86b59da909 | ||
|
|
6a9068a151 | ||
|
|
b31ffa5318 | ||
|
|
71dd998390 | ||
|
|
102a9f1484 | ||
|
|
581ec9382b | ||
|
|
cebe9350f7 | ||
|
|
406518e0a5 | ||
|
|
c458d5fb65 | ||
|
|
ba0d685c6d | ||
|
|
912f1f960d | ||
|
|
1de0edf7e3 | ||
|
|
8c9cb4be77 | ||
|
|
727b904ca9 | ||
|
|
5c0279a6a7 | ||
|
|
9d72d20dd2 | ||
|
|
62d0630702 | ||
|
|
2260b4a4d2 | ||
|
|
cfb50afe42 | ||
|
|
b14b2142e3 | ||
|
|
896d73f12e | ||
|
|
c64c4cc5ff | ||
|
|
7071de294f | ||
|
|
5eb16fdbdb | ||
|
|
131fd1a6b0 | ||
|
|
f9bc02ed1c | ||
|
|
1035e397e0 | ||
|
|
9c788561a7 | ||
|
|
449e96d842 | ||
|
|
c7626c669a | ||
|
|
ede5bc4ecd | ||
|
|
8bb60bfd03 | ||
|
|
f009aa17d9 | ||
|
|
ef7ac8bef8 | ||
|
|
b75f895a1f | ||
|
|
53f5b61e8e | ||
|
|
64a2c69540 | ||
|
|
c4ec14db5d | ||
|
|
4560820649 | ||
|
|
3051bc7edc | ||
|
|
c9b99eabb4 |
No files matched your search
@@ -92,7 +92,7 @@ JWT_SECRET="$(openssl rand -hex 32)" docker compose up -d
|
||||
Then open http://localhost:52345. If the app is reached at another origin (domain or LAN IP), set `CLIENT_HOST` accordingly. To build the image yourself, run `docker build -f docker/Dockerfile -t checkmate .` from a checkout. For TLS, put any reverse proxy (Caddy, Traefik, nginx) in front of port 52345.
|
||||
|
||||
There are also 1-click installation options like [Repocloud](https://repocloud.io/details/Checkmate),
|
||||
[Pikapods](https://www.pikapods.com/), [Coolify](https://coolify.io/), [Elestio](https://elest.io/open-source/checkmate), [K8s](../../charts/helm/checkmate/INSTALLATION.md), [Sive Host](https://sive.host) or [Cloudzy](https://cloudzy.com/marketplace/checkmate).
|
||||
[Pikapods](https://www.pikapods.com/), [Coolify](https://coolify.io/), [Elestio](https://elest.io/open-source/checkmate), [Easypanel](https://easypanel.io/templates/checkmate), [K8s](./charts/helm/checkmate/INSTALLATION.md), [Sive Host](https://sive.host) or [Cloudzy](https://cloudzy.com/marketplace/checkmate). Note that the Helm chart has not yet been migrated to the all-in-one image: it still deploys the legacy `checkmate-client`, `checkmate-backend` and `checkmate-mongo` images, pinned at v3.8.1.
|
||||
|
||||
|
||||
### Configuration
|
||||
@@ -104,7 +104,15 @@ The image is configured entirely through environment variables on the server con
|
||||
| `DB_CONNECTION_STRING` | Yes | MongoDB connection string, e.g. `mongodb://mongodb:27017/uptime_db` |
|
||||
| `JWT_SECRET` | Yes | Secret used to sign auth tokens; generate one with `openssl rand -hex 32` |
|
||||
| `CLIENT_HOST` | Yes | The URL users reach the app at, e.g. `https://checkmate.example.com`; used for CORS and for links in notifications and emails |
|
||||
| `ENCRYPTION_KEY` | No | Encrypts stored Docker TLS client keys at rest; generate one with `openssl rand -base64 32`. Comma-separated list: the first key encrypts, every key decrypts. Must be identical on the API and every worker. To rotate without downtime, deploy `OLD_KEY,NEW_KEY` everywhere, then `NEW_KEY,OLD_KEY` everywhere, wait for the worker to re-encrypt every row, then drop `OLD_KEY`. |
|
||||
| `PORT` | No | Port the API and web client are served on (default `52345`) |
|
||||
| `HEALTH_PORT` | No | Port for the `/livez`, `/readyz` and `/metrics` endpoints, served by any process that runs the job worker (default `52346`) |
|
||||
| `NODE_ENV` | No | `development`, `production` or `test` (default `development`). `development` disables the general API rate limiter; set `production` on real deployments |
|
||||
| `LOG_LEVEL` | No | Server log level: `error`, `warn`, `info`, or `debug` (default `debug`) |
|
||||
| `TOKEN_TTL` | No | Lifetime of issued auth tokens, e.g. `12h` or `7d` (default `99d`) |
|
||||
| `QUEUE_MODE` | No | `primary` (default) runs the API, the web client and the job scheduler; `worker` runs a job-processing worker only, with no API |
|
||||
| `QUEUE_PRIMARY_PROCESSES` | No | `true` (default) or `false`. Whether a `primary` node also processes monitoring jobs itself; set `false` when dedicated `worker` nodes handle all checks. Ignored in `worker` mode |
|
||||
| `STATUS_PAGE_THEMES_ENABLED` | No | `true` (default) or `false`. When `false`, status pages ignore theme settings and always render the default theme |
|
||||
|
||||
The web client needs no configuration by default: it calls the API on the same origin it was served from (`/api/v1`). For setups where the defaults don't apply — for example, the API is reached through a different origin than the page — the server renders overrides into the client at runtime via these optional variables:
|
||||
|
||||
@@ -118,12 +126,119 @@ The web client needs no configuration by default: it calls the API on the same o
|
||||
|
||||
See full installation instructions in the [Checkmate documentation portal](https://checkmate.so/docs).
|
||||
|
||||
Alternatively, you can also use [Coolify](https://coolify.io/), [Elestio](https://elest.io/open-source/checkmate), [K8s](./charts/helm/checkmate/INSTALLATION.md), [Sive Host](https://sive.host) (South Africa), [Cloudzy](https://cloudzy.com/marketplace/checkmate) or [Pikapods](https://www.pikapods.com/) to quickly spin off a Checkmate instance. If you would like to monitor your server infrastructure, you'll need [Capture agent](https://github.com/bluewave-labs/capture). Capture repository also contains the installation instructions.
|
||||
Alternatively, you can also use [Coolify](https://coolify.io/), [Elestio](https://elest.io/open-source/checkmate), [K8s](./charts/helm/checkmate/INSTALLATION.md) (legacy images, pinned at v3.8.1), [Sive Host](https://sive.host) (South Africa), [Cloudzy](https://cloudzy.com/marketplace/checkmate) or [Pikapods](https://www.pikapods.com/) to quickly spin off a Checkmate instance. If you would like to monitor your server infrastructure, you'll need [Capture agent](https://github.com/bluewave-labs/capture). Capture repository also contains the installation instructions.
|
||||
|
||||
### Using a Custom CA
|
||||
|
||||
If you need to monitor internal HTTPS endpoints with certificates from private Certificate Authorities (like Smallstep), see our [Custom CA Trust Guide](./docs/custom-ca-trust.md) for Docker configuration options.
|
||||
|
||||
### Docker monitors
|
||||
|
||||
A Docker monitor connects to a Docker daemon and reports on every container it runs. The daemon's ping response decides whether the monitor is up or down and its latency is the response time. Each check also records every container's state, health, CPU and memory usage, restart count, published ports and mounts. Enabling **Collect container logs** additionally stores the latest 200 log lines per container on every check; logs are kept for 7 days.
|
||||
|
||||
The **Docker host** field accepts two forms:
|
||||
|
||||
| Host | Example | Notes |
|
||||
|---|---|---|
|
||||
| Local socket | `unix:///var/run/docker.sock` | Also accepts a bare absolute path such as `/var/run/docker.sock`. Use this for the daemon on the same machine Checkmate runs on. |
|
||||
| Remote daemon | `tcp://docker.example.com:2376` | Always uses mutual TLS; the port defaults to `2376`. Unencrypted daemons on `2375` are not supported. |
|
||||
|
||||
**Monitoring the local socket.** The reference Compose file does not mount the socket, so add it and grant the container the host's `docker` group. The image runs as an unprivileged user and cannot read the socket otherwise. Find the group id with `stat -c %g /var/run/docker.sock`, then:
|
||||
|
||||
```yaml
|
||||
services:
|
||||
checkmate:
|
||||
volumes:
|
||||
- /var/run/docker.sock:/var/run/docker.sock:ro
|
||||
group_add:
|
||||
- "989" # the gid printed by stat
|
||||
```
|
||||
|
||||
**Monitoring a remote daemon.** Set the host to `tcp://host:port` and fill in the **TLS credentials** section of the monitor form with the same PEM files you would pass to `docker --tlsverify`:
|
||||
|
||||
- **CA certificate**: the CA that signed the daemon's server certificate. Required unless **Ignore TLS/SSL errors** is on, which skips verifying the daemon's identity.
|
||||
- **Client certificate**: the certificate the daemon uses to authenticate Checkmate.
|
||||
- **Client key**: the matching private key, unencrypted (no passphrase). Checkmate checks that it matches the certificate, encrypts it with `ENCRYPTION_KEY`, and never shows it again. Leave the field blank when editing to keep the stored key.
|
||||
|
||||
TLS Docker monitors require `ENCRYPTION_KEY` to be set on the server (see [Configuration](#configuration)). Saving one without it fails with an error telling you so. If the key is ever removed or rotated incorrectly, affected checks fail with a decryption error until it is restored.
|
||||
|
||||
If the daemon's certificate is signed by a private CA that you also want the rest of Checkmate to trust, see the [Custom CA Trust Guide](./docs/custom-ca-trust.md); for Docker monitors alone, the **CA certificate** field is enough.
|
||||
|
||||
### Securing the Docker daemon with TLS
|
||||
|
||||
The Docker daemon does not enable TLS by default. Follow these steps on the Docker host to generate a CA, a server certificate, and a client certificate for Checkmate. This is the procedure from [Docker's own guide](https://docs.docker.com/engine/security/protect-access/), condensed. Replace `docker.example.com` and `203.0.113.10` with your daemon's DNS name and IP.
|
||||
|
||||
**1. Create a CA.** The CA key gets a passphrase; keep it offline once the certificates are issued.
|
||||
|
||||
```bash
|
||||
openssl genrsa -aes256 -out ca-key.pem 4096
|
||||
openssl req -new -x509 -days 365 -key ca-key.pem -sha256 -subj "/CN=docker-ca" -out ca.pem
|
||||
```
|
||||
|
||||
**2. Create the server certificate.** The subject alternative names must cover every name or address Checkmate will use to reach the daemon.
|
||||
|
||||
```bash
|
||||
openssl genrsa -out server-key.pem 4096
|
||||
openssl req -subj "/CN=docker.example.com" -sha256 -new -key server-key.pem -out server.csr
|
||||
cat > server-ext.cnf <<EOF
|
||||
subjectAltName = DNS:docker.example.com,IP:203.0.113.10
|
||||
extendedKeyUsage = serverAuth
|
||||
EOF
|
||||
openssl x509 -req -days 365 -sha256 -in server.csr -CA ca.pem -CAkey ca-key.pem -CAcreateserial \
|
||||
-out server-cert.pem -extfile server-ext.cnf
|
||||
```
|
||||
|
||||
**3. Create the client certificate for Checkmate.** Do not add a passphrase to this key; Checkmate cannot use encrypted private keys.
|
||||
|
||||
```bash
|
||||
openssl genrsa -out key.pem 4096
|
||||
openssl req -subj "/CN=checkmate" -new -key key.pem -out client.csr
|
||||
echo "extendedKeyUsage = clientAuth" > client-ext.cnf
|
||||
openssl x509 -req -days 365 -sha256 -in client.csr -CA ca.pem -CAkey ca-key.pem -CAcreateserial \
|
||||
-out cert.pem -extfile client-ext.cnf
|
||||
rm client.csr server.csr server-ext.cnf client-ext.cnf
|
||||
chmod 0400 ca-key.pem key.pem server-key.pem
|
||||
chmod 0444 ca.pem server-cert.pem cert.pem
|
||||
```
|
||||
|
||||
**4. Point the daemon at the certificates.** Move `ca.pem`, `server-cert.pem` and `server-key.pem` to `/etc/docker/certs/` and configure `/etc/docker/daemon.json`:
|
||||
|
||||
```json
|
||||
{
|
||||
"hosts": ["unix:///var/run/docker.sock", "tcp://0.0.0.0:2376"],
|
||||
"tls": true,
|
||||
"tlsverify": true,
|
||||
"tlscacert": "/etc/docker/certs/ca.pem",
|
||||
"tlscert": "/etc/docker/certs/server-cert.pem",
|
||||
"tlskey": "/etc/docker/certs/server-key.pem"
|
||||
}
|
||||
```
|
||||
|
||||
On distributions where the systemd unit already passes `-H fd://` (Debian, Ubuntu and derivatives), the daemon refuses to start with `hosts` set in both places. Remove the flag from the unit with an override, then restart:
|
||||
|
||||
```bash
|
||||
sudo systemctl edit docker.service
|
||||
```
|
||||
|
||||
```ini
|
||||
[Service]
|
||||
ExecStart=
|
||||
ExecStart=/usr/bin/dockerd
|
||||
```
|
||||
|
||||
```bash
|
||||
sudo systemctl daemon-reload && sudo systemctl restart docker
|
||||
```
|
||||
|
||||
Open port `2376` on the host firewall only to the machine running Checkmate.
|
||||
|
||||
**5. Verify from the Checkmate host**, then paste `ca.pem`, `cert.pem` and `key.pem` into the monitor form:
|
||||
|
||||
```bash
|
||||
docker --tlsverify --tlscacert=ca.pem --tlscert=cert.pem --tlskey=key.pem \
|
||||
-H=docker.example.com:2376 version
|
||||
```
|
||||
|
||||
For more documentation, see the [docs directory](./docs/).
|
||||
|
||||
<a id="performance"></a>
|
||||
@@ -133,7 +248,7 @@ Thanks to extensive optimizations, Checkmate operates with an exceptionally smal
|
||||
|
||||

|
||||
|
||||
You can see the memory footprint of MongoDB and Redis on the same server (398Mb and 15Mb) for the same amount of servers:
|
||||
You can see the memory footprint of MongoDB on the same server (398Mb) for the same amount of servers:
|
||||
|
||||

|
||||
|
||||
@@ -151,16 +266,16 @@ Feel free to ask questions or share your ideas - we'd love to hear from you!
|
||||
## Features
|
||||
|
||||
- Completely open source, deployable on your servers or home devices (e.g Raspberry Pi 4 or 5)
|
||||
- Several monitoring options: Uptime, Docker, Ping, SSL, Port, Game server
|
||||
- Several monitoring options: HTTP (with SSL certificate expiry), Ping, Port, DNS, Docker, gRPC, WebSocket, Game server
|
||||
- Page speed monitoring
|
||||
- Infrastructure monitoring (memory, disk usage, CPU performance, network etc) - requires [Capture](https://github.com/bluewave-labs/capture) agent
|
||||
- Selective disk monitoring with mountpoint selection
|
||||
- Incidents at a glance
|
||||
- Status pages with 4 beautiful themes
|
||||
- E-mail, Webhooks, Discord, Slack, PagerDuty, Matrix, Rocket.Chat, Microsoft Teams, Telegram, Pushover, Twilio (SMS) notifications
|
||||
- Status pages with 5 beautiful themes
|
||||
- E-mail, Webhooks, Discord, Slack, PagerDuty, Matrix, Rocket.Chat, Microsoft Teams, Telegram, Pushover, ntfy, SignalGrid, Twilio (SMS) notifications
|
||||
- Scheduled maintenance
|
||||
- JSON query monitoring
|
||||
- Multi-language support for Arabic, Chinese (Simplified), Chinese (Traditional, Taiwan), Czech, English, Finnish, French, German, Japanese, Portuguese (Brazil), Russian, Spanish, Thai, Turkish, Ukrainian, and Vietnamese
|
||||
- Multi-language support for Arabic, Catalan, Chinese (Simplified), Chinese (Traditional, Taiwan), Czech, English, Finnish, French, German, Italian, Japanese, Polish, Portuguese (Brazil), Russian, Spanish, Thai, Turkish, Ukrainian, and Vietnamese
|
||||
|
||||
|
||||
## Monitor Lifecycle
|
||||
@@ -213,7 +328,7 @@ Feel free to ask questions or share your ideas - we'd love to hear from you!
|
||||
|
||||
We are [Alex](http://github.com/ajhollid) (team lead), [Gorkem](http://github.com/gorkem-bwl/), [Aryaman](https://github.com/Br0wnHammer), [Malena](https://github.com/malenacaroline) and [Mert](https://github.com/mertssmnoglu) helping individuals and businesses monitor their infra and servers.
|
||||
|
||||
We pride ourselves on building strong connections with contributors at every level. Despite being a young project, Checkmate has already earned almost 11K+ stars and attracted 150+ contributors from around the globe.
|
||||
We pride ourselves on building strong connections with contributors at every level. Despite being a young project, Checkmate has already earned almost 11K stars and attracted 150+ contributors from around the globe.
|
||||
|
||||
Our repo is starred by employees from **Google, Microsoft, Intel, Cisco, Tencent, Electronic Arts, ByteDance, JP Morgan Chase, Deloitte, Accenture, Foxconn, Broadcom, China Telecom, Barclays, Capgemini, Wipro, Cloudflare, Dassault Systèmes and NEC**, so don’t hold back — jump in, contribute and learn with us!
|
||||
|
||||
|
||||
Generated
+2
-2
@@ -1,12 +1,12 @@
|
||||
{
|
||||
"name": "client",
|
||||
"version": "3.11.0",
|
||||
"version": "3.12.0",
|
||||
"lockfileVersion": 3,
|
||||
"requires": true,
|
||||
"packages": {
|
||||
"": {
|
||||
"name": "client",
|
||||
"version": "3.11.0",
|
||||
"version": "3.12.0",
|
||||
"dependencies": {
|
||||
"@csstools/css-tokenizer": "^3.0.4",
|
||||
"@emotion/react": "^11.13.3",
|
||||
|
||||
+1
-1
@@ -1,7 +1,7 @@
|
||||
{
|
||||
"name": "client",
|
||||
"private": true,
|
||||
"version": "3.11.0",
|
||||
"version": "3.12.0",
|
||||
"type": "module",
|
||||
"scripts": {
|
||||
"lockfile": "docker run --rm -v \"$PWD\":/w -w /w $(grep -m1 -o 'node:[^ ]*' ../docker/Dockerfile) npm install --package-lock-only",
|
||||
|
||||
@@ -8,8 +8,11 @@ import { typographyLevels } from "@/Utils/Theme/Palette";
|
||||
import useMediaQuery from "@mui/material/useMediaQuery";
|
||||
import { LAYOUT } from "@/Utils/Theme/constants";
|
||||
import { formatDuration } from "@/Utils/TimeUtils";
|
||||
import { useTranslation } from "react-i18next";
|
||||
|
||||
export const MonitorStatus = ({ monitor }: { monitor: Monitor }) => {
|
||||
const theme = useTheme();
|
||||
const { t, i18n } = useTranslation();
|
||||
const isSmall = useMediaQuery(theme.breakpoints.down("md"));
|
||||
|
||||
if (!monitor) {
|
||||
@@ -52,7 +55,12 @@ export const MonitorStatus = ({ monitor }: { monitor: Monitor }) => {
|
||||
<>
|
||||
<Dot />
|
||||
<Typography>
|
||||
Checking every {formatDuration(monitor?.interval, true)}
|
||||
{t("pages.common.monitors.statBoxes.checkingEvery", {
|
||||
interval: formatDuration(monitor.interval, {
|
||||
long: true,
|
||||
locale: i18n.resolvedLanguage,
|
||||
}),
|
||||
})}
|
||||
</Typography>
|
||||
</>
|
||||
)}
|
||||
|
||||
@@ -5,9 +5,10 @@ import {
|
||||
type FieldValues,
|
||||
} from "react-hook-form";
|
||||
import { Autocomplete } from "@/Components/inputs";
|
||||
import type { AutocompleteProps } from "@mui/material/Autocomplete";
|
||||
|
||||
export interface AutocompleteOption {
|
||||
id: string;
|
||||
id: string | number;
|
||||
name: string;
|
||||
}
|
||||
|
||||
@@ -16,6 +17,9 @@ interface FormAutocompleteProps<T extends FieldValues, O extends AutocompleteOpt
|
||||
options: O[];
|
||||
fieldLabel?: string;
|
||||
placeholder?: string;
|
||||
filterOptions?: AutocompleteProps<O, false, false, false>["filterOptions"];
|
||||
getOptionDisabled?: AutocompleteProps<O, false, false, false>["getOptionDisabled"];
|
||||
disableClearable?: boolean;
|
||||
}
|
||||
|
||||
export const FormAutocompleteField = <
|
||||
@@ -26,6 +30,9 @@ export const FormAutocompleteField = <
|
||||
options,
|
||||
fieldLabel,
|
||||
placeholder,
|
||||
filterOptions,
|
||||
getOptionDisabled,
|
||||
disableClearable,
|
||||
}: FormAutocompleteProps<T, O>) => {
|
||||
const { control } = useFormContext<T>();
|
||||
return (
|
||||
@@ -39,10 +46,14 @@ export const FormAutocompleteField = <
|
||||
isOptionEqualToValue={(option, value) => option.id === value.id}
|
||||
value={options.find((o) => o.id === field.value) ?? null}
|
||||
onChange={(_: unknown, newValue: O | null) => {
|
||||
if (newValue === null && disableClearable) return;
|
||||
field.onChange(newValue?.id ?? "");
|
||||
}}
|
||||
fieldLabel={fieldLabel}
|
||||
placeholder={placeholder}
|
||||
filterOptions={filterOptions}
|
||||
getOptionDisabled={getOptionDisabled}
|
||||
disableClearable={disableClearable}
|
||||
error={!!fieldState.error}
|
||||
helperText={fieldState.error?.message ?? ""}
|
||||
/>
|
||||
|
||||
@@ -21,7 +21,7 @@ export const MonitorStatBoxes = ({
|
||||
domainExpiry,
|
||||
}: MonitorStatBoxesProps) => {
|
||||
const theme = useTheme();
|
||||
const { t } = useTranslation();
|
||||
const { t, i18n } = useTranslation();
|
||||
if (!monitorStats || !monitor) {
|
||||
return null;
|
||||
}
|
||||
@@ -29,13 +29,16 @@ export const MonitorStatBoxes = ({
|
||||
const timeOfLastFailure = monitorStats?.timeOfLastFailure || 0;
|
||||
const timeSinceLastFailure = timeOfLastFailure > 0 ? Date.now() - timeOfLastFailure : 0;
|
||||
|
||||
// Determine time since last check
|
||||
const timeOfLastCheck = monitorStats?.lastCheckTimestamp || 0;
|
||||
const timeSinceLastCheck = Date.now() - timeOfLastCheck || 0;
|
||||
|
||||
const streakTime = formatDuration(timeSinceLastFailure);
|
||||
|
||||
const lastCheckTime = formatDuration(timeSinceLastCheck);
|
||||
const lastCheckTime = timeOfLastCheck
|
||||
? formatDuration(Date.now() - timeOfLastCheck, {
|
||||
long: true,
|
||||
relative: true,
|
||||
locale: i18n.resolvedLanguage,
|
||||
})
|
||||
: t("common.labels.na");
|
||||
const isActive =
|
||||
monitor?.status === "up" ||
|
||||
monitor?.status === "paused" ||
|
||||
|
||||
@@ -1,15 +1,58 @@
|
||||
import { BaseChart } from "@/Components/design-elements";
|
||||
import { Area, AreaChart, CartesianGrid, ResponsiveContainer } from "recharts";
|
||||
import { BaseChart, BaseBox } from "@/Components/design-elements";
|
||||
import { Area, AreaChart, CartesianGrid, ResponsiveContainer, Tooltip } from "recharts";
|
||||
import Typography from "@mui/material/Typography";
|
||||
|
||||
// Types
|
||||
import type { DockerContainerStats } from "@/Types/Monitor";
|
||||
import type { TooltipProps } from "recharts";
|
||||
import type { NameType, ValueType } from "recharts/types/component/DefaultTooltipContent";
|
||||
import type { Theme } from "@mui/material/styles";
|
||||
import type { RootState } from "@/Types/state";
|
||||
|
||||
// Hooks
|
||||
import { useTheme } from "@mui/material";
|
||||
import { Fragment, useId } from "react";
|
||||
import { useTranslation } from "react-i18next";
|
||||
import { useSelector } from "react-redux";
|
||||
|
||||
// Utils
|
||||
import { createGradient } from "@/Components/monitors/charts/ChartUtils";
|
||||
import { tooltipDateFormatLookup, formatDateWithTz } from "@/Utils/TimeUtils";
|
||||
import type { DateRange } from "@/Types/Query";
|
||||
|
||||
type DockerHistogramToolTipProps = TooltipProps<ValueType, NameType> & {
|
||||
dataKey: string;
|
||||
range: string;
|
||||
theme: Theme;
|
||||
uiTimezone: string;
|
||||
valueFormatter: (value: number) => string;
|
||||
labelKey: string;
|
||||
};
|
||||
|
||||
const DockerHistogramToolTip = ({
|
||||
dataKey,
|
||||
active,
|
||||
payload,
|
||||
range,
|
||||
theme,
|
||||
uiTimezone,
|
||||
valueFormatter,
|
||||
labelKey,
|
||||
}: DockerHistogramToolTipProps) => {
|
||||
const { t } = useTranslation();
|
||||
if (!payload) return null;
|
||||
if (!active) return null;
|
||||
const format = tooltipDateFormatLookup(range);
|
||||
const ts = payload[0].payload?._id ?? null;
|
||||
const dataPoint = payload[0].payload?.[dataKey] ?? null;
|
||||
|
||||
return (
|
||||
<BaseBox sx={{ py: theme.spacing(2), px: theme.spacing(4) }}>
|
||||
<Typography>{formatDateWithTz(ts, format, uiTimezone)}</Typography>
|
||||
<Typography>{t(labelKey, { value: valueFormatter(dataPoint) })}</Typography>
|
||||
</BaseBox>
|
||||
);
|
||||
};
|
||||
interface HistogramDockerContainerProps {
|
||||
title: string;
|
||||
rightTitle: string;
|
||||
@@ -17,6 +60,9 @@ interface HistogramDockerContainerProps {
|
||||
dataKey: string;
|
||||
strokeColor: string;
|
||||
gradientStartColor: string;
|
||||
range: DateRange;
|
||||
formatter: (value: number) => string;
|
||||
labelKey: string;
|
||||
}
|
||||
|
||||
export const HistogramDockerContainer = ({
|
||||
@@ -26,11 +72,14 @@ export const HistogramDockerContainer = ({
|
||||
dataKey,
|
||||
strokeColor,
|
||||
gradientStartColor,
|
||||
range,
|
||||
formatter,
|
||||
labelKey,
|
||||
}: HistogramDockerContainerProps) => {
|
||||
const uniqueId = useId();
|
||||
const theme = useTheme();
|
||||
const gradientId = `gradient-${uniqueId}`;
|
||||
|
||||
const uiTimezone = useSelector((state: RootState) => state.ui.timezone);
|
||||
return (
|
||||
<BaseChart
|
||||
icon="null"
|
||||
@@ -56,6 +105,19 @@ export const HistogramDockerContainer = ({
|
||||
endColor: "transparent",
|
||||
direction: "vertical",
|
||||
})}
|
||||
<Tooltip
|
||||
content={(props) => (
|
||||
<DockerHistogramToolTip
|
||||
{...props}
|
||||
range={range}
|
||||
theme={theme}
|
||||
uiTimezone={uiTimezone}
|
||||
dataKey={dataKey}
|
||||
valueFormatter={formatter}
|
||||
labelKey={labelKey}
|
||||
/>
|
||||
)}
|
||||
/>
|
||||
<Area
|
||||
dataKey={dataKey}
|
||||
type="monotone"
|
||||
|
||||
@@ -20,12 +20,18 @@ import {
|
||||
Tag,
|
||||
Waypoints,
|
||||
Container,
|
||||
// LayoutDashboard,
|
||||
} from "lucide-react";
|
||||
export const getMenu = (t: Function) => {
|
||||
return [
|
||||
{
|
||||
group: t("components.sidebar.menu.groups.monitoring"),
|
||||
items: [
|
||||
// {
|
||||
// name: t("components.sidebar.menu.dashboard"),
|
||||
// path: "dashboard",
|
||||
// icon: <Icon icon={LayoutDashboard} />,
|
||||
// },
|
||||
{
|
||||
name: t("components.sidebar.menu.uptime"),
|
||||
path: "uptime",
|
||||
|
||||
@@ -77,6 +77,12 @@ export const getMonitorDefaults = (
|
||||
...base,
|
||||
type: "docker",
|
||||
url: data?.url || "",
|
||||
ignoreTlsErrors: data?.ignoreTlsErrors || false,
|
||||
dockerTlsCa: data?.dockerTlsCa || "",
|
||||
dockerTlsCert: data?.dockerTlsCert || "",
|
||||
// The key is write-only; the API never returns it.
|
||||
dockerTlsKey: "",
|
||||
dockerTlsKeySet: data?.dockerTlsKeySet ?? false,
|
||||
dockerLogsEnabled: data?.dockerLogsEnabled ?? false,
|
||||
};
|
||||
break;
|
||||
|
||||
@@ -38,6 +38,7 @@ import {
|
||||
PageSpeedStrategies,
|
||||
DnsRecordTypes,
|
||||
MonitorIntervalOptions,
|
||||
MIN_MONITOR_INTERVAL_MS,
|
||||
DefaultMonitorMatchMethod,
|
||||
MonitorMatchMethods,
|
||||
GeoCheckIntervalOptions,
|
||||
@@ -50,6 +51,7 @@ import type { AppSettingsResponse } from "@/Types/Settings";
|
||||
import {
|
||||
stepFieldsFor,
|
||||
monitorStepCount,
|
||||
isDockerTlsUrl,
|
||||
type MonitorFormData,
|
||||
} from "@/Validation/monitor";
|
||||
import { FormNumberField } from "@/Components/inputs/forms/FormNumberField";
|
||||
@@ -57,15 +59,62 @@ import { FormTextField } from "@/Components/inputs/forms/FormTextField";
|
||||
import { FormRadioGroup } from "@/Components/inputs/forms/FormRadioGroupField";
|
||||
import { FormMultiSelectField } from "@/Components/inputs/forms/FormMultiSelectField";
|
||||
import { FormSelectField } from "@/Components/inputs/forms/FormSelectField";
|
||||
import {
|
||||
FormAutocompleteField,
|
||||
type AutocompleteOption,
|
||||
} from "@/Components/inputs/forms/FormAutoCompleteField";
|
||||
import { FormSliderField } from "@/Components/inputs/forms/FormSliderField";
|
||||
import { FormSwitchField } from "@/Components/inputs/forms/FormSwitchField";
|
||||
import type { ProxyResponse } from "@/Types/Proxy";
|
||||
import { formatDuration } from "@/Utils/TimeUtils";
|
||||
|
||||
const httpMethodOptions = HttpMethods.map((method) => ({
|
||||
value: method,
|
||||
label: method,
|
||||
}));
|
||||
|
||||
const PEM_FIELD_ROWS = 6;
|
||||
|
||||
// Hides the pasted key while the field is not focused. Ignored by browsers
|
||||
// without -webkit-text-security, which then show the key as plain text.
|
||||
const maskedInputSx = {
|
||||
"&:not(.Mui-focused) textarea": { WebkitTextSecurity: "disc" },
|
||||
};
|
||||
|
||||
interface IntervalOption extends AutocompleteOption {
|
||||
id: number;
|
||||
}
|
||||
|
||||
const INTERVAL_UNIT_MULTIPLIERS = [1000, 60000, 3600000, 86400000] as const;
|
||||
|
||||
const toIntervalOption = (
|
||||
intervalMs: number,
|
||||
locale: string | undefined
|
||||
): IntervalOption => ({
|
||||
id: intervalMs,
|
||||
name: formatDuration(intervalMs, { long: true, locale }),
|
||||
});
|
||||
|
||||
const filterIntervalOptions = (
|
||||
options: IntervalOption[],
|
||||
inputValue: string,
|
||||
locale: string | undefined
|
||||
): IntervalOption[] => {
|
||||
const input = inputValue.trim();
|
||||
if (input === "" || options.some((option) => option.name === input)) {
|
||||
return options;
|
||||
}
|
||||
|
||||
const amount = Number(input);
|
||||
if (!Number.isFinite(amount) || amount <= 0) {
|
||||
return [];
|
||||
}
|
||||
|
||||
return INTERVAL_UNIT_MULTIPLIERS.map((multiplier) => amount * multiplier)
|
||||
.filter(Number.isSafeInteger)
|
||||
.map((ms) => toIntervalOption(ms, locale));
|
||||
};
|
||||
|
||||
interface GeneralSettingsConfig {
|
||||
urlLabel: string;
|
||||
urlPlaceholder: string;
|
||||
@@ -243,7 +292,7 @@ const getGeneralSettingsConfig = (
|
||||
|
||||
const CreateMonitorPage = () => {
|
||||
const theme = useTheme();
|
||||
const { t } = useTranslation();
|
||||
const { t, i18n } = useTranslation();
|
||||
const { monitorId } = useParams();
|
||||
const location = useLocation();
|
||||
const navigate = useNavigate();
|
||||
@@ -300,10 +349,12 @@ const CreateMonitorPage = () => {
|
||||
const showStep = (step: number) => isEditMode || currentStep === step;
|
||||
|
||||
const watchedType = watch("type") as MonitorType;
|
||||
const watchedUrl = watch("url") as string;
|
||||
const watchedMethod = watch("method") as HttpMethod | undefined;
|
||||
const watchedProxyMode = watch("proxyMode") as ProxyMode | undefined;
|
||||
const watchedUseAdvancedMatching = watch("useAdvancedMatching") as boolean;
|
||||
const watchGeoCheckEnabled = watch("geoCheckEnabled") as boolean;
|
||||
const watchedInterval = watch("interval");
|
||||
|
||||
// Steps without an advanced section drop it, so the last step is the form's.
|
||||
const totalSteps = monitorStepCount(watchedType);
|
||||
@@ -334,6 +385,8 @@ const CreateMonitorPage = () => {
|
||||
[watchedType, t]
|
||||
);
|
||||
|
||||
const pemInputSx = { fontFamily: theme.typography.fontFamilyMonospace };
|
||||
|
||||
const { post, loading: isCreating } = usePost<MonitorFormData, Monitor>();
|
||||
const { patch, loading: isUpdating } = usePatch<MonitorFormData, Monitor>();
|
||||
const isSubmitting = isCreating || isUpdating;
|
||||
@@ -459,16 +512,18 @@ const CreateMonitorPage = () => {
|
||||
[t]
|
||||
);
|
||||
|
||||
const intervalOptions = useMemo(
|
||||
() =>
|
||||
MonitorIntervalOptions.map((option) => ({
|
||||
value: option.value,
|
||||
label: t(
|
||||
`pages.createMonitor.form.frequency.option.frequency.value.${option.labelKey}`
|
||||
),
|
||||
})),
|
||||
[t]
|
||||
);
|
||||
const intervalOptions = useMemo(() => {
|
||||
const options = MonitorIntervalOptions.map(({ value }) =>
|
||||
toIntervalOption(value, i18n.resolvedLanguage)
|
||||
);
|
||||
if (
|
||||
typeof watchedInterval === "number" &&
|
||||
!options.some((option) => option.id === watchedInterval)
|
||||
) {
|
||||
options.push(toIntervalOption(watchedInterval, i18n.resolvedLanguage));
|
||||
}
|
||||
return options;
|
||||
}, [watchedInterval, i18n.resolvedLanguage]);
|
||||
|
||||
const matchMethodOptions = useMemo(
|
||||
() =>
|
||||
@@ -689,17 +744,80 @@ const CreateMonitorPage = () => {
|
||||
/>
|
||||
)}
|
||||
|
||||
{showStep(0) && watchedType === "docker" && isDockerTlsUrl(watchedUrl) && (
|
||||
<ConfigBox
|
||||
title={t("pages.createMonitor.form.dockerTls.title")}
|
||||
subtitle={t("pages.createMonitor.form.dockerTls.description")}
|
||||
rightContent={
|
||||
<Stack spacing={theme.spacing(LAYOUT.MD)}>
|
||||
<FormTextField
|
||||
name="dockerTlsCa"
|
||||
multiline
|
||||
rows={PEM_FIELD_ROWS}
|
||||
fieldLabel={t("pages.createMonitor.form.dockerTls.option.ca.label")}
|
||||
placeholder={t(
|
||||
"pages.createMonitor.form.dockerTls.option.ca.placeholder"
|
||||
)}
|
||||
slotProps={{ input: { sx: pemInputSx } }}
|
||||
/>
|
||||
<FormTextField
|
||||
name="dockerTlsCert"
|
||||
multiline
|
||||
rows={PEM_FIELD_ROWS}
|
||||
fieldLabel={t("pages.createMonitor.form.dockerTls.option.cert.label")}
|
||||
placeholder={t(
|
||||
"pages.createMonitor.form.dockerTls.option.cert.placeholder"
|
||||
)}
|
||||
slotProps={{ input: { sx: pemInputSx } }}
|
||||
/>
|
||||
<FormTextField
|
||||
name="dockerTlsKey"
|
||||
multiline
|
||||
rows={PEM_FIELD_ROWS}
|
||||
fieldLabel={t("pages.createMonitor.form.dockerTls.option.key.label")}
|
||||
placeholder={t(
|
||||
"pages.createMonitor.form.dockerTls.option.key.placeholder"
|
||||
)}
|
||||
helperText={
|
||||
existingMonitor?.dockerTlsKeySet
|
||||
? t("pages.createMonitor.form.dockerTls.option.key.stored")
|
||||
: undefined
|
||||
}
|
||||
autoComplete="off"
|
||||
slotProps={{ input: { sx: { ...pemInputSx, ...maskedInputSx } } }}
|
||||
/>
|
||||
<FormSwitchField
|
||||
name="ignoreTlsErrors"
|
||||
label={t("pages.createMonitor.form.dockerTls.option.ignoreTls.label")}
|
||||
/>
|
||||
<Typography
|
||||
component="span"
|
||||
color={theme.palette.text.secondary}
|
||||
sx={{ opacity: 0.8 }}
|
||||
>
|
||||
{t("pages.createMonitor.form.dockerTls.option.ignoreTls.description")}
|
||||
</Typography>
|
||||
</Stack>
|
||||
}
|
||||
/>
|
||||
)}
|
||||
|
||||
{showStep(1) && (
|
||||
<ConfigBox
|
||||
title={t("pages.createMonitor.form.frequency.title")}
|
||||
subtitle={t("pages.createMonitor.form.frequency.description")}
|
||||
rightContent={
|
||||
<FormSelectField
|
||||
<FormAutocompleteField
|
||||
name="interval"
|
||||
fieldLabel={t(
|
||||
"pages.createMonitor.form.frequency.option.frequency.label"
|
||||
)}
|
||||
options={intervalOptions}
|
||||
filterOptions={(options, { inputValue }) =>
|
||||
filterIntervalOptions(options, inputValue, i18n.resolvedLanguage)
|
||||
}
|
||||
getOptionDisabled={(option) => option.id < MIN_MONITOR_INTERVAL_MS}
|
||||
disableClearable
|
||||
/>
|
||||
}
|
||||
/>
|
||||
|
||||
@@ -0,0 +1,54 @@
|
||||
import Box from "@mui/material/Box";
|
||||
import Stack from "@mui/material/Stack";
|
||||
import Typography from "@mui/material/Typography";
|
||||
import { useTheme } from "@mui/material/styles";
|
||||
import { LAYOUT } from "@/Utils/Theme/constants";
|
||||
import { typographyLevels } from "@/Utils/Theme/Palette";
|
||||
|
||||
interface DashboardCardProps {
|
||||
title: string;
|
||||
topRight?: React.ReactNode;
|
||||
children: React.ReactNode;
|
||||
}
|
||||
|
||||
export const DashboardCard = ({ title, topRight, children }: DashboardCardProps) => {
|
||||
const theme = useTheme();
|
||||
|
||||
return (
|
||||
<Box
|
||||
bgcolor={theme.palette.background.paper}
|
||||
border={1}
|
||||
borderColor={theme.palette.divider}
|
||||
borderRadius={`${theme.shape.borderRadius}px`}
|
||||
padding={theme.spacing(LAYOUT.MD)}
|
||||
display="flex"
|
||||
flexDirection="column"
|
||||
gap={theme.spacing(LAYOUT.SM)}
|
||||
height="100%"
|
||||
boxSizing="border-box"
|
||||
>
|
||||
<Stack
|
||||
direction="row"
|
||||
alignItems="center"
|
||||
justifyContent="space-between"
|
||||
>
|
||||
<Typography
|
||||
fontSize={typographyLevels.m}
|
||||
fontWeight={500}
|
||||
color={theme.palette.text.primary}
|
||||
>
|
||||
{title}
|
||||
</Typography>
|
||||
{topRight && (
|
||||
<Typography
|
||||
fontSize={typographyLevels.s}
|
||||
color={theme.palette.text.secondary}
|
||||
>
|
||||
{topRight}
|
||||
</Typography>
|
||||
)}
|
||||
</Stack>
|
||||
{children}
|
||||
</Box>
|
||||
);
|
||||
};
|
||||
@@ -0,0 +1,28 @@
|
||||
import { LAYOUT } from "@/Utils/Theme/constants";
|
||||
import Box from "@mui/material/Box";
|
||||
import { useTheme } from "@mui/material/styles";
|
||||
|
||||
interface ProgressBarProps {
|
||||
fillRatio: number;
|
||||
}
|
||||
|
||||
export const ProgressBar = ({ fillRatio }: ProgressBarProps) => {
|
||||
const theme = useTheme();
|
||||
const clamped = Math.min(1, Math.max(0, fillRatio));
|
||||
|
||||
return (
|
||||
<Box
|
||||
height={theme.spacing(LAYOUT.XS)}
|
||||
borderRadius={theme.shape.borderRadius}
|
||||
bgcolor={theme.palette.action.disabledBackground}
|
||||
overflow="hidden"
|
||||
>
|
||||
<Box
|
||||
height="100%"
|
||||
width={`${clamped * 100}%`}
|
||||
borderRadius={theme.shape.borderRadius}
|
||||
bgcolor={theme.palette.primary.main}
|
||||
/>
|
||||
</Box>
|
||||
);
|
||||
};
|
||||
@@ -0,0 +1,42 @@
|
||||
import Stack from "@mui/material/Stack";
|
||||
import Typography from "@mui/material/Typography";
|
||||
import { useTheme } from "@mui/material/styles";
|
||||
import { typographyLevels } from "@/Utils/Theme/Palette";
|
||||
import { SPACING } from "@/Utils/Theme/constants";
|
||||
import { ProgressBar } from "@/Pages/Dashboard/components/ProgressBar";
|
||||
|
||||
interface BarRowProps {
|
||||
label: string;
|
||||
value: string;
|
||||
fillRatio: number;
|
||||
valueColor?: string;
|
||||
}
|
||||
|
||||
export const BarRow = ({ label, value, fillRatio, valueColor }: BarRowProps) => {
|
||||
const theme = useTheme();
|
||||
|
||||
return (
|
||||
<Stack gap={theme.spacing(SPACING.LG)}>
|
||||
<Stack
|
||||
direction="row"
|
||||
justifyContent="space-between"
|
||||
alignItems="center"
|
||||
>
|
||||
<Typography
|
||||
fontSize={typographyLevels.m}
|
||||
color={theme.palette.text.primary}
|
||||
>
|
||||
{label}
|
||||
</Typography>
|
||||
<Typography
|
||||
fontSize={typographyLevels.m}
|
||||
color={valueColor ?? theme.palette.primary.main}
|
||||
fontWeight={500}
|
||||
>
|
||||
{value}
|
||||
</Typography>
|
||||
</Stack>
|
||||
<ProgressBar fillRatio={fillRatio} />
|
||||
</Stack>
|
||||
);
|
||||
};
|
||||
@@ -0,0 +1,49 @@
|
||||
import Stack from "@mui/material/Stack";
|
||||
import Typography from "@mui/material/Typography";
|
||||
import { useTheme } from "@mui/material/styles";
|
||||
import { DashboardCard } from "@/Pages/Dashboard/components/DashboardCard";
|
||||
import { ProgressBar } from "@/Pages/Dashboard/components/ProgressBar";
|
||||
import { typographyLevels } from "@/Utils/Theme/Palette";
|
||||
import { SPACING } from "@/Utils/Theme/constants";
|
||||
import { useTranslation } from "react-i18next";
|
||||
import type { MonitorsSummary } from "@/Types/Monitor";
|
||||
|
||||
interface MonitorStatusCardProps {
|
||||
summary: MonitorsSummary;
|
||||
}
|
||||
|
||||
export const MonitorStatusCard = ({ summary }: MonitorStatusCardProps) => {
|
||||
const theme = useTheme();
|
||||
const { t } = useTranslation();
|
||||
|
||||
const up = summary.upMonitors;
|
||||
const total = summary.totalMonitors;
|
||||
const ratio = total > 0 ? up / total : 0;
|
||||
|
||||
return (
|
||||
<DashboardCard title={t("pages.dashboard.cards.monitorStatus")}>
|
||||
<Stack gap={theme.spacing(SPACING.SM)}>
|
||||
<Stack
|
||||
direction="row"
|
||||
alignItems="baseline"
|
||||
gap={theme.spacing(SPACING.XS)}
|
||||
>
|
||||
<Typography
|
||||
fontSize={typographyLevels.xxl}
|
||||
fontWeight={500}
|
||||
color={theme.palette.text.primary}
|
||||
>
|
||||
{up}
|
||||
</Typography>
|
||||
<Typography
|
||||
fontSize={typographyLevels.m}
|
||||
color={theme.palette.text.secondary}
|
||||
>
|
||||
{t("pages.dashboard.monitorStatus.ofUp", { total })}
|
||||
</Typography>
|
||||
</Stack>
|
||||
<ProgressBar fillRatio={ratio} />
|
||||
</Stack>
|
||||
</DashboardCard>
|
||||
);
|
||||
};
|
||||
@@ -0,0 +1,35 @@
|
||||
import Stack from "@mui/material/Stack";
|
||||
import { DashboardCard } from "@/Pages/Dashboard/components/DashboardCard";
|
||||
import { BarRow } from "@/Pages/Dashboard/components/cards/BarRow";
|
||||
import { useTranslation } from "react-i18next";
|
||||
import { useTheme } from "@mui/material/styles";
|
||||
import type { DashboardTypeCount } from "@/Pages/Dashboard/types";
|
||||
import { getMonitorTypeLabel } from "@/Types/Monitor";
|
||||
import { LAYOUT } from "@/Utils/Theme/constants";
|
||||
|
||||
interface MonitorsByTypeCardProps {
|
||||
monitorsByType: DashboardTypeCount[];
|
||||
}
|
||||
|
||||
export const MonitorsByTypeCard = ({ monitorsByType }: MonitorsByTypeCardProps) => {
|
||||
const { t } = useTranslation();
|
||||
const theme = useTheme();
|
||||
|
||||
const maxCount = monitorsByType[0]?.count ?? 1;
|
||||
|
||||
return (
|
||||
<DashboardCard title={t("pages.dashboard.cards.monitorsByType")}>
|
||||
<Stack gap={theme.spacing(LAYOUT.SM)}>
|
||||
{monitorsByType.map(({ type, count }) => (
|
||||
<BarRow
|
||||
key={type}
|
||||
label={getMonitorTypeLabel(type, t)}
|
||||
value={String(count)}
|
||||
fillRatio={count / maxCount}
|
||||
valueColor={theme.palette.text.primary}
|
||||
/>
|
||||
))}
|
||||
</Stack>
|
||||
</DashboardCard>
|
||||
);
|
||||
};
|
||||
@@ -0,0 +1,45 @@
|
||||
import Box from "@mui/material/Box";
|
||||
import { useTheme } from "@mui/material/styles";
|
||||
|
||||
import { BasePage } from "@/Components/design-elements/BasePage";
|
||||
import { LAYOUT } from "@/Utils/Theme/constants";
|
||||
|
||||
import { MonitorStatusCard } from "@/Pages/Dashboard/components/cards/MonitorStatusCard";
|
||||
import { MonitorsByTypeCard } from "@/Pages/Dashboard/components/cards/MonitorsByTypeCard";
|
||||
import type { MonitorsSummary } from "@/Types/Monitor";
|
||||
import type { DashboardTypeCount } from "@/Pages/Dashboard/types";
|
||||
|
||||
const MOCK_SUMMARY: MonitorsSummary = {
|
||||
totalMonitors: 12,
|
||||
upMonitors: 10,
|
||||
downMonitors: 1,
|
||||
pausedMonitors: 1,
|
||||
initializingMonitors: 0,
|
||||
maintenanceMonitors: 0,
|
||||
breachedMonitors: 0,
|
||||
};
|
||||
|
||||
const MOCK_MONITORS_BY_TYPE: DashboardTypeCount[] = [
|
||||
{ type: "http", count: 7 },
|
||||
{ type: "ping", count: 3 },
|
||||
{ type: "port", count: 2 },
|
||||
];
|
||||
|
||||
const Dashboard = () => {
|
||||
const theme = useTheme();
|
||||
|
||||
return (
|
||||
<BasePage headerKey="dashboard">
|
||||
<Box
|
||||
display="grid"
|
||||
gridTemplateColumns={{ xs: "1fr", md: "1fr 1fr" }}
|
||||
gap={theme.spacing(LAYOUT.MD)}
|
||||
>
|
||||
<MonitorStatusCard summary={MOCK_SUMMARY} />
|
||||
<MonitorsByTypeCard monitorsByType={MOCK_MONITORS_BY_TYPE} />
|
||||
</Box>
|
||||
</BasePage>
|
||||
);
|
||||
};
|
||||
|
||||
export default Dashboard;
|
||||
@@ -0,0 +1,6 @@
|
||||
import type { MonitorType } from "@/Types/Monitor";
|
||||
|
||||
export interface DashboardTypeCount {
|
||||
type: MonitorType;
|
||||
count: number;
|
||||
}
|
||||
Whitespace-only changes.
@@ -17,6 +17,7 @@ import { formatPercentage } from "@/Utils/FormatUtils";
|
||||
import prettyBytes from "pretty-bytes";
|
||||
import { BaseBox } from "@/Components/design-elements";
|
||||
import { dedupeDockerPorts, getDockerMountLabel } from "@/Utils/MonitorUtils";
|
||||
import type { DateRange } from "@/Types/Query";
|
||||
|
||||
const getChartConfigs = (theme: Theme, t: TFunction, stats: DockerContainerStats) => [
|
||||
{
|
||||
@@ -24,20 +25,29 @@ const getChartConfigs = (theme: Theme, t: TFunction, stats: DockerContainerStats
|
||||
key: "avgCpuPct",
|
||||
color: theme.palette.success.main,
|
||||
rightTitle: formatPercentage(stats?.latest?.container.cpuPct || 0),
|
||||
formatter: formatPercentage,
|
||||
labelKey: "common.charts.histogram.avgCpuUsage",
|
||||
},
|
||||
{
|
||||
title: t("common.charts.labels.memoryUsage"),
|
||||
key: "avgMemoryUsedBytes",
|
||||
color: theme.palette.primary.main,
|
||||
rightTitle: prettyBytes(stats?.latest?.container.memoryUsedBytes || 0),
|
||||
formatter: prettyBytes,
|
||||
labelKey: "common.charts.histogram.avgMemoryUsage",
|
||||
},
|
||||
];
|
||||
|
||||
export const TabOverview = ({ stats }: { stats: DockerContainerStats }) => {
|
||||
export const TabOverview = ({
|
||||
stats,
|
||||
range,
|
||||
}: {
|
||||
stats: DockerContainerStats;
|
||||
range: DateRange;
|
||||
}) => {
|
||||
const theme = useTheme();
|
||||
const { t } = useTranslation();
|
||||
const isSmall = useMediaQuery(theme.breakpoints.down("md"));
|
||||
|
||||
const { ports, mounts } = stats?.latest?.container ?? {};
|
||||
const dedupedPorts = dedupeDockerPorts(ports ?? []);
|
||||
return (
|
||||
@@ -58,6 +68,9 @@ export const TabOverview = ({ stats }: { stats: DockerContainerStats }) => {
|
||||
dataKey={config.key}
|
||||
strokeColor={config.color}
|
||||
gradientStartColor={config.color}
|
||||
range={range}
|
||||
formatter={config.formatter}
|
||||
labelKey={config.labelKey}
|
||||
/>
|
||||
</Grid>
|
||||
);
|
||||
|
||||
@@ -21,7 +21,7 @@ const TAB_MAP = {
|
||||
logs: 1,
|
||||
} as const;
|
||||
|
||||
const DockerDetailsPage = ({ open = "logs" }: DockerContainerDetailsProps) => {
|
||||
const DockerDetailsPage = ({ open = "overview" }: DockerContainerDetailsProps) => {
|
||||
const [activeTab, setActiveTab] = useState<number>(TAB_MAP[open]);
|
||||
const [dateRange, setDateRange] = useState<DateRange>("recent");
|
||||
const { monitorId, containerName } = useParams<{
|
||||
@@ -69,7 +69,12 @@ const DockerDetailsPage = ({ open = "logs" }: DockerContainerDetailsProps) => {
|
||||
<Tab label={t("pages.docker.container.tabs.overview")} />
|
||||
<Tab label={t("pages.docker.container.tabs.logs")} />
|
||||
</Tabs>
|
||||
{activeTab === 0 && <TabOverview stats={stats} />}
|
||||
{activeTab === 0 && (
|
||||
<TabOverview
|
||||
stats={stats}
|
||||
range={dateRange}
|
||||
/>
|
||||
)}
|
||||
{activeTab === 1 && (
|
||||
<TabLogs
|
||||
monitorId={monitorId}
|
||||
|
||||
@@ -8,7 +8,7 @@ import { useTranslation } from "react-i18next";
|
||||
|
||||
const LogsPage = () => {
|
||||
const { t } = useTranslation();
|
||||
const [activeTab, setActiveTab] = useState<number>(1);
|
||||
const [activeTab, setActiveTab] = useState<number>(0);
|
||||
return (
|
||||
<BasePage headerKey="logs">
|
||||
<Tabs
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
import NotFoundSvg from "@/assets/Images/sushi_404.svg";
|
||||
import NotFoundSvg from "@/assets/Images/sushi_404.svg?react";
|
||||
import { Button } from "@/Components/inputs";
|
||||
import Box from "@mui/material/Box";
|
||||
import Stack from "@mui/material/Stack";
|
||||
@@ -36,10 +36,11 @@ const NotFoundPage = ({ title, desc }: NotFoundProps) => {
|
||||
alignItems="center"
|
||||
>
|
||||
<Box
|
||||
component="img"
|
||||
src={NotFoundSvg}
|
||||
alt="404"
|
||||
maxHeight={"25rem"}
|
||||
component={NotFoundSvg}
|
||||
role="img"
|
||||
aria-label="404"
|
||||
width="100%"
|
||||
maxHeight="25rem"
|
||||
/>
|
||||
<Typography
|
||||
component="h1"
|
||||
|
||||
@@ -8,6 +8,9 @@ import AuthRegister from "@/Pages/Auth/Register";
|
||||
import AuthForgotPassword from "@/Pages/Auth/Recovery";
|
||||
import AuthSetNewPassword from "@/Pages/Auth/SetNewPassword";
|
||||
|
||||
// Dashboard
|
||||
import Dashboard from "@/Pages/Dashboard";
|
||||
|
||||
// Uptime
|
||||
import Uptime from "@/Pages/Uptime/Monitors";
|
||||
import UptimeDetails from "@/Pages/Uptime/Details";
|
||||
@@ -92,6 +95,10 @@ const Routes = () => {
|
||||
path="/"
|
||||
element={<Navigate to="/uptime" />}
|
||||
/>
|
||||
<Route
|
||||
path="dashboard"
|
||||
element={<Dashboard />}
|
||||
/>
|
||||
<Route
|
||||
path="/uptime"
|
||||
element={<Uptime />}
|
||||
|
||||
@@ -60,6 +60,15 @@ export const monitorTypeLabelKey: Record<SelectableMonitorType, string> &
|
||||
pagespeed: "optionPagespeed",
|
||||
};
|
||||
|
||||
export const getMonitorTypeLabel = (
|
||||
type: MonitorType,
|
||||
t: (key: string) => string
|
||||
): string => {
|
||||
const monitorLabel = monitorTypeLabelKey[type];
|
||||
return monitorLabel ? t(`pages.common.monitors.monitorTypes.${monitorLabel}`) : type;
|
||||
};
|
||||
export const MIN_MONITOR_INTERVAL_MS = 10000;
|
||||
|
||||
export const MonitorIntervalOptions = [
|
||||
{ value: 15000, labelKey: "fifteenSeconds" },
|
||||
{ value: 30000, labelKey: "thirtySeconds" },
|
||||
@@ -160,6 +169,9 @@ export interface Monitor {
|
||||
geoCheckLocations?: GeoContinent[];
|
||||
geoCheckInterval?: number;
|
||||
dockerLogsEnabled?: boolean;
|
||||
dockerTlsCa?: string;
|
||||
dockerTlsCert?: string;
|
||||
dockerTlsKeySet?: boolean;
|
||||
dnsServer?: string;
|
||||
dnsRecordType?: DnsRecordType;
|
||||
recentChecks: CheckSnapshot[];
|
||||
|
||||
@@ -104,9 +104,52 @@ export const formatMs = (ms: number, hasSpace: boolean = true): string => {
|
||||
return formatted;
|
||||
};
|
||||
|
||||
export const formatDuration = (ms: number, verbose: boolean = false, hasSpace = true) => {
|
||||
if (verbose) {
|
||||
return prettyMilliseconds(ms, { verbose: true });
|
||||
type DurationUnit = "day" | "hour" | "minute" | "second";
|
||||
|
||||
interface FormatDurationOptions {
|
||||
long?: boolean;
|
||||
relative?: boolean;
|
||||
hasSpace?: boolean;
|
||||
locale?: string;
|
||||
}
|
||||
|
||||
const DURATION_UNITS: Array<{ unit: DurationUnit; milliseconds: number }> = [
|
||||
{ unit: "day", milliseconds: MS_PER_DAY },
|
||||
{ unit: "hour", milliseconds: MS_PER_HOUR },
|
||||
{ unit: "minute", milliseconds: MS_PER_MINUTE },
|
||||
{ unit: "second", milliseconds: MS_PER_SECOND },
|
||||
];
|
||||
|
||||
const FALLBACK_DURATION_UNIT = DURATION_UNITS[DURATION_UNITS.length - 1];
|
||||
|
||||
const getDurationUnit = (ms: number) =>
|
||||
DURATION_UNITS.find(({ milliseconds }) => Math.abs(ms) >= milliseconds) ??
|
||||
FALLBACK_DURATION_UNIT;
|
||||
|
||||
const getExactDurationUnit = (ms: number) =>
|
||||
DURATION_UNITS.find(({ milliseconds }) => ms !== 0 && ms % milliseconds === 0) ??
|
||||
FALLBACK_DURATION_UNIT;
|
||||
|
||||
export const formatDuration = (
|
||||
ms: number,
|
||||
{ long = false, relative = false, hasSpace = true, locale }: FormatDurationOptions = {}
|
||||
): string => {
|
||||
if (relative) {
|
||||
const { unit, milliseconds } = getDurationUnit(ms);
|
||||
const value = Math.sign(ms) * Math.floor(Math.abs(ms) / milliseconds);
|
||||
return new Intl.RelativeTimeFormat(locale, {
|
||||
numeric: "always",
|
||||
style: long ? "long" : "narrow",
|
||||
}).format(-value, unit);
|
||||
}
|
||||
|
||||
if (long) {
|
||||
const { unit, milliseconds } = getExactDurationUnit(ms);
|
||||
return new Intl.NumberFormat(locale, {
|
||||
style: "unit",
|
||||
unit,
|
||||
unitDisplay: "long",
|
||||
}).format(ms / milliseconds);
|
||||
}
|
||||
|
||||
const formatted = prettyMilliseconds(ms, {
|
||||
|
||||
@@ -6,6 +6,7 @@ import {
|
||||
HttpMethods,
|
||||
PageSpeedStrategies,
|
||||
ProxyModes,
|
||||
MIN_MONITOR_INTERVAL_MS,
|
||||
type MonitorType,
|
||||
} from "@/Types/Monitor";
|
||||
import { ALL_HTTP_STATUS_CODES } from "@/Utils/statusCode";
|
||||
@@ -27,7 +28,8 @@ const baseSchema = z.object({
|
||||
description: z.string().optional(),
|
||||
interval: z
|
||||
.number()
|
||||
.min(15000, "Interval must be at least 15 seconds")
|
||||
.int("Interval must be a whole number of milliseconds")
|
||||
.min(MIN_MONITOR_INTERVAL_MS, "Interval must be at least 10 seconds")
|
||||
.register(monitorStepRegistry, { step: 1 }),
|
||||
notifications: z.array(z.string()).register(monitorStepRegistry, { step: 1 }),
|
||||
tags: z.array(z.string()).register(monitorStepRegistry, { step: 1 }),
|
||||
@@ -108,10 +110,30 @@ const portSchema = baseSchema.extend({
|
||||
.max(65535, "Port must be at most 65535"),
|
||||
});
|
||||
|
||||
// Docker monitor schema
|
||||
// Docker monitor schema. The url forms mirror DOCKER_TLS_URL and
|
||||
// DOCKER_SOCKET_URL in server/src/utils/dockerHost.ts.
|
||||
const dockerTlsUrlRegex = /^(tcp|https):\/\/([^/\s:]+)(?::(\d+))?\/?$/;
|
||||
const dockerSocketUrlRegex = /^(?:unix:\/\/\/\S+|\/\S+)$/;
|
||||
|
||||
export const isDockerTlsUrl = (url: string | undefined): boolean =>
|
||||
dockerTlsUrlRegex.test(url?.trim() ?? "");
|
||||
|
||||
const dockerSchema = baseSchema.extend({
|
||||
type: z.literal("docker"),
|
||||
url: z.string().min(1, "Docker host URL is required"),
|
||||
url: z
|
||||
.string()
|
||||
.min(1, "Docker host URL is required")
|
||||
.refine(
|
||||
(url) => dockerSocketUrlRegex.test(url.trim()) || isDockerTlsUrl(url),
|
||||
"Docker host must be unix:///path, an absolute socket path, or tcp://host[:port]"
|
||||
),
|
||||
ignoreTlsErrors: z.boolean(),
|
||||
dockerTlsCa: z.string(),
|
||||
dockerTlsCert: z.string(),
|
||||
// blank on edit keeps the stored key.
|
||||
dockerTlsKey: z.string(),
|
||||
// Read only mirror of the server flag, stripped on submit
|
||||
dockerTlsKeySet: z.boolean(),
|
||||
dockerLogsEnabled: z.boolean().register(monitorStepRegistry, { step: 1 }),
|
||||
});
|
||||
|
||||
@@ -217,7 +239,8 @@ const monitorSchemaUnion = z.discriminatedUnion("type", [
|
||||
dnsSchema,
|
||||
]);
|
||||
|
||||
// Mirrors the server's refineProxySelection (monitorValidation.ts)
|
||||
// Mirrors the server's refineProxySelection and refineDockerTls
|
||||
// (monitorValidation.ts). PEM contents are only parsed on the server.
|
||||
export const monitorSchema = monitorSchemaUnion.superRefine((data, ctx) => {
|
||||
if (data.type === "http" && data.proxyMode === "custom" && !data.proxyId) {
|
||||
ctx.addIssue({
|
||||
@@ -226,6 +249,30 @@ export const monitorSchema = monitorSchemaUnion.superRefine((data, ctx) => {
|
||||
message: "A proxy must be selected when proxy mode is custom",
|
||||
});
|
||||
}
|
||||
|
||||
if (data.type === "docker" && isDockerTlsUrl(data.url)) {
|
||||
if (!data.dockerTlsCert.trim()) {
|
||||
ctx.addIssue({
|
||||
code: "custom",
|
||||
path: ["dockerTlsCert"],
|
||||
message: "Client certificate is required for a TLS Docker host",
|
||||
});
|
||||
}
|
||||
if (!data.dockerTlsKeySet && !data.dockerTlsKey.trim()) {
|
||||
ctx.addIssue({
|
||||
code: "custom",
|
||||
path: ["dockerTlsKey"],
|
||||
message: "Client key is required for a TLS Docker host",
|
||||
});
|
||||
}
|
||||
if (!data.ignoreTlsErrors && !data.dockerTlsCa.trim()) {
|
||||
ctx.addIssue({
|
||||
code: "custom",
|
||||
path: ["dockerTlsCa"],
|
||||
message: "CA certificate is required unless TLS errors are ignored",
|
||||
});
|
||||
}
|
||||
}
|
||||
});
|
||||
|
||||
export type MonitorFormData = z.infer<typeof monitorSchema>;
|
||||
|
||||
File diff suppressed because it is too large.
Load diff
|
Before Width: | Height: | Size: 536 KiB After Width: | Height: | Size: 286 KiB |
@@ -448,6 +448,7 @@
|
||||
},
|
||||
"statBoxes": {
|
||||
"activeFor": "Aktiv seit",
|
||||
"checkingEvery": "Check alle {{interval}}",
|
||||
"certificateExpiry": "Zertifikatablauf",
|
||||
"lastCheck": "Letzter Check",
|
||||
"lastResponseTime": "Letzte Antwortzeit",
|
||||
|
||||
@@ -80,7 +80,9 @@
|
||||
"avg": "Avg: {{value}}",
|
||||
"max": "Max: {{value}}",
|
||||
"responseTime": "Response time: {{value}}",
|
||||
"total": "Total: {{value}}"
|
||||
"total": "Total: {{value}}",
|
||||
"avgCpuUsage": "Average CPU usage: {{value}}",
|
||||
"avgMemoryUsage": "Average memory usage: {{value}}"
|
||||
},
|
||||
"phases": {
|
||||
"dns": "DNS",
|
||||
@@ -224,7 +226,8 @@
|
||||
"alerting": "Alerting",
|
||||
"reporting": "Reporting",
|
||||
"configuration": "Configuration"
|
||||
}
|
||||
},
|
||||
"dashboard": "Dashboard"
|
||||
},
|
||||
"bottomMenu": {
|
||||
"support": "Support",
|
||||
@@ -545,6 +548,7 @@
|
||||
},
|
||||
"statBoxes": {
|
||||
"activeFor": "Active for",
|
||||
"checkingEvery": "Checking every {{interval}}",
|
||||
"certificateExpiry": "Certificate expiry",
|
||||
"domainExpiry": "Domain expiry",
|
||||
"lastCheck": "Last check",
|
||||
@@ -716,7 +720,7 @@
|
||||
"http": "Enter the URL or IP to monitor (e.g., https://example.com/ or 192.168.1.100) and add a clear display name that appears on the dashboard. Set an optional proxy server to connect through.",
|
||||
"ping": "Enter the IP address or hostname to ping (e.g., 192.168.1.100 or example.com) and add a clear display name that appears on the dashboard.",
|
||||
"port": "Enter the URL or IP of the server, the port number and a clear display name that appears on the dashboard.",
|
||||
"docker": "Enter the Docker socket to monitor, as unix:///var/run/docker.sock or an absolute path such as /var/run/docker.sock. Every container on the host is monitored.",
|
||||
"docker": "Enter the Docker socket to monitor, as unix:///var/run/docker.sock or an absolute path such as /var/run/docker.sock, or a remote daemon as tcp://host:2376 to connect with TLS client certificates. Every container on the host is monitored.",
|
||||
"game": "Enter the IP address or hostname and the port number to ping (e.g., 192.168.1.100 or example.com) and choose game type. Some games require enabling server queries in their config file — check your game server's documentation. See the <gamedigLink>supported games list</gamedigLink> for protocol details.",
|
||||
"pagespeed": "Track page load performance, Core Web Vitals, and optimization scores for your website.",
|
||||
"grpc": "Enter the hostname and port of the gRPC server, optionally specify a service name for the health check, and add a display name.",
|
||||
@@ -832,6 +836,29 @@
|
||||
"description": "This will permanently delete all monitoring history, statistics, and check data for your team. This action cannot be undone.",
|
||||
"confirm": "Yes, clear all stats"
|
||||
}
|
||||
},
|
||||
"dockerTls": {
|
||||
"title": "TLS credentials",
|
||||
"description": "Connect to a remote Docker daemon over mutual TLS. Paste the PEM files you would pass to docker --tlsverify. The client key is encrypted before it is stored and is never shown again.",
|
||||
"option": {
|
||||
"ca": {
|
||||
"label": "CA certificate",
|
||||
"placeholder": "-----BEGIN CERTIFICATE-----"
|
||||
},
|
||||
"cert": {
|
||||
"label": "Client certificate",
|
||||
"placeholder": "-----BEGIN CERTIFICATE-----"
|
||||
},
|
||||
"key": {
|
||||
"label": "Client key",
|
||||
"placeholder": "-----BEGIN PRIVATE KEY-----",
|
||||
"stored": "A key is stored. Leave blank to keep it."
|
||||
},
|
||||
"ignoreTls": {
|
||||
"label": "Ignore TLS/SSL errors",
|
||||
"description": "Skips verifying the daemon's certificate, so the CA certificate becomes optional. The client certificate and key are still required because the daemon verifies them."
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
@@ -1882,6 +1909,19 @@
|
||||
"title": "Uptime monitors",
|
||||
"description": "Watch HTTP endpoints, pings, containers, and ports — and get alerted the moment something goes down."
|
||||
}
|
||||
},
|
||||
"dashboard": {
|
||||
"monitorStatus": {
|
||||
"ofUp": "of {{total}} up"
|
||||
},
|
||||
"cards": {
|
||||
"monitorStatus": "Monitor status",
|
||||
"monitorsByType": "Monitors by type"
|
||||
},
|
||||
"header": {
|
||||
"title": "Dashboard",
|
||||
"description": "See every monitor type in one place and spot what needs attention."
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -9,6 +9,7 @@ services:
|
||||
- DB_CONNECTION_STRING=mongodb://mongodb:27017/uptime_db
|
||||
- CLIENT_HOST=${CLIENT_HOST:-http://localhost:52345}
|
||||
- JWT_SECRET=${JWT_SECRET:?set JWT_SECRET in your environment}
|
||||
- ENCRYPTION_KEY=${ENCRYPTION_KEY:-}
|
||||
stop_grace_period: 60s
|
||||
healthcheck:
|
||||
test:
|
||||
|
||||
@@ -27,6 +27,17 @@ CLIENT_HOST=http://localhost:5173
|
||||
# ----------------------
|
||||
ORIGIN=localhost
|
||||
|
||||
# Encryption
|
||||
# ------------------
|
||||
# Encrypts stored Docker TLS client keys. Not needed for socket-only Docker monitors.
|
||||
# Generate a key: openssl rand -base64 32
|
||||
# Key list: comma-separated; the first key encrypts, every key decrypts.
|
||||
# Split deployments: set the same value on the API and every worker.
|
||||
# Rotate a key: 1. deploy OLD_KEY,NEW_KEY everywhere so every process can read both
|
||||
# 2. deploy NEW_KEY,OLD_KEY everywhere so new writes use the new key
|
||||
# 3. wait for the worker to re-encrypt every row, then drop OLD_KEY
|
||||
# ENCRYPTION_KEY=<output of openssl rand -base64 32>
|
||||
|
||||
# Feature Flags
|
||||
# -------------
|
||||
# Set to "false" to disable the selectable theme feature on status pages.
|
||||
|
||||
+409
-21
@@ -422,9 +422,6 @@
|
||||
"secret": {
|
||||
"type": "string"
|
||||
},
|
||||
"sshPrivateKey": {
|
||||
"type": "string"
|
||||
},
|
||||
"cpuAlertThreshold": {
|
||||
"type": "number"
|
||||
},
|
||||
@@ -466,6 +463,18 @@
|
||||
"geoCheckInterval": {
|
||||
"type": "number"
|
||||
},
|
||||
"dockerLogsEnabled": {
|
||||
"type": "boolean"
|
||||
},
|
||||
"dockerTlsCa": {
|
||||
"type": "string"
|
||||
},
|
||||
"dockerTlsCert": {
|
||||
"type": "string"
|
||||
},
|
||||
"dockerTlsKeySet": {
|
||||
"type": "boolean"
|
||||
},
|
||||
"dnsServer": {
|
||||
"type": "string"
|
||||
},
|
||||
@@ -513,6 +522,7 @@
|
||||
"geoCheckEnabled",
|
||||
"geoCheckLocations",
|
||||
"geoCheckInterval",
|
||||
"dockerLogsEnabled",
|
||||
"teamId",
|
||||
"userId",
|
||||
"createdAt",
|
||||
@@ -640,9 +650,6 @@
|
||||
"secret": {
|
||||
"type": "string"
|
||||
},
|
||||
"sshPrivateKey": {
|
||||
"type": "string"
|
||||
},
|
||||
"cpuAlertThreshold": {
|
||||
"type": "number"
|
||||
},
|
||||
@@ -684,6 +691,18 @@
|
||||
"geoCheckInterval": {
|
||||
"type": "number"
|
||||
},
|
||||
"dockerLogsEnabled": {
|
||||
"type": "boolean"
|
||||
},
|
||||
"dockerTlsCa": {
|
||||
"type": "string"
|
||||
},
|
||||
"dockerTlsCert": {
|
||||
"type": "string"
|
||||
},
|
||||
"dockerTlsKeySet": {
|
||||
"type": "boolean"
|
||||
},
|
||||
"dnsServer": {
|
||||
"type": "string"
|
||||
},
|
||||
@@ -733,6 +752,7 @@
|
||||
"geoCheckEnabled",
|
||||
"geoCheckLocations",
|
||||
"geoCheckInterval",
|
||||
"dockerLogsEnabled",
|
||||
"teamId",
|
||||
"userId",
|
||||
"createdAt",
|
||||
@@ -929,6 +949,9 @@
|
||||
{
|
||||
"$ref": "#/components/schemas/PushoverNotification"
|
||||
},
|
||||
{
|
||||
"$ref": "#/components/schemas/SignalgridNotification"
|
||||
},
|
||||
{
|
||||
"$ref": "#/components/schemas/TwilioNotification"
|
||||
},
|
||||
@@ -949,6 +972,7 @@
|
||||
"teams": "#/components/schemas/TeamsNotification",
|
||||
"telegram": "#/components/schemas/TelegramNotification",
|
||||
"pushover": "#/components/schemas/PushoverNotification",
|
||||
"signalgrid": "#/components/schemas/SignalgridNotification",
|
||||
"twilio": "#/components/schemas/TwilioNotification",
|
||||
"ntfy": "#/components/schemas/NtfyNotification"
|
||||
}
|
||||
@@ -1380,6 +1404,34 @@
|
||||
"accessToken": "a1234567890abcdef"
|
||||
}
|
||||
},
|
||||
"SignalgridNotification": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"notificationName": {
|
||||
"type": "string",
|
||||
"minLength": 1
|
||||
},
|
||||
"type": {
|
||||
"type": "string",
|
||||
"enum": ["signalgrid"]
|
||||
},
|
||||
"address": {
|
||||
"type": "string",
|
||||
"minLength": 1
|
||||
},
|
||||
"accessToken": {
|
||||
"type": "string",
|
||||
"minLength": 1
|
||||
}
|
||||
},
|
||||
"required": ["notificationName", "type", "address", "accessToken"],
|
||||
"example": {
|
||||
"notificationName": "Signalgrid",
|
||||
"type": "signalgrid",
|
||||
"address": "your-channel",
|
||||
"accessToken": "your-client-key"
|
||||
}
|
||||
},
|
||||
"TwilioNotification": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
@@ -7090,9 +7142,6 @@
|
||||
"secret": {
|
||||
"type": "string"
|
||||
},
|
||||
"sshPrivateKey": {
|
||||
"type": "string"
|
||||
},
|
||||
"cpuAlertThreshold": {
|
||||
"type": "number"
|
||||
},
|
||||
@@ -7134,6 +7183,18 @@
|
||||
"geoCheckInterval": {
|
||||
"type": "number"
|
||||
},
|
||||
"dockerLogsEnabled": {
|
||||
"type": "boolean"
|
||||
},
|
||||
"dockerTlsCa": {
|
||||
"type": "string"
|
||||
},
|
||||
"dockerTlsCert": {
|
||||
"type": "string"
|
||||
},
|
||||
"dockerTlsKeySet": {
|
||||
"type": "boolean"
|
||||
},
|
||||
"dnsServer": {
|
||||
"type": "string"
|
||||
},
|
||||
@@ -7183,6 +7244,7 @@
|
||||
"geoCheckEnabled",
|
||||
"geoCheckLocations",
|
||||
"geoCheckInterval",
|
||||
"dockerLogsEnabled",
|
||||
"teamId",
|
||||
"userId",
|
||||
"createdAt",
|
||||
@@ -7655,9 +7717,6 @@
|
||||
"secret": {
|
||||
"type": "string"
|
||||
},
|
||||
"sshPrivateKey": {
|
||||
"type": "string"
|
||||
},
|
||||
"cpuAlertThreshold": {
|
||||
"type": "number"
|
||||
},
|
||||
@@ -7699,6 +7758,18 @@
|
||||
"geoCheckInterval": {
|
||||
"type": "number"
|
||||
},
|
||||
"dockerLogsEnabled": {
|
||||
"type": "boolean"
|
||||
},
|
||||
"dockerTlsCa": {
|
||||
"type": "string"
|
||||
},
|
||||
"dockerTlsCert": {
|
||||
"type": "string"
|
||||
},
|
||||
"dockerTlsKeySet": {
|
||||
"type": "boolean"
|
||||
},
|
||||
"dnsServer": {
|
||||
"type": "string"
|
||||
},
|
||||
@@ -7748,6 +7819,7 @@
|
||||
"geoCheckEnabled",
|
||||
"geoCheckLocations",
|
||||
"geoCheckInterval",
|
||||
"dockerLogsEnabled",
|
||||
"teamId",
|
||||
"userId",
|
||||
"createdAt",
|
||||
@@ -7984,6 +8056,231 @@
|
||||
}
|
||||
}
|
||||
},
|
||||
"/monitors/docker/details/{monitorId}/containers/{containerName}/logs": {
|
||||
"get": {
|
||||
"tags": ["monitors"],
|
||||
"summary": "Get stored log lines for one container on a Docker host monitor, newest first",
|
||||
"security": [
|
||||
{
|
||||
"bearerAuth": []
|
||||
}
|
||||
],
|
||||
"parameters": [
|
||||
{
|
||||
"schema": {
|
||||
"type": "string",
|
||||
"minLength": 1
|
||||
},
|
||||
"required": true,
|
||||
"name": "monitorId",
|
||||
"in": "path"
|
||||
},
|
||||
{
|
||||
"schema": {
|
||||
"type": "string",
|
||||
"minLength": 1
|
||||
},
|
||||
"required": true,
|
||||
"name": "containerName",
|
||||
"in": "path"
|
||||
},
|
||||
{
|
||||
"schema": {
|
||||
"type": "string",
|
||||
"format": "date-time"
|
||||
},
|
||||
"required": false,
|
||||
"name": "before",
|
||||
"in": "query"
|
||||
},
|
||||
{
|
||||
"schema": {
|
||||
"type": "string",
|
||||
"format": "date-time"
|
||||
},
|
||||
"required": false,
|
||||
"name": "after",
|
||||
"in": "query"
|
||||
},
|
||||
{
|
||||
"schema": {
|
||||
"type": "integer",
|
||||
"minimum": 1,
|
||||
"maximum": 50,
|
||||
"default": 20
|
||||
},
|
||||
"required": false,
|
||||
"name": "limit",
|
||||
"in": "query"
|
||||
}
|
||||
],
|
||||
"responses": {
|
||||
"200": {
|
||||
"description": "OK",
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"success": {
|
||||
"type": "boolean",
|
||||
"enum": [true]
|
||||
},
|
||||
"msg": {
|
||||
"type": "string"
|
||||
},
|
||||
"data": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"logs": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"id": {
|
||||
"type": "string"
|
||||
},
|
||||
"metadata": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"monitorId": {
|
||||
"type": "string"
|
||||
},
|
||||
"teamId": {
|
||||
"type": "string"
|
||||
},
|
||||
"containerId": {
|
||||
"type": "string"
|
||||
},
|
||||
"containerName": {
|
||||
"type": "string"
|
||||
}
|
||||
},
|
||||
"required": ["monitorId", "teamId", "containerId", "containerName"]
|
||||
},
|
||||
"lines": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"ts": {
|
||||
"type": "string"
|
||||
},
|
||||
"stream": {
|
||||
"type": "string",
|
||||
"enum": ["stdout", "stderr"]
|
||||
},
|
||||
"text": {
|
||||
"type": "string"
|
||||
}
|
||||
},
|
||||
"required": ["ts", "stream", "text"]
|
||||
}
|
||||
},
|
||||
"gap": {
|
||||
"type": "boolean"
|
||||
},
|
||||
"checkedAt": {
|
||||
"type": "string"
|
||||
},
|
||||
"expiry": {
|
||||
"type": "string"
|
||||
},
|
||||
"createdAt": {
|
||||
"type": "string"
|
||||
},
|
||||
"updatedAt": {
|
||||
"type": "string"
|
||||
}
|
||||
},
|
||||
"required": ["id", "metadata", "lines", "gap", "checkedAt", "expiry", "createdAt", "updatedAt"]
|
||||
}
|
||||
},
|
||||
"nextCursor": {
|
||||
"type": "string",
|
||||
"nullable": true
|
||||
}
|
||||
},
|
||||
"required": ["logs", "nextCursor"]
|
||||
}
|
||||
},
|
||||
"required": ["success", "msg", "data"]
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"401": {
|
||||
"description": "Unauthorized",
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"success": {
|
||||
"type": "boolean",
|
||||
"enum": [false]
|
||||
},
|
||||
"msg": {
|
||||
"type": "string"
|
||||
},
|
||||
"data": {
|
||||
"nullable": true
|
||||
}
|
||||
},
|
||||
"required": ["success", "msg"]
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"403": {
|
||||
"description": "Forbidden",
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"success": {
|
||||
"type": "boolean",
|
||||
"enum": [false]
|
||||
},
|
||||
"msg": {
|
||||
"type": "string"
|
||||
},
|
||||
"data": {
|
||||
"nullable": true
|
||||
}
|
||||
},
|
||||
"required": ["success", "msg"]
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"500": {
|
||||
"description": "Internal server error",
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"success": {
|
||||
"type": "boolean",
|
||||
"enum": [false]
|
||||
},
|
||||
"msg": {
|
||||
"type": "string"
|
||||
},
|
||||
"data": {
|
||||
"nullable": true
|
||||
}
|
||||
},
|
||||
"required": ["success", "msg"]
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"/monitors/pagespeed/details/{monitorId}": {
|
||||
"get": {
|
||||
"tags": ["monitors"],
|
||||
@@ -8908,9 +9205,6 @@
|
||||
"secret": {
|
||||
"type": "string"
|
||||
},
|
||||
"sshPrivateKey": {
|
||||
"type": "string"
|
||||
},
|
||||
"jsonPath": {
|
||||
"anyOf": [
|
||||
{
|
||||
@@ -9011,6 +9305,42 @@
|
||||
"type": "number",
|
||||
"minimum": 300000
|
||||
},
|
||||
"dockerLogsEnabled": {
|
||||
"type": "boolean"
|
||||
},
|
||||
"dockerTlsCa": {
|
||||
"anyOf": [
|
||||
{
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"type": "string",
|
||||
"enum": [""]
|
||||
}
|
||||
]
|
||||
},
|
||||
"dockerTlsCert": {
|
||||
"anyOf": [
|
||||
{
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"type": "string",
|
||||
"enum": [""]
|
||||
}
|
||||
]
|
||||
},
|
||||
"dockerTlsKey": {
|
||||
"anyOf": [
|
||||
{
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"type": "string",
|
||||
"enum": [""]
|
||||
}
|
||||
]
|
||||
},
|
||||
"dnsServer": {
|
||||
"type": "string",
|
||||
"minLength": 1
|
||||
@@ -9604,9 +9934,6 @@
|
||||
"secret": {
|
||||
"type": "string"
|
||||
},
|
||||
"sshPrivateKey": {
|
||||
"type": "string"
|
||||
},
|
||||
"cpuAlertThreshold": {
|
||||
"type": "number",
|
||||
"default": 100
|
||||
@@ -9702,6 +10029,10 @@
|
||||
"minimum": 300000,
|
||||
"default": 300000
|
||||
},
|
||||
"dockerLogsEnabled": {
|
||||
"type": "boolean",
|
||||
"default": false
|
||||
},
|
||||
"dnsServer": {
|
||||
"type": "string",
|
||||
"minLength": 1
|
||||
@@ -10137,9 +10468,6 @@
|
||||
"secret": {
|
||||
"type": "string"
|
||||
},
|
||||
"sshPrivateKey": {
|
||||
"type": "string"
|
||||
},
|
||||
"ignoreTlsErrors": {
|
||||
"type": "boolean"
|
||||
},
|
||||
@@ -10269,6 +10597,42 @@
|
||||
"type": "number",
|
||||
"minimum": 300000
|
||||
},
|
||||
"dockerLogsEnabled": {
|
||||
"type": "boolean"
|
||||
},
|
||||
"dockerTlsCa": {
|
||||
"anyOf": [
|
||||
{
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"type": "string",
|
||||
"enum": [""]
|
||||
}
|
||||
]
|
||||
},
|
||||
"dockerTlsCert": {
|
||||
"anyOf": [
|
||||
{
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"type": "string",
|
||||
"enum": [""]
|
||||
}
|
||||
]
|
||||
},
|
||||
"dockerTlsKey": {
|
||||
"anyOf": [
|
||||
{
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"type": "string",
|
||||
"enum": [""]
|
||||
}
|
||||
]
|
||||
},
|
||||
"dnsServer": {
|
||||
"type": "string",
|
||||
"minLength": 1
|
||||
@@ -12930,6 +13294,30 @@
|
||||
"globalProxyId": {
|
||||
"type": "string",
|
||||
"nullable": true
|
||||
},
|
||||
"egressCheckEnabled": {
|
||||
"type": "boolean"
|
||||
},
|
||||
"egressCheckTargets": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"type": "string",
|
||||
"minLength": 1,
|
||||
"maxLength": 253
|
||||
},
|
||||
"maxItems": 10
|
||||
},
|
||||
"egressPollIntervalSeconds": {
|
||||
"type": "integer",
|
||||
"minimum": 5,
|
||||
"maximum": 600
|
||||
},
|
||||
"egressNotifications": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"type": "string",
|
||||
"pattern": "^[0-9a-fA-F]{24}$"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -5,6 +5,7 @@ import { requireTeamId, requireUserEmail, requireUserId, requireUserRoles } from
|
||||
|
||||
import {
|
||||
registrationBodyValidation,
|
||||
registerInviteTokenValidation,
|
||||
loginValidation,
|
||||
recoveryValidation,
|
||||
recoveryTokenBodyValidation,
|
||||
@@ -48,7 +49,7 @@ class AuthController implements IAuthController {
|
||||
|
||||
registerUser = catchAsync(async (req: Request, res: Response) => {
|
||||
const newUser = req.body.user;
|
||||
const newUserToken = req.body.token;
|
||||
const newUserToken = registerInviteTokenValidation.parse(req.body.token);
|
||||
if (newUser?.email) {
|
||||
const newUserEmail = requireUserEmail(newUser.email);
|
||||
newUser.email = newUserEmail.toLowerCase();
|
||||
|
||||
@@ -6,6 +6,7 @@ import { AppError } from "@/utils/AppError.js";
|
||||
import { ISettingsService } from "@/domain/app-settings/app-settings.service.js";
|
||||
import { IEmailService } from "@/service/emailService.js";
|
||||
import { IProxiesService } from "@/domain/proxies/proxy.service.js";
|
||||
import { IEgressStateService } from "@/domain/egress/egress-state.service.js";
|
||||
import { Settings } from "@/domain/app-settings/app-settings.type.js";
|
||||
|
||||
export interface ISettingsController {
|
||||
@@ -18,10 +19,17 @@ class SettingsController implements ISettingsController {
|
||||
private settingsService: ISettingsService;
|
||||
private emailService: IEmailService;
|
||||
private proxiesService: IProxiesService;
|
||||
constructor(settingsService: ISettingsService, emailService: IEmailService, proxiesService: IProxiesService) {
|
||||
private egressStateService: IEgressStateService;
|
||||
constructor(
|
||||
settingsService: ISettingsService,
|
||||
emailService: IEmailService,
|
||||
proxiesService: IProxiesService,
|
||||
egressStateService: IEgressStateService
|
||||
) {
|
||||
this.settingsService = settingsService;
|
||||
this.emailService = emailService;
|
||||
this.proxiesService = proxiesService;
|
||||
this.egressStateService = egressStateService;
|
||||
}
|
||||
|
||||
buildAppSettings = async (dbSettings: Settings) => {
|
||||
@@ -70,7 +78,14 @@ class SettingsController implements ISettingsController {
|
||||
}
|
||||
}
|
||||
|
||||
const previousSettings = await this.settingsService.getDBSettings();
|
||||
const updatedSettings = await this.settingsService.updateDbSettings(validatedBody);
|
||||
|
||||
// Switching the egress check on or off starts from a clean state: no degraded episode, no pending recovery job
|
||||
if (validatedBody.egressCheckEnabled !== undefined && validatedBody.egressCheckEnabled !== previousSettings.egressCheckEnabled) {
|
||||
await this.egressStateService.reset();
|
||||
}
|
||||
|
||||
const returnSettings = await this.buildAppSettings(updatedSettings);
|
||||
return res.status(200).json({
|
||||
success: true,
|
||||
|
||||
@@ -23,6 +23,8 @@ export const registrationBodyValidation = z.object({
|
||||
inviteToken: z.string().optional().default(""),
|
||||
});
|
||||
|
||||
export const registerInviteTokenValidation = z.string().optional().default("");
|
||||
|
||||
export const recoveryValidation = z.object({
|
||||
email: z.email("Must be a valid email address"),
|
||||
});
|
||||
|
||||
@@ -15,6 +15,9 @@ import {
|
||||
import { DateRanges, SortOrders } from "@/types/query.js";
|
||||
import { DockerContainerStates, DockerHealthStatuses, DockerLogStreams, DockerPortProtocols } from "@/domain/docker/docker.type.js";
|
||||
import { DOCKER_LOG_PAGE_DEFAULT, DOCKER_LOG_PAGE_MAX } from "@/domain/docker/docker-log.type.js";
|
||||
import { isDockerSocketUrl, isDockerTlsUrl } from "@/utils/dockerHost.js";
|
||||
import { X509Certificate } from "node:crypto";
|
||||
import { keyMatchesCertificate, parseCertificates, parsePrivateKey } from "@/utils/pem.js";
|
||||
|
||||
const httpStatusCode = z.number().refine((code) => HttpStatusCodeSet.has(code), { message: "Must be a valid HTTP status code" });
|
||||
|
||||
@@ -120,19 +123,60 @@ const refineProxySelection = (body: { proxyMode?: string; proxyId?: string }, ct
|
||||
}
|
||||
};
|
||||
|
||||
const dockerUrlRegex = /^(?:unix:\/\/\/\S+|\/\S+)$/;
|
||||
|
||||
const refineDockerUrl = (data: { type?: string; url?: string }, ctx: z.RefinementCtx) => {
|
||||
if (data.type !== "docker" || data.url === undefined) return;
|
||||
if (!dockerUrlRegex.test(data.url)) {
|
||||
if (!isDockerSocketUrl(data.url) && !isDockerTlsUrl(data.url)) {
|
||||
ctx.addIssue({
|
||||
code: "custom",
|
||||
path: ["url"],
|
||||
message: "Docker host must be unix:///path or an absolute socket path",
|
||||
message: "Docker host must be unix:///path, an absolute socket path, or tcp://host[:port]",
|
||||
});
|
||||
}
|
||||
};
|
||||
|
||||
type DockerTlsFields = {
|
||||
type?: string;
|
||||
url?: string;
|
||||
ignoreTlsErrors?: boolean;
|
||||
dockerTlsCa?: string;
|
||||
dockerTlsCert?: string;
|
||||
dockerTlsKey?: string;
|
||||
};
|
||||
|
||||
const refineDockerTls = (mode: "create" | "edit") => (data: DockerTlsFields, ctx: z.RefinementCtx) => {
|
||||
if (data.type !== "docker" || !isDockerTlsUrl(data.url)) return;
|
||||
const issue = (path: string, message: string) => ctx.addIssue({ code: "custom", path: [path], message });
|
||||
|
||||
if (!data.dockerTlsCert) issue("dockerTlsCert", "TLS certificate is required for a TLS Docker host");
|
||||
|
||||
if (mode === "create" && !data.dockerTlsKey) issue("dockerTlsKey", "TLS key is required for a TLS Docker host");
|
||||
|
||||
if (!data.ignoreTlsErrors && !data.dockerTlsCa) issue("dockerTlsCa", "CA certificate is required unless TLS errors are ignored");
|
||||
|
||||
let certificate: X509Certificate | undefined;
|
||||
|
||||
try {
|
||||
if (data.dockerTlsCa) parseCertificates(data.dockerTlsCa);
|
||||
} catch (error: unknown) {
|
||||
issue("dockerTlsCa", error instanceof Error ? error.message : "Docker TLS CA error");
|
||||
}
|
||||
|
||||
try {
|
||||
if (data.dockerTlsCert) [certificate] = parseCertificates(data.dockerTlsCert);
|
||||
} catch (error) {
|
||||
issue("dockerTlsCert", error instanceof Error ? error.message : "Docker TLS cert error");
|
||||
}
|
||||
|
||||
try {
|
||||
if (data.dockerTlsKey) {
|
||||
const key = parsePrivateKey(data.dockerTlsKey);
|
||||
if (certificate && !keyMatchesCertificate(key, certificate)) issue("dockerTlsKey", "Key does not match certificate");
|
||||
}
|
||||
} catch (error: unknown) {
|
||||
issue("dockerTlsKey", error instanceof Error ? error.message : "Docker TLS key error");
|
||||
}
|
||||
};
|
||||
|
||||
export const createMonitorBodyValidation = z
|
||||
.object({
|
||||
_id: z.string().optional(),
|
||||
@@ -170,6 +214,9 @@ export const createMonitorBodyValidation = z
|
||||
geoCheckLocations: z.array(z.enum(GeoContinents)).optional(),
|
||||
geoCheckInterval: z.number().min(300000).optional(),
|
||||
dockerLogsEnabled: z.boolean().optional(),
|
||||
dockerTlsCa: z.union([z.string(), z.literal("")]).optional(),
|
||||
dockerTlsCert: z.union([z.string(), z.literal("")]).optional(),
|
||||
dockerTlsKey: z.union([z.string(), z.literal("")]).optional(),
|
||||
dnsServer: dnsServerValidation.optional(),
|
||||
dnsRecordType: z.enum(DnsRecordTypes).optional(),
|
||||
})
|
||||
@@ -178,7 +225,8 @@ export const createMonitorBodyValidation = z
|
||||
.superRefine(refineHeadMatching)
|
||||
.superRefine(refineRegexPattern)
|
||||
.superRefine(refineProxySelection)
|
||||
.superRefine(refineDockerUrl);
|
||||
.superRefine(refineDockerUrl)
|
||||
.superRefine(refineDockerTls("create"));
|
||||
|
||||
export const editMonitorBodyValidation = z
|
||||
.object({
|
||||
@@ -215,6 +263,9 @@ export const editMonitorBodyValidation = z
|
||||
geoCheckLocations: z.array(z.enum(GeoContinents)).optional(),
|
||||
geoCheckInterval: z.number().min(300000).optional(),
|
||||
dockerLogsEnabled: z.boolean().optional(),
|
||||
dockerTlsCa: z.union([z.string(), z.literal("")]).optional(),
|
||||
dockerTlsCert: z.union([z.string(), z.literal("")]).optional(),
|
||||
dockerTlsKey: z.union([z.string(), z.literal("")]).optional(),
|
||||
dnsServer: dnsServerValidation.optional(),
|
||||
dnsRecordType: z.enum(DnsRecordTypes).optional(),
|
||||
})
|
||||
@@ -223,7 +274,8 @@ export const editMonitorBodyValidation = z
|
||||
.superRefine(refineHeadMatching)
|
||||
.superRefine(refineRegexPattern)
|
||||
.superRefine(refineProxySelection)
|
||||
.superRefine(refineDockerUrl);
|
||||
.superRefine(refineDockerUrl)
|
||||
.superRefine(refineDockerTls("edit"));
|
||||
|
||||
export const pauseMonitorParamValidation = z.object({
|
||||
monitorId: z.string().min(1, "Monitor ID is required"),
|
||||
@@ -291,7 +343,6 @@ const importedMonitorSchema = z
|
||||
geoCheckLocations: z.array(z.enum(GeoContinents)).default([]),
|
||||
geoCheckInterval: z.number().min(300000).default(300000),
|
||||
dockerLogsEnabled: z.boolean().default(false),
|
||||
|
||||
dnsServer: dnsServerValidation.optional(),
|
||||
dnsRecordType: z.enum(DnsRecordTypes).optional(),
|
||||
createdAt: z.string().optional(),
|
||||
@@ -381,6 +432,9 @@ export const monitorResponseSchema = z
|
||||
geoCheckLocations: z.array(z.enum(GeoContinents)),
|
||||
geoCheckInterval: z.number(),
|
||||
dockerLogsEnabled: z.boolean(),
|
||||
dockerTlsCa: z.string().optional(),
|
||||
dockerTlsCert: z.string().optional(),
|
||||
dockerTlsKeySet: z.boolean().optional(),
|
||||
dnsServer: z.string().optional(),
|
||||
dnsRecordType: z.enum(DnsRecordTypes).optional(),
|
||||
teamId: z.string(),
|
||||
|
||||
@@ -1,6 +1,17 @@
|
||||
import { CHECK_TTL_SENTINEL } from "@/domain/checks/check.type.js";
|
||||
import { MAX_EGRESS_POLL_INTERVAL_SECONDS, MAX_EGRESS_TARGETS, MIN_EGRESS_POLL_INTERVAL_SECONDS } from "@/domain/egress/egress.type.js";
|
||||
import { parseEgressTarget } from "@/utils/egressTarget.js";
|
||||
import { z } from "zod";
|
||||
|
||||
// A reliability target is a bare host/IP (ICMP ping), host:port (TCP connect) or an http(s) URL.
|
||||
// Parsed with the same function the probe uses, so nothing the validator accepts can fail to probe.
|
||||
const egressTargetValidation = z
|
||||
.string()
|
||||
.trim()
|
||||
.min(1)
|
||||
.max(253)
|
||||
.refine((target) => parseEgressTarget(target) !== null, "Enter a hostname or IP address, host:port, or http(s) URL");
|
||||
|
||||
//****************************************
|
||||
// Settings Validations
|
||||
//****************************************
|
||||
@@ -42,6 +53,11 @@ export const updateAppSettingsBodyValidation = z
|
||||
.optional(),
|
||||
globalProxyEnabled: z.boolean().optional(),
|
||||
globalProxyId: z.string().nullable().optional(),
|
||||
|
||||
egressCheckEnabled: z.boolean().optional(),
|
||||
egressCheckTargets: z.array(egressTargetValidation).max(MAX_EGRESS_TARGETS).optional(), // empty list falls back to DEFAULT_EGRESS_TARGETS
|
||||
egressPollIntervalSeconds: z.number().int().min(MIN_EGRESS_POLL_INTERVAL_SECONDS).max(MAX_EGRESS_POLL_INTERVAL_SECONDS).optional(),
|
||||
egressNotifications: z.array(z.string().regex(/^[0-9a-fA-F]{24}$/, "Invalid notification id")).optional(),
|
||||
})
|
||||
.strip()
|
||||
.superRefine((body, ctx) => {
|
||||
|
||||
@@ -36,7 +36,12 @@ export const initializeControllers = (apiServices: ApiServices): InitializedCont
|
||||
return {
|
||||
authController: new AuthController(apiServices.userService),
|
||||
monitorController: new MonitorController(apiServices.monitorService, apiServices.notificationsService),
|
||||
settingsController: new SettingsController(apiServices.settingsService, apiServices.emailService, apiServices.proxiesService),
|
||||
settingsController: new SettingsController(
|
||||
apiServices.settingsService,
|
||||
apiServices.emailService,
|
||||
apiServices.proxiesService,
|
||||
apiServices.egressStateService
|
||||
),
|
||||
checkController: new CheckController(apiServices.checkService),
|
||||
geoCheckController: new GeoCheckController(apiServices.geoChecksService),
|
||||
inviteController: new InviteController(apiServices.inviteService),
|
||||
|
||||
@@ -3,6 +3,39 @@ import { DbTypes, LogLevels, QueueModes } from "@/domain/app-settings/app-settin
|
||||
import { booleanCoercion } from "@/api/validation/shared.js";
|
||||
import { ILogger } from "@/utils/logger.js";
|
||||
|
||||
// Padded standard base64 of exactly 32 bytes, as produced by `openssl rand -base64 32`
|
||||
const ENCRYPTION_KEY_ENTRY = /^[A-Za-z0-9+/]{43}=$/;
|
||||
|
||||
export const encryptionKeyList = z
|
||||
.string()
|
||||
.optional()
|
||||
.transform((raw, ctx) => {
|
||||
if (raw === undefined) return [];
|
||||
const entries = raw
|
||||
.split(",")
|
||||
.map((entry, index) => ({ entry: entry.trim(), slot: index + 1 }))
|
||||
.filter(({ entry }) => entry.length > 0);
|
||||
const seen = new Set<string>();
|
||||
const keys: Buffer[] = [];
|
||||
entries.forEach(({ entry, slot }) => {
|
||||
if (!ENCRYPTION_KEY_ENTRY.test(entry)) {
|
||||
ctx.addIssue({
|
||||
code: "custom",
|
||||
message: `ENCRYPTION_KEY entry ${slot} must be 32 bytes as padded standard base64 (openssl rand -base64 32)`,
|
||||
});
|
||||
return;
|
||||
}
|
||||
const decoded = Buffer.from(entry, "base64");
|
||||
const fingerprint = decoded.toString("hex");
|
||||
if (seen.has(fingerprint)) {
|
||||
ctx.addIssue({ code: "custom", message: `ENCRYPTION_KEY entry ${slot} duplicates an earlier entry` });
|
||||
}
|
||||
seen.add(fingerprint);
|
||||
keys.push(decoded);
|
||||
});
|
||||
return keys;
|
||||
});
|
||||
|
||||
const envSchema = z.object({
|
||||
// Server Configuration
|
||||
PORT: z.string().default("52345"),
|
||||
@@ -36,6 +69,9 @@ const envSchema = z.object({
|
||||
|
||||
// Feature flags
|
||||
STATUS_PAGE_THEMES_ENABLED: booleanCoercion.default(true),
|
||||
|
||||
// Encryption
|
||||
ENCRYPTION_KEY: encryptionKeyList,
|
||||
});
|
||||
|
||||
export type ValidatedEnv = z.infer<typeof envSchema>;
|
||||
|
||||
@@ -7,6 +7,7 @@ import { ITagsService, TagsService } from "@/domain/tags/tag.service.js";
|
||||
import { IUserService, UserService } from "@/domain/users/user.service.js";
|
||||
import { IJobScheduler } from "@/worker/worker.interface.js";
|
||||
import { ProxiesService, IProxiesService } from "@/domain/proxies/proxy.service.js";
|
||||
import { EgressStateService, IEgressStateService } from "@/domain/egress/egress-state.service.js";
|
||||
import { SharedServices } from "@/config/services.shared.js";
|
||||
|
||||
// Third-party
|
||||
@@ -24,6 +25,7 @@ export interface ApiServices extends SharedServices {
|
||||
tagsService: ITagsService;
|
||||
diagnosticService: IDiagnosticService;
|
||||
proxiesService: IProxiesService;
|
||||
egressStateService: IEgressStateService;
|
||||
}
|
||||
|
||||
export const buildApi = (shared: SharedServices, jobScheduler: IJobScheduler): ApiServices => {
|
||||
@@ -48,6 +50,7 @@ export const buildApi = (shared: SharedServices, jobScheduler: IJobScheduler): A
|
||||
maintenanceWindowsRepository,
|
||||
jobsRepository,
|
||||
proxiesRepository,
|
||||
egressStateRepository,
|
||||
} = shared;
|
||||
|
||||
const userService = new UserService({
|
||||
@@ -80,6 +83,7 @@ export const buildApi = (shared: SharedServices, jobScheduler: IJobScheduler): A
|
||||
monitorStatsRepository,
|
||||
statusPagesRepository,
|
||||
incidentsRepository,
|
||||
encryptionService: shared.encryptionService,
|
||||
});
|
||||
|
||||
const maintenanceWindowService = new MaintenanceWindowService({
|
||||
@@ -99,6 +103,7 @@ export const buildApi = (shared: SharedServices, jobScheduler: IJobScheduler): A
|
||||
const tagsService = new TagsService(tagsRepository, monitorsRepository);
|
||||
const diagnosticService = new DiagnosticService(db);
|
||||
const proxiesService = new ProxiesService(proxiesRepository, monitorsRepository, settingsService);
|
||||
const egressStateService = new EgressStateService(egressStateRepository, jobsRepository);
|
||||
return {
|
||||
...shared,
|
||||
worker: jobScheduler,
|
||||
@@ -110,5 +115,6 @@ export const buildApi = (shared: SharedServices, jobScheduler: IJobScheduler): A
|
||||
tagsService,
|
||||
diagnosticService,
|
||||
proxiesService,
|
||||
egressStateService,
|
||||
};
|
||||
};
|
||||
@@ -1,6 +1,6 @@
|
||||
import { Mongoose } from "mongoose";
|
||||
import { hostname } from "node:os";
|
||||
import { randomUUID } from "node:crypto";
|
||||
import * as nodeCrypto from "node:crypto";
|
||||
import fs from "fs";
|
||||
import path from "path";
|
||||
import nodemailer from "nodemailer";
|
||||
@@ -20,6 +20,7 @@ import { INotificationsService, NotificationsService } from "@/domain/notificati
|
||||
import { IEmailService, EmailService } from "@/service/emailService.js";
|
||||
import { GlobalPingService } from "@/service/globalPingService.js";
|
||||
import { ILogger } from "@/utils/logger.js";
|
||||
import { IEncryptionService, EncryptionService } from "@/service/encryption/encryptionService.js";
|
||||
|
||||
// Notification providers
|
||||
import type { NotificationProviderRegistry } from "@/domain/notifications/notification.service.js";
|
||||
@@ -55,6 +56,7 @@ import { ITagsRepository } from "@/domain/tags/tag.repository.interface.js";
|
||||
import { ITeamsRepository } from "@/domain/teams/team.repository.interface.js";
|
||||
import { IUsersRepository } from "@/domain/users/user.repository.interface.js";
|
||||
import { IProxiesRepository } from "@/domain/proxies/proxy.repository.interface.js";
|
||||
import { IEgressStateRepository } from "@/domain/egress/egress-state.repository.interface.js";
|
||||
|
||||
// Mongo repository implementations
|
||||
import MongoSettingsRepository from "@/domain/app-settings/app-settings.repository.mongo.js";
|
||||
@@ -75,6 +77,7 @@ import MongoTagsRepository from "@/domain/tags/tag.repository.mongo.js";
|
||||
import MongoTeamsRepository from "@/domain/teams/team.repository.model.js";
|
||||
import MongoUsersRepository from "@/domain/users/user.repository.mongo.js";
|
||||
import MongoProxiesRepository from "@/domain/proxies/proxy.repository.mongo.js";
|
||||
import MongoEgressStateRepository from "@/domain/egress/egress-state.repository.mongo.js";
|
||||
import { IDockerLogsRepository } from "@/domain/docker/docker-log.repository.interface.js";
|
||||
|
||||
// Shared infrastructure + business services that both the API and the worker process construct.
|
||||
@@ -83,6 +86,7 @@ export interface SharedServices {
|
||||
db: IDb<Mongoose>;
|
||||
settingsService: ISettingsService;
|
||||
emailService: IEmailService;
|
||||
encryptionService: IEncryptionService;
|
||||
notificationMessageBuilder: INotificationMessageBuilder;
|
||||
incidentService: IIncidentService;
|
||||
checkService: ICheckService;
|
||||
@@ -112,19 +116,25 @@ export interface SharedServices {
|
||||
teamsRepository: ITeamsRepository;
|
||||
maintenanceWindowsRepository: IMaintenanceWindowsRepository;
|
||||
proxiesRepository: IProxiesRepository;
|
||||
egressStateRepository: IEgressStateRepository;
|
||||
}
|
||||
|
||||
export const buildShared = async ({
|
||||
logger,
|
||||
envSettings,
|
||||
settingsService,
|
||||
encryptionKeys,
|
||||
}: {
|
||||
logger: ILogger;
|
||||
envSettings: EnvConfig;
|
||||
settingsService: ISettingsService;
|
||||
encryptionKeys: Buffer[];
|
||||
}): Promise<SharedServices> => {
|
||||
// Shared worker ID
|
||||
const workerId = `${hostname()}:${process.pid}:${randomUUID()}`;
|
||||
const workerId = `${hostname()}:${process.pid}:${nodeCrypto.randomUUID()}`;
|
||||
|
||||
// Create the encryption service
|
||||
const encryptionService = new EncryptionService(encryptionKeys, logger, nodeCrypto);
|
||||
|
||||
// Create DB
|
||||
let db: IDb<Mongoose> | null = null;
|
||||
@@ -150,6 +160,7 @@ export const buildShared = async ({
|
||||
const teamsRepository = new MongoTeamsRepository();
|
||||
const maintenanceWindowsRepository = new MongoMaintenanceWindowsRepository();
|
||||
const proxiesRepository = new MongoProxiesRepository();
|
||||
const egressStateRepository = new MongoEgressStateRepository();
|
||||
|
||||
// Inject settings repository into settings service (now that DB is connected)
|
||||
(settingsService as SettingsService).setRepository(settingsRepository);
|
||||
@@ -213,6 +224,7 @@ export const buildShared = async ({
|
||||
db,
|
||||
settingsService,
|
||||
emailService,
|
||||
encryptionService,
|
||||
notificationMessageBuilder,
|
||||
incidentService,
|
||||
checkService,
|
||||
@@ -238,6 +250,7 @@ export const buildShared = async ({
|
||||
teamsRepository,
|
||||
maintenanceWindowsRepository,
|
||||
proxiesRepository,
|
||||
egressStateRepository,
|
||||
};
|
||||
return sharedServices;
|
||||
};
|
||||
@@ -16,6 +16,7 @@ import { SharedServices } from "@/config/services.shared.js";
|
||||
import { INetworkService, NetworkService } from "@/service/networkService.js";
|
||||
import { IBufferService, BufferService } from "@/service/bufferService.js";
|
||||
import { IStatusService, StatusService } from "@/service/statusService.js";
|
||||
import { SecretsRotationService } from "@/service/encryption/secretsRotationService.js";
|
||||
import { IQueueWorker } from "@/worker/worker.interface.js";
|
||||
import { MonitorStatusPolicy } from "@/worker/worker.monitor-status-policy.js";
|
||||
import { WorkerHelper } from "@/worker/worker.helper.js";
|
||||
@@ -27,6 +28,7 @@ import { IncidentReactor } from "@/worker/reactors/reactor.incident.js";
|
||||
import { ReactorDispatcher } from "@/worker/reactors/reactor.dispatcher.js";
|
||||
import { DBQueueWorker } from "@/worker/worker.db-queue.js";
|
||||
import { ProxyResolver } from "@/service/network/ProxyResolver.js";
|
||||
import { IEgressService, EgressService } from "@/domain/egress/egress.service.js";
|
||||
// Network providers
|
||||
import { PingProvider } from "@/service/network/PingProvider.js";
|
||||
import { HttpProvider } from "@/service/network/HttpProvider.js";
|
||||
@@ -39,11 +41,13 @@ import { GameProvider } from "@/service/network/GameProvider.js";
|
||||
import { GrpcProvider } from "@/service/network/GrpcProvider.js";
|
||||
import { WebSocketProvider } from "@/service/network/WebSocketProvider.js";
|
||||
import { DNSProvider } from "@/service/network/DNSProvider.js";
|
||||
import { AppError } from "@/utils/AppError.js";
|
||||
export interface WorkerServices {
|
||||
worker: IQueueWorker;
|
||||
networkService: INetworkService;
|
||||
bufferService: IBufferService;
|
||||
statusService: IStatusService;
|
||||
egressService: IEgressService;
|
||||
}
|
||||
|
||||
export const buildWorker = async (shared: SharedServices, envSettings: EnvConfig): Promise<WorkerServices> => {
|
||||
@@ -59,6 +63,7 @@ export const buildWorker = async (shared: SharedServices, envSettings: EnvConfig
|
||||
jobsRepository,
|
||||
queueWorkersRepository,
|
||||
monitorsRepository,
|
||||
encryptionService,
|
||||
checksRepository,
|
||||
geoChecksRepository,
|
||||
dockerLogsRepository,
|
||||
@@ -67,16 +72,17 @@ export const buildWorker = async (shared: SharedServices, envSettings: EnvConfig
|
||||
teamsRepository,
|
||||
maintenanceWindowsRepository,
|
||||
proxiesRepository,
|
||||
egressStateRepository,
|
||||
} = shared;
|
||||
|
||||
// ***********************
|
||||
// Network providers
|
||||
// ***********************
|
||||
const pingProvider = new PingProvider(ping);
|
||||
const pingProvider = new PingProvider(ping, net);
|
||||
const httpProvider = new HttpProvider(got, new AdvancedMatcher(jmespath));
|
||||
const pageSpeedProvider = new PageSpeedProvider(httpProvider, settingsService, logger);
|
||||
const hardwareProvider = new HardwareProvider(httpProvider);
|
||||
const dockerProvider = new DockerProvider(logger, Docker);
|
||||
const dockerProvider = new DockerProvider(logger, Docker, encryptionService);
|
||||
const portProvider = new PortProvider(net);
|
||||
const gameProvider = new GameProvider(logger, GameDig);
|
||||
const grpcProvider = new GrpcProvider(grpc, protoLoader);
|
||||
@@ -100,6 +106,7 @@ export const buildWorker = async (shared: SharedServices, envSettings: EnvConfig
|
||||
const bufferService = new BufferService(logger, checkService, geoChecksService, dockerLogsService, settingsService, jobsRepository);
|
||||
const statusService = new StatusService(logger, monitorsRepository, monitorStatsRepository);
|
||||
const monitorStatusPolicy = new MonitorStatusPolicy();
|
||||
const egressService = new EgressService(settingsService, egressStateRepository, jobsRepository, networkService, proxyResolver, logger);
|
||||
|
||||
// ***********************
|
||||
// Reactors and dispatcher
|
||||
@@ -142,7 +149,8 @@ export const buildWorker = async (shared: SharedServices, envSettings: EnvConfig
|
||||
checksRepository,
|
||||
incidentsRepository,
|
||||
geoChecksRepository,
|
||||
dockerLogsRepository
|
||||
dockerLogsRepository,
|
||||
egressService
|
||||
);
|
||||
|
||||
const worker = await DBQueueWorker.create({
|
||||
@@ -164,5 +172,23 @@ export const buildWorker = async (shared: SharedServices, envSettings: EnvConfig
|
||||
workerId,
|
||||
});
|
||||
|
||||
return { worker, networkService, bufferService, statusService };
|
||||
const secretsRotationService = new SecretsRotationService(monitorsRepository, encryptionService, logger);
|
||||
try {
|
||||
const result = await secretsRotationService.run();
|
||||
logger.debug({
|
||||
message: "Secrets rotation run successfully",
|
||||
service: "WorkerServices",
|
||||
method: "buildWorker",
|
||||
details: { ...result },
|
||||
});
|
||||
} catch (error: unknown) {
|
||||
logger.warn({
|
||||
message: `Could not rotate Docker TLS secrets: ${error instanceof Error ? error.message : String(error)}`,
|
||||
service: "WorkerServices",
|
||||
method: "buildWorker",
|
||||
details: { ...(error instanceof AppError ? error.details : {}) },
|
||||
});
|
||||
}
|
||||
|
||||
return { worker, networkService, bufferService, statusService, egressService };
|
||||
};
|
||||
@@ -6,6 +6,7 @@ export interface ISettingsRepository {
|
||||
findSingleton(): Promise<Settings | null>;
|
||||
// update
|
||||
update(settings: SettingsUpdate): Promise<Settings>;
|
||||
removeEgressNotification(notificationId: string): Promise<void>; // $pull, so it cannot race a concurrent settings update
|
||||
// delete
|
||||
deleteLegacy: () => Promise<boolean>;
|
||||
}
|
||||
@@ -1,5 +1,6 @@
|
||||
import { Schema, model, type Types } from "mongoose";
|
||||
import type { Settings, SettingsThresholds } from "@/domain/app-settings/app-settings.type.js";
|
||||
import { DEFAULT_EGRESS_POLL_INTERVAL_SECONDS, DEFAULT_EGRESS_TARGETS } from "@/domain/egress/egress.type.js";
|
||||
|
||||
interface AppSettingsDocument extends Omit<Settings, "id" | "createdAt" | "updatedAt"> {
|
||||
_id: Types.ObjectId;
|
||||
@@ -42,6 +43,10 @@ const AppSettingsSchema = new Schema<AppSettingsDocument>(
|
||||
globalThresholds: { type: thresholdsSchema },
|
||||
globalProxyEnabled: { type: Boolean, default: false },
|
||||
globalProxyId: { type: String, default: null },
|
||||
egressCheckEnabled: { type: Boolean, default: false },
|
||||
egressCheckTargets: { type: [String], default: () => [...DEFAULT_EGRESS_TARGETS] },
|
||||
egressPollIntervalSeconds: { type: Number, default: DEFAULT_EGRESS_POLL_INTERVAL_SECONDS },
|
||||
egressNotifications: { type: [String], default: () => [] }, // Notification ids, stored as strings like globalProxyId
|
||||
},
|
||||
{ timestamps: true }
|
||||
);
|
||||
|
||||
@@ -3,6 +3,7 @@ import { ISettingsRepository } from "@/domain/app-settings/app-settings-reposito
|
||||
import type { Settings, SettingsUpdate } from "@/domain/app-settings/app-settings.type.js";
|
||||
import { AppSettingsModel, type AppSettingsDocument } from "@/domain/app-settings/app-settings.model.js";
|
||||
import { toStringId, toDateString } from "@/utils/mongoMappers.js";
|
||||
import { DEFAULT_EGRESS_POLL_INTERVAL_SECONDS, DEFAULT_EGRESS_TARGETS } from "@/domain/egress/egress.type.js";
|
||||
|
||||
class MongoSettingsRepository implements ISettingsRepository {
|
||||
private toEntity = (doc: AppSettingsDocument): Settings => {
|
||||
@@ -30,6 +31,10 @@ class MongoSettingsRepository implements ISettingsRepository {
|
||||
globalThresholds: doc.globalThresholds ?? undefined,
|
||||
globalProxyEnabled: doc.globalProxyEnabled ?? false,
|
||||
globalProxyId: doc.globalProxyId ?? undefined,
|
||||
egressCheckEnabled: doc.egressCheckEnabled ?? false,
|
||||
egressCheckTargets: doc.egressCheckTargets ?? [...DEFAULT_EGRESS_TARGETS],
|
||||
egressPollIntervalSeconds: doc.egressPollIntervalSeconds ?? DEFAULT_EGRESS_POLL_INTERVAL_SECONDS,
|
||||
egressNotifications: doc.egressNotifications ?? [],
|
||||
createdAt: toDateString(doc.createdAt),
|
||||
updatedAt: toDateString(doc.updatedAt),
|
||||
};
|
||||
@@ -79,6 +84,10 @@ class MongoSettingsRepository implements ISettingsRepository {
|
||||
return this.toEntity(updatedSettings);
|
||||
};
|
||||
|
||||
removeEgressNotification = async (notificationId: string) => {
|
||||
await AppSettingsModel.updateMany({ egressNotifications: notificationId }, { $pull: { egressNotifications: notificationId } });
|
||||
};
|
||||
|
||||
deleteLegacy = async () => {
|
||||
const res = await AppSettingsModel.deleteMany({ version: { $exists: false } });
|
||||
return res.deletedCount > 0;
|
||||
|
||||
@@ -23,15 +23,27 @@ export interface ISettingsService {
|
||||
getSettings(): EnvConfig;
|
||||
areStatusPageThemesEnabled(): boolean;
|
||||
getDBSettings(): Promise<Settings>;
|
||||
getCachedDBSettings(): Promise<Settings>;
|
||||
updateDbSettings(newSettings: SettingsUpdate): Promise<Settings>;
|
||||
removeEgressNotification(notificationId: string): Promise<void>;
|
||||
}
|
||||
|
||||
const DB_SETTINGS_CACHE_TTL_MS = 60_000;
|
||||
|
||||
type CachedSettings = {
|
||||
value: Promise<Settings>;
|
||||
expiresAt: number;
|
||||
};
|
||||
export class SettingsService implements ISettingsService {
|
||||
static SERVICE_NAME = SERVICE_NAME;
|
||||
private settings: EnvConfig;
|
||||
private settingsRepository: ISettingsRepository | null = null;
|
||||
private dbSettingsCache: CachedSettings | null = null;
|
||||
|
||||
constructor(env: ValidatedEnv) {
|
||||
constructor(
|
||||
env: ValidatedEnv,
|
||||
private dbSettingsTTLMs: number = DB_SETTINGS_CACHE_TTL_MS
|
||||
) {
|
||||
this.settings = {
|
||||
jwtSecret: env.JWT_SECRET,
|
||||
jwtTTL: env.TOKEN_TTL as StringValue,
|
||||
@@ -75,7 +87,14 @@ export class SettingsService implements ISettingsService {
|
||||
}
|
||||
|
||||
updateDbSettings = async (newSettings: SettingsUpdate) => {
|
||||
return await this.getRepository().update(newSettings);
|
||||
const updated = await this.getRepository().update(newSettings);
|
||||
this.dbSettingsCache = null;
|
||||
return updated;
|
||||
};
|
||||
|
||||
removeEgressNotification = async (notificationId: string) => {
|
||||
await this.getRepository().removeEgressNotification(notificationId);
|
||||
this.dbSettingsCache = null;
|
||||
};
|
||||
|
||||
getDBSettings = async () => {
|
||||
@@ -95,4 +114,17 @@ export class SettingsService implements ISettingsService {
|
||||
|
||||
return settings;
|
||||
};
|
||||
|
||||
getCachedDBSettings = async () => {
|
||||
const hit = this.dbSettingsCache;
|
||||
if (hit && hit.expiresAt > Date.now()) {
|
||||
return hit.value;
|
||||
}
|
||||
const entry: CachedSettings = { value: this.getDBSettings(), expiresAt: Date.now() + this.dbSettingsTTLMs };
|
||||
this.dbSettingsCache = entry;
|
||||
entry.value.catch(() => {
|
||||
if (this.dbSettingsCache === entry) this.dbSettingsCache = null;
|
||||
});
|
||||
return entry.value;
|
||||
};
|
||||
}
|
||||
@@ -51,6 +51,11 @@ export interface Settings {
|
||||
globalThresholds?: SettingsThresholds;
|
||||
globalProxyEnabled: boolean;
|
||||
globalProxyId?: string | null;
|
||||
// Egress self-check (see domain/egress)
|
||||
egressCheckEnabled: boolean;
|
||||
egressCheckTargets: string[];
|
||||
egressPollIntervalSeconds: number; // recovery poll cadence while degraded, not a general schedule
|
||||
egressNotifications: string[]; // Notification ids to receive the "egress recovered" alert
|
||||
createdAt: string;
|
||||
updatedAt: string;
|
||||
}
|
||||
|
||||
@@ -0,0 +1,40 @@
|
||||
import { Schema, model, type Types } from "mongoose";
|
||||
import { EgressStatuses, type EgressProbeResult, type EgressState } from "@/domain/egress/egress.type.js";
|
||||
|
||||
interface EgressStateDocument extends Omit<EgressState, "id" | "degradedSince" | "lastRecoveredAt" | "lastProbeAt" | "createdAt" | "updatedAt"> {
|
||||
_id: Types.ObjectId;
|
||||
singleton: boolean;
|
||||
degradedSince: Date | null;
|
||||
lastRecoveredAt: Date | null;
|
||||
lastProbeAt: Date | null;
|
||||
createdAt: Date;
|
||||
updatedAt: Date;
|
||||
}
|
||||
|
||||
const probeResultSchema = new Schema<EgressProbeResult>(
|
||||
{
|
||||
target: { type: String, required: true },
|
||||
reachable: { type: Boolean, required: true },
|
||||
responseTime: { type: Number, default: 0 },
|
||||
message: { type: String },
|
||||
},
|
||||
{ _id: false }
|
||||
);
|
||||
|
||||
const EgressStateSchema = new Schema<EgressStateDocument>(
|
||||
{
|
||||
singleton: { type: Boolean, required: true, unique: true, default: true },
|
||||
status: { type: String, enum: EgressStatuses, required: true, default: "ok" },
|
||||
degradedSince: { type: Date, default: null },
|
||||
lastRecoveredAt: { type: Date, default: null },
|
||||
lastProbeAt: { type: Date, default: null },
|
||||
lastProbeResults: { type: [probeResultSchema], default: () => [] },
|
||||
},
|
||||
{ timestamps: true }
|
||||
);
|
||||
|
||||
const EgressStateModel = model<EgressStateDocument>("EgressState", EgressStateSchema);
|
||||
|
||||
export type { EgressStateDocument };
|
||||
export { EgressStateModel };
|
||||
export default EgressStateModel;
|
||||
@@ -0,0 +1,13 @@
|
||||
import type { EgressProbeResult, EgressState } from "@/domain/egress/egress.type.js";
|
||||
|
||||
export interface IEgressStateRepository {
|
||||
// read
|
||||
findSingleton(): Promise<EgressState>; // creates the "ok" singleton on first call, a plain read after that
|
||||
// update
|
||||
recordProbe(results: EgressProbeResult[], now: Date): Promise<EgressState>;
|
||||
reset(): Promise<EgressState>; // back to "ok" with no episode; used when the feature is toggled
|
||||
// Atomic transitions. Each returns the updated state when this caller performed the transition,
|
||||
// or null when the state was already on the target side (another worker got there first).
|
||||
markDegraded(results: EgressProbeResult[], now: Date): Promise<EgressState | null>;
|
||||
markRecovered(results: EgressProbeResult[], now: Date): Promise<EgressState | null>;
|
||||
}
|
||||
@@ -0,0 +1,88 @@
|
||||
import { IEgressStateRepository } from "@/domain/egress/egress-state.repository.interface.js";
|
||||
import type { EgressProbeResult, EgressState } from "@/domain/egress/egress.type.js";
|
||||
import { EgressStateModel, type EgressStateDocument } from "@/domain/egress/egress-state.model.js";
|
||||
import { toStringId, toDateString } from "@/utils/mongoMappers.js";
|
||||
|
||||
const SERVICE_NAME = "EgressStateRepository";
|
||||
|
||||
class MongoEgressStateRepository implements IEgressStateRepository {
|
||||
static SERVICE_NAME = SERVICE_NAME;
|
||||
|
||||
private toEntity = (doc: EgressStateDocument): EgressState => {
|
||||
return {
|
||||
id: toStringId(doc._id),
|
||||
status: doc.status,
|
||||
degradedSince: doc.degradedSince ? toDateString(doc.degradedSince) : null,
|
||||
lastRecoveredAt: doc.lastRecoveredAt ? toDateString(doc.lastRecoveredAt) : null,
|
||||
lastProbeAt: doc.lastProbeAt ? toDateString(doc.lastProbeAt) : null,
|
||||
lastProbeResults: (doc.lastProbeResults ?? []).map((result) => ({
|
||||
target: result.target,
|
||||
reachable: result.reachable,
|
||||
responseTime: result.responseTime ?? 0,
|
||||
...(result.message !== undefined && { message: result.message }),
|
||||
})),
|
||||
createdAt: toDateString(doc.createdAt),
|
||||
updatedAt: toDateString(doc.updatedAt),
|
||||
};
|
||||
};
|
||||
|
||||
findSingleton = async () => {
|
||||
// Read first: with timestamps on, every findOneAndUpdate would $set updatedAt, so a read would write.
|
||||
const existing = await EgressStateModel.findOne({ singleton: true }).lean<EgressStateDocument>();
|
||||
if (existing) return this.toEntity(existing);
|
||||
try {
|
||||
const doc = await EgressStateModel.findOneAndUpdate(
|
||||
{ singleton: true },
|
||||
{ $setOnInsert: { singleton: true, status: "ok" } },
|
||||
{ upsert: true, new: true, setDefaultsOnInsert: true }
|
||||
).lean<EgressStateDocument>();
|
||||
return this.toEntity(doc);
|
||||
} catch (error: unknown) {
|
||||
// Two processes can race the very first insert; the unique index rejects the loser, who reads the winner's row.
|
||||
if ((error as { code?: number }).code === 11000) {
|
||||
const doc = await EgressStateModel.findOne({ singleton: true }).lean<EgressStateDocument>();
|
||||
if (doc) return this.toEntity(doc);
|
||||
}
|
||||
throw error;
|
||||
}
|
||||
};
|
||||
|
||||
reset = async () => {
|
||||
const doc = await EgressStateModel.findOneAndUpdate(
|
||||
{ singleton: true },
|
||||
{ $set: { status: "ok", degradedSince: null, lastProbeAt: null, lastProbeResults: [] }, $setOnInsert: { singleton: true } },
|
||||
{ upsert: true, new: true, setDefaultsOnInsert: true }
|
||||
).lean<EgressStateDocument>();
|
||||
return this.toEntity(doc);
|
||||
};
|
||||
|
||||
recordProbe = async (results: EgressProbeResult[], now: Date) => {
|
||||
const doc = await EgressStateModel.findOneAndUpdate(
|
||||
{ singleton: true },
|
||||
{ $set: { lastProbeAt: now, lastProbeResults: results }, $setOnInsert: { singleton: true, status: "ok" } },
|
||||
{ upsert: true, new: true, setDefaultsOnInsert: true }
|
||||
).lean<EgressStateDocument>();
|
||||
return this.toEntity(doc);
|
||||
};
|
||||
|
||||
markDegraded = async (results: EgressProbeResult[], now: Date) => {
|
||||
// Only the caller that flips ok -> degraded sees a document back; everyone else sees null.
|
||||
const doc = await EgressStateModel.findOneAndUpdate(
|
||||
{ singleton: true, status: "ok" },
|
||||
{ $set: { status: "degraded", degradedSince: now, lastProbeAt: now, lastProbeResults: results } },
|
||||
{ new: true }
|
||||
).lean<EgressStateDocument>();
|
||||
return doc ? this.toEntity(doc) : null;
|
||||
};
|
||||
|
||||
markRecovered = async (results: EgressProbeResult[], now: Date) => {
|
||||
const doc = await EgressStateModel.findOneAndUpdate(
|
||||
{ singleton: true, status: "degraded" },
|
||||
{ $set: { status: "ok", lastRecoveredAt: now, lastProbeAt: now, lastProbeResults: results } },
|
||||
{ new: true }
|
||||
).lean<EgressStateDocument>();
|
||||
return doc ? this.toEntity(doc) : null;
|
||||
};
|
||||
}
|
||||
|
||||
export default MongoEgressStateRepository;
|
||||
@@ -0,0 +1,31 @@
|
||||
import { IEgressStateRepository } from "@/domain/egress/egress-state.repository.interface.js";
|
||||
import { IJobsRepository } from "@/domain/jobs/job.repository.interface.js";
|
||||
import type { EgressState } from "@/domain/egress/egress.type.js";
|
||||
|
||||
export interface IEgressStateService {
|
||||
getState(): Promise<EgressState>;
|
||||
reset(): Promise<EgressState>;
|
||||
}
|
||||
|
||||
const SERVICE_NAME = "EgressStateService";
|
||||
|
||||
// API-process view of the instance-global egress state. Probe-driven transitions are owned by the
|
||||
// worker (egress.service.ts); this exists so the controller never touches the repositories.
|
||||
export class EgressStateService implements IEgressStateService {
|
||||
static SERVICE_NAME = SERVICE_NAME;
|
||||
|
||||
constructor(
|
||||
private egressStateRepository: IEgressStateRepository,
|
||||
private jobsRepository: IJobsRepository
|
||||
) {}
|
||||
|
||||
getState = async (): Promise<EgressState> => {
|
||||
return this.egressStateRepository.findSingleton();
|
||||
};
|
||||
|
||||
// Called when the feature is switched on or off: drop any pending recovery job and start from "ok".
|
||||
reset = async (): Promise<EgressState> => {
|
||||
await this.jobsRepository.deleteGlobalJob("egress");
|
||||
return this.egressStateRepository.reset();
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,283 @@
|
||||
import type { Monitor } from "@/domain/monitors/monitor.type.js";
|
||||
import type { ISettingsService } from "@/domain/app-settings/app-settings.service.js";
|
||||
import type { IEgressStateRepository } from "@/domain/egress/egress-state.repository.interface.js";
|
||||
import type { IJobsRepository } from "@/domain/jobs/job.repository.interface.js";
|
||||
import { jobId, type Job, type JobSeed } from "@/domain/jobs/job.type.js";
|
||||
import type { INetworkService } from "@/service/networkService.js";
|
||||
import type { IProxyResolver } from "@/service/network/ProxyResolver.js";
|
||||
import type { ILogger } from "@/utils/logger.js";
|
||||
import {
|
||||
DEFAULT_EGRESS_POLL_INTERVAL_SECONDS,
|
||||
DEFAULT_EGRESS_TARGETS,
|
||||
type EgressAssessment,
|
||||
type EgressProbeResult,
|
||||
type EgressStatus,
|
||||
} from "@/domain/egress/egress.type.js";
|
||||
import { timeRequest } from "@/service/network/utils.js";
|
||||
import { parseEgressTarget, type EgressTarget } from "@/utils/egressTarget.js";
|
||||
|
||||
const SERVICE_NAME = "EgressService";
|
||||
const PROBE_TIMEOUT_MS = 5000;
|
||||
// How long a settled assessment answers for later failures. An assessment still in flight is shared
|
||||
// for as long as it takes, so this only bounds how stale a reused answer can be.
|
||||
const ASSESSMENT_REUSE_MS = 5000;
|
||||
// Global job row (refId null) that re-probes while degraded. Inserted when egress degrades, removed on recovery.
|
||||
const RECOVERY_JOB_TYPE = "egress" as const;
|
||||
const RECOVERY_JOB_ID = jobId(RECOVERY_JOB_TYPE, null);
|
||||
|
||||
// HttpProvider reports transport failures with NETWORK_ERROR (outside the HTTP range) and everything else with the real status code.
|
||||
const isHttpStatusCode = (code: unknown): boolean => typeof code === "number" && code >= 100 && code <= 599;
|
||||
|
||||
// Synthetic identity stamped on the probe monitors so provider responses are recognisable in logs.
|
||||
const PROBE_MONITOR_ID = "egress-probe";
|
||||
const PROBE_TEAM_ID = "system";
|
||||
|
||||
export interface IEgressService {
|
||||
probeTargets(targets: string[]): Promise<EgressProbeResult[]>;
|
||||
// Called by the check producer when a check fails. Null when the feature is off or the assessment itself failed.
|
||||
assessAfterFailure(): Promise<EgressStatus | null>;
|
||||
// Handler for the "egress" job: re-probes while degraded and removes the job once egress is back.
|
||||
checkRecovery(job: Job): Promise<void>;
|
||||
}
|
||||
|
||||
type CachedAssessment = {
|
||||
value: Promise<EgressAssessment | null>;
|
||||
expiresAt: number | null; // null while the assessment is still in flight
|
||||
};
|
||||
|
||||
export class EgressService implements IEgressService {
|
||||
static SERVICE_NAME = SERVICE_NAME;
|
||||
|
||||
private assessment: CachedAssessment | null = null;
|
||||
|
||||
constructor(
|
||||
private settingsService: ISettingsService,
|
||||
private egressStateRepository: IEgressStateRepository,
|
||||
private jobsRepository: IJobsRepository,
|
||||
private networkService: INetworkService,
|
||||
private proxyResolver: IProxyResolver,
|
||||
private logger: ILogger
|
||||
) {}
|
||||
|
||||
// ****************************
|
||||
// Probing
|
||||
// ****************************
|
||||
|
||||
// Reuses the existing ping/port/http providers by dressing each target up as a minimal monitor.
|
||||
// proxyMode "inherit" lets the resolver apply the global proxy to http probes, as it does for monitors.
|
||||
private toProbeMonitor = (target: EgressTarget, raw: string): Monitor => {
|
||||
const base = {
|
||||
id: PROBE_MONITOR_ID,
|
||||
teamId: PROBE_TEAM_ID,
|
||||
name: `Egress probe: ${raw}`,
|
||||
method: "GET",
|
||||
proxyMode: "inherit",
|
||||
useAdvancedMatching: false,
|
||||
ignoreTlsErrors: false,
|
||||
customUpCodes: [],
|
||||
};
|
||||
|
||||
switch (target.kind) {
|
||||
case "http":
|
||||
return { ...base, type: "http", url: target.url } as unknown as Monitor;
|
||||
case "port":
|
||||
return { ...base, type: "port", url: target.host, port: target.port } as unknown as Monitor;
|
||||
case "ping":
|
||||
return { ...base, type: "ping", url: target.host } as unknown as Monitor;
|
||||
}
|
||||
};
|
||||
|
||||
private probeTarget = async (raw: string): Promise<EgressProbeResult> => {
|
||||
const target = parseEgressTarget(raw);
|
||||
if (!target) {
|
||||
// Validation rejects these; a stored value can still be stale, and it must not count as reachable.
|
||||
return { target: raw, reachable: false, responseTime: 0, message: "Invalid egress target" };
|
||||
}
|
||||
|
||||
const monitor = this.toProbeMonitor(target, raw);
|
||||
let timer: NodeJS.Timeout | undefined;
|
||||
|
||||
const { response, responseTime, error } = await timeRequest(async () => {
|
||||
const proxyUrl = await this.proxyResolver.resolve(monitor);
|
||||
return Promise.race([
|
||||
this.networkService.requestStatus(monitor, { proxyUrl }),
|
||||
new Promise<never>((_, reject) => {
|
||||
timer = setTimeout(() => reject(new Error(`Egress probe timed out after ${PROBE_TIMEOUT_MS}ms`)), PROBE_TIMEOUT_MS);
|
||||
}),
|
||||
]).finally(() => clearTimeout(timer));
|
||||
});
|
||||
|
||||
if (error || !response) {
|
||||
return {
|
||||
target: raw,
|
||||
reachable: false,
|
||||
responseTime,
|
||||
message: error instanceof Error ? error.message : error ? String(error) : "No response",
|
||||
};
|
||||
}
|
||||
|
||||
return {
|
||||
target: raw,
|
||||
// Any HTTP response, including 4xx/5xx, proves the instance can reach the target; only a transport
|
||||
// failure counts against egress. Ping and port providers have no such distinction.
|
||||
reachable: response.status === true || (target.kind === "http" && isHttpStatusCode(response.code)),
|
||||
responseTime: response.responseTime ?? responseTime,
|
||||
message: response.message,
|
||||
};
|
||||
};
|
||||
|
||||
probeTargets = async (targets: string[]): Promise<EgressProbeResult[]> => {
|
||||
return await Promise.all(targets.map((target) => this.probeTarget(target)));
|
||||
};
|
||||
|
||||
private resolveTargets = (configured: string[] | undefined): string[] => {
|
||||
const targets = (configured ?? []).map((target) => target.trim()).filter((target) => target.length > 0);
|
||||
// An empty list would make "all unreachable" vacuously true, so fall back to the defaults.
|
||||
return targets.length > 0 ? targets : [...DEFAULT_EGRESS_TARGETS];
|
||||
};
|
||||
|
||||
private toIntervalMs = (seconds: number | undefined): number => {
|
||||
const valid = typeof seconds === "number" && Number.isFinite(seconds) && seconds > 0;
|
||||
return (valid ? seconds : DEFAULT_EGRESS_POLL_INTERVAL_SECONDS) * 1000;
|
||||
};
|
||||
|
||||
// ****************************
|
||||
// Event-triggered assessment
|
||||
// ****************************
|
||||
|
||||
assessAfterFailure = async (): Promise<EgressStatus | null> => {
|
||||
const assessment = await this.sharedAssessment();
|
||||
return assessment === "disabled" ? null : assessment;
|
||||
};
|
||||
|
||||
// When egress is down every monitor fails at once. Concurrent failures share the in-flight assessment
|
||||
// however long it takes, and a settled one answers for ASSESSMENT_REUSE_MS. Same shape as ProxyResolver's cache.
|
||||
private sharedAssessment = (): Promise<EgressAssessment | null> => {
|
||||
const cached = this.assessment;
|
||||
if (cached && (cached.expiresAt === null || cached.expiresAt > Date.now())) {
|
||||
return cached.value;
|
||||
}
|
||||
|
||||
const entry: CachedAssessment = { value: this.assess(), expiresAt: null };
|
||||
this.assessment = entry;
|
||||
void entry.value.then((result) => {
|
||||
if (this.assessment !== entry) return;
|
||||
if (result === null) {
|
||||
this.assessment = null; // an internal failure is not cached, so the next failing check retries
|
||||
} else {
|
||||
entry.expiresAt = Date.now() + ASSESSMENT_REUSE_MS;
|
||||
}
|
||||
});
|
||||
return entry.value;
|
||||
};
|
||||
|
||||
private assess = async (): Promise<EgressAssessment | null> => {
|
||||
try {
|
||||
const settings = await this.settingsService.getCachedDBSettings();
|
||||
if (!settings.egressCheckEnabled) {
|
||||
return "disabled";
|
||||
}
|
||||
|
||||
const state = await this.egressStateRepository.findSingleton();
|
||||
if (state.status === "degraded") {
|
||||
// Recovery is detected by the scheduled job. Make sure the row exists without touching its schedule.
|
||||
await this.scheduleRecoveryCheck(settings.egressPollIntervalSeconds, false);
|
||||
return "degraded";
|
||||
}
|
||||
|
||||
const results = await this.probeTargets(this.resolveTargets(settings.egressCheckTargets));
|
||||
const now = new Date();
|
||||
|
||||
if (results.some((result) => result.reachable)) {
|
||||
await this.egressStateRepository.recordProbe(results, now);
|
||||
return "ok";
|
||||
}
|
||||
|
||||
// Schedule the recovery job before recording the transition: a crash between the two then leaves a stray row
|
||||
// while ok, which the job's first run removes, rather than a degraded state that nothing is polling.
|
||||
await this.scheduleRecoveryCheck(settings.egressPollIntervalSeconds, true);
|
||||
const degraded = await this.egressStateRepository.markDegraded(results, now);
|
||||
this.logger.warn({
|
||||
message: degraded
|
||||
? "Instance egress degraded: every reliability target is unreachable. Monitor failures will not be counted until it recovers"
|
||||
: "Instance egress degraded (transition already recorded by another worker)",
|
||||
service: SERVICE_NAME,
|
||||
method: "assessAfterFailure",
|
||||
details: { results },
|
||||
});
|
||||
return "degraded";
|
||||
} catch (error: unknown) {
|
||||
// The egress check must never break check production; treat an internal failure as "unknown" and carry on.
|
||||
this.logger.error({
|
||||
message: error instanceof Error ? error.message : String(error),
|
||||
service: SERVICE_NAME,
|
||||
method: "assessAfterFailure",
|
||||
stack: error instanceof Error ? error.stack : undefined,
|
||||
});
|
||||
return null;
|
||||
}
|
||||
};
|
||||
|
||||
// ****************************
|
||||
// Recovery job
|
||||
// ****************************
|
||||
|
||||
// reschedule=true sets nextScheduledAt (start of an episode); false only inserts the row if it is missing,
|
||||
// so re-arming while degraded never pushes a pending run back.
|
||||
private scheduleRecoveryCheck = async (pollIntervalSeconds: number | undefined, reschedule: boolean) => {
|
||||
const intervalMs = this.toIntervalMs(pollIntervalSeconds);
|
||||
const seed: JobSeed = {
|
||||
id: RECOVERY_JOB_ID,
|
||||
type: RECOVERY_JOB_TYPE,
|
||||
refId: null,
|
||||
isActive: true,
|
||||
nextScheduledAt: Date.now() + intervalMs,
|
||||
intervalMs,
|
||||
};
|
||||
if (reschedule) {
|
||||
await this.jobsRepository.upsertCleanupJob(seed); // global-row upsert that also sets the schedule
|
||||
} else {
|
||||
await this.jobsRepository.upsertJob(seed);
|
||||
}
|
||||
};
|
||||
|
||||
// Removes the row only if nobody has rescheduled it since this run claimed it. Another worker may have seen
|
||||
// egress fail again in the meantime and started a new episode; its schedule must survive.
|
||||
private releaseRecoveryJob = async (job: Job) => {
|
||||
await this.jobsRepository.deleteGlobalJobIfUnchanged(RECOVERY_JOB_TYPE, job.nextScheduledAt);
|
||||
};
|
||||
|
||||
// Runs on the queue at the configured interval while degraded. Errors propagate so the queue records
|
||||
// the failure and retries with its usual backoff.
|
||||
checkRecovery = async (job: Job): Promise<void> => {
|
||||
const settings = await this.settingsService.getCachedDBSettings();
|
||||
const state = await this.egressStateRepository.findSingleton();
|
||||
if (!settings.egressCheckEnabled || state.status !== "degraded") {
|
||||
// Feature switched off, recovery recorded by another worker, or state reset: nothing left to poll for.
|
||||
await this.releaseRecoveryJob(job);
|
||||
return;
|
||||
}
|
||||
|
||||
const results = await this.probeTargets(this.resolveTargets(settings.egressCheckTargets));
|
||||
const now = new Date();
|
||||
|
||||
if (!results.some((result) => result.reachable)) {
|
||||
await this.egressStateRepository.recordProbe(results, now);
|
||||
return;
|
||||
}
|
||||
|
||||
const recovered = await this.egressStateRepository.markRecovered(results, now);
|
||||
await this.releaseRecoveryJob(job);
|
||||
if (!recovered) {
|
||||
// Another process performed the transition.
|
||||
return;
|
||||
}
|
||||
|
||||
this.logger.info({
|
||||
message: `Instance egress recovered (degraded since ${recovered.degradedSince ?? "unknown"})`,
|
||||
service: SERVICE_NAME,
|
||||
method: "checkRecovery",
|
||||
details: { results },
|
||||
});
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,35 @@
|
||||
// Egress self-check: detects loss of the instance's own outbound connectivity so that
|
||||
// a burst of simultaneous monitor failures is not reported as a burst of real outages.
|
||||
|
||||
export const EgressStatuses = ["ok", "degraded"] as const;
|
||||
export type EgressStatus = (typeof EgressStatuses)[number];
|
||||
// Outcome of one assessment: a status, or "disabled" when the feature is switched off (distinct from an internal failure).
|
||||
export type EgressAssessment = EgressStatus | "disabled";
|
||||
|
||||
// Sensible defaults for a fresh install. Bare hosts are ICMP-pinged, host:port is a TCP connect,
|
||||
// http(s):// URLs are fetched. Any one target being reachable means egress is fine.
|
||||
export const DEFAULT_EGRESS_TARGETS = ["1.1.1.1", "8.8.8.8"] as const;
|
||||
export const DEFAULT_EGRESS_POLL_INTERVAL_SECONDS = 30;
|
||||
export const MIN_EGRESS_POLL_INTERVAL_SECONDS = 5;
|
||||
export const MAX_EGRESS_POLL_INTERVAL_SECONDS = 600;
|
||||
export const MAX_EGRESS_TARGETS = 10;
|
||||
|
||||
// Instance-global singleton persisted in MongoDB so that every worker process and the API
|
||||
// process share one view of the state (the queue worker may run separately from the API).
|
||||
export interface EgressState {
|
||||
id: string;
|
||||
status: EgressStatus;
|
||||
degradedSince: string | null; // ISO timestamp of the current or most recent degraded entry
|
||||
lastRecoveredAt: string | null; // ISO timestamp of the most recent recovery
|
||||
lastProbeAt: string | null; // ISO timestamp of the most recent reliability probe
|
||||
lastProbeResults: EgressProbeResult[];
|
||||
createdAt: string;
|
||||
updatedAt: string;
|
||||
}
|
||||
|
||||
export interface EgressProbeResult {
|
||||
target: string;
|
||||
reachable: boolean;
|
||||
responseTime: number; // ms
|
||||
message?: string;
|
||||
}
|
||||
@@ -61,6 +61,11 @@ export interface IJobsRepository {
|
||||
// Delete a single job of type
|
||||
deleteByIdAndType(refId: string, type: JobType): Promise<boolean>;
|
||||
|
||||
// Global rows (refId null) are addressed by their canonical _id, see jobId
|
||||
deleteGlobalJob(type: JobType): Promise<boolean>;
|
||||
// Delete a global row only if nobody has rescheduled it since it was claimed (nextScheduledAt unchanged)
|
||||
deleteGlobalJobIfUnchanged(type: JobType, nextScheduledAt: number): Promise<boolean>;
|
||||
|
||||
// Delete jobs that reference monitors that no longer exist
|
||||
deleteByMonitorIdsNotIn(monitorIds: string[]): Promise<number>;
|
||||
|
||||
|
||||
@@ -107,6 +107,8 @@ class MongoJobsRepository implements IJobsRepository {
|
||||
lockedBy: null, // Remove lock
|
||||
lockedUntil: null,
|
||||
lastFinishedAt: now,
|
||||
failCount: 0,
|
||||
lastFailReason: null,
|
||||
},
|
||||
$inc: {
|
||||
runCount: 1, // Increment run count
|
||||
@@ -150,6 +152,8 @@ class MongoJobsRepository implements IJobsRepository {
|
||||
lockedBy: null, // Remove lock
|
||||
lockedUntil: null,
|
||||
lastFinishedAt: now,
|
||||
failCount: 0,
|
||||
lastFailReason: null,
|
||||
},
|
||||
$inc: {
|
||||
runCount: 1, // Increment run count
|
||||
@@ -278,6 +282,16 @@ class MongoJobsRepository implements IJobsRepository {
|
||||
return res.deletedCount > 0;
|
||||
};
|
||||
|
||||
deleteGlobalJob = async (type: JobType) => {
|
||||
const res = await JobModel.deleteOne({ _id: jobId(type, null) });
|
||||
return res.deletedCount > 0;
|
||||
};
|
||||
|
||||
deleteGlobalJobIfUnchanged = async (type: JobType, nextScheduledAt: number) => {
|
||||
const res = await JobModel.deleteOne({ _id: jobId(type, null), nextScheduledAt });
|
||||
return res.deletedCount > 0;
|
||||
};
|
||||
|
||||
deleteByMonitorIdsNotIn = async (monitorIds: string[]): Promise<number> => {
|
||||
// remove jobs who'se refId is not null (cleanup jobs) and not in the list of monitor IDs (orphaned monitor jobs)
|
||||
const res = await JobModel.deleteMany({ refId: { $ne: null, $nin: monitorIds } });
|
||||
|
||||
@@ -1,10 +1,10 @@
|
||||
export const JOB_TYPES = ["check", "geo-check", "evaluate", "cleanup-orphaned", "cleanup-retention"] as const;
|
||||
export const JOB_TYPES = ["check", "geo-check", "evaluate", "cleanup-orphaned", "cleanup-retention", "egress"] as const;
|
||||
export type JobType = (typeof JOB_TYPES)[number];
|
||||
export const LOCK_MS = 60_000;
|
||||
export const BACKOFF_MS = 5_000;
|
||||
export const PARKED = Number.MAX_SAFE_INTEGER;
|
||||
|
||||
// Canonical _id scheme: monitor-bound rows are `${type}:${refId}`; global rows (refId null) are just the type.
|
||||
// Canonical _id scheme: monitor-bound rows are `${type}:${refId}`; global rows (refId null: cleanup, egress) are just the type.
|
||||
export const jobId = (type: JobType, refId: string | null): string => (refId === null ? type : `${type}:${refId}`);
|
||||
|
||||
export type Job = {
|
||||
|
||||
@@ -307,6 +307,20 @@ const MonitorSchema = new Schema<MonitorDocument>(
|
||||
type: Boolean,
|
||||
default: false,
|
||||
},
|
||||
dockerTlsCa: {
|
||||
type: String,
|
||||
},
|
||||
dockerTlsCert: {
|
||||
type: String,
|
||||
},
|
||||
dockerTlsKey: {
|
||||
type: String,
|
||||
select: false,
|
||||
},
|
||||
dockerTlsKeySet: {
|
||||
type: Boolean,
|
||||
default: false,
|
||||
},
|
||||
dnsServer: {
|
||||
type: String,
|
||||
},
|
||||
|
||||
@@ -55,6 +55,11 @@ export interface IMonitorsRepository {
|
||||
findMonitorCountByTeamIdAndType(teamId: string, config: TeamQueryConfig): Promise<number>;
|
||||
findMonitorCountByProxyId(proxyId: string): Promise<number>;
|
||||
|
||||
// Docker TLS
|
||||
findDockerTlsKeyById(monitorId: string): Promise<string | null>;
|
||||
findAllDockerTlsKeys(): Promise<{ id: string; dockerTlsKey: string }[]>;
|
||||
updateDockerTlsKey(monitorId: string, dockerTlsKey: string): Promise<void>; // Needed for worker to rotate keys, no teamId/userId
|
||||
|
||||
// other
|
||||
findMonitorsSummaryByTeamId(teamId: string, config?: SummaryConfig): Promise<MonitorsSummary>;
|
||||
removeNotificationFromMonitors(notificationId: string): Promise<void>;
|
||||
|
||||
@@ -298,6 +298,20 @@ class MongoMonitorsRepository implements IMonitorsRepository {
|
||||
return { monitors: this.mapDocuments(monitors), deletedCount };
|
||||
};
|
||||
|
||||
findDockerTlsKeyById = async (monitorId: string): Promise<string | null> => {
|
||||
const doc = await MonitorModel.findOne({ _id: monitorId }, { dockerTlsKey: 1 }).lean();
|
||||
return doc?.dockerTlsKey ?? null;
|
||||
};
|
||||
|
||||
findAllDockerTlsKeys = async (): Promise<{ id: string; dockerTlsKey: string }[]> => {
|
||||
const docs = await MonitorModel.find({ dockerTlsKeySet: true }, { dockerTlsKey: 1 }).lean();
|
||||
return docs.flatMap((doc) => (doc.dockerTlsKey ? [{ id: doc._id.toString(), dockerTlsKey: doc.dockerTlsKey }] : []));
|
||||
};
|
||||
|
||||
updateDockerTlsKey = async (monitorId: string, dockerTlsKey: string): Promise<void> => {
|
||||
await MonitorModel.updateOne({ _id: monitorId }, { $set: { dockerTlsKey } });
|
||||
};
|
||||
|
||||
findMonitorsSummaryByTeamId = async (teamId: string, config: SummaryConfig): Promise<MonitorsSummary> => {
|
||||
const match = this.queryBuilder(config, teamId);
|
||||
const pipeline = [
|
||||
@@ -454,6 +468,9 @@ class MongoMonitorsRepository implements IMonitorsRepository {
|
||||
geoCheckLocations: doc.geoCheckLocations ?? [],
|
||||
geoCheckInterval: doc.geoCheckInterval ?? 300000,
|
||||
dockerLogsEnabled: doc.dockerLogsEnabled ?? false,
|
||||
dockerTlsCa: doc.dockerTlsCa ?? undefined,
|
||||
dockerTlsCert: doc.dockerTlsCert ?? undefined,
|
||||
dockerTlsKeySet: doc.dockerTlsKeySet ?? false,
|
||||
dnsServer: doc.dnsServer ?? undefined,
|
||||
dnsRecordType: doc.dnsRecordType ?? undefined,
|
||||
createdAt: toDateString(doc.createdAt),
|
||||
|
||||
@@ -33,6 +33,8 @@ import { ILogger } from "@/utils/logger.js";
|
||||
import { IJobScheduler } from "@/worker/worker.interface.js";
|
||||
import { DateRange } from "@/types/query.js";
|
||||
import { IDockerLogsRepository } from "@/domain/docker/docker-log.repository.interface.js";
|
||||
import { IEncryptionService } from "@/service/encryption/encryptionService.js";
|
||||
import { isDockerTlsUrl } from "@/utils/dockerHost.js";
|
||||
|
||||
const SERVICE_NAME = "MonitorService";
|
||||
|
||||
@@ -144,6 +146,7 @@ export class MonitorService implements IMonitorService {
|
||||
private monitorStatsRepository: IMonitorStatsRepository;
|
||||
private statusPagesRepository: IStatusPagesRepository;
|
||||
private incidentsRepository: IIncidentsRepository;
|
||||
private encryptionService: IEncryptionService;
|
||||
|
||||
constructor({
|
||||
scheduler,
|
||||
@@ -156,6 +159,7 @@ export class MonitorService implements IMonitorService {
|
||||
monitorStatsRepository,
|
||||
statusPagesRepository,
|
||||
incidentsRepository,
|
||||
encryptionService,
|
||||
}: {
|
||||
scheduler: IJobScheduler;
|
||||
logger: ILogger;
|
||||
@@ -167,6 +171,7 @@ export class MonitorService implements IMonitorService {
|
||||
monitorStatsRepository: IMonitorStatsRepository;
|
||||
statusPagesRepository: IStatusPagesRepository;
|
||||
incidentsRepository: IIncidentsRepository;
|
||||
encryptionService: IEncryptionService;
|
||||
}) {
|
||||
this.scheduler = scheduler;
|
||||
this.logger = logger;
|
||||
@@ -178,6 +183,7 @@ export class MonitorService implements IMonitorService {
|
||||
this.monitorStatsRepository = monitorStatsRepository;
|
||||
this.statusPagesRepository = statusPagesRepository;
|
||||
this.incidentsRepository = incidentsRepository;
|
||||
this.encryptionService = encryptionService;
|
||||
}
|
||||
|
||||
createMonitor = async (teamId: string, userId: string, body: Monitor): Promise<Monitor> => {
|
||||
@@ -185,6 +191,10 @@ export class MonitorService implements IMonitorService {
|
||||
if (body.proxyMode !== "custom") {
|
||||
delete body.proxyId;
|
||||
}
|
||||
|
||||
// Encrypt docker TLS keys
|
||||
this.encryptDockerTls(body, "create");
|
||||
|
||||
const monitor = await this.monitorsRepository.create(body, teamId, userId);
|
||||
if (!monitor) {
|
||||
throw new AppError({ message: "Failed to create monitor", status: 500, service: SERVICE_NAME, method: "createMonitor" });
|
||||
@@ -537,6 +547,20 @@ export class MonitorService implements IMonitorService {
|
||||
if (unsetProxyId) {
|
||||
delete body.proxyId;
|
||||
}
|
||||
|
||||
// Handle Docker TLS
|
||||
this.encryptDockerTls(body, "edit");
|
||||
if (body.type === "docker" && isDockerTlsUrl(body.url) && body.dockerTlsKey === undefined) {
|
||||
const stored = await this.monitorsRepository.findById(monitorId, teamId);
|
||||
if (!stored.dockerTlsKeySet) {
|
||||
throw new AppError({
|
||||
message: "Key is required for a TLS Docker host",
|
||||
status: 422,
|
||||
service: SERVICE_NAME,
|
||||
method: "editMonitor",
|
||||
});
|
||||
}
|
||||
}
|
||||
const editedMonitor = await this.monitorsRepository.updateById(monitorId, teamId, body, { unsetProxyId });
|
||||
await this.scheduler.updateJob(editedMonitor);
|
||||
return editedMonitor;
|
||||
@@ -724,4 +748,34 @@ export class MonitorService implements IMonitorService {
|
||||
|
||||
return { imported: createdMonitors!.length, errors };
|
||||
};
|
||||
|
||||
private encryptDockerTls = (body: Partial<Monitor>, mode: "create" | "edit") => {
|
||||
if (body.type !== "docker" && mode === "create") return;
|
||||
|
||||
if (body.url !== undefined && !isDockerTlsUrl(body.url)) {
|
||||
body.dockerTlsCa = undefined;
|
||||
body.dockerTlsCert = undefined;
|
||||
body.dockerTlsKey = undefined;
|
||||
body.dockerTlsKeySet = false;
|
||||
return;
|
||||
}
|
||||
|
||||
if (!body.dockerTlsKey) {
|
||||
delete body.dockerTlsKey; // empty field means keeps stored key
|
||||
return;
|
||||
}
|
||||
|
||||
if (!this.encryptionService.isConfigured()) {
|
||||
throw new AppError({
|
||||
message:
|
||||
'Docker TLS credentials require ENCRYPTION_KEY to be set on the server. Generate one with "openssl rand -base64 32", set the same value for both the API and worker, and restart.',
|
||||
status: 422,
|
||||
service: SERVICE_NAME,
|
||||
method: "encryptDockerTls",
|
||||
});
|
||||
}
|
||||
|
||||
body.dockerTlsKey = this.encryptionService.encrypt(body.dockerTlsKey);
|
||||
body.dockerTlsKeySet = true;
|
||||
};
|
||||
}
|
||||
@@ -109,6 +109,11 @@ export interface Monitor {
|
||||
geoCheckLocations?: GeoContinent[];
|
||||
geoCheckInterval?: number;
|
||||
dockerLogsEnabled?: boolean;
|
||||
dockerTlsCa?: string;
|
||||
dockerTlsCert?: string;
|
||||
// EncryptionService ciphertext. Normally excluded
|
||||
dockerTlsKey?: string;
|
||||
dockerTlsKeySet?: boolean;
|
||||
dnsServer?: string;
|
||||
dnsRecordType?: DnsRecordType;
|
||||
recentChecks: CheckSnapshot[];
|
||||
|
||||
@@ -179,6 +179,9 @@ export class NotificationsService implements INotificationsService {
|
||||
deleteById = async (id: string, teamId: string): Promise<Notification> => {
|
||||
await this.monitorsRepository.removeNotificationFromMonitors(id);
|
||||
const deleted = await this.notificationsRepository.deleteById(id, teamId);
|
||||
// App settings hold notification ids for the egress recovered alert; drop the deleted one so it never dangles.
|
||||
// After the team-scoped delete, so a request for another team's id (404 above) cannot touch instance settings.
|
||||
await this.settingsService.removeEgressNotification(id);
|
||||
return deleted;
|
||||
};
|
||||
}
|
||||
+1
-1
@@ -45,7 +45,7 @@ const startApp = async () => {
|
||||
// ***********************
|
||||
// Build shared services
|
||||
// ***********************
|
||||
const shared = await buildShared({ logger, envSettings, settingsService });
|
||||
const shared = await buildShared({ logger, envSettings, settingsService, encryptionKeys: env.ENCRYPTION_KEY });
|
||||
|
||||
// ***********************
|
||||
// Worker node path, don't need API
|
||||
|
||||
@@ -0,0 +1,176 @@
|
||||
import { ILogger } from "@/utils/logger.js";
|
||||
import type * as nodeCrypto from "node:crypto";
|
||||
import { AppError } from "@/utils/AppError.js";
|
||||
|
||||
const SERVICE_NAME = "EncryptionService";
|
||||
|
||||
const ENCRYPTION_KEY_BYTES = 32;
|
||||
|
||||
// Encryption consts
|
||||
const FORMAT_VERSION = "v1";
|
||||
const ALGORITHM = "aes-256-gcm";
|
||||
const IV_BYTES = 12;
|
||||
const TAG_BYTES = 16;
|
||||
const KEY_ID_BYTES = 4;
|
||||
const SEGMENT_COUNT = 5;
|
||||
type V1Segments = [string, string, string, string, string];
|
||||
const isV1Segments = (segments: string[]): segments is V1Segments => segments.length === SEGMENT_COUNT;
|
||||
const BASE64URL = /^[A-Za-z0-9_-]*$/;
|
||||
|
||||
type KeyEntry = { id: string; key: Buffer };
|
||||
|
||||
export type EncryptionCryptoLib = Pick<typeof nodeCrypto, "createCipheriv" | "createDecipheriv" | "createHash" | "randomBytes">;
|
||||
|
||||
export interface IEncryptionService {
|
||||
isConfigured(): boolean;
|
||||
currentKeyId(): string;
|
||||
keyIdOf(ciphertext: string): string;
|
||||
needsReencryption(ciphertext: string): boolean;
|
||||
encrypt(plaintext: string): string;
|
||||
decrypt(ciphertext: string): string;
|
||||
}
|
||||
|
||||
export class EncryptionService implements IEncryptionService {
|
||||
static SERVICE_NAME = SERVICE_NAME;
|
||||
|
||||
private readonly entries: KeyEntry[];
|
||||
|
||||
constructor(
|
||||
keys: Buffer[],
|
||||
private logger: ILogger,
|
||||
private cryptoLib: EncryptionCryptoLib
|
||||
) {
|
||||
this.entries = keys.map((key, index) => this.toEntry(key, index));
|
||||
this.entries.forEach((entry, index) => {
|
||||
if (this.entries.findIndex((other) => other.id === entry.id) !== index) {
|
||||
throw this.fail("constructor", "ENCRYPTION_KEY key ids collide; regenerate one key", { index });
|
||||
}
|
||||
});
|
||||
|
||||
const current = this.entries[0];
|
||||
|
||||
this.logger.info({
|
||||
message: current
|
||||
? `Encryption configured with ${this.entries.length} key(s); current key id ${current.id}`
|
||||
: "ENCRYPTION_KEY not set; encrypted fields are unavailable",
|
||||
service: SERVICE_NAME,
|
||||
method: "constructor",
|
||||
});
|
||||
}
|
||||
|
||||
isConfigured = () => {
|
||||
return this.entries.length > 0;
|
||||
};
|
||||
|
||||
currentKeyId = () => {
|
||||
return this.requireCurrentKey("currentKeyId").id;
|
||||
};
|
||||
|
||||
keyIdOf = (ciphertext: string) => {
|
||||
return this.parse(ciphertext, "keyIdOf").keyId;
|
||||
};
|
||||
|
||||
// Check if the ciphertext was encrypted with the current key
|
||||
needsReencryption = (ciphertext: string) => {
|
||||
const currentKey = this.requireCurrentKey("needsReencryption");
|
||||
return this.parse(ciphertext, "needsReencryption").keyId !== currentKey.id;
|
||||
};
|
||||
|
||||
encrypt = (plaintext: string) => {
|
||||
const currentKey = this.requireCurrentKey("encrypt"); // Get the current key for encryption
|
||||
const iv = this.cryptoLib.randomBytes(IV_BYTES); // Get IV_BYTES worth of random bytes for nonce
|
||||
const cipher = this.cryptoLib.createCipheriv(ALGORITHM, currentKey.key, iv, { authTagLength: TAG_BYTES }); // Create the cipher
|
||||
cipher.setAAD(this.aad(currentKey.id)); // set signing key as additional data so it is protected by the verification tag
|
||||
const data = Buffer.concat([cipher.update(plaintext, "utf8"), cipher.final()]); // Encode plaintext as utf8 and run through cipher
|
||||
const tag = cipher.getAuthTag(); // Get verification tag
|
||||
return [FORMAT_VERSION, currentKey.id, iv.toString("base64url"), tag.toString("base64url"), data.toString("base64url")].join(".");
|
||||
};
|
||||
|
||||
decrypt = (ciphertext: string) => {
|
||||
this.requireCurrentKey("decrypt"); // Config check
|
||||
const { keyId, iv, tag, data } = this.parse(ciphertext, "decrypt");
|
||||
const key = this.entries.find((entry) => entry.id === keyId)?.key;
|
||||
if (!key) {
|
||||
throw this.fail("decrypt", "No encryption key matches ciphertext", { keyId });
|
||||
}
|
||||
const decipher = this.cryptoLib.createDecipheriv(ALGORITHM, key, iv, { authTagLength: TAG_BYTES });
|
||||
decipher.setAAD(this.aad(keyId)); // reconstruct the additional data, must be done BEFORE auth tag
|
||||
decipher.setAuthTag(tag); // Give the decipher the tag to verify
|
||||
try {
|
||||
return Buffer.concat([decipher.update(data), decipher.final()]).toString("utf8");
|
||||
} catch {
|
||||
throw this.fail("decrypt", "Ciphertext failed authentication");
|
||||
}
|
||||
};
|
||||
|
||||
private fail = (method: string, message: string, details?: Record<string, unknown>): AppError => {
|
||||
return new AppError({
|
||||
message,
|
||||
status: 500,
|
||||
service: SERVICE_NAME,
|
||||
method,
|
||||
details,
|
||||
});
|
||||
};
|
||||
|
||||
// Create an ID:Key pair for each key.
|
||||
private toEntry = (key: Buffer, keyIdx: number): KeyEntry => {
|
||||
if (key.length !== ENCRYPTION_KEY_BYTES) throw this.fail("constructor", "ENCRYPTION_KEY is not 32 bytes", { keyIdx });
|
||||
|
||||
const id = this.cryptoLib
|
||||
.createHash("sha256") // Create a sha256 hasher
|
||||
.update(key) // Feed it the key
|
||||
.digest() // Digest a hash
|
||||
.subarray(0, KEY_ID_BYTES) // Keep first KEY_ID_BYTES, enough to avoid collision
|
||||
.toString("base64url");
|
||||
return { id, key };
|
||||
};
|
||||
|
||||
private requireCurrentKey = (method: string): KeyEntry => {
|
||||
const current = this.entries[0];
|
||||
if (!current) {
|
||||
throw this.fail(method, "ENCRYPTION_KEY is not configured");
|
||||
}
|
||||
return current;
|
||||
};
|
||||
|
||||
// v1 Ciphertext stored as a string with five segments:
|
||||
// <version>.<keyId>.<iv>.<tag>.<encryptedData>
|
||||
// iv and tag belong to aes-256-gcm standard
|
||||
|
||||
private parse = (ciphertext: string, method: string) => {
|
||||
const segments = ciphertext.split(".");
|
||||
const version = segments[0];
|
||||
if (version !== FORMAT_VERSION) {
|
||||
throw this.fail(method, "Unsupported ciphertext version", { version });
|
||||
}
|
||||
|
||||
// **************************
|
||||
// v1 parser
|
||||
// **************************
|
||||
if (!isV1Segments(segments)) {
|
||||
throw this.fail(method, "Ciphertext is malformed");
|
||||
}
|
||||
|
||||
const [, keyId, ivText, tagText, dataText] = segments; // Skip first segment, we already have version
|
||||
|
||||
if (![keyId, ivText, tagText, dataText].every((segment) => BASE64URL.test(segment))) {
|
||||
throw this.fail(method, "Ciphertext is malformed");
|
||||
}
|
||||
|
||||
const iv = Buffer.from(ivText, "base64url");
|
||||
const tag = Buffer.from(tagText, "base64url");
|
||||
|
||||
if (keyId.length === 0 || iv.length !== IV_BYTES || tag.length !== TAG_BYTES) {
|
||||
throw this.fail(method, "Ciphertext is malformed");
|
||||
}
|
||||
return { keyId, iv, tag, data: Buffer.from(dataText, "base64url") };
|
||||
// **************************
|
||||
// End v1 parser
|
||||
// **************************
|
||||
};
|
||||
|
||||
private aad = (keyId: string) => {
|
||||
return Buffer.from(`${FORMAT_VERSION}.${keyId}`, "utf8");
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,60 @@
|
||||
import { IMonitorsRepository } from "@/domain/monitors/monitor.repository.interface.js";
|
||||
import { IEncryptionService } from "@/service/encryption/encryptionService.js";
|
||||
import { AppError } from "@/utils/AppError.js";
|
||||
import { ILogger } from "@/utils/logger.js";
|
||||
|
||||
const SERVICE_NAME = "SecretsRotationService";
|
||||
|
||||
export interface ISecretsRotationService {
|
||||
run(): Promise<{ scanned: number; reencrypted: number; skipped: number }>;
|
||||
}
|
||||
|
||||
export class SecretsRotationService implements ISecretsRotationService {
|
||||
static SERVICE_NAME = SERVICE_NAME;
|
||||
|
||||
constructor(
|
||||
private monitorsRepository: IMonitorsRepository,
|
||||
private encryptionService: IEncryptionService,
|
||||
private logger: ILogger
|
||||
) {}
|
||||
|
||||
run = async () => {
|
||||
const rows = await this.monitorsRepository.findAllDockerTlsKeys();
|
||||
const result = { scanned: rows.length, reencrypted: 0, skipped: 0 };
|
||||
if (rows.length === 0) return result;
|
||||
|
||||
if (!this.encryptionService.isConfigured()) {
|
||||
this.logger.warn({
|
||||
message: `${rows.length} monitor(s) have a stored Docker TLS key but ENCRYPTION_KEY is not set; their checks will fail until it is`,
|
||||
service: SERVICE_NAME,
|
||||
method: "run",
|
||||
});
|
||||
result.skipped = rows.length;
|
||||
return result;
|
||||
}
|
||||
|
||||
for (const monitor of rows) {
|
||||
const { id, dockerTlsKey } = monitor;
|
||||
try {
|
||||
if (!this.encryptionService.needsReencryption(dockerTlsKey)) continue;
|
||||
const plainText = this.encryptionService.decrypt(dockerTlsKey);
|
||||
await this.monitorsRepository.updateDockerTlsKey(id, this.encryptionService.encrypt(plainText));
|
||||
result.reencrypted += 1;
|
||||
} catch (error: unknown) {
|
||||
result.skipped += 1;
|
||||
this.logger.warn({
|
||||
message: `Could not reencrypt Docker TLS key for monitor ${id}: ${error instanceof Error ? error.message : String(error)}`,
|
||||
service: SERVICE_NAME,
|
||||
method: "run",
|
||||
details: { monitorId: id, ...(error instanceof AppError ? error.details : {}) },
|
||||
});
|
||||
}
|
||||
}
|
||||
this.logger.info({
|
||||
message: `Docker TLS key rotation: ${result.reencrypted} re-encrypted, ${result.skipped} skipped, ${result.scanned} scanned`,
|
||||
service: SERVICE_NAME,
|
||||
method: "run",
|
||||
});
|
||||
return result;
|
||||
};
|
||||
}
|
||||
@@ -1,5 +1,7 @@
|
||||
import https from "node:https";
|
||||
|
||||
import { IStatusProvider } from "@/service/network/IStatusProvider.js";
|
||||
import { DockerStatusPayload, MonitorStatusResponse } from "@/types/network.js";
|
||||
import { CheckContext, DockerStatusPayload, MonitorStatusResponse } from "@/types/network.js";
|
||||
|
||||
import {
|
||||
DOCKER_LOG_TAIL_LINES,
|
||||
@@ -23,13 +25,21 @@ import { AppError } from "@/utils/AppError.js";
|
||||
import Dockerode from "dockerode";
|
||||
import { timeRequest } from "@/service/network/utils.js";
|
||||
import { NETWORK_ERROR } from "@/types/network.js";
|
||||
import { IEncryptionService } from "@/service/encryption/encryptionService.js";
|
||||
import { DOCKER_TLS_URL, isDockerTlsUrl } from "@/utils/dockerHost.js";
|
||||
import { splitCertificateBundle } from "@/utils/pem.js";
|
||||
|
||||
type DockerodeType = typeof Dockerode;
|
||||
type DockerOptions = Dockerode.DockerOptions & { agent?: https.Agent; connectionTimeout?: number };
|
||||
type TlsCredentials = { ok: true; key: string } | { ok: false; message: string };
|
||||
|
||||
const SERVICE_NAME = "DockerProvider";
|
||||
const TIMEOUT_MS = 10000;
|
||||
const STATS_CONCURRENCY = 5;
|
||||
const DOCKER_LOG_MAX_LINE_BYTES = 4096;
|
||||
const DOCKER_LOG_TRUNCATION_MARKER = " …[truncated]";
|
||||
const DOCKER_LOG_TS_REGEX = /^(\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2})(?:\.(\d{1,9}))?Z /;
|
||||
const DOCKER_TLS_DEFAULT_PORT = 2376;
|
||||
|
||||
export interface DockerError extends Error {
|
||||
statusCode?: number;
|
||||
@@ -42,28 +52,73 @@ export class DockerProvider implements IStatusProvider<DockerStatusPayload> {
|
||||
|
||||
constructor(
|
||||
private logger: ILogger,
|
||||
private DockerLib: DockerodeType
|
||||
private DockerLib: DockerodeType,
|
||||
private encryptionService: IEncryptionService
|
||||
) {}
|
||||
|
||||
supports(type: MonitorType): boolean {
|
||||
return type === "docker";
|
||||
}
|
||||
|
||||
private resolveTlsCredentials = (monitor: Monitor, ctx?: CheckContext): TlsCredentials | undefined => {
|
||||
if (!isDockerTlsUrl(monitor.url)) return undefined;
|
||||
if (!ctx?.dockerTlsKey) return { ok: false, message: "Docker TLS key is missing" };
|
||||
try {
|
||||
return { ok: true, key: this.encryptionService.decrypt(ctx.dockerTlsKey) };
|
||||
} catch (error) {
|
||||
if (error instanceof AppError) {
|
||||
const keyId = typeof error.details?.keyId === "string" ? ` (key id ${error.details.keyId})` : "";
|
||||
return { ok: false, message: `${error.message}${keyId}` };
|
||||
}
|
||||
return { ok: false, message: "Docker TLS key could not be decrypted" };
|
||||
}
|
||||
};
|
||||
|
||||
private isDockerError(error: unknown): error is DockerError {
|
||||
return error instanceof Error && ("statusCode" in error || "reason" in error || "json" in error);
|
||||
}
|
||||
|
||||
private toDockerOptions = (monitor: Monitor): Dockerode.DockerOptions => {
|
||||
private splitCaBundle = (pem: string | undefined): string[] | undefined => {
|
||||
const blocks = pem ? splitCertificateBundle(pem) : [];
|
||||
return blocks.length > 0 ? blocks : undefined;
|
||||
};
|
||||
|
||||
private invalidUrl = (message: string) => new AppError({ message, status: 422, service: SERVICE_NAME, method: "toDockerOptions" });
|
||||
|
||||
private toDockerOptions = (monitor: Monitor, credentials?: TlsCredentials): DockerOptions => {
|
||||
const url = monitor.url?.trim();
|
||||
if (!url) throw new AppError({ message: "Docker host URL is required", status: 422, service: SERVICE_NAME, method: "toDockerOptions" });
|
||||
if (!url) throw this.invalidUrl("Docker host URL is required");
|
||||
|
||||
// Unix socket
|
||||
if (url.startsWith("unix://")) {
|
||||
const socketPath = url.slice("unix://".length);
|
||||
if (!socketPath.startsWith("/"))
|
||||
throw new AppError({ message: `Invalid Docker host URL: ${url}`, status: 422, service: SERVICE_NAME, method: "toDockerOptions" });
|
||||
return { socketPath };
|
||||
if (!socketPath.startsWith("/")) throw this.invalidUrl(`Invalid Docker host URL: ${url}`);
|
||||
return { socketPath, timeout: TIMEOUT_MS, connectionTimeout: TIMEOUT_MS };
|
||||
}
|
||||
if (url.startsWith("/")) return { socketPath: url };
|
||||
throw new AppError({ message: `Invalid Docker host URL: ${url}`, status: 422, service: SERVICE_NAME, method: "toDockerOptions" });
|
||||
if (url.startsWith("/")) return { socketPath: url, timeout: TIMEOUT_MS, connectionTimeout: TIMEOUT_MS };
|
||||
|
||||
// Docker TLS
|
||||
const match = DOCKER_TLS_URL.exec(url);
|
||||
if (!match) throw this.invalidUrl(`Invalid Docker host URL: ${url}`);
|
||||
if (!credentials?.ok) throw this.invalidUrl(credentials?.message ?? "Docker TLS key is missing");
|
||||
const [, , host, port] = match;
|
||||
const ca = monitor.ignoreTlsErrors ? undefined : this.splitCaBundle(monitor.dockerTlsCa);
|
||||
const cert = monitor.dockerTlsCert;
|
||||
const key = credentials.key;
|
||||
return {
|
||||
host,
|
||||
port: port ? Number(port) : DOCKER_TLS_DEFAULT_PORT,
|
||||
protocol: "https",
|
||||
ca,
|
||||
cert,
|
||||
key,
|
||||
agent: new https.Agent({ ca, cert, key, rejectUnauthorized: !monitor.ignoreTlsErrors }),
|
||||
socketPath: undefined,
|
||||
username: undefined,
|
||||
sshOptions: undefined,
|
||||
timeout: TIMEOUT_MS,
|
||||
connectionTimeout: TIMEOUT_MS,
|
||||
};
|
||||
};
|
||||
|
||||
private async mapWithConcurrency<T, R>(items: T[], limit: number, fn: (item: T) => Promise<R>): Promise<R[]> {
|
||||
@@ -260,9 +315,23 @@ export class DockerProvider implements IStatusProvider<DockerStatusPayload> {
|
||||
return rawLogs.filter((log) => log !== null);
|
||||
};
|
||||
|
||||
handle = async (monitor: Monitor): Promise<MonitorStatusResponse<DockerStatusPayload>> => {
|
||||
private failed = (monitor: Monitor, code: number, message: string, responseTime: number): MonitorStatusResponse<DockerStatusPayload> => ({
|
||||
monitorId: monitor.id,
|
||||
teamId: monitor.teamId,
|
||||
type: monitor.type,
|
||||
status: false,
|
||||
code,
|
||||
message,
|
||||
responseTime,
|
||||
payload: null,
|
||||
});
|
||||
|
||||
handle = async (monitor: Monitor, ctx?: CheckContext): Promise<MonitorStatusResponse<DockerStatusPayload>> => {
|
||||
try {
|
||||
const docker = new this.DockerLib(this.toDockerOptions(monitor));
|
||||
const credentials = this.resolveTlsCredentials(monitor, ctx);
|
||||
if (credentials && !credentials.ok) return this.failed(monitor, NETWORK_ERROR, credentials.message, 0);
|
||||
|
||||
const docker = new this.DockerLib(this.toDockerOptions(monitor, credentials));
|
||||
// Host reachability is the monitor's status; ping latency is the responseTime
|
||||
const { responseTime, error } = await timeRequest(() => docker.ping());
|
||||
if (error) {
|
||||
@@ -274,16 +343,7 @@ export class DockerProvider implements IStatusProvider<DockerStatusPayload> {
|
||||
} else if (error instanceof Error) {
|
||||
message = error.message;
|
||||
}
|
||||
return {
|
||||
monitorId: monitor.id,
|
||||
teamId: monitor.teamId,
|
||||
type: monitor.type,
|
||||
status: false,
|
||||
code,
|
||||
message,
|
||||
responseTime,
|
||||
payload: null,
|
||||
};
|
||||
return this.failed(monitor, code, message, responseTime);
|
||||
}
|
||||
|
||||
const summaries = await docker.listContainers({ all: true });
|
||||
|
||||
@@ -22,7 +22,7 @@ export class PageSpeedProvider implements IStatusProvider<PageSpeedStatusPayload
|
||||
const { url, strategy } = monitor;
|
||||
try {
|
||||
if (!url) throw new Error("URL is required for PageSpeed monitor");
|
||||
const dbSettings = await this.settingsService.getDBSettings();
|
||||
const dbSettings = await this.settingsService.getCachedDBSettings();
|
||||
const apiKey = dbSettings?.pagespeedApiKey;
|
||||
const resolvedStrategy = strategy ?? DefaultPageSpeedStrategy;
|
||||
let pageSpeedUrl = `https://pagespeedonline.googleapis.com/pagespeedonline/v5/runPagespeed?url=${encodeURIComponent(
|
||||
|
||||
@@ -4,25 +4,33 @@ import { MonitorType, Monitor } from "@/domain/monitors/monitor.type.js";
|
||||
import { MonitorStatusResponse } from "@/types/network.js";
|
||||
import { AppError } from "@/utils/AppError.js";
|
||||
import ping from "ping";
|
||||
import * as net from "net";
|
||||
import { timeRequest } from "@/service/network/utils.js";
|
||||
const SERVICE_NAME = "PingProvider";
|
||||
|
||||
type Ping = typeof ping;
|
||||
type NetType = typeof net;
|
||||
|
||||
export class PingProvider implements IStatusProvider<PingStatusPayload> {
|
||||
readonly type = "ping";
|
||||
|
||||
constructor(private ping: Ping) {}
|
||||
constructor(
|
||||
private ping: Ping,
|
||||
private net: NetType
|
||||
) {}
|
||||
|
||||
supports(type: MonitorType): boolean {
|
||||
return type === "ping";
|
||||
}
|
||||
|
||||
private sanitizeHost(url: string): string {
|
||||
return url
|
||||
.replace(/^https?:\/\//, "")
|
||||
.replace(/\/.*$/, "")
|
||||
.replace(/:.*/, "");
|
||||
const host = url.replace(/^https?:\/\//, "").replace(/\/.*$/, "");
|
||||
// "[2001:db8::1]" or "[2001:db8::1]:53": the address is inside the brackets
|
||||
const bracketed = /^\[([^\]]+)\](?::\d+)?$/.exec(host);
|
||||
if (bracketed) return bracketed[1]!;
|
||||
// A bare IPv6 address contains colons that are not a port separator
|
||||
if (this.net.isIPv6(host)) return host;
|
||||
return host.replace(/:.*/, "");
|
||||
}
|
||||
|
||||
async handle(monitor: Monitor): Promise<MonitorStatusResponse<PingStatusPayload>> {
|
||||
|
||||
@@ -73,7 +73,7 @@ export class ProxyResolver implements IProxyResolver {
|
||||
};
|
||||
|
||||
private resolveInherit = async (monitor: Monitor): Promise<string | undefined> => {
|
||||
const settings = await this.getCached("settings", () => this.settingsService.getDBSettings());
|
||||
const settings = await this.settingsService.getCachedDBSettings();
|
||||
if (!settings.globalProxyEnabled) {
|
||||
return undefined;
|
||||
}
|
||||
|
||||
@@ -162,4 +162,5 @@ export type MonitorStatusResponseOverrides<T> = Partial<Omit<MonitorStatusRespon
|
||||
|
||||
export type CheckContext = {
|
||||
proxyUrl?: string;
|
||||
dockerTlsKey?: string;
|
||||
};
|
||||
@@ -0,0 +1,5 @@
|
||||
export const DOCKER_TLS_URL = /^(tcp|https):\/\/([^/\s:]+)(?::(\d+))?\/?$/;
|
||||
export const DOCKER_SOCKET_URL = /^(?:unix:\/\/\/\S+|\/\S+)$/;
|
||||
|
||||
export const isDockerTlsUrl = (url: string | undefined): boolean => DOCKER_TLS_URL.test(url?.trim() ?? "");
|
||||
export const isDockerSocketUrl = (url: string | undefined): boolean => DOCKER_SOCKET_URL.test(url?.trim() ?? "");
|
||||
@@ -0,0 +1,48 @@
|
||||
import { z } from "zod";
|
||||
import { dnsHostnameRegex } from "@/api/validation/shared.js";
|
||||
|
||||
// A reliability target for the egress self-check, in one of three shapes. Parsed here so the
|
||||
// settings validator and the probe agree on exactly what is accepted.
|
||||
export type EgressTarget =
|
||||
| { kind: "http"; url: string } // http(s):// URL, fetched
|
||||
| { kind: "port"; host: string; port: number } // host:port or [ipv6]:port, TCP connect
|
||||
| { kind: "ping"; host: string }; // bare hostname or IP address, ICMP
|
||||
|
||||
// "[ipv6]:port" or "host:port" where host has no colons, so a bare IPv6 address is never read as host:port.
|
||||
const HOST_PORT_PATTERN = /^(?:\[([^\]]+)\]|([^:/\s[\]]+)):(\d{1,5})$/;
|
||||
const BRACKETED_PATTERN = /^\[([^\]]+)\]$/;
|
||||
|
||||
const isIpv6 = (value: string): boolean => z.ipv6().safeParse(value).success;
|
||||
const isHostOrIpv4 = (value: string): boolean => value === "localhost" || z.ipv4().safeParse(value).success || dnsHostnameRegex.test(value);
|
||||
|
||||
const parseHttp = (target: string): EgressTarget | null => {
|
||||
try {
|
||||
const { protocol, hostname } = new URL(target);
|
||||
return (protocol === "http:" || protocol === "https:") && hostname.length > 0 ? { kind: "http", url: target } : null;
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
};
|
||||
|
||||
export const parseEgressTarget = (raw: string): EgressTarget | null => {
|
||||
const target = raw.trim();
|
||||
if (target.length === 0) return null;
|
||||
|
||||
if (/^https?:\/\//i.test(target)) return parseHttp(target);
|
||||
|
||||
const hostPort = HOST_PORT_PATTERN.exec(target);
|
||||
if (hostPort) {
|
||||
const port = Number(hostPort[3]);
|
||||
if (port < 1 || port > 65535) return null;
|
||||
const bracketedHost = hostPort[1];
|
||||
if (bracketedHost !== undefined) return isIpv6(bracketedHost) ? { kind: "port", host: bracketedHost, port } : null;
|
||||
const host = hostPort[2]!;
|
||||
return isHostOrIpv4(host) ? { kind: "port", host, port } : null;
|
||||
}
|
||||
|
||||
const bracketed = BRACKETED_PATTERN.exec(target);
|
||||
if (bracketed) return isIpv6(bracketed[1]!) ? { kind: "ping", host: bracketed[1]! } : null;
|
||||
|
||||
if (isIpv6(target) || isHostOrIpv4(target)) return { kind: "ping", host: target };
|
||||
return null;
|
||||
};
|
||||
@@ -0,0 +1,28 @@
|
||||
import { createPrivateKey, KeyObject, X509Certificate } from "node:crypto";
|
||||
|
||||
const PEM_CERT_BLOCK = /-----BEGIN CERTIFICATE-----[\s\S]*?-----END CERTIFICATE-----/g;
|
||||
|
||||
export const splitCertificateBundle = (pem: string): string[] => pem.match(PEM_CERT_BLOCK) ?? [];
|
||||
|
||||
export const parseCertificates = (pem: string): X509Certificate[] => {
|
||||
const blocks = splitCertificateBundle(pem);
|
||||
if (blocks.length === 0) throw new Error("No certificate found; expected one or more PEM CERTIFICATE blocks");
|
||||
try {
|
||||
return blocks.map((block) => new X509Certificate(block));
|
||||
} catch {
|
||||
throw new Error("Certificate is not valid PEM");
|
||||
}
|
||||
};
|
||||
|
||||
export const parsePrivateKey = (pem: string): KeyObject => {
|
||||
if (/^-----BEGIN ENCRYPTED PRIVATE KEY-----/m.test(pem) || /^Proc-Type: 4,ENCRYPTED/m.test(pem)) {
|
||||
throw new Error("Encrypted private keys are not supported; remove the passphrase first");
|
||||
}
|
||||
try {
|
||||
return createPrivateKey(pem);
|
||||
} catch {
|
||||
throw new Error("Private key is not valid PEM");
|
||||
}
|
||||
};
|
||||
|
||||
export const keyMatchesCertificate = (key: KeyObject, certificate: X509Certificate): boolean => certificate.checkPrivateKey(key);
|
||||
@@ -36,6 +36,12 @@ export class CheckProducer implements ICheckProducer {
|
||||
return windows.some((w) => isWindowActive(w, now));
|
||||
}
|
||||
|
||||
private async resolveDockerTlsKey(monitor: Monitor): Promise<string | undefined> {
|
||||
if (monitor.type !== "docker" || !monitor.dockerTlsKeySet || !monitor.id) return undefined;
|
||||
const dockerTlsKey = await this.monitorsRepository.findDockerTlsKeyById(monitor.id);
|
||||
return dockerTlsKey ?? undefined;
|
||||
}
|
||||
|
||||
produce = async (monitor: Monitor) => {
|
||||
if (!monitor.id) {
|
||||
throw new AppError({ message: "No monitor id", service: SERVICE_NAME, method: "produce" });
|
||||
@@ -61,7 +67,9 @@ export class CheckProducer implements ICheckProducer {
|
||||
|
||||
// Step 1b: Acquire status
|
||||
const proxyUrl = await this.proxyResolver.resolve(monitor);
|
||||
const status = await this.networkService.requestStatus(monitor, { proxyUrl });
|
||||
const dockerTlsKey = await this.resolveDockerTlsKey(monitor);
|
||||
|
||||
const status = await this.networkService.requestStatus(monitor, { proxyUrl, dockerTlsKey });
|
||||
if (!status) {
|
||||
throw new Error("No network response");
|
||||
}
|
||||
|
||||
@@ -28,6 +28,7 @@ const CONCURRENCY: Record<JobType, number> = {
|
||||
evaluate: 20,
|
||||
"cleanup-orphaned": 1,
|
||||
"cleanup-retention": 1,
|
||||
egress: 1,
|
||||
};
|
||||
|
||||
export interface DBQueueWorkerDependencies {
|
||||
@@ -61,6 +62,7 @@ export class DBQueueWorker extends JobScheduler implements IQueueWorker {
|
||||
evaluate: 0,
|
||||
"cleanup-orphaned": 0,
|
||||
"cleanup-retention": 0,
|
||||
egress: 0,
|
||||
};
|
||||
|
||||
private checksRepository: IChecksRepository;
|
||||
@@ -175,6 +177,9 @@ export class DBQueueWorker extends JobScheduler implements IQueueWorker {
|
||||
case "cleanup-retention":
|
||||
await this.helper.getCleanupRetentionJob()(); // Get job and execute
|
||||
break;
|
||||
case "egress":
|
||||
await this.helper.getEgressRecoveryJob()(job); // Re-probe while degraded; removes its own row on recovery
|
||||
break;
|
||||
}
|
||||
|
||||
if (job.intervalMs === null) {
|
||||
|
||||
@@ -11,10 +11,13 @@ import { IJobsRepository } from "@/domain/jobs/job.repository.interface.js";
|
||||
import { ITeamsRepository } from "@/domain/teams/team.repository.interface.js";
|
||||
import { ILogger } from "@/utils/logger.js";
|
||||
import { IDockerLogsRepository } from "@/domain/docker/docker-log.repository.interface.js";
|
||||
import { IEgressService } from "@/domain/egress/egress.service.js";
|
||||
import type { Job } from "@/domain/jobs/job.type.js";
|
||||
|
||||
export interface IWorkerHelper {
|
||||
getCleanupOrphanedJob(): () => Promise<void>;
|
||||
getCleanupRetentionJob(): () => Promise<void>;
|
||||
getEgressRecoveryJob(): (job: Job) => Promise<void>;
|
||||
}
|
||||
|
||||
export interface MonitorActionDecision {
|
||||
@@ -45,6 +48,7 @@ export class WorkerHelper implements IWorkerHelper {
|
||||
private incidentsRepository: IIncidentsRepository;
|
||||
private geoChecksRepository: IGeoChecksRepository;
|
||||
private dockerLogsRepository: IDockerLogsRepository;
|
||||
private egressService: IEgressService;
|
||||
|
||||
constructor(
|
||||
logger: ILogger,
|
||||
@@ -57,7 +61,8 @@ export class WorkerHelper implements IWorkerHelper {
|
||||
checksRepository: IChecksRepository,
|
||||
incidentsRepository: IIncidentsRepository,
|
||||
geoChecksRepository: IGeoChecksRepository,
|
||||
dockerLogsRepository: IDockerLogsRepository
|
||||
dockerLogsRepository: IDockerLogsRepository,
|
||||
egressService: IEgressService
|
||||
) {
|
||||
this.logger = logger;
|
||||
this.checkService = checkService;
|
||||
@@ -70,8 +75,17 @@ export class WorkerHelper implements IWorkerHelper {
|
||||
this.incidentsRepository = incidentsRepository;
|
||||
this.geoChecksRepository = geoChecksRepository;
|
||||
this.dockerLogsRepository = dockerLogsRepository;
|
||||
this.egressService = egressService;
|
||||
}
|
||||
|
||||
// Runs while instance egress is degraded; the service removes the row once egress is back.
|
||||
// Errors propagate so the queue records the failure and retries with its usual backoff.
|
||||
getEgressRecoveryJob = () => {
|
||||
return async (job: Job) => {
|
||||
await this.egressService.checkRecovery(job);
|
||||
};
|
||||
};
|
||||
|
||||
getCleanupOrphanedJob = () => {
|
||||
return async () => {
|
||||
try {
|
||||
|
||||
@@ -41,6 +41,7 @@ export class JobScheduler implements IJobScheduler {
|
||||
evaluate: POLL_MS,
|
||||
"cleanup-orphaned": POLL_MS,
|
||||
"cleanup-retention": POLL_MS,
|
||||
egress: POLL_MS,
|
||||
};
|
||||
// Helpers, builds job seeds. immediate=true runs on the next tick; otherwise jitter spreads the herd.
|
||||
protected toCheckJob = (monitor: MonitorScheduleFields, now: number, immediate = false): JobSeed => ({
|
||||
|
||||
+10
@@ -0,0 +1,10 @@
|
||||
Test fixtures for Docker TLS validation. Self-signed, 100-year, not trusted anywhere.
|
||||
|
||||
Regenerate with:
|
||||
|
||||
openssl req -x509 -newkey ec -pkeyopt ec_paramgen_curve:prime256v1 -nodes -days 36500 -subj "/CN=Checkmate Test CA" -keyout ca-key.pem -out ca.pem
|
||||
openssl req -newkey ec -pkeyopt ec_paramgen_curve:prime256v1 -nodes -subj "/CN=checkmate-test-client" -keyout client-key.pem -out client.csr
|
||||
openssl x509 -req -in client.csr -CA ca.pem -CAkey ca-key.pem -CAcreateserial -days 36500 -out client-cert.pem
|
||||
openssl ecparam -name prime256v1 -genkey -noout -out other-key.pem
|
||||
openssl pkcs8 -topk8 -in client-key.pem -out encrypted-key.pem -passout pass:fixture
|
||||
rm client.csr ca.srl ca-key.pem
|
||||
+11
@@ -0,0 +1,11 @@
|
||||
-----BEGIN CERTIFICATE-----
|
||||
MIIBkDCCATWgAwIBAgIUL3SODzLIekF+Tc3iHhHr/7uxvdIwCgYIKoZIzj0EAwIw
|
||||
HDEaMBgGA1UEAwwRQ2hlY2ttYXRlIFRlc3QgQ0EwIBcNMjYwOTEwMTkxNDUyWhgP
|
||||
MjEyNjA4MTcxOTE0NTJaMBwxGjAYBgNVBAMMEUNoZWNrbWF0ZSBUZXN0IENBMFkw
|
||||
EwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEXRJ0Y0jc+pKwXbuioDmbaHOqr5n3oU9x
|
||||
Y16vYajhUiFeWohEqGRG8lHA9S82KPNmpjthLWUykHvrTsKosBb6IKNTMFEwHQYD
|
||||
VR0OBBYEFNfGlRFC2MFZrCkoF4UxktbZUve1MB8GA1UdIwQYMBaAFNfGlRFC2MFZ
|
||||
rCkoF4UxktbZUve1MA8GA1UdEwEB/wQFMAMBAf8wCgYIKoZIzj0EAwIDSQAwRgIh
|
||||
APOnYaaKk5ED6NV/AkdpNXOEqlWcXwIQzh5Ycj3cjby+AiEApuAGTnk8k4dtVNmB
|
||||
aVV5bRmq7f3BjhxwZorP+KiIrdI=
|
||||
-----END CERTIFICATE-----
|
||||
+11
@@ -0,0 +1,11 @@
|
||||
-----BEGIN CERTIFICATE-----
|
||||
MIIBgjCCASigAwIBAgIUBx7r8tVOUFNX8c3YYyBxXTNltuUwCgYIKoZIzj0EAwIw
|
||||
HDEaMBgGA1UEAwwRQ2hlY2ttYXRlIFRlc3QgQ0EwIBcNMjYwOTEwMTkxNDUyWhgP
|
||||
MjEyNjA4MTcxOTE0NTJaMCAxHjAcBgNVBAMMFWNoZWNrbWF0ZS10ZXN0LWNsaWVu
|
||||
dDBZMBMGByqGSM49AgEGCCqGSM49AwEHA0IABKeUUNe9kE57nK7wTfnnvmqgzVj1
|
||||
ngNPajQYIAh9SAR+pVLAoTWLinWq5YarW1LFAzxyRKl6p2NdKtyLpVYV5BKjQjBA
|
||||
MB0GA1UdDgQWBBReuZBD6K0kWfgGDkt05zzM24wPejAfBgNVHSMEGDAWgBTXxpUR
|
||||
QtjBWawpKBeFMZLW2VL3tTAKBggqhkjOPQQDAgNIADBFAiAvfzha4dEVcJlKjH45
|
||||
yOkbFXiS/+tU+uTvn+7uutn6QAIhAI2qCXVvxX+i3Jk1xTS7R0mzJNXZXqabjODi
|
||||
cisS1VJG
|
||||
-----END CERTIFICATE-----
|
||||
@@ -0,0 +1,5 @@
|
||||
-----BEGIN PRIVATE KEY-----
|
||||
MIGHAgEAMBMGByqGSM49AgEGCCqGSM49AwEHBG0wawIBAQQgIHAuDMyYVMOi5PpA
|
||||
yUPjuNjvy/QOI9dnjQZ+AKhuQO+hRANCAASnlFDXvZBOe5yu8E35575qoM1Y9Z4D
|
||||
T2o0GCAIfUgEfqVSwKE1i4p1quWGq1tSxQM8ckSpeqdjXSrci6VWFeQS
|
||||
-----END PRIVATE KEY-----
|
||||
@@ -0,0 +1,8 @@
|
||||
-----BEGIN ENCRYPTED PRIVATE KEY-----
|
||||
MIH0MF8GCSqGSIb3DQEFDTBSMDEGCSqGSIb3DQEFDDAkBBDak7FU8VAgafGR1iEl
|
||||
eFHbAgIIADAMBggqhkiG9w0CCQUAMB0GCWCGSAFlAwQBKgQQCUTF7P/8uj9a5kLU
|
||||
3nBJdQSBkNpsmjC1FczojCBOn0g+2PlmMGI6MfR3vFvmPRAAF7SHLzQC1lXWhTU0
|
||||
3v5yCCdjlC5gTGINKf8RhZQjKym+lIpf+16EAPFLzFbRJPvt/eKq65YiixvQP2PS
|
||||
vysOG71YZJt1kSHrCOu5EPcCLvD2Kjyt48c/LJssCB81TOdqP1jV9iH29SCv4bnx
|
||||
KtjXulGg3w==
|
||||
-----END ENCRYPTED PRIVATE KEY-----
|
||||
@@ -0,0 +1,5 @@
|
||||
-----BEGIN EC PRIVATE KEY-----
|
||||
MHcCAQEEIGnUNFbPMxfa3m7xo9J3Lu9qiss2ZxWpqp91B6X9rQbfoAoGCCqGSM49
|
||||
AwEHoUQDQgAEGtbNBALMRIEROLtiEdNda4r/X0Hq1hcY/0V20gW3hEHhKRKkSilH
|
||||
0LyFWtlZ5uILZh6zXCGWlOPm0PFO9rN/hQ==
|
||||
-----END EC PRIVATE KEY-----
|
||||
@@ -0,0 +1,130 @@
|
||||
import { describe, expect, it, beforeAll, afterAll, beforeEach } from "@jest/globals";
|
||||
import mongoose from "mongoose";
|
||||
import { MongoMemoryServer } from "mongodb-memory-server";
|
||||
import MongoEgressStateRepository from "../../src/domain/egress/egress-state.repository.mongo.ts";
|
||||
import { EgressStateModel } from "../../src/domain/egress/egress-state.model.ts";
|
||||
|
||||
// ── Real-Mongo harness ─────────────────────────────────────────────────────────
|
||||
// The transitions here are the guarantees several worker processes rely on: exactly one
|
||||
// caller flips ok -> degraded (and degraded -> ok), and a plain read never writes. Both are
|
||||
// MongoDB filter/update semantics, so they need a live engine.
|
||||
|
||||
let mongod: MongoMemoryServer;
|
||||
|
||||
beforeAll(async () => {
|
||||
mongod = await MongoMemoryServer.create();
|
||||
await mongoose.connect(mongod.getUri());
|
||||
await EgressStateModel.init(); // unique index on singleton
|
||||
}, 120_000);
|
||||
|
||||
afterAll(async () => {
|
||||
await mongoose.disconnect();
|
||||
await mongod.stop();
|
||||
});
|
||||
|
||||
beforeEach(async () => {
|
||||
await EgressStateModel.deleteMany({});
|
||||
});
|
||||
|
||||
const probe = (target: string, reachable: boolean) => ({ target, reachable, responseTime: 5, message: reachable ? "Success" : "timeout" });
|
||||
const NOW = new Date("2026-01-01T10:00:00.000Z");
|
||||
const LATER = new Date("2026-01-01T10:05:00.000Z");
|
||||
|
||||
describe("MongoEgressStateRepository", () => {
|
||||
const repo = new MongoEgressStateRepository();
|
||||
|
||||
describe("findSingleton", () => {
|
||||
it("creates the ok singleton on first call and returns the same document afterwards", async () => {
|
||||
const first = await repo.findSingleton();
|
||||
const second = await repo.findSingleton();
|
||||
|
||||
expect(first.status).toBe("ok");
|
||||
expect(second.id).toBe(first.id);
|
||||
expect(await EgressStateModel.countDocuments()).toBe(1);
|
||||
});
|
||||
|
||||
it("does not bump updatedAt on a read", async () => {
|
||||
const created = await repo.findSingleton();
|
||||
await new Promise((resolve) => setTimeout(resolve, 20));
|
||||
|
||||
const read = await repo.findSingleton();
|
||||
|
||||
expect(read.updatedAt).toBe(created.updatedAt);
|
||||
});
|
||||
|
||||
it("survives two processes racing the first insert", async () => {
|
||||
const [a, b] = await Promise.all([repo.findSingleton(), repo.findSingleton()]);
|
||||
|
||||
expect(a.id).toBe(b.id);
|
||||
expect(await EgressStateModel.countDocuments()).toBe(1);
|
||||
});
|
||||
});
|
||||
|
||||
describe("markDegraded / markRecovered", () => {
|
||||
it("markDegraded does nothing when the singleton has never been created (callers read it first)", async () => {
|
||||
expect(await repo.markDegraded([probe("1.1.1.1", false)], NOW)).toBeNull();
|
||||
expect(await EgressStateModel.countDocuments()).toBe(0);
|
||||
});
|
||||
|
||||
it("lets exactly one caller perform each transition", async () => {
|
||||
await repo.findSingleton();
|
||||
|
||||
const [d1, d2] = await Promise.all([repo.markDegraded([probe("1.1.1.1", false)], NOW), repo.markDegraded([probe("1.1.1.1", false)], NOW)]);
|
||||
expect([d1, d2].filter((result) => result !== null)).toHaveLength(1);
|
||||
expect((await repo.findSingleton()).status).toBe("degraded");
|
||||
|
||||
const [r1, r2] = await Promise.all([repo.markRecovered([probe("1.1.1.1", true)], LATER), repo.markRecovered([probe("1.1.1.1", true)], LATER)]);
|
||||
expect([r1, r2].filter((result) => result !== null)).toHaveLength(1);
|
||||
const state = await repo.findSingleton();
|
||||
expect(state.status).toBe("ok");
|
||||
expect(state.degradedSince).toBe(NOW.toISOString());
|
||||
expect(state.lastRecoveredAt).toBe(LATER.toISOString());
|
||||
});
|
||||
|
||||
it("markRecovered is a no-op while ok and markDegraded is a no-op while degraded", async () => {
|
||||
await repo.findSingleton();
|
||||
expect(await repo.markRecovered([probe("1.1.1.1", true)], NOW)).toBeNull();
|
||||
|
||||
await repo.markDegraded([probe("1.1.1.1", false)], NOW);
|
||||
expect(await repo.markDegraded([probe("1.1.1.1", false)], LATER)).toBeNull();
|
||||
expect((await repo.findSingleton()).degradedSince).toBe(NOW.toISOString());
|
||||
});
|
||||
});
|
||||
|
||||
describe("recordProbe", () => {
|
||||
it("stores the probe without changing the status", async () => {
|
||||
await repo.findSingleton();
|
||||
await repo.markDegraded([probe("1.1.1.1", false)], NOW);
|
||||
|
||||
const state = await repo.recordProbe([probe("1.1.1.1", false), probe("8.8.8.8", false)], LATER);
|
||||
|
||||
expect(state.status).toBe("degraded");
|
||||
expect(state.lastProbeAt).toBe(LATER.toISOString());
|
||||
expect(state.lastProbeResults.map((result) => result.target)).toEqual(["1.1.1.1", "8.8.8.8"]);
|
||||
});
|
||||
});
|
||||
|
||||
describe("reset", () => {
|
||||
it("returns a degraded singleton to ok with no episode and keeps the recovery history", async () => {
|
||||
await repo.findSingleton();
|
||||
await repo.markDegraded([probe("1.1.1.1", false)], NOW);
|
||||
await repo.markRecovered([probe("1.1.1.1", true)], LATER);
|
||||
await repo.markDegraded([probe("1.1.1.1", false)], LATER);
|
||||
|
||||
const state = await repo.reset();
|
||||
|
||||
expect(state.status).toBe("ok");
|
||||
expect(state.degradedSince).toBeNull();
|
||||
expect(state.lastProbeAt).toBeNull();
|
||||
expect(state.lastProbeResults).toEqual([]);
|
||||
expect(state.lastRecoveredAt).toBe(LATER.toISOString());
|
||||
});
|
||||
|
||||
it("creates the singleton when none exists", async () => {
|
||||
const state = await repo.reset();
|
||||
|
||||
expect(state.status).toBe("ok");
|
||||
expect(await EgressStateModel.countDocuments()).toBe(1);
|
||||
});
|
||||
});
|
||||
});
|
||||
@@ -3,7 +3,7 @@ import mongoose from "mongoose";
|
||||
import { MongoMemoryServer } from "mongodb-memory-server";
|
||||
import MongoJobsRepository from "../../src/domain/jobs/job.repository.mongo.ts";
|
||||
import JobModel, { type JobDocument } from "../../src/domain/jobs/job.model.ts";
|
||||
import { BACKOFF_MS, LOCK_MS, type Job, type JobType } from "../../src/domain/jobs/job.type.ts";
|
||||
import { BACKOFF_MS, LOCK_MS, PARKED, type Job, type JobType } from "../../src/domain/jobs/job.type.ts";
|
||||
|
||||
// ── Real-Mongo harness ─────────────────────────────────────────────────────────
|
||||
// The headline guarantee of the queue — "exactly one claimer wins" — IS MongoDB's
|
||||
@@ -180,7 +180,7 @@ describe("MongoJobsRepository", () => {
|
||||
|
||||
it("does not release a lock owned by another worker (fenced no-op)", async () => {
|
||||
// A worker whose lease expired and was reclaimed must not clobber the new owner's lock.
|
||||
await seedJob({ lockedBy: "other-worker", lockedUntil: NOW + LOCK_MS, runCount: 7 });
|
||||
await seedJob({ lockedBy: "other-worker", lockedUntil: NOW + LOCK_MS, runCount: 7, failCount: 2, lastFailReason: "timeout" });
|
||||
|
||||
const ok = await repo.recordSuccess("check:mon-1", NOW, 60_000, NOW + 1_000);
|
||||
|
||||
@@ -189,6 +189,55 @@ describe("MongoJobsRepository", () => {
|
||||
expect(row?.lockedBy).toBe("other-worker"); // lock untouched
|
||||
expect(row?.lockedUntil).toBe(NOW + LOCK_MS);
|
||||
expect(row?.runCount).toBe(7); // not double-counted
|
||||
expect(row?.failCount).toBe(2); // failure state untouched
|
||||
expect(row?.lastFailReason).toBe("timeout");
|
||||
});
|
||||
|
||||
it("clears the failure state so failCount counts consecutive failures", async () => {
|
||||
await seedJob({ lockedBy: ownedBy(repo), lockedUntil: NOW + LOCK_MS, failCount: 3, lastFailReason: "connection refused" });
|
||||
|
||||
await repo.recordSuccess("check:mon-1", NOW, 60_000, NOW + 1_000);
|
||||
|
||||
const row = await readRow("check:mon-1");
|
||||
expect(row?.failCount).toBe(0);
|
||||
expect(row?.lastFailReason).toBeNull();
|
||||
});
|
||||
});
|
||||
|
||||
// ── recordOneShot (park after a single run) ─────────────────────────────────
|
||||
describe("recordOneShot", () => {
|
||||
it("parks the job, releases the lease, counts the run and clears the failure state", async () => {
|
||||
await seedJob({
|
||||
intervalMs: null,
|
||||
lockedBy: ownedBy(repo),
|
||||
lockedUntil: NOW + LOCK_MS,
|
||||
runCount: 1,
|
||||
failCount: 2,
|
||||
lastFailReason: "connection refused",
|
||||
});
|
||||
|
||||
const ok = await repo.recordOneShot("check:mon-1", NOW + 1_000);
|
||||
|
||||
expect(ok).toBe(true);
|
||||
const row = await readRow("check:mon-1");
|
||||
expect(row?.nextScheduledAt).toBe(PARKED);
|
||||
expect(row?.lockedBy).toBeNull();
|
||||
expect(row?.lockedUntil).toBeNull();
|
||||
expect(row?.lastFinishedAt).toBe(NOW + 1_000);
|
||||
expect(row?.runCount).toBe(2);
|
||||
expect(row?.failCount).toBe(0);
|
||||
expect(row?.lastFailReason).toBeNull();
|
||||
});
|
||||
|
||||
it("does not touch a lock owned by another worker (fenced no-op)", async () => {
|
||||
await seedJob({ intervalMs: null, lockedBy: "other-worker", lockedUntil: NOW + LOCK_MS, failCount: 2, lastFailReason: "timeout" });
|
||||
|
||||
expect(await repo.recordOneShot("check:mon-1", NOW + 1_000)).toBe(false);
|
||||
const row = await readRow("check:mon-1");
|
||||
expect(row?.nextScheduledAt).toBe(NOW); // not parked
|
||||
expect(row?.lockedBy).toBe("other-worker");
|
||||
expect(row?.failCount).toBe(2);
|
||||
expect(row?.lastFailReason).toBe("timeout");
|
||||
});
|
||||
});
|
||||
|
||||
@@ -533,4 +582,47 @@ describe("MongoJobsRepository", () => {
|
||||
expect(await repo.countDueBacklog(NOW)).toBe(0);
|
||||
});
|
||||
});
|
||||
|
||||
// ── global rows: egress recovery job ──────────────────────────────────────
|
||||
|
||||
describe("global rows", () => {
|
||||
const repo = new MongoJobsRepository(WORKER_ID);
|
||||
const egressRow = (nextScheduledAt: number) =>
|
||||
seedJob({ _id: "egress", type: "egress" as JobType, refId: null, nextScheduledAt, intervalMs: 30_000 });
|
||||
|
||||
it("deleteGlobalJob removes the row by its canonical _id and leaves other global rows alone", async () => {
|
||||
await egressRow(NOW);
|
||||
await seedJob({ _id: "cleanup-orphaned", type: "cleanup-orphaned" as JobType, refId: null });
|
||||
|
||||
expect(await repo.deleteGlobalJob("egress")).toBe(true);
|
||||
|
||||
expect(await readRow("egress")).toBeNull();
|
||||
expect(await readRow("cleanup-orphaned")).not.toBeNull();
|
||||
expect(await repo.deleteGlobalJob("egress")).toBe(false);
|
||||
});
|
||||
|
||||
it("deleteGlobalJobIfUnchanged removes the row only while nextScheduledAt matches the claimed value", async () => {
|
||||
await egressRow(NOW);
|
||||
|
||||
// Another worker started a new episode and rescheduled the row after this run claimed it
|
||||
await JobModel.updateOne({ _id: "egress" }, { $set: { nextScheduledAt: NOW + 30_000 } });
|
||||
expect(await repo.deleteGlobalJobIfUnchanged("egress", NOW)).toBe(false);
|
||||
expect(await readRow("egress")).not.toBeNull();
|
||||
|
||||
// Nobody touched it: the row goes
|
||||
expect(await repo.deleteGlobalJobIfUnchanged("egress", NOW + 30_000)).toBe(true);
|
||||
expect(await readRow("egress")).toBeNull();
|
||||
});
|
||||
|
||||
it("recordSuccess after a refused conditional delete is a no-op once the lock has moved on", async () => {
|
||||
await egressRow(NOW);
|
||||
const [claimed] = await repo.claimDueBatch("egress", 1, NOW);
|
||||
expect(claimed?.id).toBe("egress");
|
||||
|
||||
// Row deleted by the handler (recovered), then recordSuccess runs from the queue: nothing to update
|
||||
expect(await repo.deleteGlobalJobIfUnchanged("egress", claimed!.nextScheduledAt)).toBe(true);
|
||||
expect(await repo.recordSuccess("egress", claimed!.nextScheduledAt, 30_000, NOW + 1_000)).toBe(false);
|
||||
expect(await readRow("egress")).toBeNull();
|
||||
});
|
||||
});
|
||||
});
|
||||
@@ -123,4 +123,70 @@ describe("MongoMonitorsRepository", () => {
|
||||
expect(monitors[0].recentChecks).toEqual([]);
|
||||
});
|
||||
});
|
||||
|
||||
describe("docker tls key access", () => {
|
||||
const seedDockerMonitor = (overrides: Record<string, unknown> = {}) =>
|
||||
MonitorModel.create({
|
||||
userId: new mongoose.Types.ObjectId(),
|
||||
teamId: new mongoose.Types.ObjectId(),
|
||||
name: "docker",
|
||||
type: "docker",
|
||||
url: "tcp://host",
|
||||
...overrides,
|
||||
});
|
||||
|
||||
it("never returns the key on a normal load, even with the flag set", async () => {
|
||||
const repo = new MongoMonitorsRepository();
|
||||
const doc = await seedDockerMonitor({ dockerTlsKey: "v1.abc123.iv.tag.data", dockerTlsKeySet: true });
|
||||
|
||||
const monitor = await repo.findByIdLean(doc._id.toString());
|
||||
|
||||
expect(monitor?.dockerTlsKeySet).toBe(true);
|
||||
expect(monitor?.dockerTlsKey).toBeUndefined();
|
||||
});
|
||||
|
||||
it("findDockerTlsKeyById returns the stored key despite select: false", async () => {
|
||||
const repo = new MongoMonitorsRepository();
|
||||
const doc = await seedDockerMonitor({ dockerTlsKey: "v1.abc123.iv.tag.data", dockerTlsKeySet: true });
|
||||
|
||||
await expect(repo.findDockerTlsKeyById(doc._id.toString())).resolves.toBe("v1.abc123.iv.tag.data");
|
||||
});
|
||||
|
||||
it("findDockerTlsKeyById returns null for a monitor without a key and for an unknown id", async () => {
|
||||
const repo = new MongoMonitorsRepository();
|
||||
const doc = await seedDockerMonitor({ url: "unix:///var/run/docker.sock" });
|
||||
|
||||
await expect(repo.findDockerTlsKeyById(doc._id.toString())).resolves.toBeNull();
|
||||
await expect(repo.findDockerTlsKeyById(new mongoose.Types.ObjectId().toString())).resolves.toBeNull();
|
||||
});
|
||||
|
||||
it("findAllDockerTlsKeys returns only rows with the flag set and a key present", async () => {
|
||||
const repo = new MongoMonitorsRepository();
|
||||
const keyed = await seedDockerMonitor({ dockerTlsKey: "v1.abc123.k1", dockerTlsKeySet: true });
|
||||
await seedDockerMonitor({ url: "unix:///var/run/docker.sock" });
|
||||
await seedDockerMonitor({ dockerTlsKeySet: true }); // flag without key: inconsistent, dropped
|
||||
await MonitorModel.create({
|
||||
userId: new mongoose.Types.ObjectId(),
|
||||
teamId: new mongoose.Types.ObjectId(),
|
||||
name: "http",
|
||||
type: "http",
|
||||
url: "https://example.com",
|
||||
});
|
||||
|
||||
await expect(repo.findAllDockerTlsKeys()).resolves.toEqual([{ id: keyed._id.toString(), dockerTlsKey: "v1.abc123.k1" }]);
|
||||
});
|
||||
|
||||
it("updateDockerTlsKey replaces the key and nothing else", async () => {
|
||||
const repo = new MongoMonitorsRepository();
|
||||
const doc = await seedDockerMonitor({ dockerTlsKey: "v1.old.k", dockerTlsKeySet: true, dockerTlsCert: "cert" });
|
||||
|
||||
await repo.updateDockerTlsKey(doc._id.toString(), "v1.new.k");
|
||||
|
||||
const after = await MonitorModel.findById(doc._id).select("+dockerTlsKey").lean();
|
||||
expect(after?.dockerTlsKey).toBe("v1.new.k");
|
||||
expect(after?.dockerTlsCert).toBe("cert");
|
||||
expect(after?.dockerTlsKeySet).toBe(true);
|
||||
expect(after?.name).toBe("docker");
|
||||
});
|
||||
});
|
||||
});
|
||||
@@ -1,5 +1,7 @@
|
||||
import { describe, expect, it, jest } from "@jest/globals";
|
||||
import https from "node:https";
|
||||
import { DockerProvider } from "../../../../src/service/network/DockerProvider.ts";
|
||||
import type { IEncryptionService } from "../../../../src/service/encryption/encryptionService.ts";
|
||||
import { testStatusProviderContract } from "../../../helpers/statusProviderContract.ts";
|
||||
import { createMockLogger } from "../../../helpers/createMockLogger.ts";
|
||||
import { NETWORK_ERROR } from "../../../../src/types/network.ts";
|
||||
@@ -9,6 +11,28 @@ import { DOCKER_LOG_TAIL_LINES } from "../../../../src/domain/docker/docker.type
|
||||
|
||||
// ── Fixtures ─────────────────────────────────────────────────────────────────
|
||||
|
||||
const CA_PEM = "-----BEGIN CERTIFICATE-----\nca-one\n-----END CERTIFICATE-----";
|
||||
const CA_BUNDLE_PEM = `${CA_PEM}\n-----BEGIN CERTIFICATE-----\nca-two\n-----END CERTIFICATE-----\n`;
|
||||
const CERT_PEM = "-----BEGIN CERTIFICATE-----\nclient\n-----END CERTIFICATE-----";
|
||||
const KEY_PEM = "-----BEGIN PRIVATE KEY-----\nstub\n-----END PRIVATE KEY-----";
|
||||
const KEY_CIPHERTEXT = "v1.abc123.iv.tag.data";
|
||||
|
||||
const makeEncryptionService = (overrides?: Partial<IEncryptionService>): IEncryptionService => ({
|
||||
isConfigured: jest.fn(() => true),
|
||||
currentKeyId: jest.fn(() => "abc123"),
|
||||
keyIdOf: jest.fn(() => "abc123"),
|
||||
needsReencryption: jest.fn(() => false),
|
||||
encrypt: jest.fn(() => KEY_CIPHERTEXT),
|
||||
decrypt: jest.fn(() => KEY_PEM),
|
||||
...overrides,
|
||||
});
|
||||
|
||||
const TIMEOUT_MS = 10000;
|
||||
const TIMEOUTS = { timeout: TIMEOUT_MS, connectionTimeout: TIMEOUT_MS };
|
||||
|
||||
const makeTlsMonitor = (overrides?: Partial<Monitor>): Monitor =>
|
||||
makeMonitor({ url: "tcp://host", dockerTlsCa: CA_PEM, dockerTlsCert: CERT_PEM, dockerTlsKeySet: true, ...overrides });
|
||||
|
||||
const makeMonitor = (overrides?: Partial<Monitor>): Monitor =>
|
||||
({
|
||||
id: "mon-1",
|
||||
@@ -60,7 +84,17 @@ const makeLogFrame = (streamType: number, text: string): Buffer => {
|
||||
return Buffer.concat([header, payload]);
|
||||
};
|
||||
|
||||
const setup = (overrides: Partial<{ ping: any; listContainers: any; getContainer: any; inspect: any; stats: any; logs: any }> = {}) => {
|
||||
const setup = (
|
||||
overrides: Partial<{
|
||||
ping: any;
|
||||
listContainers: any;
|
||||
getContainer: any;
|
||||
inspect: any;
|
||||
stats: any;
|
||||
logs: any;
|
||||
encryptionService: IEncryptionService;
|
||||
}> = {}
|
||||
) => {
|
||||
const inspect = overrides.inspect ?? jest.fn().mockResolvedValue(makeInspect());
|
||||
const stats = overrides.stats ?? jest.fn().mockResolvedValue(makeStats());
|
||||
const logs = overrides.logs ?? jest.fn().mockResolvedValue(Buffer.alloc(0));
|
||||
@@ -72,8 +106,20 @@ const setup = (overrides: Partial<{ ping: any; listContainers: any; getContainer
|
||||
};
|
||||
const DockerLib = jest.fn().mockReturnValue(instance) as any;
|
||||
const logger = createMockLogger();
|
||||
const provider = new DockerProvider(logger as any, DockerLib);
|
||||
return { provider, logger, DockerLib, ping: instance.ping, listContainers: instance.listContainers, getContainer, inspect, stats, logs };
|
||||
const encryptionService = overrides.encryptionService ?? makeEncryptionService();
|
||||
const provider = new DockerProvider(logger as any, DockerLib, encryptionService);
|
||||
return {
|
||||
provider,
|
||||
logger,
|
||||
DockerLib,
|
||||
encryptionService,
|
||||
ping: instance.ping,
|
||||
listContainers: instance.listContainers,
|
||||
getContainer,
|
||||
inspect,
|
||||
stats,
|
||||
logs,
|
||||
};
|
||||
};
|
||||
|
||||
// ── Contract ─────────────────────────────────────────────────────────────────
|
||||
@@ -96,7 +142,7 @@ describe("DockerProvider", () => {
|
||||
|
||||
await provider.handle(makeMonitor({ url: "unix:///run/docker.sock" }));
|
||||
|
||||
expect(DockerLib).toHaveBeenCalledWith({ socketPath: "/run/docker.sock" });
|
||||
expect(DockerLib).toHaveBeenCalledWith({ socketPath: "/run/docker.sock", ...TIMEOUTS });
|
||||
});
|
||||
|
||||
it("parses bare absolute paths into socketPath", async () => {
|
||||
@@ -104,7 +150,7 @@ describe("DockerProvider", () => {
|
||||
|
||||
await provider.handle(makeMonitor({ url: "/var/run/docker.sock" }));
|
||||
|
||||
expect(DockerLib).toHaveBeenCalledWith({ socketPath: "/var/run/docker.sock" });
|
||||
expect(DockerLib).toHaveBeenCalledWith({ socketPath: "/var/run/docker.sock", ...TIMEOUTS });
|
||||
});
|
||||
|
||||
it("trims surrounding whitespace", async () => {
|
||||
@@ -112,7 +158,130 @@ describe("DockerProvider", () => {
|
||||
|
||||
await provider.handle(makeMonitor({ url: " /var/run/docker.sock " }));
|
||||
|
||||
expect(DockerLib).toHaveBeenCalledWith({ socketPath: "/var/run/docker.sock" });
|
||||
expect(DockerLib).toHaveBeenCalledWith({ socketPath: "/var/run/docker.sock", ...TIMEOUTS });
|
||||
});
|
||||
});
|
||||
|
||||
// ── TLS host urls ────────────────────────────────────────────────────
|
||||
|
||||
describe("tls host urls", () => {
|
||||
const ctx = { dockerTlsKey: KEY_CIPHERTEXT };
|
||||
const optionsPassedTo = (DockerLib: jest.Mock) => DockerLib.mock.calls[0]?.[0] as Record<string, unknown>;
|
||||
const agentOf = (options: Record<string, unknown>) => options.agent as https.Agent & { options: Record<string, unknown> };
|
||||
|
||||
it("parses tcp:// into an https connection with the monitor's ca and cert and the decrypted key", async () => {
|
||||
const { provider, DockerLib, encryptionService } = setup();
|
||||
|
||||
await provider.handle(makeTlsMonitor(), ctx);
|
||||
|
||||
expect(encryptionService.decrypt).toHaveBeenCalledWith(KEY_CIPHERTEXT);
|
||||
const options = optionsPassedTo(DockerLib);
|
||||
expect(options).toMatchObject({ host: "host", port: 2376, protocol: "https", ca: [CA_PEM], cert: CERT_PEM, key: KEY_PEM });
|
||||
expect(agentOf(options)).toBeInstanceOf(https.Agent);
|
||||
expect(agentOf(options).options).toMatchObject({ ca: [CA_PEM], cert: CERT_PEM, key: KEY_PEM, rejectUnauthorized: true });
|
||||
});
|
||||
|
||||
it("parses https:// with an explicit port", async () => {
|
||||
const { provider, DockerLib } = setup();
|
||||
|
||||
await provider.handle(makeTlsMonitor({ url: "https://host:2377" }), ctx);
|
||||
|
||||
expect(optionsPassedTo(DockerLib)).toMatchObject({ host: "host", port: 2377, protocol: "https" });
|
||||
});
|
||||
|
||||
it("accepts a trailing slash", async () => {
|
||||
const { provider, DockerLib } = setup();
|
||||
|
||||
await provider.handle(makeTlsMonitor({ url: "tcp://host:2376/" }), ctx);
|
||||
|
||||
expect(optionsPassedTo(DockerLib)).toMatchObject({ host: "host", port: 2376 });
|
||||
});
|
||||
|
||||
it("splits a ca bundle into one entry per certificate on the options and the agent", async () => {
|
||||
const { provider, DockerLib } = setup();
|
||||
|
||||
await provider.handle(makeTlsMonitor({ dockerTlsCa: CA_BUNDLE_PEM }), ctx);
|
||||
|
||||
const options = optionsPassedTo(DockerLib);
|
||||
expect(options.ca).toHaveLength(2);
|
||||
expect(agentOf(options).options.ca).toEqual(options.ca);
|
||||
});
|
||||
|
||||
it("omits the ca and stops rejecting unauthorized certificates when ignoreTlsErrors is set", async () => {
|
||||
const { provider, DockerLib } = setup();
|
||||
|
||||
await provider.handle(makeTlsMonitor({ ignoreTlsErrors: true }), ctx);
|
||||
|
||||
const options = optionsPassedTo(DockerLib);
|
||||
expect(options.ca).toBeUndefined();
|
||||
expect(agentOf(options).options).toMatchObject({ ca: undefined, rejectUnauthorized: false });
|
||||
});
|
||||
|
||||
it("passes explicit undefined for every field docker-modem would otherwise fill from the environment", async () => {
|
||||
const { provider, DockerLib } = setup();
|
||||
|
||||
await provider.handle(makeTlsMonitor(), ctx);
|
||||
|
||||
const options = optionsPassedTo(DockerLib);
|
||||
for (const field of ["socketPath", "username", "sshOptions"]) {
|
||||
expect(options).toHaveProperty(field, undefined);
|
||||
}
|
||||
});
|
||||
|
||||
it("bounds every request with the provider timeout so a filtered port fails fast", async () => {
|
||||
const { provider, DockerLib } = setup();
|
||||
|
||||
await provider.handle(makeTlsMonitor(), ctx);
|
||||
|
||||
expect(optionsPassedTo(DockerLib)).toMatchObject(TIMEOUTS);
|
||||
});
|
||||
|
||||
it("returns a down check and never constructs a client when the key is missing from the context", async () => {
|
||||
const { provider, DockerLib, encryptionService } = setup();
|
||||
|
||||
const result = await provider.handle(makeTlsMonitor());
|
||||
|
||||
expect(result).toMatchObject({ status: false, code: NETWORK_ERROR, message: "Docker TLS key is missing", payload: null });
|
||||
expect(encryptionService.decrypt).not.toHaveBeenCalled();
|
||||
expect(DockerLib).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("returns a down check naming the key id when the stored key cannot be decrypted", async () => {
|
||||
const decrypt = jest.fn(() => {
|
||||
throw new AppError({
|
||||
message: "No encryption key matches ciphertext",
|
||||
status: 500,
|
||||
service: "EncryptionService",
|
||||
method: "decrypt",
|
||||
details: { keyId: "abc123" },
|
||||
});
|
||||
});
|
||||
const { provider, DockerLib } = setup({ encryptionService: makeEncryptionService({ decrypt }) });
|
||||
|
||||
const result = await provider.handle(makeTlsMonitor(), ctx);
|
||||
|
||||
expect(result).toMatchObject({ status: false, code: NETWORK_ERROR, message: "No encryption key matches ciphertext (key id abc123)" });
|
||||
expect(DockerLib).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("returns a generic down check when decryption fails with a non-AppError", async () => {
|
||||
const decrypt = jest.fn(() => {
|
||||
throw new Error("boom");
|
||||
});
|
||||
const { provider, DockerLib } = setup({ encryptionService: makeEncryptionService({ decrypt }) });
|
||||
|
||||
const result = await provider.handle(makeTlsMonitor(), ctx);
|
||||
|
||||
expect(result).toMatchObject({ status: false, message: "Docker TLS key could not be decrypted" });
|
||||
expect(DockerLib).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("does not touch the encryption service for socket monitors", async () => {
|
||||
const { provider, encryptionService } = setup();
|
||||
|
||||
await provider.handle(makeMonitor(), ctx);
|
||||
|
||||
expect(encryptionService.decrypt).not.toHaveBeenCalled();
|
||||
});
|
||||
});
|
||||
|
||||
@@ -125,8 +294,9 @@ describe("DockerProvider", () => {
|
||||
["unix:// with a relative path", "unix://run/docker.sock", "Invalid Docker host URL"],
|
||||
["garbage", "not-a-url", "Invalid Docker host URL"],
|
||||
["a container name (old semantics)", "my-container", "Invalid Docker host URL"],
|
||||
["tcp engine urls (unsupported)", "tcp://host:2375", "Invalid Docker host URL"],
|
||||
["https engine urls (unsupported)", "https://host", "Invalid Docker host URL"],
|
||||
["http engine urls (plaintext tcp is unsupported)", "http://host", "Invalid Docker host URL"],
|
||||
["a bare host and port", "host:2376", "Invalid Docker host URL"],
|
||||
["a tcp url with a path", "tcp://host/engine", "Invalid Docker host URL"],
|
||||
["ssh engine urls (unsupported)", "ssh://deploy@host", "Invalid Docker host URL"],
|
||||
];
|
||||
|
||||
|
||||
@@ -33,7 +33,7 @@ const createMockHttpProvider = () =>
|
||||
|
||||
const createMockSettingsService = (apiKey?: string) =>
|
||||
({
|
||||
getDBSettings: jest.fn().mockResolvedValue({ pagespeedApiKey: apiKey }),
|
||||
getCachedDBSettings: jest.fn().mockResolvedValue({ pagespeedApiKey: apiKey }),
|
||||
}) as unknown as jest.Mocked<ISettingsService>;
|
||||
|
||||
const createProvider = (opts?: { apiKey?: string }) => {
|
||||
@@ -154,7 +154,7 @@ describe("PageSpeedProvider", () => {
|
||||
const logger = createMockLogger();
|
||||
const httpProvider = createMockHttpProvider();
|
||||
const settingsService = {
|
||||
getDBSettings: jest.fn().mockResolvedValue(null),
|
||||
getCachedDBSettings: jest.fn().mockResolvedValue(null),
|
||||
} as unknown as jest.Mocked<ISettingsService>;
|
||||
const provider = new PageSpeedProvider(httpProvider, settingsService, logger as any);
|
||||
|
||||
|
||||
@@ -2,6 +2,7 @@ import { describe, expect, it, jest } from "@jest/globals";
|
||||
import { PingProvider } from "../../../../src/service/network/PingProvider.ts";
|
||||
import { testStatusProviderContract } from "../../../helpers/statusProviderContract.ts";
|
||||
import type { Monitor } from "../../../../src/domain/monitors/monitor.type.ts";
|
||||
import * as net from "net";
|
||||
|
||||
// ── Helpers ──────────────────────────────────────────────────────────────────
|
||||
|
||||
@@ -27,7 +28,7 @@ const createMockPing = (result?: Partial<{ alive: boolean; time: number | string
|
||||
// ── Contract ─────────────────────────────────────────────────────────────────
|
||||
|
||||
testStatusProviderContract("PingProvider", {
|
||||
create: () => new PingProvider(createMockPing() as any),
|
||||
create: () => new PingProvider(createMockPing() as any, net),
|
||||
supportedType: "ping",
|
||||
unsupportedType: "http",
|
||||
makeMonitor: () => makeMonitor(),
|
||||
@@ -37,7 +38,7 @@ testStatusProviderContract("PingProvider", {
|
||||
|
||||
describe("PingProvider", () => {
|
||||
it("returns success when host is alive", async () => {
|
||||
const provider = new PingProvider(createMockPing({ alive: true, time: 30 }) as any);
|
||||
const provider = new PingProvider(createMockPing({ alive: true, time: 30 }) as any, net);
|
||||
|
||||
const result = await provider.handle(makeMonitor());
|
||||
|
||||
@@ -55,7 +56,7 @@ describe("PingProvider", () => {
|
||||
});
|
||||
|
||||
it("returns failure when host is not alive", async () => {
|
||||
const provider = new PingProvider(createMockPing({ alive: false, time: 0 }) as any);
|
||||
const provider = new PingProvider(createMockPing({ alive: false, time: 0 }) as any, net);
|
||||
|
||||
const result = await provider.handle(makeMonitor());
|
||||
|
||||
@@ -69,7 +70,7 @@ describe("PingProvider", () => {
|
||||
});
|
||||
|
||||
it("parses string time values", async () => {
|
||||
const provider = new PingProvider(createMockPing({ alive: true, time: "42.5" }) as any);
|
||||
const provider = new PingProvider(createMockPing({ alive: true, time: "42.5" }) as any, net);
|
||||
|
||||
const result = await provider.handle(makeMonitor());
|
||||
|
||||
@@ -77,7 +78,7 @@ describe("PingProvider", () => {
|
||||
});
|
||||
|
||||
it("defaults responseTime to 0 for non-numeric time", async () => {
|
||||
const provider = new PingProvider(createMockPing({ alive: true, time: "unknown" }) as any);
|
||||
const provider = new PingProvider(createMockPing({ alive: true, time: "unknown" }) as any, net);
|
||||
|
||||
const result = await provider.handle(makeMonitor());
|
||||
|
||||
@@ -85,7 +86,7 @@ describe("PingProvider", () => {
|
||||
});
|
||||
|
||||
it("defaults status to false when alive is undefined", async () => {
|
||||
const provider = new PingProvider(createMockPing({ alive: undefined as any }) as any);
|
||||
const provider = new PingProvider(createMockPing({ alive: undefined as any }) as any, net);
|
||||
|
||||
const result = await provider.handle(makeMonitor());
|
||||
|
||||
@@ -94,7 +95,7 @@ describe("PingProvider", () => {
|
||||
|
||||
it("sanitizes URL by stripping protocol and path", async () => {
|
||||
const mockPing = createMockPing();
|
||||
const provider = new PingProvider(mockPing as any);
|
||||
const provider = new PingProvider(mockPing as any, net);
|
||||
|
||||
await provider.handle(makeMonitor({ url: "https://example.com/path?query=1" }));
|
||||
|
||||
@@ -103,36 +104,54 @@ describe("PingProvider", () => {
|
||||
|
||||
it("sanitizes URL by stripping port", async () => {
|
||||
const mockPing = createMockPing();
|
||||
const provider = new PingProvider(mockPing as any);
|
||||
const provider = new PingProvider(mockPing as any, net);
|
||||
|
||||
await provider.handle(makeMonitor({ url: "example.com:8080" }));
|
||||
|
||||
expect(mockPing.promise.probe).toHaveBeenCalledWith("example.com");
|
||||
});
|
||||
|
||||
it("keeps a bare IPv6 address intact", async () => {
|
||||
const mockPing = createMockPing();
|
||||
const provider = new PingProvider(mockPing as any, net);
|
||||
|
||||
await provider.handle(makeMonitor({ url: "2606:4700:4700::1111" }));
|
||||
|
||||
expect(mockPing.promise.probe).toHaveBeenCalledWith("2606:4700:4700::1111");
|
||||
});
|
||||
|
||||
it("unwraps a bracketed IPv6 address and drops its port", async () => {
|
||||
const mockPing = createMockPing();
|
||||
const provider = new PingProvider(mockPing as any, net);
|
||||
|
||||
await provider.handle(makeMonitor({ url: "[2606:4700:4700::1111]:53" }));
|
||||
|
||||
expect(mockPing.promise.probe).toHaveBeenCalledWith("2606:4700:4700::1111");
|
||||
});
|
||||
|
||||
it("throws AppError when url is missing", async () => {
|
||||
const provider = new PingProvider(createMockPing() as any);
|
||||
const provider = new PingProvider(createMockPing() as any, net);
|
||||
|
||||
await expect(provider.handle(makeMonitor({ url: "" }))).rejects.toThrow("URL is required for ping monitor");
|
||||
});
|
||||
|
||||
it("throws AppError when ping.probe rejects", async () => {
|
||||
const mockPing = { promise: { probe: jest.fn().mockRejectedValue(new Error("Host unreachable")) } };
|
||||
const provider = new PingProvider(mockPing as any);
|
||||
const provider = new PingProvider(mockPing as any, net);
|
||||
|
||||
await expect(provider.handle(makeMonitor())).rejects.toThrow("Host unreachable");
|
||||
});
|
||||
|
||||
it("throws AppError when probe returns no response", async () => {
|
||||
const mockPing = { promise: { probe: jest.fn().mockResolvedValue(null) } };
|
||||
const provider = new PingProvider(mockPing as any);
|
||||
const provider = new PingProvider(mockPing as any, net);
|
||||
|
||||
await expect(provider.handle(makeMonitor())).rejects.toThrow("No response from ping");
|
||||
});
|
||||
|
||||
it("throws AppError with stringified message for non-Error throws", async () => {
|
||||
const mockPing = { promise: { probe: jest.fn().mockRejectedValue(42) } };
|
||||
const provider = new PingProvider(mockPing as any);
|
||||
const provider = new PingProvider(mockPing as any, net);
|
||||
|
||||
await expect(provider.handle(makeMonitor())).rejects.toThrow("42");
|
||||
});
|
||||
|
||||
@@ -22,7 +22,7 @@ const activeWindow = () => {
|
||||
const createProducer = (overrides?: Record<string, any>) => {
|
||||
const defaults = {
|
||||
logger: createMockLogger(),
|
||||
monitorsRepository: { updateById: jest.fn().mockResolvedValue({}) },
|
||||
monitorsRepository: { updateById: jest.fn().mockResolvedValue({}), findDockerTlsKeyById: jest.fn().mockResolvedValue(null) },
|
||||
maintenanceWindowsRepository: { findByMonitorId: jest.fn().mockResolvedValue([]) },
|
||||
checkService: { toCheck: jest.fn().mockReturnValue({ id: "check-1" }) },
|
||||
networkService: { requestStatus: jest.fn().mockResolvedValue({ monitorId: "m1", status: true, code: 200, message: "OK" }) },
|
||||
@@ -111,6 +111,48 @@ describe("CheckProducer", () => {
|
||||
|
||||
// ── acquire / record ──────────────────────────────────────────────────────
|
||||
|
||||
// ── docker tls key resolution ─────────────────────────────────────────────
|
||||
|
||||
it("fetches the stored key for a docker monitor with a key set and passes it in the context", async () => {
|
||||
const { producer, defaults } = createProducer({
|
||||
monitorsRepository: { updateById: jest.fn(), findDockerTlsKeyById: jest.fn().mockResolvedValue("v1.abc123.iv.tag.data") },
|
||||
});
|
||||
const monitor = makeMonitor({ type: "docker", url: "tcp://host", dockerTlsKeySet: true });
|
||||
|
||||
await producer.produce(monitor);
|
||||
|
||||
expect(defaults.monitorsRepository.findDockerTlsKeyById).toHaveBeenCalledWith("m1");
|
||||
expect(defaults.networkService.requestStatus).toHaveBeenCalledWith(monitor, { proxyUrl: undefined, dockerTlsKey: "v1.abc123.iv.tag.data" });
|
||||
});
|
||||
|
||||
it("does not fetch a key for a docker monitor without one set", async () => {
|
||||
const { producer, defaults } = createProducer();
|
||||
const monitor = makeMonitor({ type: "docker", url: "unix:///var/run/docker.sock", dockerTlsKeySet: false });
|
||||
|
||||
await producer.produce(monitor);
|
||||
|
||||
expect(defaults.monitorsRepository.findDockerTlsKeyById).not.toHaveBeenCalled();
|
||||
expect(defaults.networkService.requestStatus).toHaveBeenCalledWith(monitor, { proxyUrl: undefined, dockerTlsKey: undefined });
|
||||
});
|
||||
|
||||
it("does not fetch a key for non-docker monitors even if the flag is set", async () => {
|
||||
const { producer, defaults } = createProducer();
|
||||
|
||||
await producer.produce(makeMonitor({ type: "http", dockerTlsKeySet: true }));
|
||||
|
||||
expect(defaults.monitorsRepository.findDockerTlsKeyById).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("passes undefined when the flag is set but the repository has no key", async () => {
|
||||
const { producer, defaults } = createProducer();
|
||||
const monitor = makeMonitor({ type: "docker", url: "tcp://host", dockerTlsKeySet: true });
|
||||
|
||||
await producer.produce(monitor);
|
||||
|
||||
expect(defaults.monitorsRepository.findDockerTlsKeyById).toHaveBeenCalledWith("m1");
|
||||
expect(defaults.networkService.requestStatus).toHaveBeenCalledWith(monitor, { proxyUrl: undefined, dockerTlsKey: undefined });
|
||||
});
|
||||
|
||||
it("throws when the network response is null", async () => {
|
||||
const { producer } = createProducer({
|
||||
networkService: { requestStatus: jest.fn().mockResolvedValue(null) },
|
||||
|
||||
@@ -9,7 +9,7 @@ import { createMockLogger } from "../../helpers/createMockLogger.ts";
|
||||
// ── Notes ──────────────────────────────────────────────────────────────────────
|
||||
//
|
||||
// DBQueueWorker is the orchestrator: it drives the polling loops, claims batches
|
||||
// sized to free capacity, runs jobs (check / evaluate / geo-check / cleanup) and
|
||||
// sized to free capacity, runs jobs (check / evaluate / geo-check / cleanup / egress) and
|
||||
// renews their locks, and registers itself in the worker heartbeat. The repository
|
||||
// claim/lock atomicity is covered in integration/jobsRepository.test.ts; here we
|
||||
// test only the orchestration around it, with the repository fully mocked.
|
||||
@@ -113,6 +113,7 @@ const createWorker = (overrides?: { queueMode?: QueueMode; queuePrimaryProcesses
|
||||
const helper = {
|
||||
getCleanupOrphanedJob: jest.fn<any>().mockReturnValue(jest.fn<any>().mockResolvedValue(undefined)),
|
||||
getCleanupRetentionJob: jest.fn<any>().mockReturnValue(jest.fn<any>().mockResolvedValue(undefined)),
|
||||
getEgressRecoveryJob: jest.fn<any>().mockReturnValue(jest.fn<any>().mockResolvedValue(undefined)),
|
||||
};
|
||||
const queueWorkersRepository = {
|
||||
upsert: jest.fn<any>().mockResolvedValue(undefined),
|
||||
@@ -226,7 +227,7 @@ describe("DBQueueWorker", () => {
|
||||
const { mocks } = await start({ queueMode: "worker" });
|
||||
|
||||
const claimedTypes = mocks.jobsRepository.claimDueBatch.mock.calls.map((c: any[]) => c[0]);
|
||||
expect(claimedTypes).toEqual(expect.arrayContaining(["check", "geo-check", "evaluate", "cleanup-orphaned", "cleanup-retention"]));
|
||||
expect(claimedTypes).toEqual(expect.arrayContaining(["check", "geo-check", "evaluate", "cleanup-orphaned", "cleanup-retention", "egress"]));
|
||||
expect(mocks.jobsRepository.claimDueBatch).toHaveBeenCalledWith("check", 500, expect.any(Number));
|
||||
expect(mocks.jobsRepository.claimDueBatch).toHaveBeenCalledWith("evaluate", 20, expect.any(Number));
|
||||
expect(mocks.jobsRepository.claimDueBatch).toHaveBeenCalledWith("cleanup-orphaned", 1, expect.any(Number));
|
||||
@@ -440,6 +441,20 @@ describe("DBQueueWorker", () => {
|
||||
expect(retentionFn).toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("egress job runs the helper's egress recovery function with the claimed job", async () => {
|
||||
const job = makeJob({ id: "egress", type: "egress", refId: null, intervalMs: 30000 });
|
||||
const egressFn = jest.fn<any>().mockResolvedValue(undefined);
|
||||
const helper = {
|
||||
getCleanupOrphanedJob: jest.fn<any>().mockReturnValue(jest.fn<any>().mockResolvedValue(undefined)),
|
||||
getCleanupRetentionJob: jest.fn<any>().mockReturnValue(jest.fn<any>().mockResolvedValue(undefined)),
|
||||
getEgressRecoveryJob: jest.fn<any>().mockReturnValue(egressFn),
|
||||
};
|
||||
const { mocks } = await start({ mocks: { jobsRepository: { ...createWorker().mocks.jobsRepository, claimDueBatch: claimOnce(job) }, helper } });
|
||||
|
||||
expect(egressFn).toHaveBeenCalledWith(expect.objectContaining({ id: "egress", type: "egress" }));
|
||||
expect(mocks.jobsRepository.recordSuccess).toHaveBeenCalledWith(job.id, job.nextScheduledAt, job.intervalMs, expect.any(Number));
|
||||
});
|
||||
|
||||
it("geo-check job runs the geo pipeline (no evaluate handoff)", async () => {
|
||||
const job = makeJob({ id: "geo-check:m1", type: "geo-check", refId: "m1", intervalMs: 300000 });
|
||||
const { mocks } = await start({ mocks: { jobsRepository: { ...createWorker().mocks.jobsRepository, claimDueBatch: claimOnce(job) } } });
|
||||
|
||||
@@ -0,0 +1,472 @@
|
||||
import { afterEach, describe, expect, it, jest } from "@jest/globals";
|
||||
import { EgressService } from "../../../src/domain/egress/egress.service.ts";
|
||||
import type { EgressState } from "../../../src/domain/egress/egress.type.ts";
|
||||
import type { Job } from "../../../src/domain/jobs/job.type.ts";
|
||||
import { createMockLogger } from "../../helpers/createMockLogger.ts";
|
||||
|
||||
// ── Helpers ──────────────────────────────────────────────────────────────────
|
||||
|
||||
const makeState = (overrides?: Partial<EgressState>): EgressState => ({
|
||||
id: "egress-1",
|
||||
status: "ok",
|
||||
degradedSince: null,
|
||||
lastRecoveredAt: null,
|
||||
lastProbeAt: null,
|
||||
lastProbeResults: [],
|
||||
createdAt: "2026-01-01T00:00:00.000Z",
|
||||
updatedAt: "2026-01-01T00:00:00.000Z",
|
||||
...overrides,
|
||||
});
|
||||
|
||||
const makeSettings = (overrides?: Record<string, unknown>) => ({
|
||||
egressCheckEnabled: true,
|
||||
egressCheckTargets: ["1.1.1.1", "8.8.8.8"],
|
||||
egressPollIntervalSeconds: 30,
|
||||
egressNotifications: ["notif-1"],
|
||||
...overrides,
|
||||
});
|
||||
|
||||
// The claimed "egress" job row as the queue hands it to the handler
|
||||
const makeJob = (overrides?: Partial<Job>): Job => ({
|
||||
id: "egress",
|
||||
type: "egress",
|
||||
refId: null,
|
||||
isActive: true,
|
||||
nextScheduledAt: 1_000,
|
||||
intervalMs: 30_000,
|
||||
lockedBy: "worker-1",
|
||||
lockedUntil: 61_000,
|
||||
runCount: 0,
|
||||
failCount: 0,
|
||||
lastFinishedAt: null,
|
||||
lastFailReason: null,
|
||||
...overrides,
|
||||
});
|
||||
|
||||
// requestStatus mock that answers per target url
|
||||
const statusFor = (reachableTargets: string[]) =>
|
||||
jest.fn().mockImplementation(async (monitor: any) => ({
|
||||
monitorId: monitor.id,
|
||||
teamId: monitor.teamId,
|
||||
type: monitor.type,
|
||||
status: reachableTargets.includes(monitor.url),
|
||||
code: reachableTargets.includes(monitor.url) ? 200 : 5000,
|
||||
message: reachableTargets.includes(monitor.url) ? "Success" : "Ping failed",
|
||||
responseTime: 12,
|
||||
}));
|
||||
|
||||
const degradedRepository = (overrides?: Record<string, any>) => ({
|
||||
findSingleton: jest.fn().mockResolvedValue(makeState({ status: "degraded", degradedSince: "2026-01-01T10:00:00.000Z" })),
|
||||
recordProbe: jest.fn().mockResolvedValue(makeState({ status: "degraded" })),
|
||||
markDegraded: jest.fn().mockResolvedValue(null),
|
||||
markRecovered: jest.fn().mockResolvedValue(makeState({ lastRecoveredAt: "2026-01-01T10:05:00.000Z", degradedSince: "2026-01-01T10:00:00.000Z" })),
|
||||
reset: jest.fn(),
|
||||
...overrides,
|
||||
});
|
||||
|
||||
const createService = (overrides?: Record<string, any>) => {
|
||||
const defaults = {
|
||||
settingsService: { getCachedDBSettings: jest.fn().mockResolvedValue(makeSettings()) },
|
||||
egressStateRepository: {
|
||||
findSingleton: jest.fn().mockResolvedValue(makeState()),
|
||||
recordProbe: jest.fn().mockResolvedValue(makeState()),
|
||||
markDegraded: jest.fn().mockResolvedValue(makeState({ status: "degraded", degradedSince: "2026-01-01T10:00:00.000Z" })),
|
||||
markRecovered: jest.fn().mockResolvedValue(makeState({ lastRecoveredAt: "2026-01-01T10:05:00.000Z" })),
|
||||
reset: jest.fn(),
|
||||
},
|
||||
jobsRepository: {
|
||||
upsertJob: jest.fn().mockResolvedValue(true),
|
||||
upsertCleanupJob: jest.fn().mockResolvedValue(true),
|
||||
deleteGlobalJobIfUnchanged: jest.fn().mockResolvedValue(true),
|
||||
},
|
||||
networkService: { requestStatus: statusFor(["1.1.1.1", "8.8.8.8"]) },
|
||||
proxyResolver: { resolve: jest.fn().mockResolvedValue(undefined) },
|
||||
logger: createMockLogger(),
|
||||
...overrides,
|
||||
};
|
||||
const service = new EgressService(
|
||||
defaults.settingsService as any,
|
||||
defaults.egressStateRepository as any,
|
||||
defaults.jobsRepository as any,
|
||||
defaults.networkService as any,
|
||||
defaults.proxyResolver as any,
|
||||
defaults.logger as any
|
||||
);
|
||||
return { service, defaults };
|
||||
};
|
||||
|
||||
const probedUrls = (defaults: { networkService: { requestStatus: unknown } }) =>
|
||||
(defaults.networkService.requestStatus as jest.Mock).mock.calls.map((call) => (call[0] as any).url);
|
||||
|
||||
// ── Tests ────────────────────────────────────────────────────────────────────
|
||||
|
||||
describe("EgressService", () => {
|
||||
afterEach(() => {
|
||||
jest.useRealTimers();
|
||||
});
|
||||
|
||||
// ── probeTargets ──────────────────────────────────────────────────────────
|
||||
|
||||
describe("probeTargets", () => {
|
||||
it("maps a bare host to a ping monitor, host:port to a port monitor and a URL to an http monitor", async () => {
|
||||
const { service, defaults } = createService();
|
||||
|
||||
await service.probeTargets(["1.1.1.1", "8.8.8.8:53", "https://example.com/health", "[2606:4700::1111]:443", "2606:4700::1111"]);
|
||||
|
||||
const monitors = (defaults.networkService.requestStatus as jest.Mock).mock.calls.map((call) => call[0] as any);
|
||||
expect(monitors).toEqual([
|
||||
expect.objectContaining({ type: "ping", url: "1.1.1.1" }),
|
||||
expect.objectContaining({ type: "port", url: "8.8.8.8", port: 53 }),
|
||||
expect.objectContaining({ type: "http", url: "https://example.com/health", method: "GET", useAdvancedMatching: false }),
|
||||
expect.objectContaining({ type: "port", url: "2606:4700::1111", port: 443 }),
|
||||
expect.objectContaining({ type: "ping", url: "2606:4700::1111" }),
|
||||
]);
|
||||
});
|
||||
|
||||
it("passes the resolved proxy to the provider so http probes honour the global proxy", async () => {
|
||||
const resolve = jest.fn().mockResolvedValue("http://proxy.internal:3128");
|
||||
const { service, defaults } = createService({ proxyResolver: { resolve } });
|
||||
|
||||
await service.probeTargets(["https://example.com/health"]);
|
||||
|
||||
expect(resolve).toHaveBeenCalledWith(expect.objectContaining({ type: "http", proxyMode: "inherit" }));
|
||||
expect(defaults.networkService.requestStatus).toHaveBeenCalledWith(expect.objectContaining({ type: "http" }), {
|
||||
proxyUrl: "http://proxy.internal:3128",
|
||||
});
|
||||
});
|
||||
|
||||
it("reports reachable only when the provider says status is true", async () => {
|
||||
const { service } = createService({ networkService: { requestStatus: statusFor(["1.1.1.1"]) } });
|
||||
|
||||
const results = await service.probeTargets(["1.1.1.1", "8.8.8.8"]);
|
||||
|
||||
expect(results).toEqual([
|
||||
expect.objectContaining({ target: "1.1.1.1", reachable: true, responseTime: 12 }),
|
||||
expect.objectContaining({ target: "8.8.8.8", reachable: false }),
|
||||
]);
|
||||
});
|
||||
|
||||
it("treats any HTTP response, even 4xx/5xx, as reachable but a transport failure as unreachable", async () => {
|
||||
const requestStatus = jest.fn().mockImplementation(async (monitor: any) => ({
|
||||
monitorId: monitor.id,
|
||||
teamId: monitor.teamId,
|
||||
type: monitor.type,
|
||||
status: false,
|
||||
code: monitor.url.includes("maintenance") ? 503 : 5000, // 5000 is HttpProvider's NETWORK_ERROR
|
||||
message: monitor.url.includes("maintenance") ? "Service Unavailable" : "ECONNREFUSED",
|
||||
responseTime: 3,
|
||||
}));
|
||||
const { service } = createService({ networkService: { requestStatus } });
|
||||
|
||||
const results = await service.probeTargets(["https://example.com/maintenance", "https://example.com/refused", "8.8.8.8:53"]);
|
||||
|
||||
expect(results).toEqual([
|
||||
expect.objectContaining({ target: "https://example.com/maintenance", reachable: true }),
|
||||
expect.objectContaining({ target: "https://example.com/refused", reachable: false }),
|
||||
expect.objectContaining({ target: "8.8.8.8:53", reachable: false }), // non-http: only status true counts
|
||||
]);
|
||||
});
|
||||
|
||||
it("counts an unparseable stored target as unreachable without probing", async () => {
|
||||
const { service, defaults } = createService();
|
||||
|
||||
const results = await service.probeTargets(["[1.1.1.1"]);
|
||||
|
||||
expect(results).toEqual([expect.objectContaining({ target: "[1.1.1.1", reachable: false, message: "Invalid egress target" })]);
|
||||
expect(defaults.networkService.requestStatus).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("counts a thrown provider error as unreachable", async () => {
|
||||
const { service } = createService({ networkService: { requestStatus: jest.fn().mockRejectedValue(new Error("ENETUNREACH")) } });
|
||||
|
||||
const results = await service.probeTargets(["1.1.1.1"]);
|
||||
|
||||
expect(results).toEqual([expect.objectContaining({ target: "1.1.1.1", reachable: false, message: "ENETUNREACH" })]);
|
||||
});
|
||||
|
||||
it("counts a hung provider as unreachable once the probe timeout elapses", async () => {
|
||||
jest.useFakeTimers();
|
||||
const { service } = createService({ networkService: { requestStatus: jest.fn().mockReturnValue(new Promise(() => {})) } });
|
||||
|
||||
const pending = service.probeTargets(["1.1.1.1"]);
|
||||
await jest.advanceTimersByTimeAsync(5000);
|
||||
const results = await pending;
|
||||
|
||||
expect(results).toEqual([expect.objectContaining({ target: "1.1.1.1", reachable: false, message: expect.stringContaining("timed out") })]);
|
||||
});
|
||||
});
|
||||
|
||||
// ── assessAfterFailure ────────────────────────────────────────────────────
|
||||
|
||||
describe("assessAfterFailure", () => {
|
||||
it("returns null and does not probe or record when the feature is disabled", async () => {
|
||||
const { service, defaults } = createService({
|
||||
settingsService: { getCachedDBSettings: jest.fn().mockResolvedValue(makeSettings({ egressCheckEnabled: false })) },
|
||||
});
|
||||
|
||||
const result = await service.assessAfterFailure();
|
||||
|
||||
expect(result).toBeNull();
|
||||
expect(defaults.networkService.requestStatus).not.toHaveBeenCalled();
|
||||
expect(defaults.egressStateRepository.findSingleton).not.toHaveBeenCalled();
|
||||
expect(defaults.egressStateRepository.recordProbe).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("returns ok and records the probe when any target is reachable", async () => {
|
||||
const { service, defaults } = createService({ networkService: { requestStatus: statusFor(["8.8.8.8"]) } });
|
||||
|
||||
const result = await service.assessAfterFailure();
|
||||
|
||||
expect(result).toBe("ok");
|
||||
expect(defaults.egressStateRepository.recordProbe).toHaveBeenCalledWith(
|
||||
[expect.objectContaining({ target: "1.1.1.1", reachable: false }), expect.objectContaining({ target: "8.8.8.8", reachable: true })],
|
||||
expect.any(Date)
|
||||
);
|
||||
expect(defaults.egressStateRepository.markDegraded).not.toHaveBeenCalled();
|
||||
expect(defaults.jobsRepository.upsertJob).not.toHaveBeenCalled();
|
||||
expect(defaults.jobsRepository.upsertCleanupJob).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("marks degraded, warns, schedules the recovery job and returns degraded when every target is unreachable", async () => {
|
||||
const { service, defaults } = createService({ networkService: { requestStatus: statusFor([]) } });
|
||||
|
||||
const result = await service.assessAfterFailure();
|
||||
|
||||
expect(result).toBe("degraded");
|
||||
expect(defaults.egressStateRepository.markDegraded).toHaveBeenCalledWith(
|
||||
[expect.objectContaining({ target: "1.1.1.1", reachable: false }), expect.objectContaining({ target: "8.8.8.8", reachable: false })],
|
||||
expect.any(Date)
|
||||
);
|
||||
expect(defaults.egressStateRepository.recordProbe).not.toHaveBeenCalled();
|
||||
expect(defaults.logger.warn).toHaveBeenCalledWith(expect.objectContaining({ message: expect.stringContaining("egress degraded") }));
|
||||
expect(defaults.jobsRepository.upsertCleanupJob).toHaveBeenCalledWith({
|
||||
id: "egress",
|
||||
type: "egress",
|
||||
refId: null,
|
||||
isActive: true,
|
||||
nextScheduledAt: expect.any(Number),
|
||||
intervalMs: 30_000,
|
||||
});
|
||||
expect(defaults.jobsRepository.upsertJob).not.toHaveBeenCalled();
|
||||
// Job first, transition second: a crash in between leaves a stray row, never an unpolled degraded state
|
||||
const scheduleOrder = (defaults.jobsRepository.upsertCleanupJob as jest.Mock).mock.invocationCallOrder[0]!;
|
||||
const transitionOrder = (defaults.egressStateRepository.markDegraded as jest.Mock).mock.invocationCallOrder[0]!;
|
||||
expect(scheduleOrder).toBeLessThan(transitionOrder);
|
||||
});
|
||||
|
||||
it("still returns degraded and leaves the job scheduled when another worker performed the transition first", async () => {
|
||||
const { service, defaults } = createService({
|
||||
networkService: { requestStatus: statusFor([]) },
|
||||
egressStateRepository: degradedRepository({
|
||||
findSingleton: jest.fn().mockResolvedValue(makeState()),
|
||||
markDegraded: jest.fn().mockResolvedValue(null),
|
||||
}),
|
||||
});
|
||||
|
||||
const result = await service.assessAfterFailure();
|
||||
|
||||
expect(result).toBe("degraded");
|
||||
expect(defaults.jobsRepository.upsertCleanupJob).toHaveBeenCalledWith(expect.objectContaining({ id: "egress", intervalMs: 30_000 }));
|
||||
expect(defaults.logger.warn).toHaveBeenCalledWith(expect.objectContaining({ message: expect.stringContaining("already recorded") }));
|
||||
});
|
||||
|
||||
it("does not mark degraded when the recovery job cannot be scheduled", async () => {
|
||||
const { service, defaults } = createService({
|
||||
networkService: { requestStatus: statusFor([]) },
|
||||
jobsRepository: {
|
||||
upsertJob: jest.fn(),
|
||||
upsertCleanupJob: jest.fn().mockRejectedValue(new Error("db down")),
|
||||
deleteGlobalJobIfUnchanged: jest.fn(),
|
||||
},
|
||||
});
|
||||
|
||||
expect(await service.assessAfterFailure()).toBeNull();
|
||||
expect(defaults.egressStateRepository.markDegraded).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("returns degraded without probing when the persisted state is already degraded, without touching the job's schedule", async () => {
|
||||
const { service, defaults } = createService({ egressStateRepository: degradedRepository() });
|
||||
|
||||
const result = await service.assessAfterFailure();
|
||||
|
||||
expect(result).toBe("degraded");
|
||||
expect(defaults.networkService.requestStatus).not.toHaveBeenCalled();
|
||||
expect(defaults.egressStateRepository.markDegraded).not.toHaveBeenCalled();
|
||||
expect(defaults.egressStateRepository.recordProbe).not.toHaveBeenCalled();
|
||||
expect(defaults.jobsRepository.upsertJob).toHaveBeenCalledWith(expect.objectContaining({ id: "egress", intervalMs: 30_000 }));
|
||||
expect(defaults.jobsRepository.upsertCleanupJob).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("uses the default poll interval for the recovery job when the setting is missing or invalid", async () => {
|
||||
const { service, defaults } = createService({
|
||||
settingsService: { getCachedDBSettings: jest.fn().mockResolvedValue(makeSettings({ egressPollIntervalSeconds: undefined })) },
|
||||
networkService: { requestStatus: statusFor([]) },
|
||||
});
|
||||
|
||||
await service.assessAfterFailure();
|
||||
|
||||
expect(defaults.jobsRepository.upsertCleanupJob).toHaveBeenCalledWith(expect.objectContaining({ intervalMs: 30_000 }));
|
||||
});
|
||||
|
||||
it("falls back to the default targets when the configured list is empty", async () => {
|
||||
const { service, defaults } = createService({
|
||||
settingsService: { getCachedDBSettings: jest.fn().mockResolvedValue(makeSettings({ egressCheckTargets: [] })) },
|
||||
});
|
||||
|
||||
await service.assessAfterFailure();
|
||||
|
||||
expect(probedUrls(defaults)).toEqual(["1.1.1.1", "8.8.8.8"]);
|
||||
});
|
||||
|
||||
it("returns null and logs when the assessment itself fails", async () => {
|
||||
const { service, defaults } = createService({
|
||||
settingsService: { getCachedDBSettings: jest.fn().mockRejectedValue(new Error("db down")) },
|
||||
});
|
||||
|
||||
const result = await service.assessAfterFailure();
|
||||
|
||||
expect(result).toBeNull();
|
||||
expect(defaults.logger.error).toHaveBeenCalledWith(expect.objectContaining({ message: "db down", method: "assessAfterFailure" }));
|
||||
});
|
||||
});
|
||||
|
||||
// ── shared assessment ─────────────────────────────────────────────────────
|
||||
|
||||
describe("shared assessment", () => {
|
||||
it("shares one probe between concurrent failures", async () => {
|
||||
const { service, defaults } = createService({ networkService: { requestStatus: statusFor(["8.8.8.8"]) } });
|
||||
|
||||
const results = await Promise.all([service.assessAfterFailure(), service.assessAfterFailure(), service.assessAfterFailure()]);
|
||||
|
||||
expect(results).toEqual(["ok", "ok", "ok"]);
|
||||
expect(defaults.settingsService.getCachedDBSettings).toHaveBeenCalledTimes(1);
|
||||
expect(defaults.networkService.requestStatus).toHaveBeenCalledTimes(2); // one call per target, once
|
||||
expect(defaults.egressStateRepository.recordProbe).toHaveBeenCalledTimes(1);
|
||||
});
|
||||
|
||||
it("keeps sharing an in-flight assessment that outlives the reuse window", async () => {
|
||||
jest.useFakeTimers();
|
||||
const { service, defaults } = createService({ networkService: { requestStatus: jest.fn().mockReturnValue(new Promise(() => {})) } });
|
||||
|
||||
const first = service.assessAfterFailure();
|
||||
await jest.advanceTimersByTimeAsync(4_900); // probes still hanging, past the 5 s reuse window from the first call
|
||||
const second = service.assessAfterFailure();
|
||||
await jest.advanceTimersByTimeAsync(200); // probe timeout fires, both settle
|
||||
|
||||
expect(await Promise.all([first, second])).toEqual(["degraded", "degraded"]);
|
||||
expect(defaults.networkService.requestStatus).toHaveBeenCalledTimes(2); // one round of two targets, not two rounds
|
||||
});
|
||||
|
||||
it("reuses a settled assessment within the reuse window and probes again after it", async () => {
|
||||
jest.useFakeTimers();
|
||||
const { service, defaults } = createService({ networkService: { requestStatus: statusFor(["8.8.8.8"]) } });
|
||||
|
||||
await service.assessAfterFailure();
|
||||
await jest.advanceTimersByTimeAsync(1_000);
|
||||
await service.assessAfterFailure();
|
||||
expect(defaults.egressStateRepository.recordProbe).toHaveBeenCalledTimes(1);
|
||||
|
||||
await jest.advanceTimersByTimeAsync(5_000);
|
||||
await service.assessAfterFailure();
|
||||
expect(defaults.egressStateRepository.recordProbe).toHaveBeenCalledTimes(2);
|
||||
});
|
||||
|
||||
it("reuses a disabled result instead of re-reading settings for every failure", async () => {
|
||||
const getCachedDBSettings = jest.fn().mockResolvedValue(makeSettings({ egressCheckEnabled: false }));
|
||||
const { service } = createService({ settingsService: { getCachedDBSettings } });
|
||||
|
||||
expect(await service.assessAfterFailure()).toBeNull();
|
||||
expect(await service.assessAfterFailure()).toBeNull();
|
||||
|
||||
expect(getCachedDBSettings).toHaveBeenCalledTimes(1);
|
||||
});
|
||||
|
||||
it("does not cache an internal failure", async () => {
|
||||
const getCachedDBSettings = jest.fn().mockRejectedValueOnce(new Error("db down")).mockResolvedValue(makeSettings());
|
||||
const { service, defaults } = createService({ settingsService: { getCachedDBSettings } });
|
||||
|
||||
expect(await service.assessAfterFailure()).toBeNull();
|
||||
expect(await service.assessAfterFailure()).toBe("ok");
|
||||
expect(defaults.egressStateRepository.recordProbe).toHaveBeenCalledTimes(1);
|
||||
});
|
||||
});
|
||||
|
||||
// ── checkRecovery (the "egress" job) ──────────────────────────────────────
|
||||
|
||||
describe("checkRecovery", () => {
|
||||
it("marks recovered, releases the job and logs when a target becomes reachable", async () => {
|
||||
const { service, defaults } = createService({
|
||||
egressStateRepository: degradedRepository(),
|
||||
networkService: { requestStatus: statusFor(["1.1.1.1"]) },
|
||||
});
|
||||
const job = makeJob({ nextScheduledAt: 123_456 });
|
||||
|
||||
await service.checkRecovery(job);
|
||||
|
||||
expect(defaults.egressStateRepository.markRecovered).toHaveBeenCalledWith(
|
||||
[expect.objectContaining({ target: "1.1.1.1", reachable: true }), expect.objectContaining({ target: "8.8.8.8", reachable: false })],
|
||||
expect.any(Date)
|
||||
);
|
||||
// Conditional on the schedule this run claimed, so a concurrent new episode keeps its job
|
||||
expect(defaults.jobsRepository.deleteGlobalJobIfUnchanged).toHaveBeenCalledWith("egress", 123_456);
|
||||
expect(defaults.logger.info).toHaveBeenCalledWith(expect.objectContaining({ message: expect.stringContaining("Instance egress recovered") }));
|
||||
});
|
||||
|
||||
it("releases the job but does not log a recovery when another process already recorded it", async () => {
|
||||
const { service, defaults } = createService({
|
||||
egressStateRepository: degradedRepository({ markRecovered: jest.fn().mockResolvedValue(null) }),
|
||||
networkService: { requestStatus: statusFor(["1.1.1.1"]) },
|
||||
});
|
||||
|
||||
await service.checkRecovery(makeJob());
|
||||
|
||||
expect(defaults.jobsRepository.deleteGlobalJobIfUnchanged).toHaveBeenCalledWith("egress", 1_000);
|
||||
expect(defaults.logger.info).not.toHaveBeenCalledWith(
|
||||
expect.objectContaining({ message: expect.stringContaining("Instance egress recovered") })
|
||||
);
|
||||
});
|
||||
|
||||
it("records the probe and keeps the job while every target stays unreachable", async () => {
|
||||
const { service, defaults } = createService({
|
||||
egressStateRepository: degradedRepository(),
|
||||
networkService: { requestStatus: statusFor([]) },
|
||||
});
|
||||
|
||||
await service.checkRecovery(makeJob());
|
||||
|
||||
expect(defaults.egressStateRepository.recordProbe).toHaveBeenCalledTimes(1);
|
||||
expect(defaults.egressStateRepository.markRecovered).not.toHaveBeenCalled();
|
||||
expect(defaults.jobsRepository.deleteGlobalJobIfUnchanged).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("releases the job without probing when the persisted state is no longer degraded", async () => {
|
||||
const { service, defaults } = createService();
|
||||
|
||||
await service.checkRecovery(makeJob());
|
||||
|
||||
expect(defaults.networkService.requestStatus).not.toHaveBeenCalled();
|
||||
expect(defaults.egressStateRepository.markRecovered).not.toHaveBeenCalled();
|
||||
expect(defaults.jobsRepository.deleteGlobalJobIfUnchanged).toHaveBeenCalledWith("egress", 1_000);
|
||||
});
|
||||
|
||||
it("releases the job without probing when the feature has been disabled mid-episode", async () => {
|
||||
const { service, defaults } = createService({
|
||||
settingsService: { getCachedDBSettings: jest.fn().mockResolvedValue(makeSettings({ egressCheckEnabled: false })) },
|
||||
egressStateRepository: degradedRepository(),
|
||||
});
|
||||
|
||||
await service.checkRecovery(makeJob());
|
||||
|
||||
expect(defaults.networkService.requestStatus).not.toHaveBeenCalled();
|
||||
expect(defaults.jobsRepository.deleteGlobalJobIfUnchanged).toHaveBeenCalledWith("egress", 1_000);
|
||||
});
|
||||
|
||||
it("lets errors propagate so the queue records the failure and retries", async () => {
|
||||
const { service } = createService({
|
||||
egressStateRepository: degradedRepository({ findSingleton: jest.fn().mockRejectedValue(new Error("db down")) }),
|
||||
});
|
||||
|
||||
await expect(service.checkRecovery(makeJob())).rejects.toThrow("db down");
|
||||
});
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,100 @@
|
||||
import { describe, expect, it, jest } from "@jest/globals";
|
||||
import { EgressStateService } from "../../../src/domain/egress/egress-state.service.ts";
|
||||
import type { EgressState } from "../../../src/domain/egress/egress.type.ts";
|
||||
|
||||
// ── Helpers ──────────────────────────────────────────────────────────────────
|
||||
|
||||
const createEgressStateRepo = () => ({
|
||||
findSingleton: jest.fn(),
|
||||
recordProbe: jest.fn(),
|
||||
markDegraded: jest.fn(),
|
||||
markRecovered: jest.fn(),
|
||||
reset: jest.fn(),
|
||||
});
|
||||
|
||||
const createJobsRepo = () => ({
|
||||
deleteGlobalJob: jest.fn().mockResolvedValue(true),
|
||||
});
|
||||
|
||||
const createService = () => {
|
||||
const egressStateRepository = createEgressStateRepo();
|
||||
const jobsRepository = createJobsRepo();
|
||||
const service = new EgressStateService(egressStateRepository as any, jobsRepository as any);
|
||||
return { service, egressStateRepository, jobsRepository };
|
||||
};
|
||||
|
||||
const makeState = (overrides?: Partial<EgressState>): EgressState => ({
|
||||
id: "egress-1",
|
||||
status: "ok",
|
||||
degradedSince: null,
|
||||
lastRecoveredAt: null,
|
||||
lastProbeAt: null,
|
||||
lastProbeResults: [],
|
||||
createdAt: "2026-01-01T00:00:00Z",
|
||||
updatedAt: "2026-01-01T00:00:00Z",
|
||||
...overrides,
|
||||
});
|
||||
|
||||
// ── Tests ────────────────────────────────────────────────────────────────────
|
||||
|
||||
describe("EgressStateService", () => {
|
||||
describe("getState", () => {
|
||||
it("returns the singleton state from the repository", async () => {
|
||||
const { service, egressStateRepository } = createService();
|
||||
const state = makeState({
|
||||
status: "degraded",
|
||||
degradedSince: "2026-01-01T00:05:00Z",
|
||||
lastProbeAt: "2026-01-01T00:05:00Z",
|
||||
lastProbeResults: [{ target: "1.1.1.1", reachable: false, responseTime: 0, message: "timeout" }],
|
||||
});
|
||||
(egressStateRepository.findSingleton as jest.Mock).mockResolvedValue(state);
|
||||
|
||||
const result = await service.getState();
|
||||
|
||||
expect(egressStateRepository.findSingleton).toHaveBeenCalledTimes(1);
|
||||
expect(result).toEqual(state);
|
||||
});
|
||||
|
||||
it("does not call any transition method", async () => {
|
||||
const { service, egressStateRepository } = createService();
|
||||
(egressStateRepository.findSingleton as jest.Mock).mockResolvedValue(makeState());
|
||||
|
||||
await service.getState();
|
||||
|
||||
expect(egressStateRepository.recordProbe).not.toHaveBeenCalled();
|
||||
expect(egressStateRepository.markDegraded).not.toHaveBeenCalled();
|
||||
expect(egressStateRepository.markRecovered).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("propagates repository errors", async () => {
|
||||
const { service, egressStateRepository } = createService();
|
||||
(egressStateRepository.findSingleton as jest.Mock).mockRejectedValue(new Error("db down"));
|
||||
|
||||
await expect(service.getState()).rejects.toThrow("db down");
|
||||
});
|
||||
});
|
||||
|
||||
describe("reset", () => {
|
||||
it("removes any pending recovery job and returns the state to ok", async () => {
|
||||
const { service, egressStateRepository, jobsRepository } = createService();
|
||||
const reset = makeState();
|
||||
(egressStateRepository.reset as jest.Mock).mockResolvedValue(reset);
|
||||
|
||||
const result = await service.reset();
|
||||
|
||||
expect(jobsRepository.deleteGlobalJob).toHaveBeenCalledWith("egress");
|
||||
expect(egressStateRepository.reset).toHaveBeenCalledTimes(1);
|
||||
expect(result).toEqual(reset);
|
||||
});
|
||||
|
||||
it("does not touch the probe-driven transitions", async () => {
|
||||
const { service, egressStateRepository } = createService();
|
||||
(egressStateRepository.reset as jest.Mock).mockResolvedValue(makeState());
|
||||
|
||||
await service.reset();
|
||||
|
||||
expect(egressStateRepository.markDegraded).not.toHaveBeenCalled();
|
||||
expect(egressStateRepository.markRecovered).not.toHaveBeenCalled();
|
||||
});
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,239 @@
|
||||
import { describe, expect, it } from "@jest/globals";
|
||||
import * as nodeCrypto from "node:crypto";
|
||||
import { EncryptionService } from "../../../src/service/encryption/encryptionService.ts";
|
||||
import type { EncryptionCryptoLib } from "../../../src/service/encryption/encryptionService.ts";
|
||||
import { AppError } from "../../../src/utils/AppError.ts";
|
||||
import { createMockLogger } from "../../helpers/createMockLogger.ts";
|
||||
|
||||
const newKey = () => nodeCrypto.randomBytes(32);
|
||||
|
||||
const setup = (keys: Buffer[], cryptoLib: EncryptionCryptoLib = nodeCrypto) => {
|
||||
const logger = createMockLogger();
|
||||
const service = new EncryptionService(keys, logger, cryptoLib);
|
||||
return { service, logger };
|
||||
};
|
||||
|
||||
const expectAppError = (fn: () => unknown, message: string, details?: Record<string, unknown>) => {
|
||||
let caught: unknown;
|
||||
try {
|
||||
fn();
|
||||
} catch (error) {
|
||||
caught = error;
|
||||
}
|
||||
expect(caught).toBeInstanceOf(AppError);
|
||||
const appError = caught as AppError;
|
||||
expect(appError.message).toBe(message);
|
||||
expect(appError.status).toBe(500);
|
||||
expect(appError.service).toBe("EncryptionService");
|
||||
if (details) expect(appError.details).toEqual(details);
|
||||
};
|
||||
|
||||
const tamper = (ciphertext: string, segment: number, replacer: (value: string) => string) => {
|
||||
const parts = ciphertext.split(".");
|
||||
parts[segment] = replacer(parts[segment] ?? "");
|
||||
return parts.join(".");
|
||||
};
|
||||
|
||||
const flipFirstChar = (value: string) => (value[0] === "A" ? "B" : "A") + value.slice(1);
|
||||
|
||||
describe("EncryptionService", () => {
|
||||
describe("construction", () => {
|
||||
it("is unconfigured with no keys and says so", () => {
|
||||
const { service, logger } = setup([]);
|
||||
expect(service.isConfigured()).toBe(false);
|
||||
expectAppError(() => service.currentKeyId(), "ENCRYPTION_KEY is not configured");
|
||||
expect(logger.info).toHaveBeenCalledWith(expect.objectContaining({ message: "ENCRYPTION_KEY not set; encrypted fields are unavailable" }));
|
||||
});
|
||||
|
||||
it("is configured with one key and exposes a 6-character base64url key id", () => {
|
||||
const { service, logger } = setup([newKey()]);
|
||||
expect(service.isConfigured()).toBe(true);
|
||||
expect(service.currentKeyId()).toMatch(/^[A-Za-z0-9_-]{6}$/);
|
||||
expect(logger.info).toHaveBeenCalledWith(expect.objectContaining({ message: expect.stringContaining("Encryption configured with 1 key(s)") }));
|
||||
});
|
||||
|
||||
it("rejects an entry that is not 32 bytes, naming its index", () => {
|
||||
const short = nodeCrypto.randomBytes(31);
|
||||
const long = nodeCrypto.randomBytes(33);
|
||||
expectAppError(() => setup([newKey(), short]), "ENCRYPTION_KEY is not 32 bytes", { keyIdx: 1 });
|
||||
expectAppError(() => setup([long]), "ENCRYPTION_KEY is not 32 bytes", { keyIdx: 0 });
|
||||
});
|
||||
|
||||
it("rejects the same key listed twice", () => {
|
||||
const key = newKey();
|
||||
expectAppError(() => setup([key, key]), "ENCRYPTION_KEY key ids collide; regenerate one key", { index: 1 });
|
||||
});
|
||||
|
||||
it("treats the first key as current", () => {
|
||||
const a = newKey();
|
||||
const b = newKey();
|
||||
const { service: onlyA } = setup([a]);
|
||||
const { service: onlyB } = setup([b]);
|
||||
const { service: bThenA } = setup([b, a]);
|
||||
expect(bThenA.currentKeyId()).toBe(onlyB.currentKeyId());
|
||||
expect(bThenA.currentKeyId()).not.toBe(onlyA.currentKeyId());
|
||||
});
|
||||
});
|
||||
|
||||
describe("round trip", () => {
|
||||
it.each([
|
||||
["ascii", "hello"],
|
||||
["pem-shaped", "-----BEGIN PRIVATE KEY-----\nMIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQC7\n-----END PRIVATE KEY-----\n"],
|
||||
["multi-byte utf-8", "clé privée — 秘密鍵 🔑"],
|
||||
["empty", ""],
|
||||
])("decrypts what it encrypted (%s)", (_label, plaintext) => {
|
||||
const { service } = setup([newKey()]);
|
||||
expect(service.decrypt(service.encrypt(plaintext))).toBe(plaintext);
|
||||
});
|
||||
|
||||
it("uses a fresh IV per call so equal plaintexts encrypt differently", () => {
|
||||
const { service } = setup([newKey()]);
|
||||
const first = service.encrypt("same");
|
||||
const second = service.encrypt("same");
|
||||
expect(first).not.toBe(second);
|
||||
expect(service.decrypt(first)).toBe("same");
|
||||
expect(service.decrypt(second)).toBe("same");
|
||||
});
|
||||
|
||||
it("produces the documented five-segment base64url form", () => {
|
||||
const { service } = setup([newKey()]);
|
||||
expect(service.encrypt("x")).toMatch(/^v1\.[A-Za-z0-9_-]{6}\.[A-Za-z0-9_-]{16}\.[A-Za-z0-9_-]{22}\.[A-Za-z0-9_-]*$/);
|
||||
});
|
||||
|
||||
it("does not leak the plaintext into the stored form", () => {
|
||||
const { service } = setup([newKey()]);
|
||||
const ciphertext = service.encrypt("-----BEGIN PRIVATE KEY-----");
|
||||
expect(ciphertext).not.toContain("PRIVATE");
|
||||
expect(Buffer.from(ciphertext.split(".")[4] ?? "", "base64url").toString("utf8")).not.toContain("PRIVATE");
|
||||
});
|
||||
|
||||
it("is deterministic for a fixed IV, pinning the format", () => {
|
||||
const fixedIv = Buffer.alloc(12, 7);
|
||||
const cryptoLib: EncryptionCryptoLib = {
|
||||
createCipheriv: nodeCrypto.createCipheriv,
|
||||
createDecipheriv: nodeCrypto.createDecipheriv,
|
||||
createHash: nodeCrypto.createHash,
|
||||
randomBytes: () => fixedIv,
|
||||
};
|
||||
const key = newKey();
|
||||
const { service } = setup([key], cryptoLib);
|
||||
const first = service.encrypt("pinned");
|
||||
const second = service.encrypt("pinned");
|
||||
expect(first).toBe(second);
|
||||
expect(first.split(".")[2]).toBe(fixedIv.toString("base64url"));
|
||||
expect(setup([key]).service.decrypt(first)).toBe("pinned");
|
||||
});
|
||||
});
|
||||
|
||||
describe("key selection and rotation", () => {
|
||||
it("decrypts with a key that is not first in the list and reports it needs re-encryption", () => {
|
||||
const a = newKey();
|
||||
const b = newKey();
|
||||
const { service: withA } = setup([a]);
|
||||
const ciphertext = withA.encrypt("secret");
|
||||
|
||||
const { service: withBThenA } = setup([b, a]);
|
||||
expect(withBThenA.decrypt(ciphertext)).toBe("secret");
|
||||
expect(withBThenA.keyIdOf(ciphertext)).toBe(withA.currentKeyId());
|
||||
expect(withBThenA.needsReencryption(ciphertext)).toBe(true);
|
||||
});
|
||||
|
||||
it("encrypts under the first key and does not flag it for re-encryption", () => {
|
||||
const { service } = setup([newKey(), newKey()]);
|
||||
const ciphertext = service.encrypt("secret");
|
||||
expect(service.keyIdOf(ciphertext)).toBe(service.currentKeyId());
|
||||
expect(service.needsReencryption(ciphertext)).toBe(false);
|
||||
});
|
||||
|
||||
it("fails with the key id when no listed key matches, but keyIdOf still answers", () => {
|
||||
const a = newKey();
|
||||
const { service: withA } = setup([a]);
|
||||
const ciphertext = withA.encrypt("secret");
|
||||
const idOfA = withA.currentKeyId();
|
||||
|
||||
const { service: withB } = setup([newKey()]);
|
||||
expectAppError(() => withB.decrypt(ciphertext), "No encryption key matches ciphertext", { keyId: idOfA });
|
||||
expect(withB.keyIdOf(ciphertext)).toBe(idOfA);
|
||||
});
|
||||
});
|
||||
|
||||
describe("tampering and malformed input", () => {
|
||||
it("rejects a modified data segment", () => {
|
||||
const { service } = setup([newKey()]);
|
||||
const ciphertext = service.encrypt("payload");
|
||||
expectAppError(() => service.decrypt(tamper(ciphertext, 4, flipFirstChar)), "Ciphertext failed authentication");
|
||||
});
|
||||
|
||||
it("rejects a modified tag segment", () => {
|
||||
const { service } = setup([newKey()]);
|
||||
const ciphertext = service.encrypt("payload");
|
||||
expectAppError(() => service.decrypt(tamper(ciphertext, 3, flipFirstChar)), "Ciphertext failed authentication");
|
||||
});
|
||||
|
||||
it("rejects a modified iv segment", () => {
|
||||
const { service } = setup([newKey()]);
|
||||
const ciphertext = service.encrypt("payload");
|
||||
expectAppError(() => service.decrypt(tamper(ciphertext, 2, flipFirstChar)), "Ciphertext failed authentication");
|
||||
});
|
||||
|
||||
it("rejects a ciphertext relabelled with another listed key's id", () => {
|
||||
const a = newKey();
|
||||
const b = newKey();
|
||||
const { service } = setup([a, b]);
|
||||
const ciphertext = service.encrypt("payload");
|
||||
const idOfB = setup([b]).service.currentKeyId();
|
||||
expectAppError(() => service.decrypt(tamper(ciphertext, 1, () => idOfB)), "Ciphertext failed authentication");
|
||||
});
|
||||
|
||||
it("rejects a ciphertext whose key id matches a listed key made from different bytes", () => {
|
||||
const a = newKey();
|
||||
const b = newKey();
|
||||
const ciphertext = setup([a]).service.encrypt("payload");
|
||||
const { service: withB } = setup([b]);
|
||||
const forged = tamper(ciphertext, 1, () => withB.currentKeyId());
|
||||
expectAppError(() => withB.decrypt(forged), "Ciphertext failed authentication");
|
||||
});
|
||||
|
||||
it.each([
|
||||
["too few segments", "v1.abc.def"],
|
||||
["too many segments", "v1.a.b.c.d.e"],
|
||||
["empty key id", "v1..AAAAAAAAAAAAAAAA.AAAAAAAAAAAAAAAAAAAAAA.AA"],
|
||||
["standard base64 padding", "v1.abcdef.AAAAAAAAAAAAAAA=.AAAAAAAAAAAAAAAAAAAAAA.AA"],
|
||||
["plus sign in a segment", "v1.abcdef.AAAAAAAAAAAAAA+A.AAAAAAAAAAAAAAAAAAAAAA.AA"],
|
||||
["iv of the wrong length", "v1.abcdef.AAAA.AAAAAAAAAAAAAAAAAAAAAA.AA"],
|
||||
["tag of the wrong length", "v1.abcdef.AAAAAAAAAAAAAAAA.AAAA.AA"],
|
||||
])("rejects malformed v1 input (%s)", (_label, input) => {
|
||||
const { service } = setup([newKey()]);
|
||||
expectAppError(() => service.decrypt(input), "Ciphertext is malformed");
|
||||
expectAppError(() => service.keyIdOf(input), "Ciphertext is malformed");
|
||||
});
|
||||
|
||||
it.each([
|
||||
["empty string", "", ""],
|
||||
["no version segment", "abcdef.AAAA.AAAA.AA", "abcdef"],
|
||||
["future version with a different layout", "v2.x.y.z.w.v.u", "v2"],
|
||||
])("reports the version before any layout rule (%s)", (_label, input, version) => {
|
||||
const { service } = setup([newKey()]);
|
||||
expectAppError(() => service.decrypt(input), "Unsupported ciphertext version", { version });
|
||||
expectAppError(() => service.keyIdOf(input), "Unsupported ciphertext version", { version });
|
||||
});
|
||||
});
|
||||
|
||||
describe("unconfigured service", () => {
|
||||
it("refuses to encrypt, decrypt, or evaluate rotation", () => {
|
||||
const { service: configured } = setup([newKey()]);
|
||||
const ciphertext = configured.encrypt("x");
|
||||
const { service } = setup([]);
|
||||
expectAppError(() => service.encrypt("x"), "ENCRYPTION_KEY is not configured");
|
||||
expectAppError(() => service.decrypt(ciphertext), "ENCRYPTION_KEY is not configured");
|
||||
expectAppError(() => service.needsReencryption(ciphertext), "ENCRYPTION_KEY is not configured");
|
||||
});
|
||||
|
||||
it("still parses a key id out of a well-formed ciphertext", () => {
|
||||
const { service: configured } = setup([newKey()]);
|
||||
const ciphertext = configured.encrypt("x");
|
||||
const { service } = setup([]);
|
||||
expect(service.keyIdOf(ciphertext)).toBe(configured.currentKeyId());
|
||||
});
|
||||
});
|
||||
});
|
||||
@@ -143,6 +143,21 @@ describe("JobScheduler", () => {
|
||||
});
|
||||
});
|
||||
|
||||
describe("addJob", () => {
|
||||
it("preserves an arbitrary monitor interval in the scheduled job", async () => {
|
||||
const { scheduler, mocks } = createScheduler();
|
||||
|
||||
await scheduler.addJob("m1", makeMonitor({ interval: 75000 }));
|
||||
|
||||
expect(mocks.jobsRepository.upsertJob).toHaveBeenCalledWith(
|
||||
expect.objectContaining({
|
||||
id: "check:m1",
|
||||
intervalMs: 75000,
|
||||
})
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
// Regression: a heartbeat upsert racing shutdown's deleteById could resurrect the
|
||||
// registry row, leaving a ghost worker that inflates the alive count until TTL.
|
||||
// shutdown() must await the in-flight beat and issue the delete as the LAST write.
|
||||
|
||||
@@ -7,6 +7,8 @@ import type { IIncidentsRepository } from "../../../src/domain/incidents/inciden
|
||||
import type { IMonitorStatsRepository } from "../../../src/domain/monitor-stats/monitor-stats.repository.interface.ts";
|
||||
import type { IMonitorsRepository } from "../../../src/domain/monitors/monitor.repository.interface.ts";
|
||||
import type { IStatusPagesRepository } from "../../../src/domain/status-pages/status-page-repository.interface.ts";
|
||||
import type { IEncryptionService } from "../../../src/service/encryption/encryptionService.ts";
|
||||
import { AppError } from "../../../src/utils/AppError.ts";
|
||||
import { createMockLogger } from "../../helpers/createMockLogger.ts";
|
||||
|
||||
const createMonitorsRepositoryMock = () =>
|
||||
@@ -70,6 +72,17 @@ const createJobQueueMock = () => ({
|
||||
deleteJob: jest.fn(),
|
||||
});
|
||||
|
||||
const createEncryptionServiceMock = (overrides: Partial<IEncryptionService> = {}) =>
|
||||
({
|
||||
isConfigured: jest.fn(() => true),
|
||||
currentKeyId: jest.fn(() => "key-1"),
|
||||
keyIdOf: jest.fn(() => "key-1"),
|
||||
needsReencryption: jest.fn(() => false),
|
||||
encrypt: jest.fn((plaintext: string) => `enc:${plaintext}`),
|
||||
decrypt: jest.fn((ciphertext: string) => ciphertext.replace(/^enc:/, "")),
|
||||
...overrides,
|
||||
}) as unknown as IEncryptionService;
|
||||
|
||||
const TEAM_ID = "team-1";
|
||||
const MONITOR_ID = "monitor-1";
|
||||
const USER_ID = "user-1";
|
||||
@@ -98,6 +111,7 @@ const createService = (
|
||||
geoChecksRepository?: IGeoChecksRepository;
|
||||
dockerLogsRepository?: IDockerLogsRepository;
|
||||
incidentsRepository?: IIncidentsRepository;
|
||||
encryptionService?: IEncryptionService;
|
||||
jobQueue?: ReturnType<typeof createJobQueueMock>;
|
||||
logger?: ReturnType<typeof createMockLogger>;
|
||||
games?: Record<string, unknown>;
|
||||
@@ -116,6 +130,7 @@ const createService = (
|
||||
monitorStatsRepository: overrides.monitorStatsRepository ?? createMonitorStatsRepositoryMock(),
|
||||
statusPagesRepository: overrides.statusPagesRepository ?? createStatusPagesRepositoryMock(),
|
||||
incidentsRepository: overrides.incidentsRepository ?? createIncidentsRepositoryMock(),
|
||||
encryptionService: overrides.encryptionService ?? createEncryptionServiceMock(),
|
||||
});
|
||||
return { service, jobQueue, logger };
|
||||
};
|
||||
@@ -1893,6 +1908,7 @@ describe("MonitorService", () => {
|
||||
monitorStatsRepository,
|
||||
statusPagesRepository,
|
||||
incidentsRepository,
|
||||
encryptionService: createEncryptionServiceMock(),
|
||||
});
|
||||
|
||||
const result = await service.deleteAllMonitors({ teamId: TEAM_ID });
|
||||
@@ -1981,3 +1997,167 @@ describe("MonitorService", () => {
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
describe("MonitorService — Docker TLS credentials", () => {
|
||||
const KEY_PEM = "-----BEGIN PRIVATE KEY-----\nkey\n-----END PRIVATE KEY-----";
|
||||
const tlsBody = { type: "docker" as const, url: "tcp://host:2376", dockerTlsCa: "ca-pem", dockerTlsCert: "cert-pem", dockerTlsKey: KEY_PEM };
|
||||
|
||||
const expectAppError = async (fn: () => Promise<unknown>, status: number, message: string | RegExp) => {
|
||||
let caught: unknown;
|
||||
try {
|
||||
await fn();
|
||||
} catch (error) {
|
||||
caught = error;
|
||||
}
|
||||
expect(caught).toBeInstanceOf(AppError);
|
||||
expect((caught as AppError).status).toBe(status);
|
||||
expect((caught as AppError).message).toMatch(message);
|
||||
};
|
||||
|
||||
describe("createMonitor", () => {
|
||||
it("encrypts the key and sets dockerTlsKeySet for a TLS url", async () => {
|
||||
const monitorsRepository = createMonitorsRepositoryMock();
|
||||
(monitorsRepository.create as jest.Mock).mockResolvedValue(makeMonitor({ type: "docker" }));
|
||||
const encryptionService = createEncryptionServiceMock();
|
||||
const { service } = createService({ monitorsRepository, encryptionService });
|
||||
|
||||
await service.createMonitor(TEAM_ID, USER_ID, makeMonitor(tlsBody) as any);
|
||||
|
||||
expect(encryptionService.encrypt).toHaveBeenCalledWith(KEY_PEM);
|
||||
expect(monitorsRepository.create).toHaveBeenCalledWith(
|
||||
expect.objectContaining({ dockerTlsCa: "ca-pem", dockerTlsCert: "cert-pem", dockerTlsKey: `enc:${KEY_PEM}`, dockerTlsKeySet: true }),
|
||||
TEAM_ID,
|
||||
USER_ID
|
||||
);
|
||||
});
|
||||
|
||||
it("does not encrypt anything for a socket url and leaves dockerTlsKeySet false", async () => {
|
||||
const monitorsRepository = createMonitorsRepositoryMock();
|
||||
(monitorsRepository.create as jest.Mock).mockResolvedValue(makeMonitor({ type: "docker" }));
|
||||
const encryptionService = createEncryptionServiceMock();
|
||||
const { service } = createService({ monitorsRepository, encryptionService });
|
||||
|
||||
await service.createMonitor(TEAM_ID, USER_ID, makeMonitor({ type: "docker", url: "unix:///var/run/docker.sock" }) as any);
|
||||
|
||||
expect(encryptionService.encrypt).not.toHaveBeenCalled();
|
||||
const [created] = (monitorsRepository.create as jest.Mock).mock.calls[0] as [Record<string, unknown>];
|
||||
expect(created.dockerTlsKeySet).toBe(false);
|
||||
expect(created.dockerTlsKey).toBeUndefined();
|
||||
});
|
||||
|
||||
it("ignores TLS fields on non-docker monitors", async () => {
|
||||
const monitorsRepository = createMonitorsRepositoryMock();
|
||||
(monitorsRepository.create as jest.Mock).mockResolvedValue(makeMonitor());
|
||||
const encryptionService = createEncryptionServiceMock();
|
||||
const { service } = createService({ monitorsRepository, encryptionService });
|
||||
|
||||
await service.createMonitor(TEAM_ID, USER_ID, makeMonitor({ dockerTlsKey: KEY_PEM }) as any);
|
||||
|
||||
expect(encryptionService.encrypt).not.toHaveBeenCalled();
|
||||
expect(monitorsRepository.create).toHaveBeenCalledWith(expect.objectContaining({ dockerTlsKey: KEY_PEM }), TEAM_ID, USER_ID);
|
||||
});
|
||||
|
||||
it("rejects with a 422 naming ENCRYPTION_KEY when encryption is not configured", async () => {
|
||||
const monitorsRepository = createMonitorsRepositoryMock();
|
||||
const encryptionService = createEncryptionServiceMock({ isConfigured: jest.fn(() => false) as any });
|
||||
const { service } = createService({ monitorsRepository, encryptionService });
|
||||
|
||||
await expectAppError(() => service.createMonitor(TEAM_ID, USER_ID, makeMonitor(tlsBody) as any), 422, /ENCRYPTION_KEY/);
|
||||
expect(encryptionService.encrypt).not.toHaveBeenCalled();
|
||||
expect(monitorsRepository.create).not.toHaveBeenCalled();
|
||||
});
|
||||
});
|
||||
|
||||
describe("editMonitor", () => {
|
||||
it("keeps the stored key when the key field is blank", async () => {
|
||||
const monitorsRepository = createMonitorsRepositoryMock();
|
||||
(monitorsRepository.findById as jest.Mock).mockResolvedValue(makeMonitor({ type: "docker", dockerTlsKeySet: true }));
|
||||
(monitorsRepository.updateById as jest.Mock).mockResolvedValue(makeMonitor({ type: "docker" }));
|
||||
const encryptionService = createEncryptionServiceMock();
|
||||
const { service } = createService({ monitorsRepository, encryptionService });
|
||||
|
||||
await service.editMonitor({ teamId: TEAM_ID, monitorId: MONITOR_ID, body: { ...tlsBody, dockerTlsKey: "" } });
|
||||
|
||||
expect(encryptionService.encrypt).not.toHaveBeenCalled();
|
||||
const [, , patch] = (monitorsRepository.updateById as jest.Mock).mock.calls[0] as [string, string, Record<string, unknown>];
|
||||
expect(patch).not.toHaveProperty("dockerTlsKey");
|
||||
expect(patch).not.toHaveProperty("dockerTlsKeySet");
|
||||
expect(patch).toMatchObject({ dockerTlsCa: "ca-pem", dockerTlsCert: "cert-pem" });
|
||||
});
|
||||
|
||||
it("rejects a blank key with a 422 when no key is stored", async () => {
|
||||
const monitorsRepository = createMonitorsRepositoryMock();
|
||||
(monitorsRepository.findById as jest.Mock).mockResolvedValue(makeMonitor({ type: "docker", dockerTlsKeySet: false }));
|
||||
const { service } = createService({ monitorsRepository });
|
||||
|
||||
await expectAppError(
|
||||
() => service.editMonitor({ teamId: TEAM_ID, monitorId: MONITOR_ID, body: { ...tlsBody, dockerTlsKey: "" } }),
|
||||
422,
|
||||
/required for a TLS Docker host/
|
||||
);
|
||||
expect(monitorsRepository.updateById).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("encrypts a replacement key without reading the stored monitor", async () => {
|
||||
const monitorsRepository = createMonitorsRepositoryMock();
|
||||
(monitorsRepository.updateById as jest.Mock).mockResolvedValue(makeMonitor({ type: "docker" }));
|
||||
const encryptionService = createEncryptionServiceMock();
|
||||
const { service } = createService({ monitorsRepository, encryptionService });
|
||||
|
||||
await service.editMonitor({ teamId: TEAM_ID, monitorId: MONITOR_ID, body: tlsBody });
|
||||
|
||||
expect(monitorsRepository.findById).not.toHaveBeenCalled();
|
||||
expect(monitorsRepository.updateById).toHaveBeenCalledWith(
|
||||
MONITOR_ID,
|
||||
TEAM_ID,
|
||||
expect.objectContaining({ dockerTlsKey: `enc:${KEY_PEM}`, dockerTlsKeySet: true }),
|
||||
{ unsetProxyId: false }
|
||||
);
|
||||
});
|
||||
|
||||
it("encrypts a replacement key when the body omits the url", async () => {
|
||||
const monitorsRepository = createMonitorsRepositoryMock();
|
||||
(monitorsRepository.updateById as jest.Mock).mockResolvedValue(makeMonitor({ type: "docker" }));
|
||||
const encryptionService = createEncryptionServiceMock();
|
||||
const { service } = createService({ monitorsRepository, encryptionService });
|
||||
|
||||
await service.editMonitor({ teamId: TEAM_ID, monitorId: MONITOR_ID, body: { type: "docker", dockerTlsKey: KEY_PEM } });
|
||||
|
||||
expect(encryptionService.encrypt).toHaveBeenCalledWith(KEY_PEM);
|
||||
expect(monitorsRepository.updateById).toHaveBeenCalledWith(
|
||||
MONITOR_ID,
|
||||
TEAM_ID,
|
||||
expect.objectContaining({ dockerTlsKey: `enc:${KEY_PEM}`, dockerTlsKeySet: true }),
|
||||
{ unsetProxyId: false }
|
||||
);
|
||||
});
|
||||
|
||||
it("clears dockerTlsKeySet when the url changes back to a socket", async () => {
|
||||
const monitorsRepository = createMonitorsRepositoryMock();
|
||||
(monitorsRepository.updateById as jest.Mock).mockResolvedValue(makeMonitor({ type: "docker" }));
|
||||
const encryptionService = createEncryptionServiceMock();
|
||||
const { service } = createService({ monitorsRepository, encryptionService });
|
||||
|
||||
await service.editMonitor({
|
||||
teamId: TEAM_ID,
|
||||
monitorId: MONITOR_ID,
|
||||
body: { type: "docker", url: "unix:///var/run/docker.sock", dockerTlsKey: KEY_PEM },
|
||||
});
|
||||
|
||||
expect(encryptionService.encrypt).not.toHaveBeenCalled();
|
||||
expect(monitorsRepository.findById).not.toHaveBeenCalled();
|
||||
const [, , patch] = (monitorsRepository.updateById as jest.Mock).mock.calls[0] as [string, string, Record<string, unknown>];
|
||||
expect(patch.dockerTlsKeySet).toBe(false);
|
||||
expect(patch.dockerTlsKey).toBeUndefined();
|
||||
});
|
||||
|
||||
it("rejects with a 422 naming ENCRYPTION_KEY when encryption is not configured", async () => {
|
||||
const monitorsRepository = createMonitorsRepositoryMock();
|
||||
const encryptionService = createEncryptionServiceMock({ isConfigured: jest.fn(() => false) as any });
|
||||
const { service } = createService({ monitorsRepository, encryptionService });
|
||||
|
||||
await expectAppError(() => service.editMonitor({ teamId: TEAM_ID, monitorId: MONITOR_ID, body: tlsBody }), 422, /ENCRYPTION_KEY/);
|
||||
expect(monitorsRepository.updateById).not.toHaveBeenCalled();
|
||||
});
|
||||
});
|
||||
});
|
||||
@@ -28,6 +28,7 @@ const createMonitorsRepo = () => ({
|
||||
|
||||
const createSettingsService = (clientHost = "https://app.example.com") => ({
|
||||
getSettings: jest.fn().mockReturnValue({ clientHost }),
|
||||
removeEgressNotification: jest.fn().mockResolvedValue(undefined),
|
||||
});
|
||||
|
||||
const createMessageBuilder = () => ({
|
||||
@@ -371,5 +372,26 @@ describe("NotificationsService", () => {
|
||||
expect(result).toBe(deleted);
|
||||
expect(monitorsRepository.removeNotificationFromMonitors).toHaveBeenCalledWith("notif-1");
|
||||
});
|
||||
|
||||
it("removes the notification from the egress recovered recipients after the team-scoped delete succeeds", async () => {
|
||||
const { service, notificationsRepository, settingsService } = createService();
|
||||
(notificationsRepository.deleteById as jest.Mock).mockResolvedValue(makeNotification());
|
||||
|
||||
await service.deleteById("notif-1", "team-1");
|
||||
|
||||
expect(settingsService.removeEgressNotification).toHaveBeenCalledWith("notif-1");
|
||||
const deleteOrder = (notificationsRepository.deleteById as jest.Mock).mock.invocationCallOrder[0]!;
|
||||
const cascadeOrder = (settingsService.removeEgressNotification as jest.Mock).mock.invocationCallOrder[0]!;
|
||||
expect(deleteOrder).toBeLessThan(cascadeOrder);
|
||||
});
|
||||
|
||||
it("does not touch app settings when the delete is refused", async () => {
|
||||
const { service, notificationsRepository, settingsService } = createService();
|
||||
(notificationsRepository.deleteById as jest.Mock).mockRejectedValue(new Error("Notification not found"));
|
||||
|
||||
await expect(service.deleteById("notif-1", "other-team")).rejects.toThrow("Notification not found");
|
||||
|
||||
expect(settingsService.removeEgressNotification).not.toHaveBeenCalled();
|
||||
});
|
||||
});
|
||||
});
|
||||
@@ -27,7 +27,7 @@ const makeProxy = (overrides?: Partial<Proxy>): Proxy =>
|
||||
const createResolver = (overrides?: Record<string, any>) => {
|
||||
const defaults = {
|
||||
proxiesRepository: { findByIdOrNull: jest.fn<() => Promise<Proxy | null>>().mockResolvedValue(makeProxy()) },
|
||||
settingsService: { getDBSettings: jest.fn<() => Promise<any>>().mockResolvedValue({ globalProxyEnabled: true, globalProxyId: "p1" }) },
|
||||
settingsService: { getCachedDBSettings: jest.fn<() => Promise<any>>().mockResolvedValue({ globalProxyEnabled: true, globalProxyId: "p1" }) },
|
||||
logger: createMockLogger(),
|
||||
ttlMs: 60_000,
|
||||
...overrides,
|
||||
@@ -46,7 +46,7 @@ describe("ProxyResolver", () => {
|
||||
|
||||
expect(result).toBeUndefined();
|
||||
expect(defaults.proxiesRepository.findByIdOrNull).not.toHaveBeenCalled();
|
||||
expect(defaults.settingsService.getDBSettings).not.toHaveBeenCalled();
|
||||
expect(defaults.settingsService.getCachedDBSettings).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("returns undefined for proxyMode 'none' without calling settings", async () => {
|
||||
@@ -55,7 +55,7 @@ describe("ProxyResolver", () => {
|
||||
const result = await resolver.resolve(makeMonitor({ proxyMode: "none" }));
|
||||
|
||||
expect(result).toBeUndefined();
|
||||
expect(defaults.settingsService.getDBSettings).not.toHaveBeenCalled();
|
||||
expect(defaults.settingsService.getCachedDBSettings).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
// ── custom mode ───────────────────────────────────────────────────────────
|
||||
@@ -94,7 +94,7 @@ describe("ProxyResolver", () => {
|
||||
|
||||
it("returns undefined when the global proxy is disabled, without a repo call", async () => {
|
||||
const { resolver, defaults } = createResolver({
|
||||
settingsService: { getDBSettings: jest.fn<() => Promise<any>>().mockResolvedValue({ globalProxyEnabled: false, globalProxyId: "p1" }) },
|
||||
settingsService: { getCachedDBSettings: jest.fn<() => Promise<any>>().mockResolvedValue({ globalProxyEnabled: false, globalProxyId: "p1" }) },
|
||||
});
|
||||
|
||||
const result = await resolver.resolve(makeMonitor({ proxyMode: "inherit" }));
|
||||
@@ -114,7 +114,7 @@ describe("ProxyResolver", () => {
|
||||
|
||||
it("returns undefined when the global proxy is enabled but globalProxyId is null, without a repo call", async () => {
|
||||
const { resolver, defaults } = createResolver({
|
||||
settingsService: { getDBSettings: jest.fn<() => Promise<any>>().mockResolvedValue({ globalProxyEnabled: true, globalProxyId: null }) },
|
||||
settingsService: { getCachedDBSettings: jest.fn<() => Promise<any>>().mockResolvedValue({ globalProxyEnabled: true, globalProxyId: null }) },
|
||||
});
|
||||
|
||||
const result = await resolver.resolve(makeMonitor({ proxyMode: "inherit" }));
|
||||
@@ -140,7 +140,7 @@ describe("ProxyResolver", () => {
|
||||
const result = await resolver.resolve(makeMonitor({ proxyMode: "garbage" as Monitor["proxyMode"] }));
|
||||
|
||||
expect(result).toBe("http://proxy.example.com:8080");
|
||||
expect(defaults.settingsService.getDBSettings).toHaveBeenCalled();
|
||||
expect(defaults.settingsService.getCachedDBSettings).toHaveBeenCalled();
|
||||
});
|
||||
|
||||
// ── never throws ──────────────────────────────────────────────────────────
|
||||
@@ -158,7 +158,7 @@ describe("ProxyResolver", () => {
|
||||
|
||||
it("returns undefined and warns when the settings service throws", async () => {
|
||||
const { resolver, defaults } = createResolver({
|
||||
settingsService: { getDBSettings: jest.fn<() => Promise<any>>().mockRejectedValue(new Error("db down")) },
|
||||
settingsService: { getCachedDBSettings: jest.fn<() => Promise<any>>().mockRejectedValue(new Error("db down")) },
|
||||
});
|
||||
|
||||
const result = await resolver.resolve(makeMonitor({ proxyMode: "inherit" }));
|
||||
@@ -175,7 +175,6 @@ describe("ProxyResolver", () => {
|
||||
await resolver.resolve(makeMonitor({ proxyMode: "inherit" }));
|
||||
await resolver.resolve(makeMonitor({ proxyMode: "inherit" }));
|
||||
|
||||
expect(defaults.settingsService.getDBSettings).toHaveBeenCalledTimes(1);
|
||||
expect(defaults.proxiesRepository.findByIdOrNull).toHaveBeenCalledTimes(1);
|
||||
});
|
||||
|
||||
@@ -185,7 +184,6 @@ describe("ProxyResolver", () => {
|
||||
await resolver.resolve(makeMonitor({ proxyMode: "inherit" }));
|
||||
await resolver.resolve(makeMonitor({ proxyMode: "inherit" }));
|
||||
|
||||
expect(defaults.settingsService.getDBSettings).toHaveBeenCalledTimes(2);
|
||||
expect(defaults.proxiesRepository.findByIdOrNull).toHaveBeenCalledTimes(2);
|
||||
});
|
||||
|
||||
@@ -209,7 +207,6 @@ describe("ProxyResolver", () => {
|
||||
resolver.resolve(makeMonitor({ proxyMode: "inherit" })),
|
||||
]);
|
||||
|
||||
expect(defaults.settingsService.getDBSettings).toHaveBeenCalledTimes(1);
|
||||
expect(defaults.proxiesRepository.findByIdOrNull).toHaveBeenCalledTimes(1);
|
||||
});
|
||||
|
||||
|
||||
@@ -0,0 +1,156 @@
|
||||
import { describe, expect, it, jest } from "@jest/globals";
|
||||
import * as nodeCrypto from "node:crypto";
|
||||
import { EncryptionService } from "../../../src/service/encryption/encryptionService.ts";
|
||||
import { SecretsRotationService } from "../../../src/service/encryption/secretsRotationService.ts";
|
||||
import type { IMonitorsRepository } from "../../../src/domain/monitors/monitor.repository.interface.ts";
|
||||
import { createMockLogger } from "../../helpers/createMockLogger.ts";
|
||||
|
||||
type Row = { id: string; dockerTlsKey: string };
|
||||
|
||||
const PLAINTEXT = "-----BEGIN PRIVATE KEY-----\nstub\n-----END PRIVATE KEY-----";
|
||||
|
||||
const newKey = () => nodeCrypto.randomBytes(32);
|
||||
|
||||
const encryptionServiceFor = (keys: Buffer[]) => new EncryptionService(keys, createMockLogger(), nodeCrypto);
|
||||
|
||||
const tamperData = (ciphertext: string) => {
|
||||
const parts = ciphertext.split(".");
|
||||
const data = parts[4] ?? "";
|
||||
parts[4] = (data[0] === "A" ? "B" : "A") + data.slice(1);
|
||||
return parts.join(".");
|
||||
};
|
||||
|
||||
const setup = (keys: Buffer[], rows: Row[]) => {
|
||||
const encryptionService = encryptionServiceFor(keys);
|
||||
const repository = {
|
||||
findAllDockerTlsKeys: jest.fn(async () => rows),
|
||||
updateDockerTlsKey: jest.fn(async () => undefined),
|
||||
};
|
||||
const logger = createMockLogger();
|
||||
const service = new SecretsRotationService(repository as unknown as IMonitorsRepository, encryptionService, logger);
|
||||
return { service, encryptionService, repository, logger };
|
||||
};
|
||||
|
||||
const updatedKeyFor = (repository: { updateDockerTlsKey: jest.Mock }, id: string): string => {
|
||||
const call = repository.updateDockerTlsKey.mock.calls.find(([calledId]) => calledId === id);
|
||||
expect(call).toBeDefined();
|
||||
return (call as [string, string])[1];
|
||||
};
|
||||
|
||||
describe("SecretsRotationService", () => {
|
||||
it("re-encrypts stale rows under the current key and leaves current rows untouched", async () => {
|
||||
const current = newKey();
|
||||
const previous = newKey();
|
||||
const stale = encryptionServiceFor([previous]);
|
||||
const rows = [
|
||||
{ id: "m-current", dockerTlsKey: encryptionServiceFor([current]).encrypt(PLAINTEXT) },
|
||||
{ id: "m-stale-1", dockerTlsKey: stale.encrypt(PLAINTEXT) },
|
||||
{ id: "m-stale-2", dockerTlsKey: stale.encrypt(PLAINTEXT) },
|
||||
];
|
||||
const { service, encryptionService, repository, logger } = setup([current, previous], rows);
|
||||
|
||||
const result = await service.run();
|
||||
|
||||
expect(result).toEqual({ scanned: 3, reencrypted: 2, skipped: 0 });
|
||||
expect(repository.updateDockerTlsKey).toHaveBeenCalledTimes(2);
|
||||
for (const id of ["m-stale-1", "m-stale-2"]) {
|
||||
const rotated = updatedKeyFor(repository, id);
|
||||
expect(encryptionService.keyIdOf(rotated)).toBe(encryptionService.currentKeyId());
|
||||
expect(encryptionService.decrypt(rotated)).toBe(PLAINTEXT);
|
||||
}
|
||||
expect(repository.updateDockerTlsKey).not.toHaveBeenCalledWith("m-current", expect.anything());
|
||||
expect(logger.warn).not.toHaveBeenCalled();
|
||||
expect(logger.info).toHaveBeenCalledWith(
|
||||
expect.objectContaining({ message: "Docker TLS key rotation: 2 re-encrypted, 0 skipped, 3 scanned", service: "SecretsRotationService" })
|
||||
);
|
||||
});
|
||||
|
||||
it("returns zero counts when every row is already under the current key", async () => {
|
||||
const current = newKey();
|
||||
const rows = [
|
||||
{ id: "m-1", dockerTlsKey: encryptionServiceFor([current]).encrypt(PLAINTEXT) },
|
||||
{ id: "m-2", dockerTlsKey: encryptionServiceFor([current]).encrypt(PLAINTEXT) },
|
||||
];
|
||||
const { service, repository, logger } = setup([current], rows);
|
||||
|
||||
const result = await service.run();
|
||||
|
||||
expect(result).toEqual({ scanned: 2, reencrypted: 0, skipped: 0 });
|
||||
expect(repository.updateDockerTlsKey).not.toHaveBeenCalled();
|
||||
expect(logger.info).toHaveBeenCalledWith(expect.objectContaining({ message: "Docker TLS key rotation: 0 re-encrypted, 0 skipped, 2 scanned" }));
|
||||
});
|
||||
|
||||
it("skips a row encrypted under an unknown key and logs its key id", async () => {
|
||||
const unknown = encryptionServiceFor([newKey()]);
|
||||
const rows = [{ id: "m-unknown", dockerTlsKey: unknown.encrypt(PLAINTEXT) }];
|
||||
const { service, repository, logger } = setup([newKey()], rows);
|
||||
|
||||
const result = await service.run();
|
||||
|
||||
expect(result).toEqual({ scanned: 1, reencrypted: 0, skipped: 1 });
|
||||
expect(repository.updateDockerTlsKey).not.toHaveBeenCalled();
|
||||
expect(logger.warn).toHaveBeenCalledWith(
|
||||
expect.objectContaining({
|
||||
message: expect.stringContaining("Could not reencrypt Docker TLS key for monitor m-unknown"),
|
||||
service: "SecretsRotationService",
|
||||
details: { monitorId: "m-unknown", keyId: unknown.currentKeyId() },
|
||||
})
|
||||
);
|
||||
});
|
||||
|
||||
it("continues past a row that fails to decrypt", async () => {
|
||||
const current = newKey();
|
||||
const previous = newKey();
|
||||
const stale = encryptionServiceFor([previous]);
|
||||
const rows = [
|
||||
{ id: "m-broken", dockerTlsKey: tamperData(stale.encrypt(PLAINTEXT)) },
|
||||
{ id: "m-stale", dockerTlsKey: stale.encrypt(PLAINTEXT) },
|
||||
];
|
||||
const { service, encryptionService, repository, logger } = setup([current, previous], rows);
|
||||
|
||||
const result = await service.run();
|
||||
|
||||
expect(result).toEqual({ scanned: 2, reencrypted: 1, skipped: 1 });
|
||||
expect(repository.updateDockerTlsKey).toHaveBeenCalledTimes(1);
|
||||
expect(encryptionService.decrypt(updatedKeyFor(repository, "m-stale"))).toBe(PLAINTEXT);
|
||||
expect(logger.warn).toHaveBeenCalledWith(
|
||||
expect.objectContaining({
|
||||
message: expect.stringContaining("Could not reencrypt Docker TLS key for monitor m-broken: Ciphertext failed authentication"),
|
||||
details: expect.objectContaining({ monitorId: "m-broken" }),
|
||||
})
|
||||
);
|
||||
expect(logger.info).toHaveBeenCalledWith(expect.objectContaining({ message: "Docker TLS key rotation: 1 re-encrypted, 1 skipped, 2 scanned" }));
|
||||
});
|
||||
|
||||
it("warns and skips every row when ENCRYPTION_KEY is not configured", async () => {
|
||||
const rows = [
|
||||
{ id: "m-1", dockerTlsKey: encryptionServiceFor([newKey()]).encrypt(PLAINTEXT) },
|
||||
{ id: "m-2", dockerTlsKey: encryptionServiceFor([newKey()]).encrypt(PLAINTEXT) },
|
||||
];
|
||||
const { service, repository, logger } = setup([], rows);
|
||||
|
||||
const result = await service.run();
|
||||
|
||||
expect(result).toEqual({ scanned: 2, reencrypted: 0, skipped: 2 });
|
||||
expect(repository.updateDockerTlsKey).not.toHaveBeenCalled();
|
||||
expect(logger.warn).toHaveBeenCalledTimes(1);
|
||||
expect(logger.warn).toHaveBeenCalledWith(
|
||||
expect.objectContaining({
|
||||
message: "2 monitor(s) have a stored Docker TLS key but ENCRYPTION_KEY is not set; their checks will fail until it is",
|
||||
service: "SecretsRotationService",
|
||||
})
|
||||
);
|
||||
expect(logger.info).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("logs nothing when ENCRYPTION_KEY is not configured and there are no rows", async () => {
|
||||
const { service, repository, logger } = setup([], []);
|
||||
|
||||
const result = await service.run();
|
||||
|
||||
expect(result).toEqual({ scanned: 0, reencrypted: 0, skipped: 0 });
|
||||
expect(repository.updateDockerTlsKey).not.toHaveBeenCalled();
|
||||
expect(logger.warn).not.toHaveBeenCalled();
|
||||
expect(logger.info).not.toHaveBeenCalled();
|
||||
});
|
||||
});
|
||||
@@ -16,6 +16,7 @@ const makeEnv = (overrides?: Partial<ValidatedEnv>): ValidatedEnv =>
|
||||
DB_CONNECTION_STRING: "mongodb://localhost:27017/test_db",
|
||||
DB_TYPE: "mongodb",
|
||||
STATUS_PAGE_THEMES_ENABLED: false,
|
||||
ENCRYPTION_KEY: [],
|
||||
...overrides,
|
||||
}) as ValidatedEnv;
|
||||
|
||||
@@ -42,12 +43,13 @@ const createSettingsRepo = () =>
|
||||
create: jest.fn().mockResolvedValue(makeSettings()),
|
||||
findSingleton: jest.fn().mockResolvedValue(makeSettings()),
|
||||
update: jest.fn().mockResolvedValue(makeSettings()),
|
||||
removeEgressNotification: jest.fn().mockResolvedValue(undefined),
|
||||
deleteLegacy: jest.fn().mockResolvedValue(true),
|
||||
}) as unknown as jest.Mocked<ISettingsRepository>;
|
||||
|
||||
const createService = (envOverrides?: Partial<ValidatedEnv>) => {
|
||||
const createService = (envOverrides?: Partial<ValidatedEnv>, dbSettingsTTLMs?: number) => {
|
||||
const env = makeEnv(envOverrides);
|
||||
const service = new SettingsService(env);
|
||||
const service = new SettingsService(env, dbSettingsTTLMs);
|
||||
const settingsRepository = createSettingsRepo();
|
||||
service.setRepository(settingsRepository);
|
||||
return { service, settingsRepository, env };
|
||||
@@ -170,6 +172,66 @@ describe("SettingsService", () => {
|
||||
});
|
||||
});
|
||||
|
||||
// ── getCachedDBSettings ─────────────────────────────────────────────────
|
||||
|
||||
describe("getCachedDBSettings", () => {
|
||||
it("serves a second call within the TTL from cache", async () => {
|
||||
const { service, settingsRepository } = createService();
|
||||
|
||||
const first = await service.getCachedDBSettings();
|
||||
const second = await service.getCachedDBSettings();
|
||||
|
||||
expect(second).toBe(first);
|
||||
expect(settingsRepository.findSingleton).toHaveBeenCalledTimes(1);
|
||||
});
|
||||
|
||||
it("refetches after the TTL expires", async () => {
|
||||
const { service, settingsRepository } = createService(undefined, 0);
|
||||
|
||||
await service.getCachedDBSettings();
|
||||
await service.getCachedDBSettings();
|
||||
|
||||
expect(settingsRepository.findSingleton).toHaveBeenCalledTimes(2);
|
||||
});
|
||||
|
||||
it("shares one in-flight fetch between concurrent calls", async () => {
|
||||
const { service, settingsRepository } = createService();
|
||||
|
||||
await Promise.all([service.getCachedDBSettings(), service.getCachedDBSettings(), service.getCachedDBSettings()]);
|
||||
|
||||
expect(settingsRepository.findSingleton).toHaveBeenCalledTimes(1);
|
||||
});
|
||||
|
||||
it("does not cache a rejected fetch, the next call retries", async () => {
|
||||
const { service, settingsRepository } = createService();
|
||||
(settingsRepository.findSingleton as jest.Mock).mockRejectedValueOnce(new Error("db down")).mockResolvedValueOnce(makeSettings());
|
||||
|
||||
await expect(service.getCachedDBSettings()).rejects.toThrow("db down");
|
||||
await expect(service.getCachedDBSettings()).resolves.toEqual(makeSettings());
|
||||
expect(settingsRepository.findSingleton).toHaveBeenCalledTimes(2);
|
||||
});
|
||||
|
||||
it("refetches after updateDbSettings so the write is visible immediately", async () => {
|
||||
const { service, settingsRepository } = createService();
|
||||
|
||||
await service.getCachedDBSettings();
|
||||
await service.updateDbSettings({ checkTTL: 60 });
|
||||
await service.getCachedDBSettings();
|
||||
|
||||
expect(settingsRepository.findSingleton).toHaveBeenCalledTimes(2);
|
||||
});
|
||||
|
||||
it("refetches after removeEgressNotification", async () => {
|
||||
const { service, settingsRepository } = createService();
|
||||
|
||||
await service.getCachedDBSettings();
|
||||
await service.removeEgressNotification("notif-1");
|
||||
await service.getCachedDBSettings();
|
||||
|
||||
expect(settingsRepository.findSingleton).toHaveBeenCalledTimes(2);
|
||||
});
|
||||
});
|
||||
|
||||
// ── updateDbSettings ────────────────────────────────────────────────────
|
||||
|
||||
describe("updateDbSettings", () => {
|
||||
@@ -191,4 +253,20 @@ describe("SettingsService", () => {
|
||||
await expect(service.updateDbSettings({ checkTTL: 60 })).rejects.toThrow("Settings repository not initialized");
|
||||
});
|
||||
});
|
||||
|
||||
describe("removeEgressNotification", () => {
|
||||
it("delegates the pull to the repository", async () => {
|
||||
const { service, settingsRepository } = createService();
|
||||
|
||||
await service.removeEgressNotification("notif-1");
|
||||
|
||||
expect(settingsRepository.removeEgressNotification).toHaveBeenCalledWith("notif-1");
|
||||
});
|
||||
|
||||
it("throws when repository is not set", async () => {
|
||||
const service = new SettingsService(makeEnv());
|
||||
|
||||
await expect(service.removeEgressNotification("notif-1")).rejects.toThrow("Settings repository not initialized");
|
||||
});
|
||||
});
|
||||
});
|
||||
Loaded 100 of 108 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user