Checkmate/charts/helm/checkmate/INSTALLATION.md

# Kubernetes Installation Guide for Checkmate

This guide walks you through deploying Checkmate on your Kubernetes cluster using Helm.

## Prerequisites

- A running Kubernetes cluster
- Helm CLI installed and configured
- `kubectl` configured to access your cluster

## Steps

### 1. Clone the repo and navigate to the Helm chart

```bash
git clone https://github.com/bluewave-labs/checkmate.git
cd checkmate/charts/helm/checkmate
```

### 2. Customize values.yaml
Edit `values.yaml` to update:
- `client.ingress.host` and `api.ingress.host` with your domain names
- `api.protocol` (usually http or https)
- **If upgrading**: Migrate persistence settings from flat structure to nested:
  - Old: `persistence.mongodbSize` → New: `persistence.mongo.size`
  - Add: `persistence.mongo.storageClass` (leave empty for default)
- Secrets under the `secrets` section (`JWT_SECRET`, email credentials, API keys, etc.) — replace all change_me values
- **For TLS/HTTPS**: Configure ingress TLS settings (see section below)

### 3. Deploy the Helm chart
```bash
helm install checkmate ./charts/helm/checkmate
```
This will deploy the client, api, and MongoDB components.

### 4. Verify the deployment
Check pods and services:
```bash
kubectl get pods
kubectl get svc
```

Once all pods are `Running` and `Ready`, you can access Checkmate via the configured ingress hosts.

## Scaling: the worker tier & autoscaling

By default the chart runs a single all-in-one API pod that both schedules **and** processes monitoring
jobs (`worker.enabled: false`). For larger deployments you can split processing onto a dedicated,
horizontally-scalable worker tier.

### Topology

| `worker.enabled` | API (`checkmate-api`) | Worker (`checkmate-worker`) |
| --- | --- | --- |
| `false` (default) | schedules **and** processes jobs (`QUEUE_PRIMARY_PROCESSES=true`) | not deployed |
| `true` | schedules + serves the API only (`QUEUE_PRIMARY_PROCESSES=false`, derived) | processes all jobs |

`QUEUE_PRIMARY_PROCESSES` is **derived** from `worker.enabled` — you never set it directly, so the API
and worker tier can never both process (or both ignore) the queue.

### Enabling the worker tier

```yaml
worker:
  enabled: true
  replicas: 2   # used only when autoscaling.enabled = false
```

The worker takes **no inbound traffic** (no Service): the kubelet probes pod IPs directly on
`HEALTH_PORT` (`/livez`, `/readyz`), and graceful shutdown drains in-flight jobs before exit.

> **Invariant:** `worker.terminationGracePeriodSeconds` (default `30`) **must stay greater than the
> server drain timeout (25s)**. Lower it and Kubernetes SIGKILLs workers mid-drain, losing in-flight
> checks.

> **Caveat:** with the worker tier enabled the API no longer processes jobs, so if **all** workers are
> down nothing processes. The `autoscaling.minReplicaCount: 1` floor plus liveness restarts mitigate
> this; full failover to the API would need leader election (not supported).

### Autoscaling with KEDA

Backlog-driven autoscaling is **off by default** and requires the
[KEDA operator](https://keda.sh/) installed in-cluster. Enable it with:

```yaml
worker:
  enabled: true
  autoscaling:
    enabled: true
    minReplicaCount: 1     # processing floor — never 0, or checks silently stall
    maxReplicaCount: 10
    backlogPerReplica: 50  # target due-job backlog per worker
    mongo:
      dbName: uptime_db    # MUST match the database in DB_CONNECTION_STRING
```

A KEDA `ScaledObject` queries MongoDB directly for the due-check backlog and scales the worker
Deployment between `minReplicaCount` and `maxReplicaCount`. When autoscaling is on, the Deployment
omits `replicas` so it doesn't fight the HPA KEDA creates. Make sure `autoscaling.mongo.dbName`
matches the database name in your `DB_CONNECTION_STRING`, or the scaler counts the wrong DB (always 0
backlog → never scales up).

#### Fallback: Prometheus-adapter HPA (not shipped)

If your KEDA/Mongo versions reject the `$$NOW`/`$toLong` query, or you already run Prometheus, scrape
the worker's `/metrics` (`checkmate_worker_due_backlog`), expose it via prometheus-adapter, and target
it with a standard `HorizontalPodAutoscaler`. The manifest is left to the operator; the app computes
`now` in code on this path, so the query coercion is unnecessary.

## Enabling TLS/HTTPS with cert-manager

If you have [cert-manager](https://cert-manager.io/) installed in your cluster, you can enable automatic TLS certificate provisioning using Let's Encrypt or other certificate issuers.

### Prerequisites
- cert-manager installed in your cluster
- A ClusterIssuer or Issuer configured (e.g., `letsencrypt-prod`)

### Configuration

Edit `values.yaml` to enable TLS (and update protocols to https):

```yaml
client:
  protocol: https
  ingress:
    enabled: true
    host: checkmate.example.com
    className: nginx
    annotations:
      cert-manager.io/cluster-issuer: "letsencrypt-prod"
    tls:
      enabled: true
      secretName: checkmate-client-tls

api:
  protocol: https
  ingress:
    enabled: true
    host: checkmate.example.com
    className: nginx
    annotations:
      cert-manager.io/cluster-issuer: "letsencrypt-prod"
    tls:
      enabled: true
      secretName: checkmate-api-tls
```

### Alternative: Using --set flags

You can also enable TLS during installation using Helm's `--set` flags:

```bash
helm install checkmate ./charts/helm/checkmate \
  --set client.protocol=https \
  --set api.protocol=https \
  --set client.ingress.annotations."cert-manager\.io/cluster-issuer"="letsencrypt-prod" \
  --set client.ingress.tls.enabled=true \
  --set client.ingress.tls.secretName=checkmate-client-tls \
  --set api.ingress.annotations."cert-manager\.io/cluster-issuer"="letsencrypt-prod" \
  --set api.ingress.tls.enabled=true \
  --set api.ingress.tls.secretName=checkmate-api-tls
```

### Verification

After deployment, cert-manager will automatically create the TLS secrets. You can verify the certificate status:

```bash
# Check certificates
kubectl get certificates

# Check certificate details
kubectl describe certificate checkmate-client-tls
kubectl describe certificate checkmate-api-tls

# Verify the secrets were created
kubectl get secrets | grep checkmate-tls
```

The ingress will automatically use these secrets to enable HTTPS access to your Checkmate instance.