Architecture & Ops

Operations

This page covers the operational maintenance of the system: backup and restore, data retention, health checks with graceful shutdown, and the pre-deploy production verification.

Backup and restore

PostgreSQL, the uploads volume, and CREDENTIALS_ENC_KEY from .env must be protected together. Without the key, the credential vault cannot be recovered.

BACKUP_DIR=/guvenli/hedef ./scripts/backup.sh
./scripts/restore.sh /guvenli/hedef/20260716T120000Z

Encrypt the backup target at the disk level, restrict it to operators only, and keep a second copy on a different machine/region. A reasonable starting target is a daily backup, 30 days of retention, and a monthly restore rehearsal in an isolated environment.

Each backup includes a SHA256SUMS integrity manifest. Before a restore begins, the manifest, the PostgreSQL dump catalog, and the uploads archive are verified; if verification fails, the existing data is left untouched. During a restore the backend is stopped and restarted even if the operation fails. environment.snapshot contains secrets and is not automatically written over the active .env; the operator must compare the keys separately.

If the credential vault key (CREDENTIALS_ENC_KEY) is lost, encrypted records cannot be recovered. Back up the key and store the backup separately from the data itself.

Data retention

By default the retention command only produces a report:

docker compose exec backend npm run db:retention:check
docker compose exec backend npm run db:retention:apply

RETENTION_CLOSED_TICKET_DAYS defaults to 365 days and cannot be lower than 30 days. Take a backup before deleting. On the first run, do not run apply until you have confirmed the file count and byte total in the check output.

For automated reporting, run only the check mode daily via a host cron/systemd timer; apply should never be scheduled without human approval:

15 3 * * * cd /opt/ticket-system && docker compose exec -T backend npm run db:retention:check >> /var/log/ticket-retention.log 2>&1

For deletion, run db:retention:apply after a fresh backup during a maintenance window. The output is JSON and can be monitored by a central log/alerting system.

apply deletes data permanently. It must not be scheduled without human approval, and only run after a fresh backup during a maintenance window.

Health and shutdown

  • /health/live: is the Node.js process responding.
  • /health/ready: are PostgreSQL and Redis accepting requests.
  • SIGTERM/SIGINT: HTTP, workers, queue, Redis, and Prisma connections are shut down gracefully within at most 30 seconds.

Alarm thresholds: readiness failing for 2 minutes, disk usage at 80 percent, the last successful backup older than 26 hours, and an unusual increase in the failed notification count.

Pre-deploy production check

./scripts/check-production-env.sh .env
docker compose config --quiet

The check rejects missing/changeme_* secrets, development mode, identical or short JWT keys, an invalid vault key, and a non-HTTPS APP_URL. It also warns if the API documentation is left open.

See Security for security requirements, and Installation for setup and hardening guidance.