CloudNativePG — Postgres on Kubernetes, the GitOps Way

The approach

CloudNativePG (CNPG) is a Kubernetes operator that runs Postgres as a first-class resource: failover, backups, and monitoring are declarative. The key decision here is everything goes through Helm + GitOps — no raw Cluster manifests, no kubectl apply, no manual helm upgrade. The CNPG cluster chart is a subchart of the app's own chart:

# Chart.yaml
dependencies:
  - name: cluster
    repository: "https://cloudnative-pg.github.io/charts"
    version: "0.7.0"
    alias: postgresql

Why a subchart: the database's full configuration lives under one postgresql: key in the app's values.yaml, versioned next to the app that uses it. Argo CD syncs both together — replicating the setup to a new environment means copying a values block, not re-running a runbook.

The moving parts

Core

CNPG Cluster

Instances, storage, resources, affinity and monitoring, all in values. The operator creates a read-write Service at <name>-rw.<namespace>:5432 — that's the hostname apps connect to.

Credentials

External Secrets

Two ExternalSecrets pull the app user and superuser credentials from the cloud secrets manager as kubernetes.io/basic-auth Secrets, referenced by initdb and superuserSecret. Apps get their connection URL the same way.

Backups

S3 via IAM role

WAL archiving plus scheduled base backups to S3, gzip-compressed and AES256-encrypted, with a retention policy. No access keys: the Postgres service account assumes an IAM role (IRSA on EKS).

Hardening

NetworkPolicy + monitoring

A NetworkPolicy restricts ingress to port 5432. A PodMonitor labeled for the Prometheus release exposes CNPG's built-in metrics — replication lag, WAL, transactions.

Secrets layout

One secrets-manager entry per environment holds everything the database stack needs:

Key	What it is
`postgres-username` / `postgres-password`	The application user — becomes the `initdb` owner of the app database.
`postgres-root-username` / `postgres-admin-password`	The superuser, wired in via `superuserSecret`.
`POSTGRES_URL`	Full connection string apps consume, e.g. `postgres://app_user:<pw>@myapp-postgresql-rw.myapp:5432/app_db`. Each component that needs the database pulls it through its own ExternalSecret.

Password rule learned in production: alphanumeric only. A password containing @ broke connection-string parsing in one component — silently. The URL parser split on the wrong @ and the failure surfaced far from the cause. Long random alphanumeric strings avoid the whole class of bug.

Backups without access keys

The backup config in values is short — the interesting part is the IAM plumbing behind inheritFromIAMRole:

backups:
  enabled: true
  provider: s3
  s3:
    region: eu-west-1
    bucket: <BACKUP_BUCKET>
    path: /cnpg
    inheritFromIAMRole: true
  wal:  { compression: gzip, encryption: AES256, maxParallel: 4 }
  data: { compression: gzip, encryption: AES256, jobs: 2 }
  scheduledBackups:
    - name: daily-backup
      schedule: "0 0 3 * * *"   # six fields — seconds first, not standard cron
      method: barmanObjectStore
  retentionPolicy: "4d"

IRSA instead of keys. An IAM role trusts the cluster's OIDC provider, scoped to the exact service account (system:serviceaccount:<ns>:<cluster-name>). The role ARN goes on the Postgres pod via serviceAccountTemplate annotations.

Least-privilege S3 policy. ListBucket on the bucket, object read/write/delete only under the /cnpg/* prefix.

CNPG schedules have six fields. 0 0 3 * * * is 3:00 AM — the first field is seconds. Copy-pasting a five-field cron expression silently means something else.

Validating the deployment

The kubectl cnpg plugin does most of the work. The one-shot psql pod is the fastest end-to-end test — it validates the secret, the URL, DNS and the NetworkPolicy in one command:

# Cluster health
kubectl cnpg status myapp-postgresql -n myapp

# Did External Secrets actually populate the credentials?
kubectl get secret myapp-postgresql -n myapp \
  -o jsonpath='{.data.username}' | base64 -d

# End-to-end connection test from inside the cluster
kubectl run -n myapp pg-test --rm -i --restart=Never \
  --image=postgres:16-alpine -- psql "$POSTGRES_URL" -c "SELECT 1;"

# Trigger and verify a backup
kubectl cnpg backup --immediate myapp-postgresql -n myapp
kubectl cnpg backups -n myapp
aws s3 ls s3://<BACKUP_BUCKET>/cnpg/

GitOps discipline: once Argo CD owns the app, never sync from the CLI in shared clusters — auto-sync or the UI only. A CLI sync with local state is how drift sneaks in.

Troubleshooting

Symptom	Likely cause & fix
Cluster pods stuck Pending	CNPG operator isn't installed in the cluster. The chart only renders the `Cluster` resource — the operator is a separate install.
`Init:Error` during bootstrap	The credentials Secret is missing or malformed. Check the ExternalSecret status and the secrets-manager keys it maps.
Backups failing	IRSA trust policy, S3 permissions, or the service-account annotation. Verify all three — the error rarely says which one.
App CrashLoopBackOff on startup	The connection URL wasn't injected — the component's ExternalSecret is missing the `POSTGRES_URL` entry.
Connection refused	Wrong hostname. Apps must use the operator's read-write Service: `<cluster>-rw.<namespace>:5432`.
Auth or parsing errors with a correct password	Special characters in the password breaking URL parsing. Regenerate alphanumeric-only.