GitOps on k3s: Getting-Started Guide (Single Repo)¶
Part of the Git Operations (GitOps) section. For the concepts behind this guide, see GitOps: A Comprehensive Guide. The alternative layout, one platform repo plus one repo per project, is in the multi-repo guide, and the single vs multi-repo comparison weighs the two.
Commands default to GitHub. If you use GitLab, every step that differs links to the GitLab appendix at the end.
Overview and assumptions¶
By the end of this guide, one Git repository will describe everything running on your k3s cluster, and Argo CD will keep the cluster matching it. Adding a new project becomes "create a folder, open a PR, merge". Nobody runs kubectl apply by hand any more.
The core idea: Git is the source of truth, a controller in the cluster pulls from Git and reconciles, and every change is a commit you can review, diff and revert.
What you will build
- A
gitopsrepository with a layout that scales from 3 namespaces to 30. - Argo CD installed on k3s, managing itself from that repo.
- A single root Application plus ApplicationSets that discover new projects automatically from folders.
- Kustomize base/overlay structure for environments, and Helm for third-party charts.
- Encrypted secrets committed safely to Git.
- Your existing hand-applied workloads adopted without downtime.
- Day-2 habits: drift detection, rollbacks, automated image updates and notifications.
Assumptions (adapt where yours differ)
| Assumption | If yours differs |
|---|---|
| Single k3s cluster, 1 to 3 nodes, recent k3s (v1.30+) | Multi-cluster works too; see the note in Next steps |
Bundled Traefik is your ingress, local-path is your storage class |
Swap the Ingress/IngressRoute and storageClassName values |
| Git hosted on GitHub (private repo) | GitLab: see the GitLab appendix. Gitea or Forgejo: only the repo URL and credentials change |
You have kubectl, helm and git locally, with cluster-admin kubeconfig |
Run commands from a node with k3s kubectl instead |
| Workloads are currently plain YAML and/or Helm releases applied by hand | Section 8 covers both |
| One environment today, maybe staging later | The overlay layout handles both from day one |
Placeholder names: repo github.com/YOUR_USER/gitops, domain home.example.com, projects blog, media, monitoring |
Search and replace with your own |
Time budget: about 1 hour to bootstrap, then roughly 15 to 30 minutes per existing project you migrate.
1. Choose a tool: use Argo CD¶
For a k3s cluster with several projects and more coming, I recommend Argo CD. Flux is an equally solid choice; the deciding factors for you are visibility and multi-project ergonomics.
| Argo CD | Flux | |
|---|---|---|
| Web UI | Built in: live resource tree, diffs, sync/rollback buttons | None built in (third-party UIs like Capacitor or Weave GitOps) |
| Many projects | ApplicationSets generate one app per folder automatically; AppProjects fence projects off | Kustomization per folder; tenancy via service-account impersonation |
| Resource cost | Roughly 400 to 700 MiB RAM in a small install | Roughly 150 to 300 MiB; lighter |
| Helm | Renders charts then applies (no Helm release secrets) | Native helm install/upgrade via helm-controller |
| Secrets | Sealed Secrets is simplest; SOPS needs a plugin (KSOPS) | SOPS decryption built in |
| Image auto-update | Argo CD Image Updater (add-on) | Built in (image-reflector + image-automation) |
| Bootstrap | helm install, then hand it its own config |
flux bootstrap CLI does it all in one command |
Why Argo CD here: with a growing number of namespaces, the UI pays for itself. You see every project's health on one page, you see drift as a diff, and you can roll back with a click while learning. ApplicationSets make "new folder = new app" a one-time setup.
When to pick Flux instead: the cluster is very memory-constrained (Raspberry Pi with 2 GB), you prefer a CLI-only workflow, or you want SOPS without plugins. The repo layout in this guide transfers almost unchanged; Flux equivalents are noted in each section.
Don't run both. Pick one controller per cluster to avoid two reconcilers fighting over the same objects.
2. Design the repo layout¶
Use one repo, three top-level areas, and one rule: folder name = Argo CD app name = namespace. That rule is what lets ApplicationSets add new projects with zero extra config.
gitops/
├── bootstrap/
│ └── root.yaml # the ONLY thing you ever kubectl apply
├── argocd/ # Argo CD manages its own config from here
│ ├── install/ # Helm values for Argo CD itself
│ │ └── values.yaml
│ ├── projects/ # one AppProject per tenant/group
│ │ ├── infrastructure.yaml
│ │ └── apps.yaml
│ └── appsets/ # ApplicationSets that discover folders
│ ├── infrastructure.yaml
│ └── apps.yaml
├── infrastructure/ # cluster-wide services, synced first
│ ├── sealed-secrets/
│ ├── cert-manager/
│ ├── traefik-config/ # HelmChartConfig, middlewares
│ └── monitoring/
└── apps/ # your projects, one folder each
├── blog/
│ ├── base/
│ │ ├── kustomization.yaml
│ │ ├── namespace.yaml
│ │ ├── deployment.yaml
│ │ ├── service.yaml
│ │ └── ingress.yaml
│ └── overlays/
│ └── prod/
│ ├── kustomization.yaml
│ └── sealed-secret.yaml
└── media/
└── overlays/prod/ ...
Why this shape
infrastructure/andapps/are separate so platform pieces (cert-manager, Sealed Secrets, CRDs) can sync before workloads that depend on them, and so you can give them different permissions.- Every project has a
base/(what the app is) andoverlays/<env>/(where and how it runs). Even with one environment, start withoverlays/prod/. Adding staging later is then a new folder, not a refactor. - Argo CD's own config lives in the repo, so upgrading Argo CD is also a PR.
Decisions to make now
- Monorepo vs repo-per-app: keep deploy manifests in this one repo even if app source code lives elsewhere. CI in the app repos builds images; this repo only says which image tag runs. Split later only if different people need different write access.
- Branching: deploy from
mainonly. Use PRs for review. Avoid branch-per-environment; use folders per environment instead, because branches drift and merges get painful. - Naming: lowercase, DNS-safe folder names (
home-assistant, notHomeAssistant), since they become namespace names.
Flux equivalent: the same tree works; replace argocd/ with clusters/home/ containing flux-system/ plus one Kustomization per area.
3. Bootstrap Argo CD on k3s¶
You install Argo CD once with Helm, point it at the repo, then apply one root Application. From then on, Argo CD manages itself and everything else from Git.
Step 3.1: Create the repo¶
GitLab: create the project with
glaband usegitlab.comURLs; see G1.
# Create an empty PRIVATE repo named gitops on GitHub first, then:
git clone [email protected]:YOUR_USER/gitops.git && cd gitops
mkdir -p bootstrap argocd/{install,projects,appsets} infrastructure apps
Step 3.2: Write Argo CD's Helm values¶
argocd/install/values.yaml:
global:
domain: argocd.home.example.com
configs:
params:
server.insecure: true # Traefik terminates TLS, Argo serves plain HTTP behind it
cm:
application.resourceTrackingMethod: annotation # safer than labels; default in v3, set it explicitly
kustomize.buildOptions: --enable-helm # lets Kustomize inflate Helm charts
timeout.reconciliation: 180s # how often Git is polled
exec.enabled: false
server:
ingress:
enabled: true
ingressClassName: traefik
tls: true
# annotations:
# cert-manager.io/cluster-issuer: letsencrypt # once cert-manager is in place
dex:
enabled: false # turn on later if you want GitHub/OIDC SSO
# Small-cluster resource hints; raise if you have many apps
controller:
resources:
requests: { cpu: 100m, memory: 256Mi }
repoServer:
resources:
requests: { cpu: 50m, memory: 128Mi }
Step 3.3: Install it¶
helm repo add argo https://argoproj.github.io/argo-helm
helm repo update
helm search repo argo/argo-cd --versions | head -5 # pick the latest and PIN it
export ARGO_CHART_VERSION=<chart version from above>
kubectl create namespace argocd
helm install argocd argo/argo-cd -n argocd \
--version "$ARGO_CHART_VERSION" \
-f argocd/install/values.yaml
kubectl -n argocd rollout status deploy/argocd-server
kubectl -n argocd get pods
Step 3.4: Log in and secure the admin account¶
# Initial password
kubectl -n argocd get secret argocd-initial-admin-secret \
-o jsonpath='{.data.password}' | base64 -d; echo
# Until DNS/TLS is sorted, use a port-forward
kubectl -n argocd port-forward svc/argocd-server 8080:80
# Install the CLI (macOS: brew install argocd; Linux: download from the GitHub releases page)
argocd login localhost:8080 --plaintext --username admin
argocd account update-password
kubectl -n argocd delete secret argocd-initial-admin-secret
The UI is at http://localhost:8080 (or https://argocd.home.example.com once DNS points at your Traefik).
Step 3.5: Give Argo CD read access to the repo¶
GitLab: deploy keys work the same way (project Settings, Repository, Deploy keys), or use a project deploy token; see G2.
Use a read-only deploy key, so a compromised cluster can't push to Git.
ssh-keygen -t ed25519 -N '' -C argocd -f ~/.ssh/argocd_gitops
# GitHub: repo Settings > Deploy keys > Add, paste ~/.ssh/argocd_gitops.pub, leave "write" unticked
argocd repo add [email protected]:YOUR_USER/gitops.git \
--ssh-private-key-path ~/.ssh/argocd_gitops
argocd repo list # STATUS should be Successful
This creates a Secret in argocd. Don't commit it in plain text; once Sealed Secrets is running (section 6) you can seal it and move it into Git.
Step 3.6: Let Argo CD manage itself¶
argocd/argocd.yaml installs the same chart version with the values file from Git, so future upgrades are a PR that bumps targetRevision:
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: argocd
namespace: argocd
spec:
project: default
sources:
- repoURL: https://argoproj.github.io/argo-helm
chart: argo-cd
targetRevision: <same as ARGO_CHART_VERSION>
helm:
releaseName: argocd
valueFiles:
- $values/argocd/install/values.yaml
- repoURL: [email protected]:YOUR_USER/gitops.git
targetRevision: main
ref: values
destination:
server: https://kubernetes.default.svc
namespace: argocd
syncPolicy:
automated:
prune: false # never let Argo prune itself by accident
selfHeal: true
syncOptions:
- ServerSideApply=true # Argo's CRDs are too large for client-side apply
argocd/kustomization.yaml ties the folder together (the projects and appsets files are written in section 4; write them before you push in step 3.7, or comment those four lines out until then):
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- argocd.yaml
- projects/infrastructure.yaml
- projects/apps.yaml
- appsets/infrastructure.yaml
- appsets/apps.yaml
Step 3.7: Apply the root Application (the last manual apply)¶
bootstrap/root.yaml:
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: root
namespace: argocd
# Deliberately NO resources-finalizer here: deleting root must not cascade-delete the cluster
spec:
project: default
source:
repoURL: [email protected]:YOUR_USER/gitops.git
targetRevision: main
path: argocd
destination:
server: https://kubernetes.default.svc
namespace: argocd
syncPolicy:
automated:
prune: true
selfHeal: true
git add . && git commit -m "Bootstrap Argo CD" && git push
kubectl apply -f bootstrap/root.yaml
argocd app list # root and argocd should go Synced / Healthy
Checkpoint: in the UI you should see root and argocd. If argocd shows OutOfSync on a few fields right after adoption, sync it once; that is Helm-installed objects being taken over.
4. Scale to many projects: AppProjects and ApplicationSets¶
The root Application creates two ApplicationSets, and each ApplicationSet creates one Application per folder it finds. A new project is therefore just a new folder in Git; you never hand-write an Application again.
root (bootstrap/root.yaml, applied once)
├── argocd Application: Argo CD manages itself
├── infrastructure ApplicationSet: one app per infrastructure/*
│ ├── sealed-secrets
│ ├── cert-manager
│ └── traefik-config
└── apps ApplicationSet: one app per apps/*
├── blog
├── media
├── monitoring
└── (your next folder)
Root also creates the two AppProjects below, which set the guardrails each generated app runs inside.
Step 4.1: AppProjects as guardrails¶
AppProjects limit what an app may deploy and where. Infrastructure gets broad rights; your workloads get only namespaced resources, so a typo in apps/blog can never touch kube-system or create a ClusterRole.
argocd/projects/infrastructure.yaml:
apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
name: infrastructure
namespace: argocd
spec:
description: Cluster-wide services under infrastructure/
sourceRepos:
- [email protected]:YOUR_USER/gitops.git
- https://* # allow upstream Helm repos (tighten to a list later)
destinations:
- server: https://kubernetes.default.svc
namespace: '*'
clusterResourceWhitelist:
- group: '*'
kind: '*'
argocd/projects/apps.yaml:
apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
name: apps
namespace: argocd
spec:
description: Workloads under apps/
sourceRepos:
- [email protected]:YOUR_USER/gitops.git
destinations:
- server: https://kubernetes.default.svc
namespace: '*'
- server: '*'
namespace: '!kube-system' # deny rules win over allows
- server: '*'
namespace: '!argocd'
clusterResourceWhitelist:
- group: ''
kind: Namespace # only cluster-scoped thing apps may create
orphanedResources:
warn: true # flags objects in the namespace that are not in Git
Step 4.2: ApplicationSet for infrastructure¶
argocd/appsets/infrastructure.yaml:
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
name: infrastructure
namespace: argocd
spec:
goTemplate: true
goTemplateOptions: ["missingkey=error"]
generators:
- git:
repoURL: [email protected]:YOUR_USER/gitops.git
revision: main
directories:
- path: infrastructure/*
template:
metadata:
name: '{{ .path.basename }}'
labels:
area: infrastructure
spec:
project: infrastructure
source:
repoURL: [email protected]:YOUR_USER/gitops.git
targetRevision: main
path: '{{ .path.path }}'
destination:
server: https://kubernetes.default.svc
namespace: '{{ .path.basename }}'
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- CreateNamespace=true
- ServerSideApply=true
- SkipDryRunOnMissingResource=true # CRDs may arrive in the same sync
retry:
limit: 5
backoff: { duration: 15s, factor: 2, maxDuration: 5m }
syncPolicy:
preserveResourcesOnDeletion: true # removing a folder won't nuke running services by surprise
Step 4.3: ApplicationSet for your projects¶
argocd/appsets/apps.yaml generates one app per apps/<name>/overlays/prod folder. The app name and namespace both come from <name>:
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
name: apps
namespace: argocd
spec:
goTemplate: true
goTemplateOptions: ["missingkey=error"]
generators:
- git:
repoURL: [email protected]:YOUR_USER/gitops.git
revision: main
directories:
- path: apps/*/overlays/prod
template:
metadata:
name: '{{ index .path.segments 1 }}'
labels:
area: apps
spec:
project: apps
source:
repoURL: [email protected]:YOUR_USER/gitops.git
targetRevision: main
path: '{{ .path.path }}'
destination:
server: https://kubernetes.default.svc
namespace: '{{ index .path.segments 1 }}'
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- CreateNamespace=true
- PruneLast=true
retry:
limit: 3
backoff: { duration: 10s, factor: 2, maxDuration: 2m }
syncPolicy:
preserveResourcesOnDeletion: true
Ordering: ApplicationSets don't wait for one app to be healthy before syncing the next. The retry blocks handle the common case (an app needs a CRD from cert-manager that isn't installed yet): it fails, backs off, and succeeds once the CRD appears. Within a single app, use argocd.argoproj.io/sync-wave annotations to order resources (for example, a database StatefulSet in wave -1, the app Deployment in wave 0).
Per-app exceptions: if one project needs different settings (say, manual sync), use templatePatch in the ApplicationSet keyed on a file in the folder, or give that project its own small Application in argocd/ and exclude its folder with exclude: true in the generator.
Flux equivalent: one Kustomization for ./infrastructure and one for ./apps with dependsOn: infrastructure; each subfolder has its own Kustomization pointing at overlays/prod.
5. Environments and overlays: Kustomize for your apps, Helm for third-party charts¶
Use Kustomize base/ + overlays/<env>/ for apps you write, and let Kustomize inflate Helm charts for software you consume (cert-manager, Grafana, Nextcloud). That gives every folder the same entry point: a kustomization.yaml.
Step 5.1: A base for one project¶
apps/blog/base/kustomization.yaml:
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- namespace.yaml
- deployment.yaml
- service.yaml
- ingress.yaml
labels:
- pairs:
app.kubernetes.io/name: blog
app.kubernetes.io/part-of: blog
includeSelectors: false
apps/blog/base/deployment.yaml (trimmed):
apiVersion: apps/v1
kind: Deployment
metadata:
name: blog
spec:
replicas: 1
selector:
matchLabels: { app: blog }
template:
metadata:
labels: { app: blog }
spec:
containers:
- name: blog
image: ghcr.io/YOUR_USER/blog:1.4.2 # always a pinned tag, never :latest
ports: [{ containerPort: 8080 }]
envFrom:
- secretRef: { name: blog-secrets }
resources:
requests: { cpu: 50m, memory: 64Mi }
limits: { memory: 256Mi }
Step 5.2: The prod overlay¶
apps/blog/overlays/prod/kustomization.yaml:
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: blog
resources:
- ../../base
- sealed-secret.yaml # section 6
images:
- name: ghcr.io/YOUR_USER/blog
newTag: 1.4.2 # the one line that changes on each release
patches:
- target: { kind: Ingress, name: blog }
patch: |
- op: replace
path: /spec/rules/0/host
value: blog.home.example.com
Check what Argo CD will render before you push:
kubectl kustomize apps/blog/overlays/prod | less
kubectl kustomize apps/blog/overlays/prod | kubectl diff -f - # vs the live cluster
Step 5.3: Adding staging later¶
- Create
apps/blog/overlays/staging/withnamespace: blog-staging, a different host and maybereplicas: 1. - Change the apps ApplicationSet generator path to
apps/*/overlays/*and the names to include the environment:
name: '{{ index .path.segments 1 }}-{{ .path.basename }}' # blog-prod, blog-staging
# destination.namespace:
namespace: '{{ index .path.segments 1 }}{{ if ne .path.basename "prod" }}-{{ .path.basename }}{{ end }}'
Renaming apps from blog to blog-prod makes Argo CD create new Applications and delete the old ones. Because preserveResourcesOnDeletion: true is set, the running workloads stay put and are adopted by the new app. Still, do it when you can tolerate a blip, or decide on the -prod suffix from day one.
Promotion is then a PR that copies the image tag from overlays/staging to overlays/prod.
Step 5.4: Third-party software via Helm¶
For upstream charts, keep the chart version and values in Git and let Kustomize render them. infrastructure/cert-manager/kustomization.yaml:
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: cert-manager
helmCharts:
- name: cert-manager
repo: https://charts.jetstack.io
version: v1.x.y # pin; Renovate can bump it (section 10)
releaseName: cert-manager
namespace: cert-manager
includeCRDs: true
valuesInline:
crds:
enabled: true
resources:
- cluster-issuer.yaml # your own manifests alongside the chart
This works because Step 3.2 set kustomize.buildOptions: --enable-helm. Preview locally with kubectl kustomize --enable-helm infrastructure/cert-manager.
Alternative: a multi-source Application with chart: + $values/... (as in Step 3.6). It's cleaner for big charts with long values files, but it can't be auto-discovered by the folder ApplicationSet. Use Kustomize helmCharts by default and switch per app only when it gets awkward.
Rule of thumb
| You need | Use |
|---|---|
| Your own app, a few manifests | Plain YAML in base/ + Kustomize overlay |
| An upstream chart with modest values | Kustomize helmCharts in its folder |
| An upstream chart with huge values or OCI-only quirks | Multi-source Application with $values |
| Same app, slightly different per env | Kustomize patches in each overlay |
6. Secrets: Sealed Secrets (recommended with Argo CD)¶
Plain Secrets must never land in Git. With Argo CD, Sealed Secrets is the least-friction option: you encrypt locally with a public key, commit the ciphertext, and only the controller in your cluster can decrypt it. No Argo CD plugins are needed.
| Option | How it works | Pick it when |
|---|---|---|
| Sealed Secrets | Controller holds a private key; kubeseal encrypts with its public cert |
Default for Argo CD on a single cluster |
| SOPS + age | Files encrypted with an age key; decrypted at render time | You use Flux (built in), or want to edit encrypted YAML in place; with Argo CD it needs the KSOPS plugin |
| External Secrets Operator | Pulls from 1Password, Bitwarden, Vault, cloud KMS | You already keep secrets in a password manager or vault; good later step |
Step 6.1: Install the controller from Git¶
infrastructure/sealed-secrets/kustomization.yaml:
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: sealed-secrets
helmCharts:
- name: sealed-secrets
repo: https://bitnami-labs.github.io/sealed-secrets
version: 2.x.y # pin the latest
releaseName: sealed-secrets
namespace: sealed-secrets
includeCRDs: true
valuesInline:
fullnameOverride: sealed-secrets-controller
Push, wait for the sealed-secrets app to go Healthy, then install the CLI (brew install kubeseal, or the binary from the GitHub releases page) and save yourself some typing:
alias kubeseal='kubeseal --controller-namespace sealed-secrets --controller-name sealed-secrets-controller'
Step 6.2: Back up the sealing key (do this today)¶
If you rebuild the cluster without this key, every sealed secret in Git becomes unreadable.
kubectl -n sealed-secrets get secret \
-l sealedsecrets.bitnami.com/sealed-secrets-key -o yaml > sealed-secrets-key.backup.yaml
# Store it in your password manager or offline. NEVER commit it.
# On a rebuilt cluster: kubectl apply -f sealed-secrets-key.backup.yaml BEFORE the controller starts (or restart it after)
The controller rotates in a new key every 30 days by default and keeps the old ones, so re-run the backup occasionally.
Step 6.3: Seal a secret¶
kubectl create secret generic blog-secrets -n blog \
--from-literal=DB_PASSWORD='s3cr3t' \
--from-literal=API_KEY='abc123' \
--dry-run=client -o yaml \
| kubeseal --format yaml > apps/blog/overlays/prod/sealed-secret.yaml
git add apps/blog/overlays/prod/sealed-secret.yaml
git commit -m "blog: add secrets" && git push
By default a sealed secret is bound to its name and namespace: copying it to blog-staging won't decrypt. Seal per environment, which is what you want.
Useful extras:
- Seal offline / in CI:
kubeseal --fetch-cert > pub-cert.pem, commit the cert, thenkubeseal --cert pub-cert.pem ...works without cluster access. - Update one key:
echo -n newvalue | kubeseal --raw --namespace blog --name blog-secrets --from-file=/dev/stdin, then paste intospec.encryptedData. - Existing secrets in the cluster:
kubectl get secret blog-secrets -n blog -o yaml | kubeseal --format yaml > ...(stripresourceVersion/uidfirst or use thekubectl neatplugin). - Argo CD repo credential from Step 3.5 can now be sealed and moved into
argocd/the same way.
Step 6.4: Guard rails against leaks¶
# .gitignore
*.backup.yaml
*.dec.yaml
*-unsealed.yaml
.env
Add a secret scanner such as gitleaks as a pre-commit hook or CI check. If a plain secret is ever pushed, rotate the credential; rewriting history isn't enough.
SOPS path (if you prefer it): generate a key with age-keygen -o age.key, add a .sops.yaml with creation_rules: [{path_regex: .*\.sops\.yaml$, encrypted_regex: ^(data|stringData)$, age: <public key>}], encrypt with sops -e -i secret.sops.yaml, store age.key as a Secret for Flux (decryption.provider: sops) or for the KSOPS sidecar in Argo CD's repo-server.
7. Migrate existing workloads without downtime¶
Argo CD adopts existing objects when the Git version has the same kind, name and namespace; it patches them in place instead of recreating. Migrate one namespace at a time, least critical first, with pruning off until each project's diff is clean.
Step 7.1: Turn pruning off for the migration period¶
In argocd/appsets/apps.yaml, set prune: false (keep selfHeal: true). This way, a missing file in Git can't delete a live object while you're still assembling the folder. Flip it back in Step 7.7.
Step 7.2: Inventory what's running¶
kubectl get ns
helm list -A # anything installed with Helm
kubectl get helmcharts.helm.cattle.io -A # anything installed via k3s HelmChart CRs
ls /var/lib/rancher/k3s/server/manifests/ # on the server node: k3s auto-deploy files
# Per namespace, what you actually need to capture:
kubectl -n blog get deploy,sts,ds,svc,ingress,ingressroute,cm,secret,pvc,cronjob
Note for each project how it was deployed: plain YAML, helm install, or a k3s HelmChart. Each has its own path below.
Step 7.3: Export clean manifests (plain-YAML projects)¶
If you still have the original YAML files, start from those; they're cleaner. Otherwise export and strip the server-generated fields:
kubectl krew install neat # one-time; or strip by hand
mkdir -p apps/blog/base apps/blog/overlays/prod
for kind in deploy svc ingress cm pvc; do
kubectl -n blog get $kind -o yaml | kubectl neat > apps/blog/base/$kind.yaml
done
Then tidy: remove namespace: fields (the overlay sets it), delete the kube-root-ca.crt ConfigMap and any ServiceAccount token secrets, and convert real Secrets to SealedSecrets (Step 6.3).
Things that must not change during adoption
- Deployment
spec.selectoris immutable. Keep the exact labels; that's why Step 5.1 usesincludeSelectors: falseinstead of the oldercommonLabels. - PVC spec (storageClassName, size, accessModes) is immutable. Copy it exactly so the existing volume and data are kept.
- Service
clusterIP: leave it out of Git; Argo CD won't try to change it.
Step 7.4: Protect data before the first sync¶
k3s's local-path provisioner creates volumes with reclaim policy Delete. If a PVC were ever pruned, the data on disk would go with it. Before migrating anything stateful:
# Make existing volumes survive PVC deletion
kubectl get pv -o custom-columns=NAME:.metadata.name,CLAIM:.spec.claimRef.name,POLICY:.spec.persistentVolumeReclaimPolicy
kubectl patch pv <pv-name> -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}'
And tell Argo CD never to prune PVCs, in each base:
# apps/blog/base/pvc.yaml
metadata:
name: blog-data
annotations:
argocd.argoproj.io/sync-options: Prune=false,Delete=false
Also take a normal backup of the volume directory (/var/lib/rancher/k3s/storage/ on the node) for anything you care about.
Step 7.5: Diff, then sync¶
kubectl kustomize apps/blog/overlays/prod | kubectl diff -f - # local preview, should be near-empty
git add apps/blog && git commit -m "blog: adopt into GitOps" && git push
argocd app get blog # appears within ~3 minutes, or: argocd app get apps --refresh
argocd app diff blog # what Argo would change
Iterate until argocd app diff blog shows only harmless changes (the tracking annotation, defaulted fields). Auto-sync then applies it. Watch that pods don't restart unexpectedly:
kubectl -n blog get pods -w
If a field keeps showing as different because a controller sets it (HPA replicas, a mutating webhook), add an ignoreDifferences entry rather than fighting it (see section 10).
Step 7.6: Projects installed with Helm or k3s HelmChart¶
Installed with helm install:
helm -n media get values media > /tmp/media-values.yaml # your custom values only
helm -n media list # note chart name and version
Put those into apps/media/overlays/prod/kustomization.yaml as a helmCharts entry with the same releaseName and chart version, so rendered object names match. After Argo CD shows Synced/Healthy, remove Helm's bookkeeping so nobody runs helm upgrade out of habit:
# Do NOT run helm uninstall: that deletes the workload.
kubectl -n media delete secret -l owner=helm,name=media
Installed via a k3s HelmChart (manifest in /var/lib/rancher/k3s/server/manifests/): deleting a HelmChart object makes k3s's helm-controller uninstall the release. Plan these as a short, controlled redeploy: set PVs to Retain (Step 7.4), back up the manifest file, move it out of the auto-deploy directory, add the equivalent helmCharts entry to Git with the same release name, and let Argo CD bring it back up. To re-bind a retained volume, clear its old claim with kubectl patch pv <pv-name> --type json -p '[{"op":"remove","path":"/spec/claimRef"}]' and set spec.volumeName: <pv-name> on the PVC in Git. How k3s reacts to a removed auto-deploy file has changed between versions, so test this on a non-critical app first.
Step 7.7: Finish the migration¶
Once every namespace you care about shows Synced/Healthy:
- Set
prune: trueagain in the apps ApplicationSet. - Check the orphaned-resources warnings (Step 4.1 enabled them) for leftovers in each namespace and either add them to Git or delete them.
- Revoke or stop using cluster-admin kubeconfigs for day-to-day work. A read-only kubeconfig is enough when Git is the way you change things.
8. Onboard a new project¶
With the ApplicationSet in place, a new project is one folder and one PR. No Argo CD config changes, no kubectl.
Step 8.1: Scaffold the folder¶
Save this as scripts/new-app.sh in the repo:
#!/usr/bin/env bash
set -euo pipefail
APP=${1:?usage: new-app.sh <name> <image:tag> [port]}
IMAGE=${2:?image required, e.g. ghcr.io/me/app:1.0.0}
PORT=${3:-8080}
DOMAIN=${DOMAIN:-home.example.com}
B=apps/$APP/base O=apps/$APP/overlays/prod
mkdir -p "$B" "$O"
cat > "$B/kustomization.yaml" <<EOF
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources: [deployment.yaml, service.yaml, ingress.yaml]
labels:
- pairs: { app.kubernetes.io/name: $APP }
includeSelectors: false
EOF
cat > "$B/deployment.yaml" <<EOF
apiVersion: apps/v1
kind: Deployment
metadata: { name: $APP }
spec:
replicas: 1
selector: { matchLabels: { app: $APP } }
template:
metadata: { labels: { app: $APP } }
spec:
containers:
- name: $APP
image: ${IMAGE%:*}
ports: [{ containerPort: $PORT }]
resources:
requests: { cpu: 25m, memory: 64Mi }
limits: { memory: 256Mi }
EOF
cat > "$B/service.yaml" <<EOF
apiVersion: v1
kind: Service
metadata: { name: $APP }
spec:
selector: { app: $APP }
ports: [{ port: 80, targetPort: $PORT }]
EOF
cat > "$B/ingress.yaml" <<EOF
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata: { name: $APP }
spec:
ingressClassName: traefik
rules:
- host: $APP.$DOMAIN
http:
paths:
- path: /
pathType: Prefix
backend: { service: { name: $APP, port: { number: 80 } } }
EOF
cat > "$O/kustomization.yaml" <<EOF
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: $APP
resources: [../../base]
images:
- name: ${IMAGE%:*}
newTag: "${IMAGE##*:}"
EOF
kubectl kustomize "$O" >/dev/null && echo "apps/$APP scaffolded and renders OK"
chmod +x scripts/new-app.sh
scripts/new-app.sh whoami traefik/whoami:v1.10.1 80
Step 8.2: The onboarding checklist¶
- Run the scaffold, or copy an existing project folder that's similar.
- Adjust resources, probes, env vars, and add a
pvc.yamlwith thePrune=false,Delete=falseannotation if it stores data. - Seal its secrets into
overlays/prod/sealed-secret.yaml(Step 6.3) and add the file to the overlay'sresources. - Render locally:
kubectl kustomize apps/<name>/overlays/prod. - Open a PR. Let CI validate (Step 8.3), review, merge.
- Within about 3 minutes, Argo CD creates the
<name>Application and namespace. Check withargocd app get <name>or in the UI. - If the project needs its own image-update automation or notifications, add those (section 10).
Removing a project is the reverse: delete the folder and merge. Because of preserveResourcesOnDeletion: true, the Application disappears but the workload keeps running until you kubectl delete ns <name>. That's a deliberate safety net; drop that setting if you'd rather have folder deletion clean up fully.
Step 8.3: Validate PRs in CI (recommended)¶
GitLab: use the
.gitlab-ci.ymljob in G3 instead of this workflow.
A cheap GitHub Actions job catches most broken manifests before Argo CD ever sees them. .github/workflows/validate.yaml:
name: validate
on: [pull_request]
jobs:
render:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: azure/setup-helm@v4
- name: Install kubeconform
run: |
curl -sL https://github.com/yannh/kubeconform/releases/latest/download/kubeconform-linux-amd64.tar.gz | tar xz
sudo mv kubeconform /usr/local/bin/
- name: Render and validate every overlay
run: |
set -e
for d in apps/*/overlays/* infrastructure/*; do
[ -f "$d/kustomization.yaml" ] || continue
echo "::group::$d"
kubectl kustomize --enable-helm "$d" | kubeconform -strict -ignore-missing-schemas -summary
echo "::endgroup::"
done
Adapt it if you're on GitLab or Gitea Actions; the loop is the part that matters.
9. k3s specifics¶
k3s ships a few components that it manages itself. The rule: don't put k3s-owned objects in Git unless you first disable them in k3s, otherwise k3s and Argo CD will overwrite each other.
| k3s component | Owned by | GitOps approach |
|---|---|---|
Traefik (HelmChart in kube-system) |
k3s | Customize with a HelmChartConfig in Git, or disable and own it fully |
| CoreDNS, metrics-server, local-path-provisioner | k3s | Leave alone; customize via k3s config files |
| ServiceLB (klipper) | k3s | Works as-is for type: LoadBalancer |
/var/lib/rancher/k3s/server/manifests/ auto-deploy |
k3s | Stop adding files here; move your own ones into Git (Step 7.6) |
HelmChart CRD (helm.cattle.io/v1) |
k3s helm-controller | Usable from Git, but prefer Kustomize helmCharts |
Traefik: option A, keep the bundled one (simplest)¶
Customize it with a HelmChartConfig, which k3s merges into its own Traefik install. Put it in infrastructure/traefik-config/:
# infrastructure/traefik-config/helmchartconfig.yaml
apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
name: traefik
namespace: kube-system # must be kube-system, overrides the app's destination namespace
spec:
valuesContent: |-
logs:
access:
enabled: true
# Add values for the Traefik chart version your k3s ships;
# check it with: kubectl -n kube-system get helmchart traefik -o yaml
# infrastructure/traefik-config/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- helmchartconfig.yaml
- middlewares.yaml # optional: shared Traefik Middlewares (auth, headers, redirects)
Traefik values keys change between major chart versions, so check them against the version k3s bundles after each k3s upgrade.
Traefik: option B, own it in Git (full control)¶
Start k3s with Traefik disabled, then install it like any other chart:
# /etc/rancher/k3s/config.yaml on each server node, then: systemctl restart k3s
disable:
- traefik
Add infrastructure/traefik/kustomization.yaml with a helmCharts entry for https://traefik.github.io/charts. You choose the version and Renovate can bump it, at the cost of one more thing to maintain. Do this during a maintenance window, since ingress is down between disabling and Argo CD reinstalling it.
Ingress vs IngressRoute: standard Ingress objects (as in the scaffold) keep apps portable. Use Traefik's IngressRoute CRD (traefik.io/v1alpha1) only when you need Traefik-specific features; it works fine with Argo CD.
local-path storage¶
- Volumes are directories on one node (
/var/lib/rancher/k3s/storage/). A pod with a local-path PVC is pinned to that node; on a multi-node cluster, expect that. - Default reclaim policy is Delete. Combine
Prune=false,Delete=falseon PVCs (Step 7.4) with a Retain storage class for anything precious:
# infrastructure/storage/local-path-retain.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: local-path-retain
provisioner: rancher.io/local-path
reclaimPolicy: Retain
volumeBindingMode: WaitForFirstConsumer
- Git doesn't back up data. GitOps restores configuration; you still need a volume backup (restic/rsync of the storage dir, Velero, or Longhorn snapshots if you move to Longhorn).
- Size requests on local-path are not enforced, and PVC specs are immutable. Pick a sensible size up front.
HelmChart CRD from Git¶
You can commit helm.cattle.io/v1 HelmChart objects and let Argo CD apply them. It works, but Argo CD then sees only the HelmChart object, not the Deployments it produces, so health, diffs and drift detection go blind. Use it only as a stopgap while migrating, then convert to Kustomize helmCharts.
Small-cluster tuning¶
- Argo CD runs on arm64 (Raspberry Pi) without changes.
- On memory-tight nodes, keep
dexdisabled, don't enable HA Redis, and raisetimeout.reconciliation(for example to300s). Add a GitHub webhook tohttps://argocd.home.example.com/api/webhookif you want instant syncs without fast polling (GitLab: G5). - Your kubeconfig on the server is
/etc/rancher/k3s/k3s.yaml; copy it to your workstation and changeserver:to the node's address.
10. Day-2 operations¶
The daily loop becomes: change Git, merge, watch Argo CD converge. These are the habits and settings that keep that loop safe.
Drift¶
With selfHeal: true, any kubectl edit is reverted within seconds. That's the point: the cluster can't silently diverge from Git.
argocd app list # OutOfSync = drift or an unsynced commit
argocd app diff blog # exactly what differs
argocd app get blog --refresh # force a re-read of Git
Expected differences (fields controllers set) should be ignored, not fought. Per app in the ApplicationSet template, or globally in values.yaml under configs.cm:
# in the ApplicationSet template.spec
ignoreDifferences:
- group: apps
kind: Deployment
jsonPointers: [/spec/replicas] # when an HPA owns replicas
Pausing one app for debugging: the ApplicationSet rewrites Application specs, so first let per-app sync settings stick by adding this to each ApplicationSet's spec:
ignoreApplicationDifferences:
- jsonPointers: [/spec/syncPolicy]
Then argocd app set blog --sync-policy none pauses it, and argocd app set blog --sync-policy automated --self-heal --auto-prune resumes it. Make the real fix in Git, then resume.
Rollbacks¶
Roll back in Git, so Git stays the truth:
git log --oneline -- apps/blog # find the bad commit
git revert <sha> && git push # Argo CD syncs the previous state
argocd app history blog shows what was deployed when. argocd app rollback blog <id> exists but only works with auto-sync off, and Git then disagrees with the cluster; use it only as an emergency brake, then revert in Git and re-enable sync.
What a rollback does not undo: database migrations and data in PVCs. Keep schema changes backward compatible, or back up first.
Image updates¶
GitLab: Renovate has no hosted GitLab app, and CI needs a project access token to push to the gitops repo; see G4.
Pick one approach per project. They can coexist across projects.
| Approach | Best for | How |
|---|---|---|
| Renovate bot | Third-party images and Helm charts (most homelab apps) | Opens PRs bumping newTag, helmCharts.version, chart targetRevision; you merge or auto-merge patches |
| CI write-back | Your own apps with their own build pipeline | App repo's CI builds the image, then commits the new tag to this repo |
| Argo CD Image Updater | Fully automatic tracking of a registry | Add-on that watches registries and writes new tags back to Git; see its docs for the current config format |
Renovate (GitHub App, free). Add renovate.json at the repo root:
{
"$schema": "https://docs.renovatebot.com/renovate-schema.json",
"extends": ["config:recommended"],
"kubernetes": { "managerFilePatterns": ["/(apps|infrastructure)/.+\\.yaml$/"] },
"argocd": { "managerFilePatterns": ["/argocd/.+\\.yaml$/"] },
"packageRules": [
{ "matchUpdateTypes": ["patch", "digest"], "automerge": true },
{ "matchUpdateTypes": ["major"], "dependencyDashboardApproval": true }
]
}
Renovate's kustomize manager picks up images: and helmCharts: in kustomization.yaml automatically. Older Renovate versions call managerFilePatterns fileMatch.
CI write-back from an app repo, after pushing ghcr.io/YOUR_USER/blog:$TAG:
git clone https://x-access-token:${GITOPS_TOKEN}@github.com/YOUR_USER/gitops.git
cd gitops/apps/blog/overlays/prod
kustomize edit set image ghcr.io/YOUR_USER/blog:${TAG}
git commit -am "blog: deploy ${TAG}" && git push # or open a PR for prod
Use a fine-grained token scoped to only the gitops repo's contents.
Notifications¶
Argo CD's notifications controller ships with the Helm chart. Configure it in argocd/install/values.yaml so it's GitOps-managed too. Discord via webhook, as an example (Slack, Telegram, ntfy, email and others are built in):
notifications:
enabled: true
secret:
create: false # supply argocd-notifications-secret as a SealedSecret with key discord-url
notifiers:
service.webhook.discord: |
url: $discord-url
headers:
- name: Content-Type
value: application/json
templates:
template.app-sync-failed: |
webhook:
discord:
method: POST
body: |
{"content": ":x: {{.app.metadata.name}} sync failed: {{.app.status.operationState.message}}"}
template.app-degraded: |
webhook:
discord:
method: POST
body: |
{"content": ":warning: {{.app.metadata.name}} is Degraded"}
triggers:
trigger.on-sync-failed: |
- when: app.status.operationState.phase in ['Error', 'Failed']
send: [app-sync-failed]
trigger.on-health-degraded: |
- when: app.status.health.status == 'Degraded'
send: [app-degraded]
subscriptions:
- recipients: [discord]
triggers: [on-sync-failed, on-health-degraded]
Test it by pushing a deliberately broken manifest to a throwaway app folder.
Upgrades¶
- Argo CD: bump
targetRevisioninargocd/argocd.yamlvia PR. Read the release notes for major versions first. - Charts and images: Renovate PRs.
- k3s itself: outside Argo CD's scope. Use the system-upgrade-controller (its
Planobjects can live ininfrastructure/) or upgrade nodes manually. Recheck the Traefik HelmChartConfig afterwards.
Disaster recovery: rebuilding from Git¶
If a node dies, your recovery is short because everything except data is in Git:
- Install k3s (same config file, same disabled components).
kubectl apply -f sealed-secrets-key.backup.yaml(from your password manager).helm install argocd ...exactly as in Step 3.3, add the repo credential (Step 3.5).kubectl apply -f bootstrap/root.yaml.- Restore PVC data from your volume backups.
Rehearse this once on a spare VM. It's the best test that your repo really is the source of truth.
11. Troubleshooting and next steps¶
| Symptom | Likely cause | Fix |
|---|---|---|
root shows ComparisonError |
A file listed in argocd/kustomization.yaml doesn't exist yet |
Add the file or comment out the line (Step 3.6) |
| Repo shows "authentication required" | Deploy key missing or wrong URL form (ssh vs https) | argocd repo list; URL in apps must match the registered repo exactly |
| New project folder doesn't appear | Path doesn't match apps/*/overlays/prod, or Git not re-polled yet |
Check the path; argocd app get apps --refresh; check argocd-applicationset-controller logs |
| "resource ... is not permitted in project apps" | AppProject blocks it (cluster-scoped kind or denied namespace) | Move it to infrastructure/ or widen the AppProject deliberately |
| Sync fails: "field is immutable" on a Deployment | Labels changed the selector | Restore the original matchLabels; use includeSelectors: false |
helmCharts ignored / "must specify --enable-helm" |
Build option missing | Check kustomize.buildOptions in argocd-cm (Step 3.2) |
| CRD "no matches for kind" on first sync | CRD-providing app not ready yet | Retries handle it; ensure SkipDryRunOnMissingResource=true |
| App flips between Synced and OutOfSync | Something else edits the same field (HPA, webhook, k3s) | ignoreDifferences for that field, or remove the competing manager |
| SealedSecret stuck, no Secret created | Wrong namespace/name when sealing, or a sealing key from another cluster | kubectl -n sealed-secrets logs deploy/sealed-secrets-controller; reseal |
| Argo CD UI via Traefik gives redirect loop | Argo serving TLS behind a TLS-terminating proxy | server.insecure: true (Step 3.2) |
Your first week, in order¶
- [ ] Create the repo and bootstrap Argo CD (section 3)
- [ ] Add AppProjects and ApplicationSets (section 4)
- [ ] Install Sealed Secrets and back up its key (section 6)
- [ ] Set local-path PVs to Retain and add the Prune=false annotations (section 7.4)
- [ ] Migrate the least critical namespace, then the rest one by one (section 7)
- [ ] Re-enable pruning and clean up orphans (Step 7.7)
- [ ] Add CI validation and Renovate (Steps 8.3 and section 10)
- [ ] Turn on notifications (section 10)
- [ ] Rehearse a rebuild on a spare VM (section 10)
Where to go later¶
- SSO: enable Dex with GitHub OAuth (GitLab: G6) so you're not using the admin account; then disable it with
admin.enabled: "false". - Private images on GitLab's registry: see G7.
- External Secrets Operator if you keep secrets in 1Password, Bitwarden or Vault.
- More clusters: register them with
argocd cluster add, and switch the ApplicationSets to a matrix generator of clusters by folders (clusters/<name>/...). - Policy: Kyverno in
infrastructure/to enforce resource limits and pinned image tags. - Progressive syncs (ApplicationSet rolling sync) once you have staging and prod.
Once you share the repo or your current namespace list, I can tailor the layout, ApplicationSets and migration order to what you actually run.
Appendix: GitLab steps¶
GitHub is the default throughout this guide. If your repos are on GitLab (gitlab.com or self-managed), everything else stays the same: Argo CD, Kustomize, Sealed Secrets, the ApplicationSets and the k3s handling. Use the steps below wherever the guide points here.
Placeholders: gitlab.com/YOUR_GROUP/gitops is your GitOps project. For self-managed GitLab, swap in your own host, for example gitlab.home.example.com.
| Guide step | GitHub (default) | GitLab |
|---|---|---|
| 3.1 Create the repo | GitHub UI or gh |
glab repo create (G1) |
| 3.5 Repo access | Deploy key | Deploy key (same idea) or project deploy token (G2) |
| 8.3 CI validation | GitHub Actions | .gitlab-ci.yml (G3) |
| 10 Image updates | Renovate app, CI with a GitHub token | Self-run Renovate, CI with a project access token (G4) |
| 9 Webhooks | Repo webhook | Project webhook with a secret token (G5) |
| SSO (Next steps) | Dex GitHub connector | Dex GitLab connector (G6) |
| Private images | GHCR | GitLab Container Registry pull secret (G7) |
G1. Create the repo¶
GitLab calls repos "projects". Install the glab CLI (brew install glab, or the binary from GitLab's releases), then run glab auth login.
glab repo create YOUR_GROUP/gitops --private
git clone [email protected]:YOUR_GROUP/gitops.git && cd gitops
mkdir -p bootstrap argocd/{install,projects,appsets} infrastructure apps
Everywhere the guide says [email protected]:YOUR_USER/gitops.git, use [email protected]:YOUR_GROUP/gitops.git (SSH, with a deploy key). If you use a deploy token, use https://gitlab.com/YOUR_GROUP/gitops.git instead. Keep the .git suffix.
G2. Give Argo CD read access¶
Replaces Step 3.5. With a single repo, a deploy key works exactly as on GitHub:
ssh-keygen -t ed25519 -N '' -C argocd -f ~/.ssh/argocd_gitops
# GitLab: project Settings > Repository > Deploy keys > Add new key,
# paste ~/.ssh/argocd_gitops.pub, leave "Grant write permissions" unticked
ssh-keyscan gitlab.com | argocd cert add-ssh --batch # only if Argo CD doesn't already know gitlab.com's host key
argocd repo add [email protected]:YOUR_GROUP/gitops.git \
--ssh-private-key-path ~/.ssh/argocd_gitops
argocd repo list # STATUS should be Successful
Alternatively, use a project deploy token over HTTPS: Settings, Repository, Deploy tokens, scope read_repository. Then run argocd repo add https://gitlab.com/YOUR_GROUP/gitops.git --username gitlab+deploy-token-<id> --password <token>, and use HTTPS URLs throughout.
Self-managed GitLab with a private CA: add it with argocd cert add-tls gitlab.home.example.com --from ca.pem. Also add its SSH host key with ssh-keyscan gitlab.home.example.com | argocd cert add-ssh --batch.
G3. Validate merge requests with GitLab CI¶
Replaces Step 8.3. In the gitops project, add .gitlab-ci.yml:
validate:
image:
name: alpine/k8s:1.31.4 # pin a current tag from Docker Hub; ships kubectl, kustomize and helm
entrypoint: [""]
rules:
- if: $CI_PIPELINE_SOURCE == "merge_request_event"
script:
- wget -qO- https://github.com/yannh/kubeconform/releases/latest/download/kubeconform-linux-amd64.tar.gz | tar xz -C /usr/local/bin
- |
set -e
for d in apps/*/overlays/* infrastructure/*; do
[ -f "$d/kustomization.yaml" ] || continue
echo "== $d"
kubectl kustomize --enable-helm "$d" | kubeconform -strict -ignore-missing-schemas -summary
done
Under Settings, Merge requests, tick "Pipelines must succeed" so a failing render blocks the merge.
G4. Image updates: Renovate and CI write-back¶
Renovate. There's no hosted Renovate app for GitLab, so you run it yourself on a schedule:
- Create a project such as
YOUR_GROUP/renovate-runner. - Create a bot token with scopes
api,read_userandwrite_repository: a project or group access token (Premium on gitlab.com, free on self-managed), or a bot user's personal access token. Add it as CI/CD variableRENOVATE_TOKEN(masked). Also add a read-only GitHub token asGITHUB_COM_TOKEN, for changelogs and to avoid rate limits. - On gitlab.com, add this
.gitlab-ci.yml:
include:
- project: renovate-bot/renovate-runner
file: /templates/renovate.gitlab-ci.yml
variables:
RENOVATE_EXTRA_FLAGS: "--autodiscover=true --autodiscover-filter=YOUR_GROUP/gitops"
On self-managed GitLab, mirror that template, and set RENOVATE_ENDPOINT: https://gitlab.home.example.com/api/v4 and RENOVATE_PLATFORM: gitlab.
- Add a pipeline schedule (CI/CD, Pipeline schedules), hourly or nightly.
The renovate.json from section 10 goes in the gitops project unchanged.
CI write-back from an app's project to the gitops project. The app's CI_JOB_TOKEN can't push to another project, so:
- In the gitops project: Settings, Access tokens, add a project access token. Role: Developer (Maintainer if
mainis protected). Scope:write_repository. On gitlab.com Free, where project access tokens need Premium, use a bot user's personal access token. - In the app's project: Settings, CI/CD, Variables, add it as
GITOPS_TOKEN(masked, protected).
# .gitlab-ci.yml in the app's project, after the image is pushed
deploy-tag:
stage: deploy
image: alpine/k8s:1.31.4
rules:
- if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
script:
- git clone "https://ci:${GITOPS_TOKEN}@${CI_SERVER_HOST}/YOUR_GROUP/gitops.git"
- cd gitops/apps/$CI_PROJECT_NAME/overlays/prod
- git config user.name "GitLab CI" && git config user.email "[email protected]"
- kustomize edit set image $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA
- git commit -am "$CI_PROJECT_NAME: deploy $CI_COMMIT_SHORT_SHA"
- git push origin HEAD:main # or push a branch and open an MR for prod
G5. Webhooks for instant syncs¶
Replaces the GitHub webhook mentioned in section 9 (Small-cluster tuning). Set a shared secret in Argo CD first:
# argocd/install/values.yaml
configs:
secret:
gitlabSecret: <random string> # or supply argocd-secret's webhook.gitlab.secret via a SealedSecret
Then, in the gitops project, go to Settings, Webhooks, Add new webhook. Set URL https://argocd.home.example.com/api/webhook, the same secret token, and trigger Push events. GitLab must be able to reach that URL. For self-managed GitLab calling a LAN address, you may need to allow local requests under Admin, Settings, Network, Outbound requests.
G6. SSO with GitLab¶
Dex has a GitLab connector. Set dex.enabled: true and add the connector in values.yaml:
configs:
cm:
url: https://argocd.home.example.com
dex.config: |
connectors:
- type: gitlab
id: gitlab
name: GitLab
config:
baseURL: https://gitlab.com # or your self-managed URL
clientID: $dex.gitlab.clientID
clientSecret: $dex.gitlab.clientSecret
groups: [YOUR_GROUP]
Create the OAuth application under the group's Settings, Applications. Use the redirect URI https://argocd.home.example.com/api/dex/callback and scopes read_user and openid. Grant roles with g, YOUR_GROUP, role:admin in configs.rbac.policy.csv.
G7. Pulling images from the GitLab Container Registry¶
If your images live in registry.gitlab.com/YOUR_GROUP/<project>, create a group deploy token with scope read_registry, then seal a pull secret per namespace:
kubectl create secret docker-registry gitlab-registry -n blog \
--docker-server=registry.gitlab.com \
--docker-username=gitlab+deploy-token-<id> \
--docker-password=<read_registry token> \
--dry-run=client -o yaml \
| kubeseal --format yaml > apps/blog/overlays/prod/gitlab-registry.sealed.yaml
Add it to the overlay's resources, and add imagePullSecrets: [{ name: gitlab-registry }] to the pod spec. Alternatively, configure it once per node in /etc/rancher/k3s/registries.yaml. That's simpler, but it lives outside Git.