Andrew Mercer
on this page

GitOps on k3s: Getting-Started Guide (Single Repo)

Part of the Git Operations (GitOps) section. For the concepts behind this guide, see GitOps: A Comprehensive Guide. The alternative layout, one platform repo plus one repo per project, is in the multi-repo guide, and the single vs multi-repo comparison weighs the two.

Commands default to GitHub. If you use GitLab, every step that differs links to the GitLab appendix at the end.

Overview and assumptions

By the end of this guide, one Git repository will describe everything running on your k3s cluster, and Argo CD will keep the cluster matching it. Adding a new project becomes "create a folder, open a PR, merge". Nobody runs kubectl apply by hand any more.

The core idea: Git is the source of truth, a controller in the cluster pulls from Git and reconciles, and every change is a commit you can review, diff and revert.

What you will build

  1. A gitops repository with a layout that scales from 3 namespaces to 30.
  2. Argo CD installed on k3s, managing itself from that repo.
  3. A single root Application plus ApplicationSets that discover new projects automatically from folders.
  4. Kustomize base/overlay structure for environments, and Helm for third-party charts.
  5. Encrypted secrets committed safely to Git.
  6. Your existing hand-applied workloads adopted without downtime.
  7. Day-2 habits: drift detection, rollbacks, automated image updates and notifications.

Assumptions (adapt where yours differ)

Assumption If yours differs
Single k3s cluster, 1 to 3 nodes, recent k3s (v1.30+) Multi-cluster works too; see the note in Next steps
Bundled Traefik is your ingress, local-path is your storage class Swap the Ingress/IngressRoute and storageClassName values
Git hosted on GitHub (private repo) GitLab: see the GitLab appendix. Gitea or Forgejo: only the repo URL and credentials change
You have kubectl, helm and git locally, with cluster-admin kubeconfig Run commands from a node with k3s kubectl instead
Workloads are currently plain YAML and/or Helm releases applied by hand Section 8 covers both
One environment today, maybe staging later The overlay layout handles both from day one
Placeholder names: repo github.com/YOUR_USER/gitops, domain home.example.com, projects blog, media, monitoring Search and replace with your own

Time budget: about 1 hour to bootstrap, then roughly 15 to 30 minutes per existing project you migrate.

1. Choose a tool: use Argo CD

For a k3s cluster with several projects and more coming, I recommend Argo CD. Flux is an equally solid choice; the deciding factors for you are visibility and multi-project ergonomics.

Argo CD Flux
Web UI Built in: live resource tree, diffs, sync/rollback buttons None built in (third-party UIs like Capacitor or Weave GitOps)
Many projects ApplicationSets generate one app per folder automatically; AppProjects fence projects off Kustomization per folder; tenancy via service-account impersonation
Resource cost Roughly 400 to 700 MiB RAM in a small install Roughly 150 to 300 MiB; lighter
Helm Renders charts then applies (no Helm release secrets) Native helm install/upgrade via helm-controller
Secrets Sealed Secrets is simplest; SOPS needs a plugin (KSOPS) SOPS decryption built in
Image auto-update Argo CD Image Updater (add-on) Built in (image-reflector + image-automation)
Bootstrap helm install, then hand it its own config flux bootstrap CLI does it all in one command

Why Argo CD here: with a growing number of namespaces, the UI pays for itself. You see every project's health on one page, you see drift as a diff, and you can roll back with a click while learning. ApplicationSets make "new folder = new app" a one-time setup.

When to pick Flux instead: the cluster is very memory-constrained (Raspberry Pi with 2 GB), you prefer a CLI-only workflow, or you want SOPS without plugins. The repo layout in this guide transfers almost unchanged; Flux equivalents are noted in each section.

Don't run both. Pick one controller per cluster to avoid two reconcilers fighting over the same objects.

2. Design the repo layout

Use one repo, three top-level areas, and one rule: folder name = Argo CD app name = namespace. That rule is what lets ApplicationSets add new projects with zero extra config.

gitops/
├── bootstrap/
│   └── root.yaml                # the ONLY thing you ever kubectl apply
├── argocd/                      # Argo CD manages its own config from here
│   ├── install/                 # Helm values for Argo CD itself
│   │   └── values.yaml
│   ├── projects/                # one AppProject per tenant/group
│   │   ├── infrastructure.yaml
│   │   └── apps.yaml
│   └── appsets/                 # ApplicationSets that discover folders
│       ├── infrastructure.yaml
│       └── apps.yaml
├── infrastructure/              # cluster-wide services, synced first
│   ├── sealed-secrets/
│   ├── cert-manager/
│   ├── traefik-config/          # HelmChartConfig, middlewares
│   └── monitoring/
└── apps/                        # your projects, one folder each
    ├── blog/
    │   ├── base/
    │   │   ├── kustomization.yaml
    │   │   ├── namespace.yaml
    │   │   ├── deployment.yaml
    │   │   ├── service.yaml
    │   │   └── ingress.yaml
    │   └── overlays/
    │       └── prod/
    │           ├── kustomization.yaml
    │           └── sealed-secret.yaml
    └── media/
        └── overlays/prod/ ...

Why this shape

  • infrastructure/ and apps/ are separate so platform pieces (cert-manager, Sealed Secrets, CRDs) can sync before workloads that depend on them, and so you can give them different permissions.
  • Every project has a base/ (what the app is) and overlays/<env>/ (where and how it runs). Even with one environment, start with overlays/prod/. Adding staging later is then a new folder, not a refactor.
  • Argo CD's own config lives in the repo, so upgrading Argo CD is also a PR.

Decisions to make now

  • Monorepo vs repo-per-app: keep deploy manifests in this one repo even if app source code lives elsewhere. CI in the app repos builds images; this repo only says which image tag runs. Split later only if different people need different write access.
  • Branching: deploy from main only. Use PRs for review. Avoid branch-per-environment; use folders per environment instead, because branches drift and merges get painful.
  • Naming: lowercase, DNS-safe folder names (home-assistant, not HomeAssistant), since they become namespace names.

Flux equivalent: the same tree works; replace argocd/ with clusters/home/ containing flux-system/ plus one Kustomization per area.

3. Bootstrap Argo CD on k3s

You install Argo CD once with Helm, point it at the repo, then apply one root Application. From then on, Argo CD manages itself and everything else from Git.

Step 3.1: Create the repo

GitLab: create the project with glab and use gitlab.com URLs; see G1.

# Create an empty PRIVATE repo named gitops on GitHub first, then:
git clone [email protected]:YOUR_USER/gitops.git && cd gitops
mkdir -p bootstrap argocd/{install,projects,appsets} infrastructure apps

Step 3.2: Write Argo CD's Helm values

argocd/install/values.yaml:

global:
  domain: argocd.home.example.com

configs:
  params:
    server.insecure: true            # Traefik terminates TLS, Argo serves plain HTTP behind it
  cm:
    application.resourceTrackingMethod: annotation   # safer than labels; default in v3, set it explicitly
    kustomize.buildOptions: --enable-helm            # lets Kustomize inflate Helm charts
    timeout.reconciliation: 180s                     # how often Git is polled
    exec.enabled: false

server:
  ingress:
    enabled: true
    ingressClassName: traefik
    tls: true
    # annotations:
    #   cert-manager.io/cluster-issuer: letsencrypt   # once cert-manager is in place

dex:
  enabled: false                     # turn on later if you want GitHub/OIDC SSO

# Small-cluster resource hints; raise if you have many apps
controller:
  resources:
    requests: { cpu: 100m, memory: 256Mi }
repoServer:
  resources:
    requests: { cpu: 50m, memory: 128Mi }

Step 3.3: Install it

helm repo add argo https://argoproj.github.io/argo-helm
helm repo update
helm search repo argo/argo-cd --versions | head -5     # pick the latest and PIN it

export ARGO_CHART_VERSION=<chart version from above>
kubectl create namespace argocd
helm install argocd argo/argo-cd -n argocd \
  --version "$ARGO_CHART_VERSION" \
  -f argocd/install/values.yaml

kubectl -n argocd rollout status deploy/argocd-server
kubectl -n argocd get pods

Step 3.4: Log in and secure the admin account

# Initial password
kubectl -n argocd get secret argocd-initial-admin-secret \
  -o jsonpath='{.data.password}' | base64 -d; echo

# Until DNS/TLS is sorted, use a port-forward
kubectl -n argocd port-forward svc/argocd-server 8080:80

# Install the CLI (macOS: brew install argocd; Linux: download from the GitHub releases page)
argocd login localhost:8080 --plaintext --username admin
argocd account update-password
kubectl -n argocd delete secret argocd-initial-admin-secret

The UI is at http://localhost:8080 (or https://argocd.home.example.com once DNS points at your Traefik).

Step 3.5: Give Argo CD read access to the repo

GitLab: deploy keys work the same way (project Settings, Repository, Deploy keys), or use a project deploy token; see G2.

Use a read-only deploy key, so a compromised cluster can't push to Git.

ssh-keygen -t ed25519 -N '' -C argocd -f ~/.ssh/argocd_gitops
# GitHub: repo Settings > Deploy keys > Add, paste ~/.ssh/argocd_gitops.pub, leave "write" unticked

argocd repo add [email protected]:YOUR_USER/gitops.git \
  --ssh-private-key-path ~/.ssh/argocd_gitops
argocd repo list      # STATUS should be Successful

This creates a Secret in argocd. Don't commit it in plain text; once Sealed Secrets is running (section 6) you can seal it and move it into Git.

Step 3.6: Let Argo CD manage itself

argocd/argocd.yaml installs the same chart version with the values file from Git, so future upgrades are a PR that bumps targetRevision:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: argocd
  namespace: argocd
spec:
  project: default
  sources:
    - repoURL: https://argoproj.github.io/argo-helm
      chart: argo-cd
      targetRevision: <same as ARGO_CHART_VERSION>
      helm:
        releaseName: argocd
        valueFiles:
          - $values/argocd/install/values.yaml
    - repoURL: [email protected]:YOUR_USER/gitops.git
      targetRevision: main
      ref: values
  destination:
    server: https://kubernetes.default.svc
    namespace: argocd
  syncPolicy:
    automated:
      prune: false          # never let Argo prune itself by accident
      selfHeal: true
    syncOptions:
      - ServerSideApply=true   # Argo's CRDs are too large for client-side apply

argocd/kustomization.yaml ties the folder together (the projects and appsets files are written in section 4; write them before you push in step 3.7, or comment those four lines out until then):

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - argocd.yaml
  - projects/infrastructure.yaml
  - projects/apps.yaml
  - appsets/infrastructure.yaml
  - appsets/apps.yaml

Step 3.7: Apply the root Application (the last manual apply)

bootstrap/root.yaml:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: root
  namespace: argocd
  # Deliberately NO resources-finalizer here: deleting root must not cascade-delete the cluster
spec:
  project: default
  source:
    repoURL: [email protected]:YOUR_USER/gitops.git
    targetRevision: main
    path: argocd
  destination:
    server: https://kubernetes.default.svc
    namespace: argocd
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
git add . && git commit -m "Bootstrap Argo CD" && git push
kubectl apply -f bootstrap/root.yaml
argocd app list          # root and argocd should go Synced / Healthy

Checkpoint: in the UI you should see root and argocd. If argocd shows OutOfSync on a few fields right after adoption, sync it once; that is Helm-installed objects being taken over.

4. Scale to many projects: AppProjects and ApplicationSets

The root Application creates two ApplicationSets, and each ApplicationSet creates one Application per folder it finds. A new project is therefore just a new folder in Git; you never hand-write an Application again.

root (bootstrap/root.yaml, applied once)
├── argocd                     Application: Argo CD manages itself
├── infrastructure             ApplicationSet: one app per infrastructure/*
│   ├── sealed-secrets
│   ├── cert-manager
│   └── traefik-config
└── apps                       ApplicationSet: one app per apps/*
    ├── blog
    ├── media
    ├── monitoring
    └── (your next folder)

Root also creates the two AppProjects below, which set the guardrails each generated app runs inside.

Step 4.1: AppProjects as guardrails

AppProjects limit what an app may deploy and where. Infrastructure gets broad rights; your workloads get only namespaced resources, so a typo in apps/blog can never touch kube-system or create a ClusterRole.

argocd/projects/infrastructure.yaml:

apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: infrastructure
  namespace: argocd
spec:
  description: Cluster-wide services under infrastructure/
  sourceRepos:
    - [email protected]:YOUR_USER/gitops.git
    - https://*                # allow upstream Helm repos (tighten to a list later)
  destinations:
    - server: https://kubernetes.default.svc
      namespace: '*'
  clusterResourceWhitelist:
    - group: '*'
      kind: '*'

argocd/projects/apps.yaml:

apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: apps
  namespace: argocd
spec:
  description: Workloads under apps/
  sourceRepos:
    - [email protected]:YOUR_USER/gitops.git
  destinations:
    - server: https://kubernetes.default.svc
      namespace: '*'
    - server: '*'
      namespace: '!kube-system'     # deny rules win over allows
    - server: '*'
      namespace: '!argocd'
  clusterResourceWhitelist:
    - group: ''
      kind: Namespace               # only cluster-scoped thing apps may create
  orphanedResources:
    warn: true                      # flags objects in the namespace that are not in Git

Step 4.2: ApplicationSet for infrastructure

argocd/appsets/infrastructure.yaml:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: infrastructure
  namespace: argocd
spec:
  goTemplate: true
  goTemplateOptions: ["missingkey=error"]
  generators:
    - git:
        repoURL: [email protected]:YOUR_USER/gitops.git
        revision: main
        directories:
          - path: infrastructure/*
  template:
    metadata:
      name: '{{ .path.basename }}'
      labels:
        area: infrastructure
    spec:
      project: infrastructure
      source:
        repoURL: [email protected]:YOUR_USER/gitops.git
        targetRevision: main
        path: '{{ .path.path }}'
      destination:
        server: https://kubernetes.default.svc
        namespace: '{{ .path.basename }}'
      syncPolicy:
        automated:
          prune: true
          selfHeal: true
        syncOptions:
          - CreateNamespace=true
          - ServerSideApply=true
          - SkipDryRunOnMissingResource=true   # CRDs may arrive in the same sync
        retry:
          limit: 5
          backoff: { duration: 15s, factor: 2, maxDuration: 5m }
  syncPolicy:
    preserveResourcesOnDeletion: true   # removing a folder won't nuke running services by surprise

Step 4.3: ApplicationSet for your projects

argocd/appsets/apps.yaml generates one app per apps/<name>/overlays/prod folder. The app name and namespace both come from <name>:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: apps
  namespace: argocd
spec:
  goTemplate: true
  goTemplateOptions: ["missingkey=error"]
  generators:
    - git:
        repoURL: [email protected]:YOUR_USER/gitops.git
        revision: main
        directories:
          - path: apps/*/overlays/prod
  template:
    metadata:
      name: '{{ index .path.segments 1 }}'
      labels:
        area: apps
    spec:
      project: apps
      source:
        repoURL: [email protected]:YOUR_USER/gitops.git
        targetRevision: main
        path: '{{ .path.path }}'
      destination:
        server: https://kubernetes.default.svc
        namespace: '{{ index .path.segments 1 }}'
      syncPolicy:
        automated:
          prune: true
          selfHeal: true
        syncOptions:
          - CreateNamespace=true
          - PruneLast=true
        retry:
          limit: 3
          backoff: { duration: 10s, factor: 2, maxDuration: 2m }
  syncPolicy:
    preserveResourcesOnDeletion: true

Ordering: ApplicationSets don't wait for one app to be healthy before syncing the next. The retry blocks handle the common case (an app needs a CRD from cert-manager that isn't installed yet): it fails, backs off, and succeeds once the CRD appears. Within a single app, use argocd.argoproj.io/sync-wave annotations to order resources (for example, a database StatefulSet in wave -1, the app Deployment in wave 0).

Per-app exceptions: if one project needs different settings (say, manual sync), use templatePatch in the ApplicationSet keyed on a file in the folder, or give that project its own small Application in argocd/ and exclude its folder with exclude: true in the generator.

Flux equivalent: one Kustomization for ./infrastructure and one for ./apps with dependsOn: infrastructure; each subfolder has its own Kustomization pointing at overlays/prod.

5. Environments and overlays: Kustomize for your apps, Helm for third-party charts

Use Kustomize base/ + overlays/<env>/ for apps you write, and let Kustomize inflate Helm charts for software you consume (cert-manager, Grafana, Nextcloud). That gives every folder the same entry point: a kustomization.yaml.

Step 5.1: A base for one project

apps/blog/base/kustomization.yaml:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - namespace.yaml
  - deployment.yaml
  - service.yaml
  - ingress.yaml
labels:
  - pairs:
      app.kubernetes.io/name: blog
      app.kubernetes.io/part-of: blog
    includeSelectors: false

apps/blog/base/deployment.yaml (trimmed):

apiVersion: apps/v1
kind: Deployment
metadata:
  name: blog
spec:
  replicas: 1
  selector:
    matchLabels: { app: blog }
  template:
    metadata:
      labels: { app: blog }
    spec:
      containers:
        - name: blog
          image: ghcr.io/YOUR_USER/blog:1.4.2     # always a pinned tag, never :latest
          ports: [{ containerPort: 8080 }]
          envFrom:
            - secretRef: { name: blog-secrets }
          resources:
            requests: { cpu: 50m, memory: 64Mi }
            limits:   { memory: 256Mi }

Step 5.2: The prod overlay

apps/blog/overlays/prod/kustomization.yaml:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: blog
resources:
  - ../../base
  - sealed-secret.yaml          # section 6
images:
  - name: ghcr.io/YOUR_USER/blog
    newTag: 1.4.2               # the one line that changes on each release
patches:
  - target: { kind: Ingress, name: blog }
    patch: |
      - op: replace
        path: /spec/rules/0/host
        value: blog.home.example.com

Check what Argo CD will render before you push:

kubectl kustomize apps/blog/overlays/prod | less
kubectl kustomize apps/blog/overlays/prod | kubectl diff -f -    # vs the live cluster

Step 5.3: Adding staging later

  1. Create apps/blog/overlays/staging/ with namespace: blog-staging, a different host and maybe replicas: 1.
  2. Change the apps ApplicationSet generator path to apps/*/overlays/* and the names to include the environment:
      name: '{{ index .path.segments 1 }}-{{ .path.basename }}'      # blog-prod, blog-staging
      # destination.namespace:
      namespace: '{{ index .path.segments 1 }}{{ if ne .path.basename "prod" }}-{{ .path.basename }}{{ end }}'

Renaming apps from blog to blog-prod makes Argo CD create new Applications and delete the old ones. Because preserveResourcesOnDeletion: true is set, the running workloads stay put and are adopted by the new app. Still, do it when you can tolerate a blip, or decide on the -prod suffix from day one.

Promotion is then a PR that copies the image tag from overlays/staging to overlays/prod.

Step 5.4: Third-party software via Helm

For upstream charts, keep the chart version and values in Git and let Kustomize render them. infrastructure/cert-manager/kustomization.yaml:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: cert-manager
helmCharts:
  - name: cert-manager
    repo: https://charts.jetstack.io
    version: v1.x.y                 # pin; Renovate can bump it (section 10)
    releaseName: cert-manager
    namespace: cert-manager
    includeCRDs: true
    valuesInline:
      crds:
        enabled: true
resources:
  - cluster-issuer.yaml             # your own manifests alongside the chart

This works because Step 3.2 set kustomize.buildOptions: --enable-helm. Preview locally with kubectl kustomize --enable-helm infrastructure/cert-manager.

Alternative: a multi-source Application with chart: + $values/... (as in Step 3.6). It's cleaner for big charts with long values files, but it can't be auto-discovered by the folder ApplicationSet. Use Kustomize helmCharts by default and switch per app only when it gets awkward.

Rule of thumb

You need Use
Your own app, a few manifests Plain YAML in base/ + Kustomize overlay
An upstream chart with modest values Kustomize helmCharts in its folder
An upstream chart with huge values or OCI-only quirks Multi-source Application with $values
Same app, slightly different per env Kustomize patches in each overlay

Plain Secrets must never land in Git. With Argo CD, Sealed Secrets is the least-friction option: you encrypt locally with a public key, commit the ciphertext, and only the controller in your cluster can decrypt it. No Argo CD plugins are needed.

Option How it works Pick it when
Sealed Secrets Controller holds a private key; kubeseal encrypts with its public cert Default for Argo CD on a single cluster
SOPS + age Files encrypted with an age key; decrypted at render time You use Flux (built in), or want to edit encrypted YAML in place; with Argo CD it needs the KSOPS plugin
External Secrets Operator Pulls from 1Password, Bitwarden, Vault, cloud KMS You already keep secrets in a password manager or vault; good later step

Step 6.1: Install the controller from Git

infrastructure/sealed-secrets/kustomization.yaml:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: sealed-secrets
helmCharts:
  - name: sealed-secrets
    repo: https://bitnami-labs.github.io/sealed-secrets
    version: 2.x.y                  # pin the latest
    releaseName: sealed-secrets
    namespace: sealed-secrets
    includeCRDs: true
    valuesInline:
      fullnameOverride: sealed-secrets-controller

Push, wait for the sealed-secrets app to go Healthy, then install the CLI (brew install kubeseal, or the binary from the GitHub releases page) and save yourself some typing:

alias kubeseal='kubeseal --controller-namespace sealed-secrets --controller-name sealed-secrets-controller'

Step 6.2: Back up the sealing key (do this today)

If you rebuild the cluster without this key, every sealed secret in Git becomes unreadable.

kubectl -n sealed-secrets get secret \
  -l sealedsecrets.bitnami.com/sealed-secrets-key -o yaml > sealed-secrets-key.backup.yaml
# Store it in your password manager or offline. NEVER commit it.
# On a rebuilt cluster: kubectl apply -f sealed-secrets-key.backup.yaml BEFORE the controller starts (or restart it after)

The controller rotates in a new key every 30 days by default and keeps the old ones, so re-run the backup occasionally.

Step 6.3: Seal a secret

kubectl create secret generic blog-secrets -n blog \
  --from-literal=DB_PASSWORD='s3cr3t' \
  --from-literal=API_KEY='abc123' \
  --dry-run=client -o yaml \
| kubeseal --format yaml > apps/blog/overlays/prod/sealed-secret.yaml

git add apps/blog/overlays/prod/sealed-secret.yaml
git commit -m "blog: add secrets" && git push

By default a sealed secret is bound to its name and namespace: copying it to blog-staging won't decrypt. Seal per environment, which is what you want.

Useful extras:

  • Seal offline / in CI: kubeseal --fetch-cert > pub-cert.pem, commit the cert, then kubeseal --cert pub-cert.pem ... works without cluster access.
  • Update one key: echo -n newvalue | kubeseal --raw --namespace blog --name blog-secrets --from-file=/dev/stdin, then paste into spec.encryptedData.
  • Existing secrets in the cluster: kubectl get secret blog-secrets -n blog -o yaml | kubeseal --format yaml > ... (strip resourceVersion/uid first or use the kubectl neat plugin).
  • Argo CD repo credential from Step 3.5 can now be sealed and moved into argocd/ the same way.

Step 6.4: Guard rails against leaks

# .gitignore
*.backup.yaml
*.dec.yaml
*-unsealed.yaml
.env

Add a secret scanner such as gitleaks as a pre-commit hook or CI check. If a plain secret is ever pushed, rotate the credential; rewriting history isn't enough.

SOPS path (if you prefer it): generate a key with age-keygen -o age.key, add a .sops.yaml with creation_rules: [{path_regex: .*\.sops\.yaml$, encrypted_regex: ^(data|stringData)$, age: <public key>}], encrypt with sops -e -i secret.sops.yaml, store age.key as a Secret for Flux (decryption.provider: sops) or for the KSOPS sidecar in Argo CD's repo-server.

7. Migrate existing workloads without downtime

Argo CD adopts existing objects when the Git version has the same kind, name and namespace; it patches them in place instead of recreating. Migrate one namespace at a time, least critical first, with pruning off until each project's diff is clean.

Step 7.1: Turn pruning off for the migration period

In argocd/appsets/apps.yaml, set prune: false (keep selfHeal: true). This way, a missing file in Git can't delete a live object while you're still assembling the folder. Flip it back in Step 7.7.

Step 7.2: Inventory what's running

kubectl get ns
helm list -A                                   # anything installed with Helm
kubectl get helmcharts.helm.cattle.io -A        # anything installed via k3s HelmChart CRs
ls /var/lib/rancher/k3s/server/manifests/      # on the server node: k3s auto-deploy files

# Per namespace, what you actually need to capture:
kubectl -n blog get deploy,sts,ds,svc,ingress,ingressroute,cm,secret,pvc,cronjob

Note for each project how it was deployed: plain YAML, helm install, or a k3s HelmChart. Each has its own path below.

Step 7.3: Export clean manifests (plain-YAML projects)

If you still have the original YAML files, start from those; they're cleaner. Otherwise export and strip the server-generated fields:

kubectl krew install neat                       # one-time; or strip by hand
mkdir -p apps/blog/base apps/blog/overlays/prod
for kind in deploy svc ingress cm pvc; do
  kubectl -n blog get $kind -o yaml | kubectl neat > apps/blog/base/$kind.yaml
done

Then tidy: remove namespace: fields (the overlay sets it), delete the kube-root-ca.crt ConfigMap and any ServiceAccount token secrets, and convert real Secrets to SealedSecrets (Step 6.3).

Things that must not change during adoption

  • Deployment spec.selector is immutable. Keep the exact labels; that's why Step 5.1 uses includeSelectors: false instead of the older commonLabels.
  • PVC spec (storageClassName, size, accessModes) is immutable. Copy it exactly so the existing volume and data are kept.
  • Service clusterIP: leave it out of Git; Argo CD won't try to change it.

Step 7.4: Protect data before the first sync

k3s's local-path provisioner creates volumes with reclaim policy Delete. If a PVC were ever pruned, the data on disk would go with it. Before migrating anything stateful:

# Make existing volumes survive PVC deletion
kubectl get pv -o custom-columns=NAME:.metadata.name,CLAIM:.spec.claimRef.name,POLICY:.spec.persistentVolumeReclaimPolicy
kubectl patch pv <pv-name> -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}'

And tell Argo CD never to prune PVCs, in each base:

# apps/blog/base/pvc.yaml
metadata:
  name: blog-data
  annotations:
    argocd.argoproj.io/sync-options: Prune=false,Delete=false

Also take a normal backup of the volume directory (/var/lib/rancher/k3s/storage/ on the node) for anything you care about.

Step 7.5: Diff, then sync

kubectl kustomize apps/blog/overlays/prod | kubectl diff -f -    # local preview, should be near-empty
git add apps/blog && git commit -m "blog: adopt into GitOps" && git push

argocd app get blog                # appears within ~3 minutes, or: argocd app get apps --refresh
argocd app diff blog               # what Argo would change

Iterate until argocd app diff blog shows only harmless changes (the tracking annotation, defaulted fields). Auto-sync then applies it. Watch that pods don't restart unexpectedly:

kubectl -n blog get pods -w

If a field keeps showing as different because a controller sets it (HPA replicas, a mutating webhook), add an ignoreDifferences entry rather than fighting it (see section 10).

Step 7.6: Projects installed with Helm or k3s HelmChart

Installed with helm install:

helm -n media get values media > /tmp/media-values.yaml     # your custom values only
helm -n media list                                         # note chart name and version

Put those into apps/media/overlays/prod/kustomization.yaml as a helmCharts entry with the same releaseName and chart version, so rendered object names match. After Argo CD shows Synced/Healthy, remove Helm's bookkeeping so nobody runs helm upgrade out of habit:

# Do NOT run helm uninstall: that deletes the workload.
kubectl -n media delete secret -l owner=helm,name=media

Installed via a k3s HelmChart (manifest in /var/lib/rancher/k3s/server/manifests/): deleting a HelmChart object makes k3s's helm-controller uninstall the release. Plan these as a short, controlled redeploy: set PVs to Retain (Step 7.4), back up the manifest file, move it out of the auto-deploy directory, add the equivalent helmCharts entry to Git with the same release name, and let Argo CD bring it back up. To re-bind a retained volume, clear its old claim with kubectl patch pv <pv-name> --type json -p '[{"op":"remove","path":"/spec/claimRef"}]' and set spec.volumeName: <pv-name> on the PVC in Git. How k3s reacts to a removed auto-deploy file has changed between versions, so test this on a non-critical app first.

Step 7.7: Finish the migration

Once every namespace you care about shows Synced/Healthy:

  1. Set prune: true again in the apps ApplicationSet.
  2. Check the orphaned-resources warnings (Step 4.1 enabled them) for leftovers in each namespace and either add them to Git or delete them.
  3. Revoke or stop using cluster-admin kubeconfigs for day-to-day work. A read-only kubeconfig is enough when Git is the way you change things.

8. Onboard a new project

With the ApplicationSet in place, a new project is one folder and one PR. No Argo CD config changes, no kubectl.

Step 8.1: Scaffold the folder

Save this as scripts/new-app.sh in the repo:

#!/usr/bin/env bash
set -euo pipefail
APP=${1:?usage: new-app.sh <name> <image:tag> [port]}
IMAGE=${2:?image required, e.g. ghcr.io/me/app:1.0.0}
PORT=${3:-8080}
DOMAIN=${DOMAIN:-home.example.com}
B=apps/$APP/base O=apps/$APP/overlays/prod
mkdir -p "$B" "$O"

cat > "$B/kustomization.yaml" <<EOF
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources: [deployment.yaml, service.yaml, ingress.yaml]
labels:
  - pairs: { app.kubernetes.io/name: $APP }
    includeSelectors: false
EOF

cat > "$B/deployment.yaml" <<EOF
apiVersion: apps/v1
kind: Deployment
metadata: { name: $APP }
spec:
  replicas: 1
  selector: { matchLabels: { app: $APP } }
  template:
    metadata: { labels: { app: $APP } }
    spec:
      containers:
        - name: $APP
          image: ${IMAGE%:*}
          ports: [{ containerPort: $PORT }]
          resources:
            requests: { cpu: 25m, memory: 64Mi }
            limits: { memory: 256Mi }
EOF

cat > "$B/service.yaml" <<EOF
apiVersion: v1
kind: Service
metadata: { name: $APP }
spec:
  selector: { app: $APP }
  ports: [{ port: 80, targetPort: $PORT }]
EOF

cat > "$B/ingress.yaml" <<EOF
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata: { name: $APP }
spec:
  ingressClassName: traefik
  rules:
    - host: $APP.$DOMAIN
      http:
        paths:
          - path: /
            pathType: Prefix
            backend: { service: { name: $APP, port: { number: 80 } } }
EOF

cat > "$O/kustomization.yaml" <<EOF
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: $APP
resources: [../../base]
images:
  - name: ${IMAGE%:*}
    newTag: "${IMAGE##*:}"
EOF

kubectl kustomize "$O" >/dev/null && echo "apps/$APP scaffolded and renders OK"
chmod +x scripts/new-app.sh
scripts/new-app.sh whoami traefik/whoami:v1.10.1 80

Step 8.2: The onboarding checklist

  1. Run the scaffold, or copy an existing project folder that's similar.
  2. Adjust resources, probes, env vars, and add a pvc.yaml with the Prune=false,Delete=false annotation if it stores data.
  3. Seal its secrets into overlays/prod/sealed-secret.yaml (Step 6.3) and add the file to the overlay's resources.
  4. Render locally: kubectl kustomize apps/<name>/overlays/prod.
  5. Open a PR. Let CI validate (Step 8.3), review, merge.
  6. Within about 3 minutes, Argo CD creates the <name> Application and namespace. Check with argocd app get <name> or in the UI.
  7. If the project needs its own image-update automation or notifications, add those (section 10).

Removing a project is the reverse: delete the folder and merge. Because of preserveResourcesOnDeletion: true, the Application disappears but the workload keeps running until you kubectl delete ns <name>. That's a deliberate safety net; drop that setting if you'd rather have folder deletion clean up fully.

GitLab: use the .gitlab-ci.yml job in G3 instead of this workflow.

A cheap GitHub Actions job catches most broken manifests before Argo CD ever sees them. .github/workflows/validate.yaml:

name: validate
on: [pull_request]
jobs:
  render:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: azure/setup-helm@v4
      - name: Install kubeconform
        run: |
          curl -sL https://github.com/yannh/kubeconform/releases/latest/download/kubeconform-linux-amd64.tar.gz | tar xz
          sudo mv kubeconform /usr/local/bin/
      - name: Render and validate every overlay
        run: |
          set -e
          for d in apps/*/overlays/* infrastructure/*; do
            [ -f "$d/kustomization.yaml" ] || continue
            echo "::group::$d"
            kubectl kustomize --enable-helm "$d" | kubeconform -strict -ignore-missing-schemas -summary
            echo "::endgroup::"
          done

Adapt it if you're on GitLab or Gitea Actions; the loop is the part that matters.

9. k3s specifics

k3s ships a few components that it manages itself. The rule: don't put k3s-owned objects in Git unless you first disable them in k3s, otherwise k3s and Argo CD will overwrite each other.

k3s component Owned by GitOps approach
Traefik (HelmChart in kube-system) k3s Customize with a HelmChartConfig in Git, or disable and own it fully
CoreDNS, metrics-server, local-path-provisioner k3s Leave alone; customize via k3s config files
ServiceLB (klipper) k3s Works as-is for type: LoadBalancer
/var/lib/rancher/k3s/server/manifests/ auto-deploy k3s Stop adding files here; move your own ones into Git (Step 7.6)
HelmChart CRD (helm.cattle.io/v1) k3s helm-controller Usable from Git, but prefer Kustomize helmCharts

Traefik: option A, keep the bundled one (simplest)

Customize it with a HelmChartConfig, which k3s merges into its own Traefik install. Put it in infrastructure/traefik-config/:

# infrastructure/traefik-config/helmchartconfig.yaml
apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: traefik
  namespace: kube-system        # must be kube-system, overrides the app's destination namespace
spec:
  valuesContent: |-
    logs:
      access:
        enabled: true
    # Add values for the Traefik chart version your k3s ships;
    # check it with: kubectl -n kube-system get helmchart traefik -o yaml
# infrastructure/traefik-config/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - helmchartconfig.yaml
  - middlewares.yaml            # optional: shared Traefik Middlewares (auth, headers, redirects)

Traefik values keys change between major chart versions, so check them against the version k3s bundles after each k3s upgrade.

Traefik: option B, own it in Git (full control)

Start k3s with Traefik disabled, then install it like any other chart:

# /etc/rancher/k3s/config.yaml on each server node, then: systemctl restart k3s
disable:
  - traefik

Add infrastructure/traefik/kustomization.yaml with a helmCharts entry for https://traefik.github.io/charts. You choose the version and Renovate can bump it, at the cost of one more thing to maintain. Do this during a maintenance window, since ingress is down between disabling and Argo CD reinstalling it.

Ingress vs IngressRoute: standard Ingress objects (as in the scaffold) keep apps portable. Use Traefik's IngressRoute CRD (traefik.io/v1alpha1) only when you need Traefik-specific features; it works fine with Argo CD.

local-path storage

  • Volumes are directories on one node (/var/lib/rancher/k3s/storage/). A pod with a local-path PVC is pinned to that node; on a multi-node cluster, expect that.
  • Default reclaim policy is Delete. Combine Prune=false,Delete=false on PVCs (Step 7.4) with a Retain storage class for anything precious:
# infrastructure/storage/local-path-retain.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: local-path-retain
provisioner: rancher.io/local-path
reclaimPolicy: Retain
volumeBindingMode: WaitForFirstConsumer
  • Git doesn't back up data. GitOps restores configuration; you still need a volume backup (restic/rsync of the storage dir, Velero, or Longhorn snapshots if you move to Longhorn).
  • Size requests on local-path are not enforced, and PVC specs are immutable. Pick a sensible size up front.

HelmChart CRD from Git

You can commit helm.cattle.io/v1 HelmChart objects and let Argo CD apply them. It works, but Argo CD then sees only the HelmChart object, not the Deployments it produces, so health, diffs and drift detection go blind. Use it only as a stopgap while migrating, then convert to Kustomize helmCharts.

Small-cluster tuning

  • Argo CD runs on arm64 (Raspberry Pi) without changes.
  • On memory-tight nodes, keep dex disabled, don't enable HA Redis, and raise timeout.reconciliation (for example to 300s). Add a GitHub webhook to https://argocd.home.example.com/api/webhook if you want instant syncs without fast polling (GitLab: G5).
  • Your kubeconfig on the server is /etc/rancher/k3s/k3s.yaml; copy it to your workstation and change server: to the node's address.

10. Day-2 operations

The daily loop becomes: change Git, merge, watch Argo CD converge. These are the habits and settings that keep that loop safe.

Drift

With selfHeal: true, any kubectl edit is reverted within seconds. That's the point: the cluster can't silently diverge from Git.

argocd app list                          # OutOfSync = drift or an unsynced commit
argocd app diff blog                     # exactly what differs
argocd app get blog --refresh            # force a re-read of Git

Expected differences (fields controllers set) should be ignored, not fought. Per app in the ApplicationSet template, or globally in values.yaml under configs.cm:

# in the ApplicationSet template.spec
ignoreDifferences:
  - group: apps
    kind: Deployment
    jsonPointers: [/spec/replicas]       # when an HPA owns replicas

Pausing one app for debugging: the ApplicationSet rewrites Application specs, so first let per-app sync settings stick by adding this to each ApplicationSet's spec:

ignoreApplicationDifferences:
  - jsonPointers: [/spec/syncPolicy]

Then argocd app set blog --sync-policy none pauses it, and argocd app set blog --sync-policy automated --self-heal --auto-prune resumes it. Make the real fix in Git, then resume.

Rollbacks

Roll back in Git, so Git stays the truth:

git log --oneline -- apps/blog           # find the bad commit
git revert <sha> && git push             # Argo CD syncs the previous state

argocd app history blog shows what was deployed when. argocd app rollback blog <id> exists but only works with auto-sync off, and Git then disagrees with the cluster; use it only as an emergency brake, then revert in Git and re-enable sync.

What a rollback does not undo: database migrations and data in PVCs. Keep schema changes backward compatible, or back up first.

Image updates

GitLab: Renovate has no hosted GitLab app, and CI needs a project access token to push to the gitops repo; see G4.

Pick one approach per project. They can coexist across projects.

Approach Best for How
Renovate bot Third-party images and Helm charts (most homelab apps) Opens PRs bumping newTag, helmCharts.version, chart targetRevision; you merge or auto-merge patches
CI write-back Your own apps with their own build pipeline App repo's CI builds the image, then commits the new tag to this repo
Argo CD Image Updater Fully automatic tracking of a registry Add-on that watches registries and writes new tags back to Git; see its docs for the current config format

Renovate (GitHub App, free). Add renovate.json at the repo root:

{
  "$schema": "https://docs.renovatebot.com/renovate-schema.json",
  "extends": ["config:recommended"],
  "kubernetes": { "managerFilePatterns": ["/(apps|infrastructure)/.+\\.yaml$/"] },
  "argocd":     { "managerFilePatterns": ["/argocd/.+\\.yaml$/"] },
  "packageRules": [
    { "matchUpdateTypes": ["patch", "digest"], "automerge": true },
    { "matchUpdateTypes": ["major"], "dependencyDashboardApproval": true }
  ]
}

Renovate's kustomize manager picks up images: and helmCharts: in kustomization.yaml automatically. Older Renovate versions call managerFilePatterns fileMatch.

CI write-back from an app repo, after pushing ghcr.io/YOUR_USER/blog:$TAG:

git clone https://x-access-token:${GITOPS_TOKEN}@github.com/YOUR_USER/gitops.git
cd gitops/apps/blog/overlays/prod
kustomize edit set image ghcr.io/YOUR_USER/blog:${TAG}
git commit -am "blog: deploy ${TAG}" && git push     # or open a PR for prod

Use a fine-grained token scoped to only the gitops repo's contents.

Notifications

Argo CD's notifications controller ships with the Helm chart. Configure it in argocd/install/values.yaml so it's GitOps-managed too. Discord via webhook, as an example (Slack, Telegram, ntfy, email and others are built in):

notifications:
  enabled: true
  secret:
    create: false        # supply argocd-notifications-secret as a SealedSecret with key discord-url
  notifiers:
    service.webhook.discord: |
      url: $discord-url
      headers:
        - name: Content-Type
          value: application/json
  templates:
    template.app-sync-failed: |
      webhook:
        discord:
          method: POST
          body: |
            {"content": ":x: {{.app.metadata.name}} sync failed: {{.app.status.operationState.message}}"}
    template.app-degraded: |
      webhook:
        discord:
          method: POST
          body: |
            {"content": ":warning: {{.app.metadata.name}} is Degraded"}
  triggers:
    trigger.on-sync-failed: |
      - when: app.status.operationState.phase in ['Error', 'Failed']
        send: [app-sync-failed]
    trigger.on-health-degraded: |
      - when: app.status.health.status == 'Degraded'
        send: [app-degraded]
  subscriptions:
    - recipients: [discord]
      triggers: [on-sync-failed, on-health-degraded]

Test it by pushing a deliberately broken manifest to a throwaway app folder.

Upgrades

  • Argo CD: bump targetRevision in argocd/argocd.yaml via PR. Read the release notes for major versions first.
  • Charts and images: Renovate PRs.
  • k3s itself: outside Argo CD's scope. Use the system-upgrade-controller (its Plan objects can live in infrastructure/) or upgrade nodes manually. Recheck the Traefik HelmChartConfig afterwards.

Disaster recovery: rebuilding from Git

If a node dies, your recovery is short because everything except data is in Git:

  1. Install k3s (same config file, same disabled components).
  2. kubectl apply -f sealed-secrets-key.backup.yaml (from your password manager).
  3. helm install argocd ... exactly as in Step 3.3, add the repo credential (Step 3.5).
  4. kubectl apply -f bootstrap/root.yaml.
  5. Restore PVC data from your volume backups.

Rehearse this once on a spare VM. It's the best test that your repo really is the source of truth.

11. Troubleshooting and next steps

Symptom Likely cause Fix
root shows ComparisonError A file listed in argocd/kustomization.yaml doesn't exist yet Add the file or comment out the line (Step 3.6)
Repo shows "authentication required" Deploy key missing or wrong URL form (ssh vs https) argocd repo list; URL in apps must match the registered repo exactly
New project folder doesn't appear Path doesn't match apps/*/overlays/prod, or Git not re-polled yet Check the path; argocd app get apps --refresh; check argocd-applicationset-controller logs
"resource ... is not permitted in project apps" AppProject blocks it (cluster-scoped kind or denied namespace) Move it to infrastructure/ or widen the AppProject deliberately
Sync fails: "field is immutable" on a Deployment Labels changed the selector Restore the original matchLabels; use includeSelectors: false
helmCharts ignored / "must specify --enable-helm" Build option missing Check kustomize.buildOptions in argocd-cm (Step 3.2)
CRD "no matches for kind" on first sync CRD-providing app not ready yet Retries handle it; ensure SkipDryRunOnMissingResource=true
App flips between Synced and OutOfSync Something else edits the same field (HPA, webhook, k3s) ignoreDifferences for that field, or remove the competing manager
SealedSecret stuck, no Secret created Wrong namespace/name when sealing, or a sealing key from another cluster kubectl -n sealed-secrets logs deploy/sealed-secrets-controller; reseal
Argo CD UI via Traefik gives redirect loop Argo serving TLS behind a TLS-terminating proxy server.insecure: true (Step 3.2)

Your first week, in order

  • [ ] Create the repo and bootstrap Argo CD (section 3)
  • [ ] Add AppProjects and ApplicationSets (section 4)
  • [ ] Install Sealed Secrets and back up its key (section 6)
  • [ ] Set local-path PVs to Retain and add the Prune=false annotations (section 7.4)
  • [ ] Migrate the least critical namespace, then the rest one by one (section 7)
  • [ ] Re-enable pruning and clean up orphans (Step 7.7)
  • [ ] Add CI validation and Renovate (Steps 8.3 and section 10)
  • [ ] Turn on notifications (section 10)
  • [ ] Rehearse a rebuild on a spare VM (section 10)

Where to go later

  • SSO: enable Dex with GitHub OAuth (GitLab: G6) so you're not using the admin account; then disable it with admin.enabled: "false".
  • Private images on GitLab's registry: see G7.
  • External Secrets Operator if you keep secrets in 1Password, Bitwarden or Vault.
  • More clusters: register them with argocd cluster add, and switch the ApplicationSets to a matrix generator of clusters by folders (clusters/<name>/...).
  • Policy: Kyverno in infrastructure/ to enforce resource limits and pinned image tags.
  • Progressive syncs (ApplicationSet rolling sync) once you have staging and prod.

Once you share the repo or your current namespace list, I can tailor the layout, ApplicationSets and migration order to what you actually run.

Appendix: GitLab steps

GitHub is the default throughout this guide. If your repos are on GitLab (gitlab.com or self-managed), everything else stays the same: Argo CD, Kustomize, Sealed Secrets, the ApplicationSets and the k3s handling. Use the steps below wherever the guide points here.

Placeholders: gitlab.com/YOUR_GROUP/gitops is your GitOps project. For self-managed GitLab, swap in your own host, for example gitlab.home.example.com.

Guide step GitHub (default) GitLab
3.1 Create the repo GitHub UI or gh glab repo create (G1)
3.5 Repo access Deploy key Deploy key (same idea) or project deploy token (G2)
8.3 CI validation GitHub Actions .gitlab-ci.yml (G3)
10 Image updates Renovate app, CI with a GitHub token Self-run Renovate, CI with a project access token (G4)
9 Webhooks Repo webhook Project webhook with a secret token (G5)
SSO (Next steps) Dex GitHub connector Dex GitLab connector (G6)
Private images GHCR GitLab Container Registry pull secret (G7)

G1. Create the repo

GitLab calls repos "projects". Install the glab CLI (brew install glab, or the binary from GitLab's releases), then run glab auth login.

glab repo create YOUR_GROUP/gitops --private
git clone [email protected]:YOUR_GROUP/gitops.git && cd gitops
mkdir -p bootstrap argocd/{install,projects,appsets} infrastructure apps

Everywhere the guide says [email protected]:YOUR_USER/gitops.git, use [email protected]:YOUR_GROUP/gitops.git (SSH, with a deploy key). If you use a deploy token, use https://gitlab.com/YOUR_GROUP/gitops.git instead. Keep the .git suffix.

G2. Give Argo CD read access

Replaces Step 3.5. With a single repo, a deploy key works exactly as on GitHub:

ssh-keygen -t ed25519 -N '' -C argocd -f ~/.ssh/argocd_gitops
# GitLab: project Settings > Repository > Deploy keys > Add new key,
#         paste ~/.ssh/argocd_gitops.pub, leave "Grant write permissions" unticked
ssh-keyscan gitlab.com | argocd cert add-ssh --batch    # only if Argo CD doesn't already know gitlab.com's host key

argocd repo add [email protected]:YOUR_GROUP/gitops.git \
  --ssh-private-key-path ~/.ssh/argocd_gitops
argocd repo list      # STATUS should be Successful

Alternatively, use a project deploy token over HTTPS: Settings, Repository, Deploy tokens, scope read_repository. Then run argocd repo add https://gitlab.com/YOUR_GROUP/gitops.git --username gitlab+deploy-token-<id> --password <token>, and use HTTPS URLs throughout.

Self-managed GitLab with a private CA: add it with argocd cert add-tls gitlab.home.example.com --from ca.pem. Also add its SSH host key with ssh-keyscan gitlab.home.example.com | argocd cert add-ssh --batch.

G3. Validate merge requests with GitLab CI

Replaces Step 8.3. In the gitops project, add .gitlab-ci.yml:

validate:
  image:
    name: alpine/k8s:1.31.4          # pin a current tag from Docker Hub; ships kubectl, kustomize and helm
    entrypoint: [""]
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
  script:
    - wget -qO- https://github.com/yannh/kubeconform/releases/latest/download/kubeconform-linux-amd64.tar.gz | tar xz -C /usr/local/bin
    - |
      set -e
      for d in apps/*/overlays/* infrastructure/*; do
        [ -f "$d/kustomization.yaml" ] || continue
        echo "== $d"
        kubectl kustomize --enable-helm "$d" | kubeconform -strict -ignore-missing-schemas -summary
      done

Under Settings, Merge requests, tick "Pipelines must succeed" so a failing render blocks the merge.

G4. Image updates: Renovate and CI write-back

Renovate. There's no hosted Renovate app for GitLab, so you run it yourself on a schedule:

  1. Create a project such as YOUR_GROUP/renovate-runner.
  2. Create a bot token with scopes api, read_user and write_repository: a project or group access token (Premium on gitlab.com, free on self-managed), or a bot user's personal access token. Add it as CI/CD variable RENOVATE_TOKEN (masked). Also add a read-only GitHub token as GITHUB_COM_TOKEN, for changelogs and to avoid rate limits.
  3. On gitlab.com, add this .gitlab-ci.yml:
include:
  - project: renovate-bot/renovate-runner
    file: /templates/renovate.gitlab-ci.yml

variables:
  RENOVATE_EXTRA_FLAGS: "--autodiscover=true --autodiscover-filter=YOUR_GROUP/gitops"

On self-managed GitLab, mirror that template, and set RENOVATE_ENDPOINT: https://gitlab.home.example.com/api/v4 and RENOVATE_PLATFORM: gitlab.

  1. Add a pipeline schedule (CI/CD, Pipeline schedules), hourly or nightly.

The renovate.json from section 10 goes in the gitops project unchanged.

CI write-back from an app's project to the gitops project. The app's CI_JOB_TOKEN can't push to another project, so:

  1. In the gitops project: Settings, Access tokens, add a project access token. Role: Developer (Maintainer if main is protected). Scope: write_repository. On gitlab.com Free, where project access tokens need Premium, use a bot user's personal access token.
  2. In the app's project: Settings, CI/CD, Variables, add it as GITOPS_TOKEN (masked, protected).
# .gitlab-ci.yml in the app's project, after the image is pushed
deploy-tag:
  stage: deploy
  image: alpine/k8s:1.31.4
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
  script:
    - git clone "https://ci:${GITOPS_TOKEN}@${CI_SERVER_HOST}/YOUR_GROUP/gitops.git"
    - cd gitops/apps/$CI_PROJECT_NAME/overlays/prod
    - git config user.name "GitLab CI" && git config user.email "[email protected]"
    - kustomize edit set image $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA
    - git commit -am "$CI_PROJECT_NAME: deploy $CI_COMMIT_SHORT_SHA"
    - git push origin HEAD:main            # or push a branch and open an MR for prod

G5. Webhooks for instant syncs

Replaces the GitHub webhook mentioned in section 9 (Small-cluster tuning). Set a shared secret in Argo CD first:

# argocd/install/values.yaml
configs:
  secret:
    gitlabSecret: <random string>      # or supply argocd-secret's webhook.gitlab.secret via a SealedSecret

Then, in the gitops project, go to Settings, Webhooks, Add new webhook. Set URL https://argocd.home.example.com/api/webhook, the same secret token, and trigger Push events. GitLab must be able to reach that URL. For self-managed GitLab calling a LAN address, you may need to allow local requests under Admin, Settings, Network, Outbound requests.

G6. SSO with GitLab

Dex has a GitLab connector. Set dex.enabled: true and add the connector in values.yaml:

configs:
  cm:
    url: https://argocd.home.example.com
    dex.config: |
      connectors:
        - type: gitlab
          id: gitlab
          name: GitLab
          config:
            baseURL: https://gitlab.com            # or your self-managed URL
            clientID: $dex.gitlab.clientID
            clientSecret: $dex.gitlab.clientSecret
            groups: [YOUR_GROUP]

Create the OAuth application under the group's Settings, Applications. Use the redirect URI https://argocd.home.example.com/api/dex/callback and scopes read_user and openid. Grant roles with g, YOUR_GROUP, role:admin in configs.rbac.policy.csv.

G7. Pulling images from the GitLab Container Registry

If your images live in registry.gitlab.com/YOUR_GROUP/<project>, create a group deploy token with scope read_registry, then seal a pull secret per namespace:

kubectl create secret docker-registry gitlab-registry -n blog \
  --docker-server=registry.gitlab.com \
  --docker-username=gitlab+deploy-token-<id> \
  --docker-password=<read_registry token> \
  --dry-run=client -o yaml \
| kubeseal --format yaml > apps/blog/overlays/prod/gitlab-registry.sealed.yaml

Add it to the overlay's resources, and add imagePullSecrets: [{ name: gitlab-registry }] to the pod spec. Alternatively, configure it once per node in /etc/rancher/k3s/registries.yaml. That's simpler, but it lives outside Git.