Andrew Mercer
on this page

GitOps on k3s: Getting-Started Guide (Multi-Repo)

Part of the Git Operations (GitOps) section. For the concepts behind this guide, see GitOps: A Comprehensive Guide. The single-repo version is the single-repo guide, and the single vs multi-repo comparison weighs the two.

Commands default to GitHub. If you use GitLab, every step that differs links to the GitLab appendix at the end.

Overview and assumptions

By the end of this guide, a platform repo will describe your cluster: Argo CD itself, cluster-wide services, and a short registration for each project. Each project has its own repo holding its Kubernetes manifests. Argo CD watches all of them and keeps the cluster matching Git. Nobody runs kubectl apply by hand any more.

Onboarding a project becomes: create its repo with a deploy/ folder, add a small registration folder to the platform repo, merge. The platform repo also decides what each project is allowed to do, so a project repo can never deploy outside its own namespace.

What you will build

  1. A platform repo, k3s-platform, holding Argo CD's config, cluster services, and one registration folder per project.
  2. One repo per project (for example blog, media-deploy), each with deploy/base and deploy/overlays/<env>.
  3. Argo CD installed on k3s, managing itself from the platform repo.
  4. One credential template, so Argo CD can read every repo you own without per-repo setup.
  5. A per-project AppProject, namespace and quota, plus Argo CD RBAC, owned by the platform repo.
  6. ApplicationSets that turn each registration into an Argo CD Application pointing at that project's repo.
  7. Encrypted secrets, a no-downtime migration of your current workloads, and day-2 habits (drift, rollbacks, image updates, notifications).

Assumptions (adapt where yours differ)

Assumption If yours differs
Single k3s cluster, 1 to 3 nodes, recent k3s (v1.30+) Multi-cluster works too; see Next steps
Bundled Traefik is your ingress, local-path is your storage class Swap the Ingress and storageClassName values
Repos on GitHub under one account or org, all private GitLab: one group holds all repos; see the GitLab appendix. Gitea or Forgejo: use a token credential template
Repo URLs use HTTPS (https://github.com/YOUR_USER/<repo>.git) Required for the GitHub App credential in Step 3.5; SSH works only with a machine-user key
You have kubectl, helm, git and ideally gh locally, with cluster-admin kubeconfig Run commands from a node with k3s kubectl instead
Workloads are currently plain YAML and/or Helm releases applied by hand Section 7 covers both
Placeholder names: platform repo k3s-platform, domain home.example.com, projects blog and media Search and replace with your own

Time budget: about 1.5 hours to bootstrap, then roughly 20 to 30 minutes per existing project you migrate.

1. Choose a tool: use Argo CD

For a k3s cluster with several projects and more coming, I recommend Argo CD. Flux is an equally solid choice; the deciding factors for you are visibility and multi-project ergonomics.

Argo CD Flux
Web UI Built in: live resource tree, diffs, sync/rollback buttons None built in (third-party UIs like Capacitor or Weave GitOps)
Many projects ApplicationSets generate one app per registered repo; AppProjects fence each project to its repo and namespace Kustomization per folder; tenancy via service-account impersonation
Resource cost Roughly 400 to 700 MiB RAM in a small install Roughly 150 to 300 MiB; lighter
Helm Renders charts then applies (no Helm release secrets) Native helm install/upgrade via helm-controller
Secrets Sealed Secrets is simplest; SOPS needs a plugin (KSOPS) SOPS decryption built in
Image auto-update Argo CD Image Updater (add-on) Built in (image-reflector + image-automation)
Bootstrap helm install, then hand it its own config flux bootstrap CLI does it all in one command

Why Argo CD here: with a growing number of namespaces, the UI pays for itself. You see every project's health on one page, you see drift as a diff, and you can roll back with a click while learning. ApplicationSets and per-project AppProjects make "new repo + registration = new app" a one-time setup.

When to pick Flux instead: the cluster is very memory-constrained (Raspberry Pi with 2 GB), you prefer a CLI-only workflow, or you want SOPS without plugins. The multi-repo idea transfers directly (Flux GitRepository per project repo plus tenant service accounts); Flux equivalents are noted where they differ.

Don't run both. Pick one controller per cluster to avoid two reconcilers fighting over the same objects.

2. Design the repos: one platform repo, one repo per project

Split by who owns what. The platform repo holds everything cluster-wide and every guardrail. Each project repo holds only that project's namespaced manifests. The rule that keeps automation simple: project name = Argo CD AppProject name = namespace.

The platform repo (k3s-platform)

k3s-platform/
├── bootstrap/
│   └── root.yaml                    # the ONLY thing you ever kubectl apply
├── argocd/                          # Argo CD manages its own config from here
│   ├── install/values.yaml          # Helm values for Argo CD (incl. RBAC)
│   ├── argocd.yaml                  # Application: Argo CD upgrades itself
│   ├── appproject-platform.yaml     # AppProject for platform-owned apps
│   ├── repo-creds.sealed.yaml       # credential template for all your repos (section 6)
│   ├── appsets/
│   │   ├── infrastructure.yaml      # one app per infrastructure/*
│   │   ├── project-config.yaml      # one app per projects/* (AppProject, namespace, quota)
│   │   └── projects.yaml            # one app per entry in projects/*/config.yaml
│   └── kustomization.yaml
├── infrastructure/                  # cluster-wide services
│   ├── sealed-secrets/
│   ├── cert-manager/
│   ├── traefik-config/
│   └── storage/
├── projects/                        # one folder per registered project
│   ├── blog/
│   │   ├── config.yaml              # registration: repo URL, path, sync options (read by an ApplicationSet)
│   │   ├── kustomization.yaml
│   │   ├── appproject.yaml          # what blog may deploy, from which repo, and who may sync it
│   │   ├── namespace.yaml
│   │   └── quota.yaml               # ResourceQuota + LimitRange
│   └── media/ ...
└── scripts/register-project.sh      # Step 8.3

A project repo (blog)

blog/                                # app source + deploy/ (or a separate blog-deploy repo)
├── src/ ...
├── Dockerfile
├── .github/workflows/
└── deploy/
    ├── base/
    │   ├── kustomization.yaml
    │   ├── deployment.yaml
    │   ├── service.yaml
    │   └── ingress.yaml
    └── overlays/
        ├── prod/
        │   ├── kustomization.yaml
        │   └── sealed-secret.yaml
        └── staging/                 # later

Who owns what

Lives in Contents Changed by
Platform repo argocd/, infrastructure/ Argo CD, cert-manager, Sealed Secrets, Traefik config, storage classes, CRDs You, as platform admin
Platform repo projects/<name>/ Registration, AppProject, namespace, quota, RBAC You, on each onboarding
Project repo deploy/ Deployments, Services, Ingress, ConfigMaps, SealedSecrets, image tags The project's owner

The namespace is deliberately created by the platform repo, not the project repo. The project's AppProject forbids cluster-scoped resources, so a project repo can't create namespaces, ClusterRoles or CRDs, or deploy into another project's namespace.

Decisions to make now

  • deploy/ inside the app repo, or a separate <name>-deploy repo? For your own apps, put deploy/ in the app repo: one PR can change code and manifests together, and history stays in one place. Use a separate -deploy repo when CI-driven tag bumps would cause noisy build loops, when different people should control code and deployment, or for third-party software with no source repo (for example media-deploy holding a Jellyfin Helm chart).
  • Fixed path convention: every project repo uses deploy/overlays/<env>. Registration then needs no special cases.
  • Branching: deploy from main only, use PRs for review, and use folders per environment rather than branches.
  • Naming: lowercase, DNS-safe project names (home-assistant, not HomeAssistant), since they become namespace and AppProject names.
  • GitHub user vs organization: a personal account works. A free GitHub organization makes access control and the auto-discovery option in Step 4.7 easier.

Flux equivalent: the platform repo becomes the Flux "fleet" repo, with a GitRepository and Kustomization per project repo. Flux's multi-tenancy model (flux create tenant, serviceAccountName on each Kustomization) plays the role of AppProjects.

3. Bootstrap Argo CD from the platform repo

You install Argo CD once with Helm, give it one credential that covers all your repos, then apply one root Application from the platform repo. From then on, Argo CD manages itself and every project from Git.

Step 3.1: Create the platform repo

GitLab: use glab repo create and https://gitlab.com/YOUR_GROUP/<name>.git URLs everywhere; see G1.

gh repo create YOUR_USER/k3s-platform --private --clone   # or create it in the GitHub UI and clone
cd k3s-platform
mkdir -p bootstrap argocd/{install,appsets} infrastructure projects scripts

Step 3.2: Write Argo CD's Helm values

argocd/install/values.yaml:

global:
  domain: argocd.home.example.com

configs:
  params:
    server.insecure: true            # Traefik terminates TLS, Argo serves plain HTTP behind it
  cm:
    application.resourceTrackingMethod: annotation   # safer than labels; default in v3, set it explicitly
    kustomize.buildOptions: --enable-helm            # lets Kustomize inflate Helm charts (any repo)
    timeout.reconciliation: 180s                     # how often Git is polled
    exec.enabled: false
  rbac:
    policy.default: role:readonly    # tightened in Step 4.6
  # The repo credential is added in Step 3.5, not here, so it never sits in a values file.

server:
  ingress:
    enabled: true
    ingressClassName: traefik
    tls: true
    # annotations:
    #   cert-manager.io/cluster-issuer: letsencrypt   # once cert-manager is in place

dex:
  enabled: false                     # turn on later for GitHub/OIDC SSO

controller:
  resources:
    requests: { cpu: 100m, memory: 256Mi }
repoServer:
  resources:
    requests: { cpu: 50m, memory: 128Mi }   # renders every repo; raise it as projects grow

Step 3.3: Install it

helm repo add argo https://argoproj.github.io/argo-helm
helm repo update
helm search repo argo/argo-cd --versions | head -5     # pick the latest and PIN it

export ARGO_CHART_VERSION=<chart version from above>
kubectl create namespace argocd
helm install argocd argo/argo-cd -n argocd \
  --version "$ARGO_CHART_VERSION" \
  -f argocd/install/values.yaml

kubectl -n argocd rollout status deploy/argocd-server

Step 3.4: Log in and secure the admin account

kubectl -n argocd get secret argocd-initial-admin-secret \
  -o jsonpath='{.data.password}' | base64 -d; echo

kubectl -n argocd port-forward svc/argocd-server 8080:80     # until DNS/TLS is sorted

argocd login localhost:8080 --plaintext --username admin
argocd account update-password
kubectl -n argocd delete secret argocd-initial-admin-secret

Step 3.5: One credential for all your repos

GitLab: use a group deploy token as the credential template instead of a GitHub App; see G2.

With many repos, give Argo CD a credential template: a Secret whose URL is a prefix (https://github.com/YOUR_USER). Every repo URL starting with that prefix is authenticated automatically, so a new project repo needs no Argo CD change.

Don't use deploy keys here. GitHub ties each deploy key to a single repo, so you'd need one key and one Secret per project.

Recommended: a GitHub App (read-only, no expiry, revocable per repo)

  1. GitHub, then Settings, Developer settings, GitHub Apps, New GitHub App. Name it something like argocd-home. Untick Webhook "Active". Under Repository permissions, set Contents: Read-only (Metadata: Read-only is added automatically). Create it and note the App ID.
  2. On the app's page, generate a private key (a .pem file downloads).
  3. Install App, choose your account, then either All repositories (zero onboarding work) or Only select repositories (pick k3s-platform plus each project repo; you'll add new repos here on onboarding). The Installation ID is the number at the end of the installation's settings URL.
  4. Create the credential template in the cluster:
kubectl -n argocd create secret generic github-creds \
  --from-literal=type=git \
  --from-literal=url=https://github.com/YOUR_USER \
  --from-literal=githubAppID=<APP_ID> \
  --from-literal=githubAppInstallationID=<INSTALLATION_ID> \
  --from-file=githubAppPrivateKey=$HOME/.secrets/argocd-home.pem
kubectl -n argocd label secret github-creds argocd.argoproj.io/secret-type=repo-creds

argocd repocreds list        # shows https://github.com/YOUR_USER

Alternative: a fine-grained personal access token. Scope it to the repos Argo CD needs, with Contents read-only. Create the same Secret with --from-literal=username=git --from-literal=password=github_pat_... instead of the three githubApp* keys. Tokens expire, so put a renewal reminder in your calendar.

The credential lives only in the cluster for now. Step 6.5 seals it into the platform repo so a rebuild can restore it.

Step 3.6: Let Argo CD manage itself

argocd/argocd.yaml:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: argocd
  namespace: argocd
spec:
  project: default
  sources:
    - repoURL: https://argoproj.github.io/argo-helm
      chart: argo-cd
      targetRevision: <same as ARGO_CHART_VERSION>
      helm:
        releaseName: argocd
        valueFiles:
          - $values/argocd/install/values.yaml
    - repoURL: https://github.com/YOUR_USER/k3s-platform.git
      targetRevision: main
      ref: values
  destination:
    server: https://kubernetes.default.svc
    namespace: argocd
  syncPolicy:
    automated:
      prune: false          # never let Argo prune itself by accident
      selfHeal: true
    syncOptions:
      - ServerSideApply=true

argocd/kustomization.yaml (the AppProject and ApplicationSets are written in section 4; write them before you push in Step 3.7, or comment those lines out until then):

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - argocd.yaml
  - appproject-platform.yaml
  - appsets/infrastructure.yaml
  - appsets/project-config.yaml
  - appsets/projects.yaml
  # - repo-creds.sealed.yaml       # uncomment in Step 6.5

Step 3.7: Apply the root Application (the last manual apply)

bootstrap/root.yaml:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: root
  namespace: argocd
  # Deliberately NO resources-finalizer: deleting root must not cascade-delete the cluster
spec:
  project: default
  source:
    repoURL: https://github.com/YOUR_USER/k3s-platform.git
    targetRevision: main
    path: argocd
  destination:
    server: https://kubernetes.default.svc
    namespace: argocd
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
git add . && git commit -m "Bootstrap Argo CD" && git push
kubectl apply -f bootstrap/root.yaml
argocd app list          # root and argocd should go Synced / Healthy

4. Wire up projects: AppProjects, registrations, ApplicationSets and RBAC

The platform repo runs three ApplicationSets. One installs cluster services. One applies each project's guardrails (AppProject, namespace, quota). One reads each project's config.yaml and creates an Application that pulls manifests from that project's own repo.

root (k3s-platform: bootstrap/root.yaml)
├── argocd                         Argo CD manages itself
├── infrastructure   (AppSet)      one app per infrastructure/*        → sealed-secrets, cert-manager, ...
├── project-config   (AppSet)      one app per projects/*              → AppProject + namespace + quota per project
└── projects         (AppSet)      one app per projects/*/config.yaml entry
    ├── blog          ← https://github.com/YOUR_USER/blog.git          deploy/overlays/prod
    ├── media         ← https://github.com/YOUR_USER/media-deploy.git  deploy/overlays/prod
    └── (next project ← its own repo)

Step 4.1: The platform AppProject

argocd/appproject-platform.yaml covers apps you own: cluster services and per-project guardrails. It has broad rights, but only from the platform repo and upstream chart repos.

apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: platform
  namespace: argocd
spec:
  description: Cluster services and project guardrails from k3s-platform
  sourceRepos:
    - https://github.com/YOUR_USER/k3s-platform.git
    - https://*                       # upstream Helm repos used by infrastructure/ (tighten to a list later)
  destinations:
    - server: https://kubernetes.default.svc
      namespace: '*'
  clusterResourceWhitelist:
    - group: '*'
      kind: '*'

Step 4.2: ApplicationSet for infrastructure

argocd/appsets/infrastructure.yaml:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: infrastructure
  namespace: argocd
spec:
  goTemplate: true
  goTemplateOptions: ["missingkey=error"]
  generators:
    - git:
        repoURL: https://github.com/YOUR_USER/k3s-platform.git
        revision: main
        directories:
          - path: infrastructure/*
  template:
    metadata:
      name: '{{ .path.basename }}'
      labels: { area: infrastructure }
    spec:
      project: platform
      source:
        repoURL: https://github.com/YOUR_USER/k3s-platform.git
        targetRevision: main
        path: '{{ .path.path }}'
      destination:
        server: https://kubernetes.default.svc
        namespace: '{{ .path.basename }}'
      syncPolicy:
        automated: { prune: true, selfHeal: true }
        syncOptions:
          - CreateNamespace=true
          - ServerSideApply=true
          - SkipDryRunOnMissingResource=true   # CRDs may arrive in the same sync
        retry:
          limit: 5
          backoff: { duration: 15s, factor: 2, maxDuration: 5m }
  syncPolicy:
    preserveResourcesOnDeletion: true

Step 4.3: Register a project in the platform repo

Each project gets a folder under projects/. Step 8.3 has a script that writes all of these; here's what it produces for blog.

projects/blog/config.yaml is the registration. It's read by the projects ApplicationSet, never applied to the cluster. It's a list, one entry per environment:

- project: blog                                   # AppProject name
  name: blog                                      # Argo CD Application name
  namespace: blog
  repoURL: https://github.com/YOUR_USER/blog.git
  sourcePath: deploy/overlays/prod
  targetRevision: main
  autoPrune: true                                 # false while migrating (section 7)

projects/blog/appproject.yaml is the guardrail. Blog may deploy only from its own repo, only into its own namespace, and only namespaced resources:

apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: blog
  namespace: argocd                 # AppProjects always live in argocd
spec:
  description: blog project
  sourceRepos:
    - https://github.com/YOUR_USER/blog.git
  destinations:
    - server: https://kubernetes.default.svc
      namespace: blog
  clusterResourceWhitelist: []      # nothing cluster-scoped: the platform repo creates the namespace
  namespaceResourceBlacklist:       # quotas are the platform's call, not the project's
    - group: ''
      kind: ResourceQuota
    - group: ''
      kind: LimitRange
  orphanedResources:
    warn: true                      # flags objects in the namespace that aren't in Git
  roles:
    - name: developer
      description: View, sync and read logs for blog apps
      policies:
        - p, proj:blog:developer, applications, get, blog/*, allow
        - p, proj:blog:developer, applications, sync, blog/*, allow
        - p, proj:blog:developer, logs, get, blog/*, allow
      groups:
        - blog-devs                 # SSO group name, once SSO is on (Step 4.6)

projects/blog/namespace.yaml and projects/blog/quota.yaml:

apiVersion: v1
kind: Namespace
metadata:
  name: blog
  labels:
    pod-security.kubernetes.io/enforce: baseline
---
apiVersion: v1
kind: ResourceQuota
metadata: { name: quota, namespace: blog }
spec:
  hard:
    requests.cpu: "2"
    requests.memory: 4Gi
    limits.memory: 8Gi
    persistentvolumeclaims: "5"
---
apiVersion: v1
kind: LimitRange
metadata: { name: defaults, namespace: blog }
spec:
  limits:
    - type: Container               # pods without requests/limits get these, so the quota doesn't reject them
      defaultRequest: { cpu: 50m, memory: 64Mi }
      default: { memory: 256Mi }

(Put the Namespace in namespace.yaml and the other two in quota.yaml.)

projects/blog/kustomization.yaml lists what gets applied. config.yaml is intentionally left out:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - namespace.yaml
  - appproject.yaml
  - quota.yaml

Step 4.4: ApplicationSet for project guardrails

argocd/appsets/project-config.yaml creates one project-<name> Application per folder under projects/, which applies that folder's AppProject, namespace and quota:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: project-config
  namespace: argocd
spec:
  goTemplate: true
  goTemplateOptions: ["missingkey=error"]
  generators:
    - git:
        repoURL: https://github.com/YOUR_USER/k3s-platform.git
        revision: main
        directories:
          - path: projects/*
  template:
    metadata:
      name: 'project-{{ .path.basename }}'
      labels: { area: project-config }
    spec:
      project: platform
      source:
        repoURL: https://github.com/YOUR_USER/k3s-platform.git
        targetRevision: main
        path: '{{ .path.path }}'
      destination:
        server: https://kubernetes.default.svc
        namespace: '{{ .path.basename }}'
      syncPolicy:
        automated: { prune: true, selfHeal: true }
  syncPolicy:
    preserveResourcesOnDeletion: true   # removing a project folder won't delete its namespace by surprise

Step 4.5: ApplicationSet that points at each project repo

argocd/appsets/projects.yaml uses the git files generator. Every entry in every projects/*/config.yaml becomes one Application, in that project's AppProject, sourced from that project's repo:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: projects
  namespace: argocd
spec:
  goTemplate: true
  goTemplateOptions: ["missingkey=error"]   # a typo in config.yaml fails loudly instead of deploying wrong
  generators:
    - git:
        repoURL: https://github.com/YOUR_USER/k3s-platform.git
        revision: main
        files:
          - path: projects/*/config.yaml
  template:
    metadata:
      name: '{{ .name }}'
      labels:
        project: '{{ .project }}'
    spec:
      project: '{{ .project }}'
      source:
        repoURL: '{{ .repoURL }}'
        targetRevision: '{{ .targetRevision }}'
        path: '{{ .sourcePath }}'
      destination:
        server: https://kubernetes.default.svc
        namespace: '{{ .namespace }}'
      syncPolicy:
        syncOptions:
          - PruneLast=true
        retry:
          limit: 3
          backoff: { duration: 10s, factor: 2, maxDuration: 2m }
  templatePatch: |
    spec:
      syncPolicy:
        automated:
          prune: {{ .autoPrune }}
          selfHeal: true
  syncPolicy:
    preserveResourcesOnDeletion: true

Two details matter here:

  • The registration key is sourcePath, not path. The files generator already injects its own path parameters (the location of config.yaml), and a key named path would collide with them.
  • templatePatch lets each project choose autoPrune in its registration. Plain template fields can't render a boolean from a parameter.

Ordering: a new project's Application can appear a few seconds before its AppProject exists. It shows an error briefly, then recovers on the next reconcile. Within one project, order resources with argocd.argoproj.io/sync-wave annotations as usual.

Step 4.6: Argo CD RBAC (who can see and sync what)

GitLab: GitLab groups and subgroups can be the SSO groups; see G9.

Git permissions decide who can change a project: whoever can merge to blog's main can deploy blog. Argo CD RBAC decides who can see apps, trigger syncs and read logs in the UI and CLI. Configure it in argocd/install/values.yaml:

configs:
  cm:
    accounts.alice: login            # a local user; set the password: argocd account update-password --account alice
  rbac:
    policy.default: ''               # no access unless granted below
    scopes: '[groups]'
    policy.csv: |
      # Platform admins
      g, admin, role:admin
      # A blog developer: see, sync and read logs for blog only
      p, role:blog-dev, applications, get,  blog/*, allow
      p, role:blog-dev, applications, sync, blog/*, allow
      p, role:blog-dev, logs, get, blog/*, allow
      g, alice, role:blog-dev

Once SSO is on (Dex with GitHub, or any OIDC provider), skip local accounts. Put people in a group such as blog-devs and the AppProject role from Step 4.3 grants them access automatically.

For CI that needs to trigger or wait for syncs, mint a token scoped to one project:

argocd proj role create-token blog developer --expires-in 90d

Step 4.7: Alternative, auto-discover project repos (SCM Provider generator)

GitLab: use the gitlab: SCM provider for a group; see G3.

If you'd rather not register projects at all, Argo CD can scan a GitHub organization and create an app for every repo with a given topic and a deploy/overlays/prod folder:

generators:
  - scmProvider:
      github:
        organization: YOUR_ORG
        tokenRef: { secretName: github-scm-token, key: token }
      filters:
        - labelMatch: k3s-app                # GitHub repo topic
          pathsExist: [deploy/overlays/prod]
template:
  metadata:
    name: '{{ .repository }}'
  spec:
    project: apps                            # one shared AppProject
    source:
      repoURL: '{{ .url }}'
      targetRevision: '{{ .branch }}'
      path: deploy/overlays/prod
    destination:
      server: https://kubernetes.default.svc
      namespace: '{{ .repository }}'
    syncPolicy:
      automated: { prune: true, selfHeal: true }
      syncOptions: [CreateNamespace=true]

The trade-off: onboarding is just adding a topic, but every project shares one AppProject (no per-project repo or namespace fence), anyone who can create a repo in the org can deploy, and the generator uses a GitHub API token with rate limits. For a homelab with a handful of projects, explicit registration (Steps 4.3 to 4.5) is the safer default.

5. Environments and overlays inside each project repo

Each project repo uses Kustomize base/ + overlays/<env>/ under deploy/. Third-party software is inflated from Helm charts with Kustomize, in either a project repo or the platform repo's infrastructure/.

Step 5.1: A base for one project (in the blog repo)

deploy/base/kustomization.yaml:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - deployment.yaml
  - service.yaml
  - ingress.yaml
labels:
  - pairs:
      app.kubernetes.io/name: blog
      app.kubernetes.io/part-of: blog
    includeSelectors: false

No namespace.yaml here: the platform repo owns the namespace, and blog's AppProject would reject one.

deploy/base/deployment.yaml (trimmed):

apiVersion: apps/v1
kind: Deployment
metadata:
  name: blog
spec:
  replicas: 1
  selector:
    matchLabels: { app: blog }
  template:
    metadata:
      labels: { app: blog }
    spec:
      containers:
        - name: blog
          image: ghcr.io/YOUR_USER/blog:1.4.2     # always a pinned tag, never :latest
          ports: [{ containerPort: 8080 }]
          envFrom:
            - secretRef: { name: blog-secrets }
          resources:
            requests: { cpu: 50m, memory: 64Mi }
            limits:   { memory: 256Mi }

Step 5.2: The prod overlay

deploy/overlays/prod/kustomization.yaml:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: blog
resources:
  - ../../base
  - sealed-secret.yaml          # section 6
images:
  - name: ghcr.io/YOUR_USER/blog
    newTag: 1.4.2               # the one line that changes on each release
patches:
  - target: { kind: Ingress, name: blog }
    patch: |
      - op: replace
        path: /spec/rules/0/host
        value: blog.home.example.com

Preview from the project repo before pushing:

kubectl kustomize deploy/overlays/prod | less
kubectl kustomize deploy/overlays/prod | kubectl diff -f -    # vs the live cluster

Step 5.3: Adding staging later

  1. In the project repo, create deploy/overlays/staging/ with namespace: blog-staging, a different host, and fewer replicas.
  2. In the platform repo, add the namespace to projects/blog/namespace.yaml (plus quota if you want), add blog-staging to the AppProject's destinations, and add a second entry to projects/blog/config.yaml:
- project: blog
  name: blog-staging
  namespace: blog-staging
  repoURL: https://github.com/YOUR_USER/blog.git
  sourcePath: deploy/overlays/staging
  targetRevision: main
  autoPrune: true

The prod app keeps its name, so nothing is recreated. Promotion is a PR in the project repo copying the image tag from overlays/staging to overlays/prod.

Step 5.4: Third-party software via Helm

Keep chart version and values in Git and let Kustomize render them. For cluster services this lives in the platform repo, for example infrastructure/cert-manager/kustomization.yaml:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: cert-manager
helmCharts:
  - name: cert-manager
    repo: https://charts.jetstack.io
    version: v1.x.y                 # pin; Renovate can bump it (section 10)
    releaseName: cert-manager
    namespace: cert-manager
    includeCRDs: true
    valuesInline:
      crds:
        enabled: true
resources:
  - cluster-issuer.yaml

For a third-party app that's a project in its own right (Jellyfin, Nextcloud), put the same helmCharts block in that project's repo, for example media-deploy/deploy/overlays/prod/kustomization.yaml. Two caveats:

  • The chart must not create cluster-scoped objects (CRDs, ClusterRoles), or the project's AppProject will block the sync. Disable them in values, or treat that software as infrastructure in the platform repo.
  • Upstream chart repos must be allowed. Add https://* (or the specific chart repo URL) to that project's AppProject sourceRepos. Kustomize fetches the chart itself, but listing it documents the dependency and is required if you ever switch to a multi-source Application.
You need Use
Your own app, a few manifests Plain YAML in the project repo's deploy/base + overlay
An upstream app chart with modest values Kustomize helmCharts in that project's repo
A cluster-wide service or anything with CRDs Kustomize helmCharts in the platform repo's infrastructure/
Same app, slightly different per env Kustomize patches in each overlay

Plain Secrets must never land in Git. With Argo CD, Sealed Secrets is the least-friction option: you encrypt locally with a public key, commit the ciphertext, and only the controller in your cluster can decrypt it. No Argo CD plugins are needed.

Option How it works Pick it when
Sealed Secrets Controller holds a private key; kubeseal encrypts with its public cert Default for Argo CD on a single cluster
SOPS + age Files encrypted with an age key; decrypted at render time You use Flux (built in), or want to edit encrypted YAML in place; with Argo CD it needs the KSOPS plugin
External Secrets Operator Pulls from 1Password, Bitwarden, Vault, cloud KMS You already keep secrets in a password manager or vault; good later step

Step 6.1: Install the controller from Git

In the platform repo, infrastructure/sealed-secrets/kustomization.yaml:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: sealed-secrets
helmCharts:
  - name: sealed-secrets
    repo: https://bitnami-labs.github.io/sealed-secrets
    version: 2.x.y                  # pin the latest
    releaseName: sealed-secrets
    namespace: sealed-secrets
    includeCRDs: true
    valuesInline:
      fullnameOverride: sealed-secrets-controller

Push, wait for the sealed-secrets app to go Healthy, then install the CLI (brew install kubeseal, or the binary from the GitHub releases page) and save yourself some typing:

alias kubeseal='kubeseal --controller-namespace sealed-secrets --controller-name sealed-secrets-controller'

Step 6.2: Back up the sealing key (do this today)

If you rebuild the cluster without this key, every sealed secret in Git becomes unreadable.

kubectl -n sealed-secrets get secret \
  -l sealedsecrets.bitnami.com/sealed-secrets-key -o yaml > sealed-secrets-key.backup.yaml
# Store it in your password manager or offline. NEVER commit it.
# On a rebuilt cluster: kubectl apply -f sealed-secrets-key.backup.yaml BEFORE the controller starts (or restart it after)

The controller rotates in a new key every 30 days by default and keeps the old ones, so re-run the backup occasionally.

Step 6.3: Seal a secret (in the project repo)

Project owners seal their own secrets and commit them to their own repo. They need the cluster's public cert, not cluster access (see the first bullet below).

kubectl create secret generic blog-secrets -n blog \
  --from-literal=DB_PASSWORD='s3cr3t' \
  --from-literal=API_KEY='abc123' \
  --dry-run=client -o yaml \
| kubeseal --format yaml > deploy/overlays/prod/sealed-secret.yaml

git add deploy/overlays/prod/sealed-secret.yaml
git commit -m "blog: add secrets" && git push

By default a sealed secret is bound to its name and namespace: copying it to blog-staging won't decrypt. Seal per environment, which is what you want.

Useful extras:

  • Seal offline / in CI: kubeseal --fetch-cert > pub-cert.pem, commit the cert to the platform repo (it's public), then anyone with read access can run kubeseal --cert pub-cert.pem ... without cluster access.
  • Update one key: echo -n newvalue | kubeseal --raw --namespace blog --name blog-secrets --from-file=/dev/stdin, then paste into spec.encryptedData.
  • Existing secrets in the cluster: kubectl get secret blog-secrets -n blog -o yaml | kubeseal --format yaml > ... (strip resourceVersion/uid first or use the kubectl neat plugin).

Step 6.4: Guard rails against leaks

# .gitignore
*.backup.yaml
*.dec.yaml
*-unsealed.yaml
.env

Put this .gitignore in the platform repo and every project repo. Add a secret scanner such as gitleaks as a pre-commit hook or CI check. If a plain secret is ever pushed, rotate the credential; rewriting history isn't enough.

SOPS path (if you prefer it): generate a key with age-keygen -o age.key, add a .sops.yaml with creation_rules: [{path_regex: .*\.sops\.yaml$, encrypted_regex: ^(data|stringData)$, age: <public key>}], encrypt with sops -e -i secret.sops.yaml, store age.key as a Secret for Flux (decryption.provider: sops) or for the KSOPS sidecar in Argo CD's repo-server.

Step 6.5: Seal the repo credential into the platform repo

GitLab: seal the gitlab-creds Secret from G2 the same way, with its four GitLab values.

The credential template from Step 3.5 exists only in the cluster. Seal it so a rebuild restores it from Git:

kubectl -n argocd create secret generic github-creds \
  --from-literal=type=git \
  --from-literal=url=https://github.com/YOUR_USER \
  --from-literal=githubAppID=<APP_ID> \
  --from-literal=githubAppInstallationID=<INSTALLATION_ID> \
  --from-file=githubAppPrivateKey=$HOME/.secrets/argocd-home.pem \
  --dry-run=client -o yaml \
| kubectl label -f - --local -o yaml argocd.argoproj.io/secret-type=repo-creds \
| kubeseal --format yaml > argocd/repo-creds.sealed.yaml

# Let the SealedSecret take over the Secret you created by hand in Step 3.5
kubectl -n argocd annotate secret github-creds sealedsecrets.bitnami.com/managed=true

Uncomment repo-creds.sealed.yaml in argocd/kustomization.yaml, commit and push. On a fresh cluster you still create the plain Secret once by hand (Step 3.5), because Argo CD needs it to read the platform repo in the first place.

7. Migrate existing workloads without downtime

Argo CD adopts existing objects when Git has the same kind, name and namespace. It patches them in place instead of recreating them. Migrate one namespace at a time, least critical first. Register each project with autoPrune: false until its diff is clean.

Step 7.1: Inventory what's running

kubectl get ns
helm list -A                                   # anything installed with Helm
kubectl get helmcharts.helm.cattle.io -A        # anything installed via k3s HelmChart CRs
ls /var/lib/rancher/k3s/server/manifests/      # on the server node: k3s auto-deploy files

# Per namespace, what you actually need to capture:
kubectl -n blog get deploy,sts,ds,svc,ingress,ingressroute,cm,secret,pvc,cronjob

For each namespace, decide which repo it goes to: the app's existing source repo (add deploy/), or a new <name>-deploy repo. Note how it was deployed (plain YAML, helm install, or a k3s HelmChart), since each has its own path below.

Step 7.2: Create the project repo and export clean manifests

cd ~/src/blog                                   # existing app repo, or: gh repo create YOUR_USER/blog-deploy --private --clone
mkdir -p deploy/base deploy/overlays/prod
kubectl krew install neat                       # one-time; or strip server fields by hand
for kind in deploy svc ingress cm pvc; do
  kubectl -n blog get $kind -o yaml | kubectl neat > deploy/base/$kind.yaml
done

If you still have the original YAML files, start from those; they're cleaner. Then tidy up:

  • Remove namespace: fields; the overlay sets it.
  • Delete the kube-root-ca.crt ConfigMap, ServiceAccount token secrets, and any Namespace object (the platform repo owns it).
  • Convert real Secrets to SealedSecrets (Step 6.3).
  • Write deploy/base/kustomization.yaml and deploy/overlays/prod/kustomization.yaml as in section 5.

Things that must not change during adoption

  • Deployment spec.selector is immutable. Keep the exact labels, which is why Step 5.1 uses includeSelectors: false.
  • PVC spec (storageClassName, size, accessModes) is immutable. Copy it exactly so the existing volume and data are kept.
  • Service clusterIP: leave it out of Git.

Step 7.3: Protect data before the first sync

k3s's local-path provisioner creates volumes with reclaim policy Delete. If a PVC were ever pruned, the data on disk would go with it. Before migrating anything stateful:

kubectl get pv -o custom-columns=NAME:.metadata.name,CLAIM:.spec.claimRef.name,POLICY:.spec.persistentVolumeReclaimPolicy
kubectl patch pv <pv-name> -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}'

And tell Argo CD never to prune PVCs, in the project repo's base:

# deploy/base/pvc.yaml
metadata:
  name: blog-data
  annotations:
    argocd.argoproj.io/sync-options: Prune=false,Delete=false

Also take a normal backup of the volume directory (/var/lib/rancher/k3s/storage/ on the node) for anything you care about.

Step 7.4: Register the project with pruning off

In the platform repo, run the register script (Step 8.3) or create projects/blog/ by hand. Set autoPrune: false in config.yaml for now, so a file you haven't exported yet can't delete a live object.

Two cautions for existing namespaces:

  • namespace.yaml adopts the existing namespace. Copy any labels or annotations it already has, so Argo CD doesn't strip them.
  • A ResourceQuota on CPU/memory rejects new pods that lack requests/limits. The LimitRange in Step 4.3 supplies defaults, but check that your current usage fits under the quota (kubectl describe quota -n blog after sync). Or leave quota.yaml out of the kustomization until the project is settled.

Step 7.5: Diff, then sync

# In the project repo
kubectl kustomize deploy/overlays/prod | kubectl diff -f -    # local preview, should be near-empty
git add deploy && git commit -m "Add GitOps manifests" && git push

# In the platform repo
git add projects/blog && git commit -m "Register blog (pruning off)" && git push

argocd app get project-blog      # AppProject, namespace, quota
argocd app get blog              # appears within ~3 minutes; or: argocd app get projects --refresh
argocd app diff blog             # what Argo would change

Iterate in the project repo until argocd app diff blog shows only harmless changes (the tracking annotation, defaulted fields). Auto-sync then applies it. Watch that pods don't restart unexpectedly:

kubectl -n blog get pods -w

If a field keeps showing as different because a controller sets it (HPA replicas, a mutating webhook), add an ignoreDifferences entry rather than fighting it (section 10).

Step 7.6: Projects installed with Helm or k3s HelmChart

Installed with helm install:

helm -n media get values media > /tmp/media-values.yaml     # your custom values only
helm -n media list                                         # note chart name and version

Put those into the project repo's deploy/overlays/prod/kustomization.yaml as a helmCharts entry with the same releaseName and chart version, so rendered object names match. Check the chart creates nothing cluster-scoped (Step 5.4). After Argo CD shows Synced/Healthy, remove Helm's bookkeeping so nobody runs helm upgrade out of habit:

# Do NOT run helm uninstall: that deletes the workload.
kubectl -n media delete secret -l owner=helm,name=media

Installed via a k3s HelmChart (manifest in /var/lib/rancher/k3s/server/manifests/): deleting a HelmChart object makes k3s's helm-controller uninstall the release. Plan these as a short, controlled redeploy:

  1. Set PVs to Retain (Step 7.3) and back up the manifest file.
  2. Add the equivalent helmCharts entry, with the same release name, to the project repo.
  3. Move the file out of the auto-deploy directory, and let Argo CD bring the app back up.
  4. To re-bind a retained volume, clear its old claim with kubectl patch pv <pv-name> --type json -p '[{"op":"remove","path":"/spec/claimRef"}]' and set spec.volumeName: <pv-name> on the PVC in Git.

How k3s reacts to a removed auto-deploy file has changed between versions, so test this on a non-critical app first.

Step 7.7: Finish each migration

Once a project shows Synced/Healthy:

  1. Set autoPrune: true in its projects/<name>/config.yaml and merge.
  2. Check the orphaned-resources warnings (enabled in its AppProject) for leftovers in the namespace, and either add them to the project repo or delete them.
  3. When all projects are done, stop using cluster-admin kubeconfigs day to day. Git is now the way you change things.

8. Onboard a new project repo

A new project takes two PRs: one creating the project repo's deploy/ folder, and one registering it in the platform repo. No Argo CD config changes, no kubectl.

Step 8.1: Scaffold deploy/ in the project repo

GitLab: the scripts are the same; the glab commands for Steps 8.1 to 8.3 are in G5.

Save this as scripts/new-app.sh in the platform repo, and run it from the root of the project repo:

#!/usr/bin/env bash
# Run from the root of a PROJECT repo. Writes deploy/base and deploy/overlays/prod.
set -euo pipefail
APP=${1:?usage: new-app.sh <name> <image:tag> [port]}
IMAGE=${2:?image required, e.g. ghcr.io/me/app:1.0.0}
PORT=${3:-8080}
DOMAIN=${DOMAIN:-home.example.com}
B=deploy/base O=deploy/overlays/prod
mkdir -p "$B" "$O"

cat > "$B/kustomization.yaml" <<EOF
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources: [deployment.yaml, service.yaml, ingress.yaml]
labels:
  - pairs: { app.kubernetes.io/name: $APP }
    includeSelectors: false
EOF

cat > "$B/deployment.yaml" <<EOF
apiVersion: apps/v1
kind: Deployment
metadata: { name: $APP }
spec:
  replicas: 1
  selector: { matchLabels: { app: $APP } }
  template:
    metadata: { labels: { app: $APP } }
    spec:
      containers:
        - name: $APP
          image: ${IMAGE%:*}
          ports: [{ containerPort: $PORT }]
          resources:
            requests: { cpu: 25m, memory: 64Mi }
            limits: { memory: 256Mi }
EOF

cat > "$B/service.yaml" <<EOF
apiVersion: v1
kind: Service
metadata: { name: $APP }
spec:
  selector: { app: $APP }
  ports: [{ port: 80, targetPort: $PORT }]
EOF

cat > "$B/ingress.yaml" <<EOF
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata: { name: $APP }
spec:
  ingressClassName: traefik
  rules:
    - host: $APP.$DOMAIN
      http:
        paths:
          - path: /
            pathType: Prefix
            backend: { service: { name: $APP, port: { number: 80 } } }
EOF

cat > "$O/kustomization.yaml" <<EOF
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: $APP
resources: [../../base]
images:
  - name: ${IMAGE%:*}
    newTag: "${IMAGE##*:}"
EOF

kubectl kustomize "$O" >/dev/null && echo "deploy/ scaffolded for $APP and renders OK"
gh repo create YOUR_USER/whoami-deploy --private --clone && cd whoami-deploy
~/src/k3s-platform/scripts/new-app.sh whoami traefik/whoami:v1.10.1 80
git add deploy && git commit -m "Add deploy manifests" && git push

Step 8.2: Give Argo CD read access to the new repo

GitLab: nothing to do for projects inside the group covered by the group deploy token; see G5.

  • GitHub App installed on All repositories: nothing to do.
  • Only select repositories: GitHub, Settings, Applications, your Argo CD app, Configure, then add the new repo.
  • Fine-grained token: edit the token and add the repo.

Check access with argocd repo add https://github.com/YOUR_USER/whoami-deploy.git; it inherits the credential template and should report Successful.

Step 8.3: Register it in the platform repo

Save this as scripts/register-project.sh in the platform repo:

#!/usr/bin/env bash
# Run from the root of the PLATFORM repo. Writes projects/<name>/.
set -euo pipefail
NAME=${1:?usage: register-project.sh <name> <repo-url> [source-path]}
REPO=${2:?repo url, e.g. https://github.com/YOUR_USER/blog.git}
SRC=${3:-deploy/overlays/prod}
D=projects/$NAME
mkdir -p "$D"

cat > "$D/config.yaml" <<EOF
# Read by the 'projects' ApplicationSet; one entry per environment
- project: $NAME
  name: $NAME
  namespace: $NAME
  repoURL: $REPO
  sourcePath: $SRC
  targetRevision: main
  autoPrune: true
EOF

cat > "$D/namespace.yaml" <<EOF
apiVersion: v1
kind: Namespace
metadata:
  name: $NAME
  labels:
    pod-security.kubernetes.io/enforce: baseline
EOF

cat > "$D/quota.yaml" <<EOF
apiVersion: v1
kind: ResourceQuota
metadata: { name: quota, namespace: $NAME }
spec:
  hard: { requests.cpu: "2", requests.memory: 4Gi, limits.memory: 8Gi, persistentvolumeclaims: "5" }
---
apiVersion: v1
kind: LimitRange
metadata: { name: defaults, namespace: $NAME }
spec:
  limits:
    - type: Container
      defaultRequest: { cpu: 50m, memory: 64Mi }
      default: { memory: 256Mi }
EOF

cat > "$D/appproject.yaml" <<EOF
apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata: { name: $NAME, namespace: argocd }
spec:
  description: $NAME project
  sourceRepos: [$REPO]
  destinations:
    - { server: https://kubernetes.default.svc, namespace: $NAME }
  clusterResourceWhitelist: []
  namespaceResourceBlacklist:
    - { group: '', kind: ResourceQuota }
    - { group: '', kind: LimitRange }
  orphanedResources: { warn: true }
  roles:
    - name: developer
      policies:
        - p, proj:$NAME:developer, applications, get, $NAME/*, allow
        - p, proj:$NAME:developer, applications, sync, $NAME/*, allow
        - p, proj:$NAME:developer, logs, get, $NAME/*, allow
      groups: [$NAME-devs]
EOF

cat > "$D/kustomization.yaml" <<EOF
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources: [namespace.yaml, appproject.yaml, quota.yaml]
EOF

kubectl kustomize "$D" >/dev/null && echo "$D written and renders OK"
cd ~/src/k3s-platform
scripts/register-project.sh whoami https://github.com/YOUR_USER/whoami-deploy.git
git checkout -b register-whoami && git add projects/whoami && git commit -m "Register whoami"
gh pr create --fill        # review, merge

Within about 3 minutes Argo CD creates project-whoami (AppProject, namespace, quota) and then the whoami Application. Check with argocd app get whoami.

Step 8.4: The onboarding checklist

GitLab: for webhooks (item 7), use a project webhook with a secret token; see G4.

  1. Create the project repo, or use the app's existing repo, and scaffold deploy/ (Step 8.1).
  2. Adjust resources and probes. Add a pvc.yaml with the Prune=false,Delete=false annotation if it stores data.
  3. Seal its secrets into deploy/overlays/prod/sealed-secret.yaml (Step 6.3) and list the file in the overlay's resources.
  4. Add the CI validation workflow to the project repo (Step 8.5).
  5. Give Argo CD read access to the repo (Step 8.2).
  6. Register it in the platform repo (Step 8.3), tune its quota, and merge.
  7. Optional: add a webhook in the project repo (Settings, Webhooks, https://argocd.home.example.com/api/webhook, JSON, push events) for instant syncs instead of ~3-minute polling. This only works if GitHub can reach your Argo CD.
  8. Optional: grant people access via the AppProject role or policy.csv (Step 4.6).

Removing a project: delete projects/<name>/ from the platform repo and merge. Because of preserveResourcesOnDeletion: true, the Application and AppProject disappear but the namespace and workload keep running until you kubectl delete ns <name>. Then archive the project repo.

Step 8.5: Validate PRs in each project repo

GitLab: use a shared .gitlab-ci.yml job pulled in with include; see G6.

Put this in each project repo as .github/workflows/validate-deploy.yaml. It only runs when deploy/ changes:

name: validate-deploy
on:
  pull_request:
    paths: ['deploy/**']
jobs:
  render:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: azure/setup-helm@v4
      - name: Install kubeconform
        run: |
          curl -sL https://github.com/yannh/kubeconform/releases/latest/download/kubeconform-linux-amd64.tar.gz | tar xz
          sudo mv kubeconform /usr/local/bin/
      - name: Render and validate every overlay
        run: |
          set -e
          for d in deploy/overlays/*; do
            echo "::group::$d"
            kubectl kustomize --enable-helm "$d" | kubeconform -strict -ignore-missing-schemas -summary
            # Fail early on objects the AppProject would reject
            if kubectl kustomize --enable-helm "$d" | grep -qE '^kind: (Namespace|ClusterRole|ClusterRoleBinding|CustomResourceDefinition)$'; then
              echo "Cluster-scoped object found in $d; these belong in the platform repo"; exit 1
            fi
            echo "::endgroup::"
          done

To avoid copying this into every repo, make it a reusable workflow in the platform repo (trigger on: workflow_call) and call it with uses: YOUR_USER/k3s-platform/.github/workflows/validate-deploy.yaml@main. If the platform repo is private, allow other repos to use it under its Settings, Actions, General, Access. The platform repo needs the same kind of check for infrastructure/* and projects/*.

9. k3s specifics

k3s ships a few components that it manages itself. The rule: don't put k3s-owned objects in Git unless you first disable them in k3s, otherwise k3s and Argo CD will overwrite each other.

k3s component Owned by GitOps approach
Traefik (HelmChart in kube-system) k3s Customize with a HelmChartConfig in Git, or disable and own it fully
CoreDNS, metrics-server, local-path-provisioner k3s Leave alone; customize via k3s config files
ServiceLB (klipper) k3s Works as-is for type: LoadBalancer
/var/lib/rancher/k3s/server/manifests/ auto-deploy k3s Stop adding files here; move your own ones into Git (Step 7.6)
HelmChart CRD (helm.cattle.io/v1) k3s helm-controller Usable from Git, but prefer Kustomize helmCharts

Traefik: option A, keep the bundled one (simplest)

Customize it with a HelmChartConfig, which k3s merges into its own Traefik install. Put it in the platform repo's infrastructure/traefik-config/:

# infrastructure/traefik-config/helmchartconfig.yaml
apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: traefik
  namespace: kube-system        # must be kube-system, overrides the app's destination namespace
spec:
  valuesContent: |-
    logs:
      access:
        enabled: true
    # Add values for the Traefik chart version your k3s ships;
    # check it with: kubectl -n kube-system get helmchart traefik -o yaml
# infrastructure/traefik-config/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - helmchartconfig.yaml
  - middlewares.yaml            # optional: shared Traefik Middlewares (auth, headers, redirects)

Traefik values keys change between major chart versions, so check them against the version k3s bundles after each k3s upgrade.

Traefik: option B, own it in Git (full control)

Start k3s with Traefik disabled, then install it like any other chart:

# /etc/rancher/k3s/config.yaml on each server node, then: systemctl restart k3s
disable:
  - traefik

Add infrastructure/traefik/kustomization.yaml with a helmCharts entry for https://traefik.github.io/charts. You choose the version and Renovate can bump it, at the cost of one more thing to maintain. Do this during a maintenance window, since ingress is down between disabling and Argo CD reinstalling it.

Ingress vs IngressRoute: standard Ingress objects (as in the scaffold) keep apps portable. Use Traefik's IngressRoute CRD (traefik.io/v1alpha1) only when you need Traefik-specific features; it works fine with Argo CD.

local-path storage

  • Volumes are directories on one node (/var/lib/rancher/k3s/storage/). A pod with a local-path PVC is pinned to that node; on a multi-node cluster, expect that.
  • Default reclaim policy is Delete. Combine Prune=false,Delete=false on PVCs (Step 7.4) with a Retain storage class for anything precious:
# infrastructure/storage/local-path-retain.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: local-path-retain
provisioner: rancher.io/local-path
reclaimPolicy: Retain
volumeBindingMode: WaitForFirstConsumer
  • Git doesn't back up data. GitOps restores configuration; you still need a volume backup (restic/rsync of the storage dir, Velero, or Longhorn snapshots if you move to Longhorn).
  • Size requests on local-path are not enforced, and PVC specs are immutable. Pick a sensible size up front.

HelmChart CRD from Git

You can commit helm.cattle.io/v1 HelmChart objects and let Argo CD apply them. It works, but Argo CD then sees only the HelmChart object, not the Deployments it produces, so health, diffs and drift detection go blind. Use it only as a stopgap while migrating, then convert to Kustomize helmCharts.

Small-cluster tuning

  • Argo CD runs on arm64 (Raspberry Pi) without changes.
  • On memory-tight nodes, keep dex disabled, don't enable HA Redis, and raise timeout.reconciliation (for example to 300s). Add a GitHub webhook to https://argocd.home.example.com/api/webhook if you want instant syncs without fast polling (GitLab: G4).
  • Your kubeconfig on the server is /etc/rancher/k3s/k3s.yaml; copy it to your workstation and change server: to the node's address.

10. Day-2 operations

The daily loop becomes: change Git, merge, watch Argo CD converge. These are the habits and settings that keep that loop safe.

Drift

With selfHeal: true, any kubectl edit is reverted within seconds. That's the point: the cluster can't silently diverge from Git.

argocd app list                          # OutOfSync = drift or an unsynced commit
argocd app diff blog                     # exactly what differs
argocd app get blog --refresh            # force a re-read of Git

Expected differences (fields controllers set) should be ignored, not fought. Per app in the ApplicationSet template, or globally in values.yaml under configs.cm:

# in the ApplicationSet template.spec
ignoreDifferences:
  - group: apps
    kind: Deployment
    jsonPointers: [/spec/replicas]       # when an HPA owns replicas

Pausing one app for debugging: the ApplicationSet rewrites Application specs, so first let per-app sync settings stick by adding this to each ApplicationSet's spec:

ignoreApplicationDifferences:
  - jsonPointers: [/spec/syncPolicy]

Then argocd app set blog --sync-policy none pauses it, and argocd app set blog --sync-policy automated --self-heal --auto-prune resumes it. Make the real fix in Git, then resume.

Rollbacks

Roll back in Git, so Git stays the truth. For an app change, revert in the project repo; for a platform change (quota, AppProject, Argo CD itself), revert in the platform repo:

git log --oneline -- deploy/             # in the project repo: find the bad commit
git revert <sha> && git push             # Argo CD syncs the previous state

argocd app history blog shows what was deployed when. argocd app rollback blog <id> exists but only works with auto-sync off, and Git then disagrees with the cluster; use it only as an emergency brake, then revert in Git and re-enable sync.

What a rollback does not undo: database migrations and data in PVCs. Keep schema changes backward compatible, or back up first.

Image updates

GitLab: CI write-back needs a project access token or write deploy key, and Renovate runs as your own scheduled pipeline; see G7 and G8. Images in GitLab's registry need a pull secret: G10.

Pick one approach per project. They can coexist across projects.

Approach Best for How
Renovate bot Third-party images and Helm charts (most homelab apps) Opens PRs bumping newTag, helmCharts.version, chart targetRevision; you merge or auto-merge patches
CI write-back Your own apps with their own build pipeline App's CI builds the image, then commits the new tag to deploy/ in the same repo
Argo CD Image Updater Fully automatic tracking of a registry Add-on that watches registries and writes new tags back to Git; see its docs for the current config format

Renovate (GitHub App, free). Install it on the platform repo and every project repo, and add a renovate.json at each repo root (or keep one shared preset in the platform repo and use "extends": ["github>YOUR_USER/k3s-platform"] in the others):

{
  "$schema": "https://docs.renovatebot.com/renovate-schema.json",
  "extends": ["config:recommended"],
  "kubernetes": { "managerFilePatterns": ["/(deploy|infrastructure|projects)/.+\\.yaml$/"] },
  "argocd":     { "managerFilePatterns": ["/argocd/.+\\.yaml$/"] },
  "packageRules": [
    { "matchUpdateTypes": ["patch", "digest"], "automerge": true },
    { "matchUpdateTypes": ["major"], "dependencyDashboardApproval": true }
  ]
}

Renovate's kustomize manager picks up images: and helmCharts: in kustomization.yaml automatically. Older Renovate versions call managerFilePatterns fileMatch.

CI write-back in the app's own repo, after pushing ghcr.io/YOUR_USER/blog:$TAG. Since deploy/ lives in the same repo, the workflow can push with its built-in token (grant permissions: contents: write):

cd deploy/overlays/prod
kustomize edit set image ghcr.io/YOUR_USER/blog:${TAG}
git commit -am "deploy: blog ${TAG} [skip ci]" && git push     # or open a PR for prod

Pushes made with the built-in GITHUB_TOKEN don't trigger new workflow runs, and [skip ci] covers other CI systems, so there's no build loop. If the manifests live in a separate blog-deploy repo, use a fine-grained token scoped to that one repo's contents.

Notifications

Argo CD's notifications controller ships with the Helm chart. Configure it in argocd/install/values.yaml so it's GitOps-managed too. Discord via webhook, as an example (Slack, Telegram, ntfy, email and others are built in):

notifications:
  enabled: true
  secret:
    create: false        # supply argocd-notifications-secret as a SealedSecret with key discord-url
  notifiers:
    service.webhook.discord: |
      url: $discord-url
      headers:
        - name: Content-Type
          value: application/json
  templates:
    template.app-sync-failed: |
      webhook:
        discord:
          method: POST
          body: |
            {"content": ":x: {{.app.metadata.name}} sync failed: {{.app.status.operationState.message}}"}
    template.app-degraded: |
      webhook:
        discord:
          method: POST
          body: |
            {"content": ":warning: {{.app.metadata.name}} is Degraded"}
  triggers:
    trigger.on-sync-failed: |
      - when: app.status.operationState.phase in ['Error', 'Failed']
        send: [app-sync-failed]
    trigger.on-health-degraded: |
      - when: app.status.health.status == 'Degraded'
        send: [app-degraded]
  subscriptions:
    - recipients: [discord]
      triggers: [on-sync-failed, on-health-degraded]

Test it by pushing a deliberately broken manifest to a throwaway app folder.

Upgrades

  • Argo CD: bump targetRevision in argocd/argocd.yaml via PR. Read the release notes for major versions first.
  • Charts and images: Renovate PRs.
  • k3s itself: outside Argo CD's scope. Use the system-upgrade-controller (its Plan objects can live in infrastructure/) or upgrade nodes manually. Recheck the Traefik HelmChartConfig afterwards.

Disaster recovery: rebuilding from Git

If a node dies, your recovery is short because everything except data is in Git (the platform repo plus each project repo):

  1. Install k3s (same config file, same disabled components).
  2. kubectl apply -f sealed-secrets-key.backup.yaml (from your password manager).
  3. helm install argocd ... exactly as in Step 3.3, and create the repo credential Secret by hand (Step 3.5).
  4. kubectl apply -f bootstrap/root.yaml.
  5. Restore PVC data from your volume backups.

Rehearse this once on a spare VM. It's the best test that your repo really is the source of truth.

11. Troubleshooting and next steps

Symptom Likely cause Fix
App shows "repository not accessible" / "authentication required" Repo not covered by the GitHub App installation or token, or the URL doesn't match the credential prefix Add the repo to the installation (Step 8.2). URLs must be https://github.com/YOUR_USER/... exactly, not SSH. GitLab: the project must sit inside the group that owns the deploy token, and the URL must end in .git (G2)
root shows ComparisonError A file listed in argocd/kustomization.yaml doesn't exist yet Add the file or comment out the line (Step 3.6)
App error "AppProject blog not found" project-blog hasn't synced yet, or the folder name and project: in config.yaml differ Wait one reconcile; check argocd app get project-blog; make the names match
"application destination ... is not permitted in project" Namespace in config.yaml isn't in the AppProject's destinations Add it to projects/<name>/appproject.yaml
"resource Namespace (or ClusterRole, CRD) is not permitted in project" The project repo contains a cluster-scoped object Remove it; namespaces come from the platform repo, cluster-wide objects go in infrastructure/
New project doesn't appear config.yaml path wrong, YAML not a list, or a missing key (with missingkey=error) kubectl -n argocd logs deploy/argocd-applicationset-controller; argocd app get projects --refresh
Pods rejected with "must specify limits/requests" ResourceQuota active without defaults Keep the LimitRange from Step 4.3, or set requests/limits in the Deployment
Sync fails: "field is immutable" on a Deployment Labels changed the selector Restore the original matchLabels; use includeSelectors: false
helmCharts ignored / "must specify --enable-helm" Build option missing Check kustomize.buildOptions in argocd-cm (Step 3.2)
App flips between Synced and OutOfSync Something else edits the same field (HPA, webhook, k3s) ignoreDifferences for that field, or remove the competing manager
SealedSecret stuck, no Secret created Wrong namespace/name when sealing, or sealed with another cluster's cert kubectl -n sealed-secrets logs deploy/sealed-secrets-controller; reseal
Changes in a project repo take minutes to appear Polling every timeout.reconciliation Add the repo webhook (Step 8.4) or argocd app get <name> --refresh

Your first week, in order

  • [ ] Create the platform repo and bootstrap Argo CD (section 3)
  • [ ] Set up the GitHub App credential template (Step 3.5)
  • [ ] Add the platform AppProject and the three ApplicationSets (section 4)
  • [ ] Install Sealed Secrets, back up its key, and seal the repo credential (section 6)
  • [ ] Set local-path PVs to Retain and add the Prune=false annotations (Step 7.3)
  • [ ] Migrate the least critical namespace into its own repo, then the rest one by one (section 7)
  • [ ] Flip autoPrune to true per project and clean up orphans (Step 7.7)
  • [ ] Add CI validation to the platform repo and each project repo, and install Renovate (Step 8.5, section 10)
  • [ ] Turn on notifications (section 10)
  • [ ] Rehearse a rebuild on a spare VM (section 10)

Where to go later

  • SSO: enable Dex with GitHub OAuth (GitLab: G9), map GitHub teams or GitLab subgroups to the <project>-devs groups, and disable the admin account with admin.enabled: "false".
  • External Secrets Operator if you keep secrets in 1Password, Bitwarden or Vault. Each project then references secrets by name instead of committing ciphertext.
  • Policy: Kyverno in infrastructure/ to enforce resource limits, pinned image tags and allowed registries across all project repos.
  • More clusters: register them with argocd cluster add and add a cluster field to each config.yaml entry, used as the destination.
  • Shared tooling: move new-app.sh, the validation workflow and renovate.json into a template repo, and create new project repos from it with gh repo create --template.

Appendix: GitLab steps

GitHub is the default throughout this guide. If your repos are on GitLab (gitlab.com or self-managed), everything else stays the same: Argo CD, Kustomize, Sealed Secrets, the ApplicationSets and the k3s handling. Use the steps below wherever the guide points here.

Placeholders: gitlab.com/YOUR_GROUP is the GitLab group that holds k3s-platform and your project repos (subgroups work too). For self-managed GitLab, swap in your own host, for example gitlab.home.example.com.

Guide step GitHub (default) GitLab
3.1 Create repos gh repo create glab repo create (G1)
3.5 Repo credential GitHub App credential template Group deploy token credential template (G2)
4.6 SSO groups GitHub teams via Dex GitLab groups via Dex or OIDC (G9)
4.7 Auto-discovery SCM Provider, github: SCM Provider, gitlab: (G3)
8.2 Repo access on onboarding Add repo to the App installation Nothing to do: the group token covers new projects (G5)
8.4 Webhooks Repo webhook Project webhook with a secret token (G4)
8.5 CI validation GitHub Actions .gitlab-ci.yml, shared via include (G6)
10 CI image write-back Built-in GITHUB_TOKEN Project access token or write deploy key (G7)
10 Renovate Hosted GitHub App Self-run Renovate pipeline (G8)
Private images GHCR GitLab Container Registry pull secret (G10)

G1. Create the repos

GitLab calls repos "projects". Install the glab CLI (brew install glab, or the binary from GitLab's releases), then run glab auth login.

glab repo create YOUR_GROUP/k3s-platform --private
git clone https://gitlab.com/YOUR_GROUP/k3s-platform.git && cd k3s-platform
mkdir -p bootstrap argocd/{install,appsets} infrastructure projects scripts

Use https://gitlab.com/YOUR_GROUP/<name>.git URLs, with the .git suffix, everywhere the guide says https://github.com/YOUR_USER/<name>.git. That covers root.yaml, argocd.yaml, the ApplicationSets, AppProjects and config.yaml. Without the suffix, GitLab answers with a redirect and Argo CD logs warnings.

G2. One credential for all repos in a group

Replaces Step 3.5. A group deploy token gives read-only access to every project in the group and its subgroups, including ones created later. It's available on all tiers, including gitlab.com Free.

  1. In GitLab, open the group (YOUR_GROUP), then Settings, Repository, Deploy tokens, Add token.
  2. Name it argocd. Leave the expiry empty, or set one and add a reminder. Tick read_repository only. (Also tick read_registry if you'll use it for image pulls, as in G10, though a separate token is cleaner.)
  3. Copy the generated username (like gitlab+deploy-token-1234567) and the token. GitLab shows the token only once.
  4. Create the credential template:
kubectl -n argocd create secret generic gitlab-creds \
  --from-literal=type=git \
  --from-literal=url=https://gitlab.com/YOUR_GROUP \
  --from-literal=username=gitlab+deploy-token-1234567 \
  --from-literal=password=<deploy-token>
kubectl -n argocd label secret gitlab-creds argocd.argoproj.io/secret-type=repo-creds

argocd repocreds list        # shows https://gitlab.com/YOUR_GROUP

Alternatives:

  • Group access token (scope read_repository, role Reporter). This is a bot user that shows up in audit logs. It needs Premium on gitlab.com but is free on self-managed. It always expires, so add a renewal reminder.
  • Self-managed GitLab with a private CA: add the CA to Argo CD with argocd cert add-tls gitlab.home.example.com --from ca.pem, or set configs.tls.certificates in values.yaml.

To seal it into the platform repo (Step 6.5), use the same kubectl create secret ... --dry-run=client -o yaml | kubectl label ... | kubeseal pipeline with the four GitLab literals above and the name gitlab-creds. In argocd/kustomization.yaml, the file then holds the GitLab credential instead of the GitHub one.

G3. Auto-discover projects in a GitLab group

Replaces the generator in Step 4.7. The same trade-offs apply: one shared AppProject, and anyone who can create a project in the group can deploy.

generators:
  - scmProvider:
      cloneProtocol: https                      # match the HTTPS credential template
      gitlab:
        group: YOUR_GROUP                       # numeric group ID also works
        includeSubgroups: true
        # api: https://gitlab.home.example.com/ # self-managed only
        tokenRef: { secretName: gitlab-scm-token, key: token }
      filters:
        - labelMatch: k3s-app                   # GitLab project topic (Settings, General, Topics)
          pathsExist: [deploy/overlays/prod]
template:
  metadata:
    name: '{{ .repository }}'
  spec:
    project: apps
    source:
      repoURL: '{{ .url }}'
      targetRevision: '{{ .branch }}'
      path: deploy/overlays/prod
    destination:
      server: https://kubernetes.default.svc
      namespace: '{{ .repository }}'
    syncPolicy:
      automated: { prune: true, selfHeal: true }
      syncOptions: [CreateNamespace=true]

The SCM provider calls the GitLab API, which a deploy token can't do. Create a group access token (Premium on gitlab.com) or a personal access token of a bot user, with scope read_api, role Reporter. Store it as gitlab-scm-token in argocd (sealed, as in section 6).

G4. Webhooks for instant syncs

Replaces the GitHub webhook in Step 8.4. GitLab signs webhooks with a shared secret token, so set one in Argo CD first:

# argocd/install/values.yaml
configs:
  secret:
    gitlabSecret: <random string>      # or supply argocd-secret's webhook.gitlab.secret via a SealedSecret

Then, in each project (and the platform repo), go to Settings, Webhooks, Add new webhook:

  • URL: https://argocd.home.example.com/api/webhook
  • Secret token: the same random string
  • Trigger: Push events (all branches, or main)
  • SSL verification: on, if Argo CD has a public certificate

Group-level webhooks (one webhook for every project in the group) need Premium. On Free, add one per project, or rely on the ~3-minute polling. As with GitHub, GitLab must be able to reach your Argo CD URL. Self-managed GitLab on the same LAN can also reach the in-cluster URL; you may need to allow it under Admin, Settings, Network, Outbound requests.

G5. Onboarding a project on GitLab

Replaces the GitHub commands in Steps 8.1 to 8.3. The scripts themselves don't change.

# 8.1 Create the project repo and scaffold deploy/
glab repo create YOUR_GROUP/whoami-deploy --private
git clone https://gitlab.com/YOUR_GROUP/whoami-deploy.git && cd whoami-deploy
~/src/k3s-platform/scripts/new-app.sh whoami traefik/whoami:v1.10.1 80
git add deploy && git commit -m "Add deploy manifests" && git push

# 8.2 Repo access: nothing to do if the project is inside YOUR_GROUP (the group deploy token covers it).
argocd repo add https://gitlab.com/YOUR_GROUP/whoami-deploy.git   # optional check; should report Successful

# 8.3 Register it in the platform repo via a merge request
cd ~/src/k3s-platform
scripts/register-project.sh whoami https://gitlab.com/YOUR_GROUP/whoami-deploy.git
git checkout -b register-whoami && git add projects/whoami && git commit -m "Register whoami"
git push -u origin register-whoami
glab mr create --fill --yes

If a project lives outside YOUR_GROUP, the credential template doesn't match its URL. Add a deploy token on that project and register it as an individual repo with argocd repo add <url> --username ... --password ..., or move the project into the group.

G6. Validate merge requests with GitLab CI

Replaces Step 8.5. Keep the job in the platform repo as ci/validate-deploy.yml:

# k3s-platform/ci/validate-deploy.yml
validate-deploy:
  image:
    name: alpine/k8s:1.31.4          # pin a current tag from Docker Hub; ships kubectl, kustomize and helm
    entrypoint: [""]
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
      changes: [deploy/**/*]
  script:
    - wget -qO- https://github.com/yannh/kubeconform/releases/latest/download/kubeconform-linux-amd64.tar.gz | tar xz -C /usr/local/bin
    - |
      set -e
      for d in deploy/overlays/*; do
        echo "== $d"
        kubectl kustomize --enable-helm "$d" | kubeconform -strict -ignore-missing-schemas -summary
        if kubectl kustomize --enable-helm "$d" | grep -qE '^kind: (Namespace|ClusterRole|ClusterRoleBinding|CustomResourceDefinition)$'; then
          echo "Cluster-scoped object found in $d; these belong in the platform repo"; exit 1
        fi
      done

Each project repo pulls it in with a one-line .gitlab-ci.yml (add your normal build stages alongside):

include:
  - project: YOUR_GROUP/k3s-platform
    ref: main
    file: ci/validate-deploy.yml

Cross-project include works when the user running the pipeline can read k3s-platform, which is true for members of the group. For the platform repo itself, add a similar job looping over infrastructure/* and projects/*.

G7. CI image write-back

Replaces the CI write-back in section 10. Unlike GitHub's GITHUB_TOKEN, GitLab's CI_JOB_TOKEN can't push to the repo by default, so give the pipeline a dedicated token:

  1. In the app's project: Settings, Access tokens, add a project access token. Role: Developer (Maintainer if main is protected against Developer pushes). Scope: write_repository.
  2. Settings, CI/CD, Variables: add it as DEPLOY_PUSH_TOKEN, Masked and Protected.

Project access tokens are free on self-managed GitLab but need Premium on gitlab.com. On gitlab.com Free, use a deploy key with write access (Settings, Repository, Deploy keys, tick "Grant write permissions") loaded through an SSH-key CI variable, or a bot user's personal access token.

# .gitlab-ci.yml in the app's project, after the job that pushes the image
deploy-tag:
  stage: deploy
  image: alpine/k8s:1.31.4             # same pinned image as G6
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH && $CI_COMMIT_MESSAGE !~ /\[skip ci\]/
  script:
    - git config user.name  "GitLab CI"
    - git config user.email "[email protected]"
    - cd deploy/overlays/prod
    - kustomize edit set image $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA
    - git commit -am "deploy: $CI_PROJECT_NAME $CI_COMMIT_SHORT_SHA [skip ci]"
    - git push "https://ci:${DEPLOY_PUSH_TOKEN}@${CI_SERVER_HOST}/${CI_PROJECT_PATH}.git" HEAD:$CI_DEFAULT_BRANCH -o ci.skip

-o ci.skip and [skip ci] stop the push from starting another pipeline. If deploy/ lives in a separate -deploy project, create the token there instead, and clone that project before running kustomize edit.

G8. Renovate on GitLab

There's no hosted Renovate app for GitLab, so you run it yourself on a schedule.

  1. Create a project such as YOUR_GROUP/renovate-runner.
  2. Create a bot token with scopes api, read_user and write_repository: a group access token (Premium on gitlab.com), or a personal access token of a bot user. Add it as CI/CD variable RENOVATE_TOKEN (masked). Also add a read-only GitHub personal access token as GITHUB_COM_TOKEN, so Renovate can read changelogs and avoid rate limits.
  3. On gitlab.com, add this .gitlab-ci.yml:
include:
  - project: renovate-bot/renovate-runner
    file: /templates/renovate.gitlab-ci.yml

variables:
  RENOVATE_EXTRA_FLAGS: "--autodiscover=true --autodiscover-filter=YOUR_GROUP/*"

On self-managed GitLab, mirror or copy that template, and set RENOVATE_ENDPOINT: https://gitlab.home.example.com/api/v4 and RENOVATE_PLATFORM: gitlab.

  1. Under CI/CD, Pipeline schedules, run it hourly or nightly.

The renovate.json from section 10 works unchanged. For a shared preset, put it in the platform repo and use "extends": ["gitlab>YOUR_GROUP/k3s-platform"] in each project.

G9. SSO and RBAC groups

For Step 4.6, Dex has a GitLab connector. In argocd/install/values.yaml, set dex.enabled: true and add the connector:

configs:
  cm:
    url: https://argocd.home.example.com
    dex.config: |
      connectors:
        - type: gitlab
          id: gitlab
          name: GitLab
          config:
            baseURL: https://gitlab.com            # or your self-managed URL
            clientID: $dex.gitlab.clientID          # from argocd-secret
            clientSecret: $dex.gitlab.clientSecret
            groups: [YOUR_GROUP]                     # only members of these groups can log in

Create the OAuth application in the group: Settings, Applications. Use the redirect URI https://argocd.home.example.com/api/dex/callback and scopes read_user and openid. GitLab groups arrive as group names, so subgroups such as YOUR_GROUP/blog-devs can be used directly in each AppProject's groups: list and in policy.csv.

G10. Pulling images from the GitLab Container Registry

If your images live in registry.gitlab.com/YOUR_GROUP/<project>, each namespace needs a pull secret. Create a group deploy token with scope read_registry only, then seal one pull secret per project into its repo:

kubectl create secret docker-registry gitlab-registry -n blog \
  --docker-server=registry.gitlab.com \
  --docker-username=gitlab+deploy-token-7654321 \
  --docker-password=<read_registry token> \
  --dry-run=client -o yaml \
| kubeseal --format yaml > deploy/overlays/prod/gitlab-registry.sealed.yaml

Add the file to the overlay's resources, and reference it from the Deployment:

spec:
  template:
    spec:
      imagePullSecrets:
        - name: gitlab-registry

Alternatively, configure it once per node in k3s's /etc/rancher/k3s/registries.yaml (an auth entry for registry.gitlab.com). That's simpler, but it lives outside Git and applies to every namespace.