GitOps on k3s: Getting-Started Guide (Multi-Repo)¶
Part of the Git Operations (GitOps) section. For the concepts behind this guide, see GitOps: A Comprehensive Guide. The single-repo version is the single-repo guide, and the single vs multi-repo comparison weighs the two.
Commands default to GitHub. If you use GitLab, every step that differs links to the GitLab appendix at the end.
Overview and assumptions¶
By the end of this guide, a platform repo will describe your cluster: Argo CD itself, cluster-wide services, and a short registration for each project. Each project has its own repo holding its Kubernetes manifests. Argo CD watches all of them and keeps the cluster matching Git. Nobody runs kubectl apply by hand any more.
Onboarding a project becomes: create its repo with a deploy/ folder, add a small registration folder to the platform repo, merge. The platform repo also decides what each project is allowed to do, so a project repo can never deploy outside its own namespace.
What you will build
- A platform repo,
k3s-platform, holding Argo CD's config, cluster services, and one registration folder per project. - One repo per project (for example
blog,media-deploy), each withdeploy/baseanddeploy/overlays/<env>. - Argo CD installed on k3s, managing itself from the platform repo.
- One credential template, so Argo CD can read every repo you own without per-repo setup.
- A per-project AppProject, namespace and quota, plus Argo CD RBAC, owned by the platform repo.
- ApplicationSets that turn each registration into an Argo CD Application pointing at that project's repo.
- Encrypted secrets, a no-downtime migration of your current workloads, and day-2 habits (drift, rollbacks, image updates, notifications).
Assumptions (adapt where yours differ)
| Assumption | If yours differs |
|---|---|
| Single k3s cluster, 1 to 3 nodes, recent k3s (v1.30+) | Multi-cluster works too; see Next steps |
Bundled Traefik is your ingress, local-path is your storage class |
Swap the Ingress and storageClassName values |
| Repos on GitHub under one account or org, all private | GitLab: one group holds all repos; see the GitLab appendix. Gitea or Forgejo: use a token credential template |
Repo URLs use HTTPS (https://github.com/YOUR_USER/<repo>.git) |
Required for the GitHub App credential in Step 3.5; SSH works only with a machine-user key |
You have kubectl, helm, git and ideally gh locally, with cluster-admin kubeconfig |
Run commands from a node with k3s kubectl instead |
| Workloads are currently plain YAML and/or Helm releases applied by hand | Section 7 covers both |
Placeholder names: platform repo k3s-platform, domain home.example.com, projects blog and media |
Search and replace with your own |
Time budget: about 1.5 hours to bootstrap, then roughly 20 to 30 minutes per existing project you migrate.
1. Choose a tool: use Argo CD¶
For a k3s cluster with several projects and more coming, I recommend Argo CD. Flux is an equally solid choice; the deciding factors for you are visibility and multi-project ergonomics.
| Argo CD | Flux | |
|---|---|---|
| Web UI | Built in: live resource tree, diffs, sync/rollback buttons | None built in (third-party UIs like Capacitor or Weave GitOps) |
| Many projects | ApplicationSets generate one app per registered repo; AppProjects fence each project to its repo and namespace | Kustomization per folder; tenancy via service-account impersonation |
| Resource cost | Roughly 400 to 700 MiB RAM in a small install | Roughly 150 to 300 MiB; lighter |
| Helm | Renders charts then applies (no Helm release secrets) | Native helm install/upgrade via helm-controller |
| Secrets | Sealed Secrets is simplest; SOPS needs a plugin (KSOPS) | SOPS decryption built in |
| Image auto-update | Argo CD Image Updater (add-on) | Built in (image-reflector + image-automation) |
| Bootstrap | helm install, then hand it its own config |
flux bootstrap CLI does it all in one command |
Why Argo CD here: with a growing number of namespaces, the UI pays for itself. You see every project's health on one page, you see drift as a diff, and you can roll back with a click while learning. ApplicationSets and per-project AppProjects make "new repo + registration = new app" a one-time setup.
When to pick Flux instead: the cluster is very memory-constrained (Raspberry Pi with 2 GB), you prefer a CLI-only workflow, or you want SOPS without plugins. The multi-repo idea transfers directly (Flux GitRepository per project repo plus tenant service accounts); Flux equivalents are noted where they differ.
Don't run both. Pick one controller per cluster to avoid two reconcilers fighting over the same objects.
2. Design the repos: one platform repo, one repo per project¶
Split by who owns what. The platform repo holds everything cluster-wide and every guardrail. Each project repo holds only that project's namespaced manifests. The rule that keeps automation simple: project name = Argo CD AppProject name = namespace.
The platform repo (k3s-platform)¶
k3s-platform/
├── bootstrap/
│ └── root.yaml # the ONLY thing you ever kubectl apply
├── argocd/ # Argo CD manages its own config from here
│ ├── install/values.yaml # Helm values for Argo CD (incl. RBAC)
│ ├── argocd.yaml # Application: Argo CD upgrades itself
│ ├── appproject-platform.yaml # AppProject for platform-owned apps
│ ├── repo-creds.sealed.yaml # credential template for all your repos (section 6)
│ ├── appsets/
│ │ ├── infrastructure.yaml # one app per infrastructure/*
│ │ ├── project-config.yaml # one app per projects/* (AppProject, namespace, quota)
│ │ └── projects.yaml # one app per entry in projects/*/config.yaml
│ └── kustomization.yaml
├── infrastructure/ # cluster-wide services
│ ├── sealed-secrets/
│ ├── cert-manager/
│ ├── traefik-config/
│ └── storage/
├── projects/ # one folder per registered project
│ ├── blog/
│ │ ├── config.yaml # registration: repo URL, path, sync options (read by an ApplicationSet)
│ │ ├── kustomization.yaml
│ │ ├── appproject.yaml # what blog may deploy, from which repo, and who may sync it
│ │ ├── namespace.yaml
│ │ └── quota.yaml # ResourceQuota + LimitRange
│ └── media/ ...
└── scripts/register-project.sh # Step 8.3
A project repo (blog)¶
blog/ # app source + deploy/ (or a separate blog-deploy repo)
├── src/ ...
├── Dockerfile
├── .github/workflows/
└── deploy/
├── base/
│ ├── kustomization.yaml
│ ├── deployment.yaml
│ ├── service.yaml
│ └── ingress.yaml
└── overlays/
├── prod/
│ ├── kustomization.yaml
│ └── sealed-secret.yaml
└── staging/ # later
Who owns what¶
| Lives in | Contents | Changed by |
|---|---|---|
Platform repo argocd/, infrastructure/ |
Argo CD, cert-manager, Sealed Secrets, Traefik config, storage classes, CRDs | You, as platform admin |
Platform repo projects/<name>/ |
Registration, AppProject, namespace, quota, RBAC | You, on each onboarding |
Project repo deploy/ |
Deployments, Services, Ingress, ConfigMaps, SealedSecrets, image tags | The project's owner |
The namespace is deliberately created by the platform repo, not the project repo. The project's AppProject forbids cluster-scoped resources, so a project repo can't create namespaces, ClusterRoles or CRDs, or deploy into another project's namespace.
Decisions to make now¶
deploy/inside the app repo, or a separate<name>-deployrepo? For your own apps, putdeploy/in the app repo: one PR can change code and manifests together, and history stays in one place. Use a separate-deployrepo when CI-driven tag bumps would cause noisy build loops, when different people should control code and deployment, or for third-party software with no source repo (for examplemedia-deployholding a Jellyfin Helm chart).- Fixed path convention: every project repo uses
deploy/overlays/<env>. Registration then needs no special cases. - Branching: deploy from
mainonly, use PRs for review, and use folders per environment rather than branches. - Naming: lowercase, DNS-safe project names (
home-assistant, notHomeAssistant), since they become namespace and AppProject names. - GitHub user vs organization: a personal account works. A free GitHub organization makes access control and the auto-discovery option in Step 4.7 easier.
Flux equivalent: the platform repo becomes the Flux "fleet" repo, with a GitRepository and Kustomization per project repo. Flux's multi-tenancy model (flux create tenant, serviceAccountName on each Kustomization) plays the role of AppProjects.
3. Bootstrap Argo CD from the platform repo¶
You install Argo CD once with Helm, give it one credential that covers all your repos, then apply one root Application from the platform repo. From then on, Argo CD manages itself and every project from Git.
Step 3.1: Create the platform repo¶
GitLab: use
glab repo createandhttps://gitlab.com/YOUR_GROUP/<name>.gitURLs everywhere; see G1.
gh repo create YOUR_USER/k3s-platform --private --clone # or create it in the GitHub UI and clone
cd k3s-platform
mkdir -p bootstrap argocd/{install,appsets} infrastructure projects scripts
Step 3.2: Write Argo CD's Helm values¶
argocd/install/values.yaml:
global:
domain: argocd.home.example.com
configs:
params:
server.insecure: true # Traefik terminates TLS, Argo serves plain HTTP behind it
cm:
application.resourceTrackingMethod: annotation # safer than labels; default in v3, set it explicitly
kustomize.buildOptions: --enable-helm # lets Kustomize inflate Helm charts (any repo)
timeout.reconciliation: 180s # how often Git is polled
exec.enabled: false
rbac:
policy.default: role:readonly # tightened in Step 4.6
# The repo credential is added in Step 3.5, not here, so it never sits in a values file.
server:
ingress:
enabled: true
ingressClassName: traefik
tls: true
# annotations:
# cert-manager.io/cluster-issuer: letsencrypt # once cert-manager is in place
dex:
enabled: false # turn on later for GitHub/OIDC SSO
controller:
resources:
requests: { cpu: 100m, memory: 256Mi }
repoServer:
resources:
requests: { cpu: 50m, memory: 128Mi } # renders every repo; raise it as projects grow
Step 3.3: Install it¶
helm repo add argo https://argoproj.github.io/argo-helm
helm repo update
helm search repo argo/argo-cd --versions | head -5 # pick the latest and PIN it
export ARGO_CHART_VERSION=<chart version from above>
kubectl create namespace argocd
helm install argocd argo/argo-cd -n argocd \
--version "$ARGO_CHART_VERSION" \
-f argocd/install/values.yaml
kubectl -n argocd rollout status deploy/argocd-server
Step 3.4: Log in and secure the admin account¶
kubectl -n argocd get secret argocd-initial-admin-secret \
-o jsonpath='{.data.password}' | base64 -d; echo
kubectl -n argocd port-forward svc/argocd-server 8080:80 # until DNS/TLS is sorted
argocd login localhost:8080 --plaintext --username admin
argocd account update-password
kubectl -n argocd delete secret argocd-initial-admin-secret
Step 3.5: One credential for all your repos¶
GitLab: use a group deploy token as the credential template instead of a GitHub App; see G2.
With many repos, give Argo CD a credential template: a Secret whose URL is a prefix (https://github.com/YOUR_USER). Every repo URL starting with that prefix is authenticated automatically, so a new project repo needs no Argo CD change.
Don't use deploy keys here. GitHub ties each deploy key to a single repo, so you'd need one key and one Secret per project.
Recommended: a GitHub App (read-only, no expiry, revocable per repo)
- GitHub, then Settings, Developer settings, GitHub Apps, New GitHub App. Name it something like
argocd-home. Untick Webhook "Active". Under Repository permissions, set Contents: Read-only (Metadata: Read-only is added automatically). Create it and note the App ID. - On the app's page, generate a private key (a
.pemfile downloads). - Install App, choose your account, then either All repositories (zero onboarding work) or Only select repositories (pick
k3s-platformplus each project repo; you'll add new repos here on onboarding). The Installation ID is the number at the end of the installation's settings URL. - Create the credential template in the cluster:
kubectl -n argocd create secret generic github-creds \
--from-literal=type=git \
--from-literal=url=https://github.com/YOUR_USER \
--from-literal=githubAppID=<APP_ID> \
--from-literal=githubAppInstallationID=<INSTALLATION_ID> \
--from-file=githubAppPrivateKey=$HOME/.secrets/argocd-home.pem
kubectl -n argocd label secret github-creds argocd.argoproj.io/secret-type=repo-creds
argocd repocreds list # shows https://github.com/YOUR_USER
Alternative: a fine-grained personal access token. Scope it to the repos Argo CD needs, with Contents read-only. Create the same Secret with --from-literal=username=git --from-literal=password=github_pat_... instead of the three githubApp* keys. Tokens expire, so put a renewal reminder in your calendar.
The credential lives only in the cluster for now. Step 6.5 seals it into the platform repo so a rebuild can restore it.
Step 3.6: Let Argo CD manage itself¶
argocd/argocd.yaml:
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: argocd
namespace: argocd
spec:
project: default
sources:
- repoURL: https://argoproj.github.io/argo-helm
chart: argo-cd
targetRevision: <same as ARGO_CHART_VERSION>
helm:
releaseName: argocd
valueFiles:
- $values/argocd/install/values.yaml
- repoURL: https://github.com/YOUR_USER/k3s-platform.git
targetRevision: main
ref: values
destination:
server: https://kubernetes.default.svc
namespace: argocd
syncPolicy:
automated:
prune: false # never let Argo prune itself by accident
selfHeal: true
syncOptions:
- ServerSideApply=true
argocd/kustomization.yaml (the AppProject and ApplicationSets are written in section 4; write them before you push in Step 3.7, or comment those lines out until then):
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- argocd.yaml
- appproject-platform.yaml
- appsets/infrastructure.yaml
- appsets/project-config.yaml
- appsets/projects.yaml
# - repo-creds.sealed.yaml # uncomment in Step 6.5
Step 3.7: Apply the root Application (the last manual apply)¶
bootstrap/root.yaml:
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: root
namespace: argocd
# Deliberately NO resources-finalizer: deleting root must not cascade-delete the cluster
spec:
project: default
source:
repoURL: https://github.com/YOUR_USER/k3s-platform.git
targetRevision: main
path: argocd
destination:
server: https://kubernetes.default.svc
namespace: argocd
syncPolicy:
automated:
prune: true
selfHeal: true
git add . && git commit -m "Bootstrap Argo CD" && git push
kubectl apply -f bootstrap/root.yaml
argocd app list # root and argocd should go Synced / Healthy
4. Wire up projects: AppProjects, registrations, ApplicationSets and RBAC¶
The platform repo runs three ApplicationSets. One installs cluster services. One applies each project's guardrails (AppProject, namespace, quota). One reads each project's config.yaml and creates an Application that pulls manifests from that project's own repo.
root (k3s-platform: bootstrap/root.yaml)
├── argocd Argo CD manages itself
├── infrastructure (AppSet) one app per infrastructure/* → sealed-secrets, cert-manager, ...
├── project-config (AppSet) one app per projects/* → AppProject + namespace + quota per project
└── projects (AppSet) one app per projects/*/config.yaml entry
├── blog ← https://github.com/YOUR_USER/blog.git deploy/overlays/prod
├── media ← https://github.com/YOUR_USER/media-deploy.git deploy/overlays/prod
└── (next project ← its own repo)
Step 4.1: The platform AppProject¶
argocd/appproject-platform.yaml covers apps you own: cluster services and per-project guardrails. It has broad rights, but only from the platform repo and upstream chart repos.
apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
name: platform
namespace: argocd
spec:
description: Cluster services and project guardrails from k3s-platform
sourceRepos:
- https://github.com/YOUR_USER/k3s-platform.git
- https://* # upstream Helm repos used by infrastructure/ (tighten to a list later)
destinations:
- server: https://kubernetes.default.svc
namespace: '*'
clusterResourceWhitelist:
- group: '*'
kind: '*'
Step 4.2: ApplicationSet for infrastructure¶
argocd/appsets/infrastructure.yaml:
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
name: infrastructure
namespace: argocd
spec:
goTemplate: true
goTemplateOptions: ["missingkey=error"]
generators:
- git:
repoURL: https://github.com/YOUR_USER/k3s-platform.git
revision: main
directories:
- path: infrastructure/*
template:
metadata:
name: '{{ .path.basename }}'
labels: { area: infrastructure }
spec:
project: platform
source:
repoURL: https://github.com/YOUR_USER/k3s-platform.git
targetRevision: main
path: '{{ .path.path }}'
destination:
server: https://kubernetes.default.svc
namespace: '{{ .path.basename }}'
syncPolicy:
automated: { prune: true, selfHeal: true }
syncOptions:
- CreateNamespace=true
- ServerSideApply=true
- SkipDryRunOnMissingResource=true # CRDs may arrive in the same sync
retry:
limit: 5
backoff: { duration: 15s, factor: 2, maxDuration: 5m }
syncPolicy:
preserveResourcesOnDeletion: true
Step 4.3: Register a project in the platform repo¶
Each project gets a folder under projects/. Step 8.3 has a script that writes all of these; here's what it produces for blog.
projects/blog/config.yaml is the registration. It's read by the projects ApplicationSet, never applied to the cluster. It's a list, one entry per environment:
- project: blog # AppProject name
name: blog # Argo CD Application name
namespace: blog
repoURL: https://github.com/YOUR_USER/blog.git
sourcePath: deploy/overlays/prod
targetRevision: main
autoPrune: true # false while migrating (section 7)
projects/blog/appproject.yaml is the guardrail. Blog may deploy only from its own repo, only into its own namespace, and only namespaced resources:
apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
name: blog
namespace: argocd # AppProjects always live in argocd
spec:
description: blog project
sourceRepos:
- https://github.com/YOUR_USER/blog.git
destinations:
- server: https://kubernetes.default.svc
namespace: blog
clusterResourceWhitelist: [] # nothing cluster-scoped: the platform repo creates the namespace
namespaceResourceBlacklist: # quotas are the platform's call, not the project's
- group: ''
kind: ResourceQuota
- group: ''
kind: LimitRange
orphanedResources:
warn: true # flags objects in the namespace that aren't in Git
roles:
- name: developer
description: View, sync and read logs for blog apps
policies:
- p, proj:blog:developer, applications, get, blog/*, allow
- p, proj:blog:developer, applications, sync, blog/*, allow
- p, proj:blog:developer, logs, get, blog/*, allow
groups:
- blog-devs # SSO group name, once SSO is on (Step 4.6)
projects/blog/namespace.yaml and projects/blog/quota.yaml:
apiVersion: v1
kind: Namespace
metadata:
name: blog
labels:
pod-security.kubernetes.io/enforce: baseline
---
apiVersion: v1
kind: ResourceQuota
metadata: { name: quota, namespace: blog }
spec:
hard:
requests.cpu: "2"
requests.memory: 4Gi
limits.memory: 8Gi
persistentvolumeclaims: "5"
---
apiVersion: v1
kind: LimitRange
metadata: { name: defaults, namespace: blog }
spec:
limits:
- type: Container # pods without requests/limits get these, so the quota doesn't reject them
defaultRequest: { cpu: 50m, memory: 64Mi }
default: { memory: 256Mi }
(Put the Namespace in namespace.yaml and the other two in quota.yaml.)
projects/blog/kustomization.yaml lists what gets applied. config.yaml is intentionally left out:
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- namespace.yaml
- appproject.yaml
- quota.yaml
Step 4.4: ApplicationSet for project guardrails¶
argocd/appsets/project-config.yaml creates one project-<name> Application per folder under projects/, which applies that folder's AppProject, namespace and quota:
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
name: project-config
namespace: argocd
spec:
goTemplate: true
goTemplateOptions: ["missingkey=error"]
generators:
- git:
repoURL: https://github.com/YOUR_USER/k3s-platform.git
revision: main
directories:
- path: projects/*
template:
metadata:
name: 'project-{{ .path.basename }}'
labels: { area: project-config }
spec:
project: platform
source:
repoURL: https://github.com/YOUR_USER/k3s-platform.git
targetRevision: main
path: '{{ .path.path }}'
destination:
server: https://kubernetes.default.svc
namespace: '{{ .path.basename }}'
syncPolicy:
automated: { prune: true, selfHeal: true }
syncPolicy:
preserveResourcesOnDeletion: true # removing a project folder won't delete its namespace by surprise
Step 4.5: ApplicationSet that points at each project repo¶
argocd/appsets/projects.yaml uses the git files generator. Every entry in every projects/*/config.yaml becomes one Application, in that project's AppProject, sourced from that project's repo:
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
name: projects
namespace: argocd
spec:
goTemplate: true
goTemplateOptions: ["missingkey=error"] # a typo in config.yaml fails loudly instead of deploying wrong
generators:
- git:
repoURL: https://github.com/YOUR_USER/k3s-platform.git
revision: main
files:
- path: projects/*/config.yaml
template:
metadata:
name: '{{ .name }}'
labels:
project: '{{ .project }}'
spec:
project: '{{ .project }}'
source:
repoURL: '{{ .repoURL }}'
targetRevision: '{{ .targetRevision }}'
path: '{{ .sourcePath }}'
destination:
server: https://kubernetes.default.svc
namespace: '{{ .namespace }}'
syncPolicy:
syncOptions:
- PruneLast=true
retry:
limit: 3
backoff: { duration: 10s, factor: 2, maxDuration: 2m }
templatePatch: |
spec:
syncPolicy:
automated:
prune: {{ .autoPrune }}
selfHeal: true
syncPolicy:
preserveResourcesOnDeletion: true
Two details matter here:
- The registration key is
sourcePath, notpath. The files generator already injects its ownpathparameters (the location ofconfig.yaml), and a key namedpathwould collide with them. templatePatchlets each project chooseautoPrunein its registration. Plain template fields can't render a boolean from a parameter.
Ordering: a new project's Application can appear a few seconds before its AppProject exists. It shows an error briefly, then recovers on the next reconcile. Within one project, order resources with argocd.argoproj.io/sync-wave annotations as usual.
Step 4.6: Argo CD RBAC (who can see and sync what)¶
GitLab: GitLab groups and subgroups can be the SSO groups; see G9.
Git permissions decide who can change a project: whoever can merge to blog's main can deploy blog. Argo CD RBAC decides who can see apps, trigger syncs and read logs in the UI and CLI. Configure it in argocd/install/values.yaml:
configs:
cm:
accounts.alice: login # a local user; set the password: argocd account update-password --account alice
rbac:
policy.default: '' # no access unless granted below
scopes: '[groups]'
policy.csv: |
# Platform admins
g, admin, role:admin
# A blog developer: see, sync and read logs for blog only
p, role:blog-dev, applications, get, blog/*, allow
p, role:blog-dev, applications, sync, blog/*, allow
p, role:blog-dev, logs, get, blog/*, allow
g, alice, role:blog-dev
Once SSO is on (Dex with GitHub, or any OIDC provider), skip local accounts. Put people in a group such as blog-devs and the AppProject role from Step 4.3 grants them access automatically.
For CI that needs to trigger or wait for syncs, mint a token scoped to one project:
argocd proj role create-token blog developer --expires-in 90d
Step 4.7: Alternative, auto-discover project repos (SCM Provider generator)¶
GitLab: use the
gitlab:SCM provider for a group; see G3.
If you'd rather not register projects at all, Argo CD can scan a GitHub organization and create an app for every repo with a given topic and a deploy/overlays/prod folder:
generators:
- scmProvider:
github:
organization: YOUR_ORG
tokenRef: { secretName: github-scm-token, key: token }
filters:
- labelMatch: k3s-app # GitHub repo topic
pathsExist: [deploy/overlays/prod]
template:
metadata:
name: '{{ .repository }}'
spec:
project: apps # one shared AppProject
source:
repoURL: '{{ .url }}'
targetRevision: '{{ .branch }}'
path: deploy/overlays/prod
destination:
server: https://kubernetes.default.svc
namespace: '{{ .repository }}'
syncPolicy:
automated: { prune: true, selfHeal: true }
syncOptions: [CreateNamespace=true]
The trade-off: onboarding is just adding a topic, but every project shares one AppProject (no per-project repo or namespace fence), anyone who can create a repo in the org can deploy, and the generator uses a GitHub API token with rate limits. For a homelab with a handful of projects, explicit registration (Steps 4.3 to 4.5) is the safer default.
5. Environments and overlays inside each project repo¶
Each project repo uses Kustomize base/ + overlays/<env>/ under deploy/. Third-party software is inflated from Helm charts with Kustomize, in either a project repo or the platform repo's infrastructure/.
Step 5.1: A base for one project (in the blog repo)¶
deploy/base/kustomization.yaml:
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- deployment.yaml
- service.yaml
- ingress.yaml
labels:
- pairs:
app.kubernetes.io/name: blog
app.kubernetes.io/part-of: blog
includeSelectors: false
No namespace.yaml here: the platform repo owns the namespace, and blog's AppProject would reject one.
deploy/base/deployment.yaml (trimmed):
apiVersion: apps/v1
kind: Deployment
metadata:
name: blog
spec:
replicas: 1
selector:
matchLabels: { app: blog }
template:
metadata:
labels: { app: blog }
spec:
containers:
- name: blog
image: ghcr.io/YOUR_USER/blog:1.4.2 # always a pinned tag, never :latest
ports: [{ containerPort: 8080 }]
envFrom:
- secretRef: { name: blog-secrets }
resources:
requests: { cpu: 50m, memory: 64Mi }
limits: { memory: 256Mi }
Step 5.2: The prod overlay¶
deploy/overlays/prod/kustomization.yaml:
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: blog
resources:
- ../../base
- sealed-secret.yaml # section 6
images:
- name: ghcr.io/YOUR_USER/blog
newTag: 1.4.2 # the one line that changes on each release
patches:
- target: { kind: Ingress, name: blog }
patch: |
- op: replace
path: /spec/rules/0/host
value: blog.home.example.com
Preview from the project repo before pushing:
kubectl kustomize deploy/overlays/prod | less
kubectl kustomize deploy/overlays/prod | kubectl diff -f - # vs the live cluster
Step 5.3: Adding staging later¶
- In the project repo, create
deploy/overlays/staging/withnamespace: blog-staging, a different host, and fewer replicas. - In the platform repo, add the namespace to
projects/blog/namespace.yaml(plus quota if you want), addblog-stagingto the AppProject'sdestinations, and add a second entry toprojects/blog/config.yaml:
- project: blog
name: blog-staging
namespace: blog-staging
repoURL: https://github.com/YOUR_USER/blog.git
sourcePath: deploy/overlays/staging
targetRevision: main
autoPrune: true
The prod app keeps its name, so nothing is recreated. Promotion is a PR in the project repo copying the image tag from overlays/staging to overlays/prod.
Step 5.4: Third-party software via Helm¶
Keep chart version and values in Git and let Kustomize render them. For cluster services this lives in the platform repo, for example infrastructure/cert-manager/kustomization.yaml:
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: cert-manager
helmCharts:
- name: cert-manager
repo: https://charts.jetstack.io
version: v1.x.y # pin; Renovate can bump it (section 10)
releaseName: cert-manager
namespace: cert-manager
includeCRDs: true
valuesInline:
crds:
enabled: true
resources:
- cluster-issuer.yaml
For a third-party app that's a project in its own right (Jellyfin, Nextcloud), put the same helmCharts block in that project's repo, for example media-deploy/deploy/overlays/prod/kustomization.yaml. Two caveats:
- The chart must not create cluster-scoped objects (CRDs, ClusterRoles), or the project's AppProject will block the sync. Disable them in values, or treat that software as infrastructure in the platform repo.
- Upstream chart repos must be allowed. Add
https://*(or the specific chart repo URL) to that project's AppProjectsourceRepos. Kustomize fetches the chart itself, but listing it documents the dependency and is required if you ever switch to a multi-source Application.
| You need | Use |
|---|---|
| Your own app, a few manifests | Plain YAML in the project repo's deploy/base + overlay |
| An upstream app chart with modest values | Kustomize helmCharts in that project's repo |
| A cluster-wide service or anything with CRDs | Kustomize helmCharts in the platform repo's infrastructure/ |
| Same app, slightly different per env | Kustomize patches in each overlay |
6. Secrets: Sealed Secrets (recommended with Argo CD)¶
Plain Secrets must never land in Git. With Argo CD, Sealed Secrets is the least-friction option: you encrypt locally with a public key, commit the ciphertext, and only the controller in your cluster can decrypt it. No Argo CD plugins are needed.
| Option | How it works | Pick it when |
|---|---|---|
| Sealed Secrets | Controller holds a private key; kubeseal encrypts with its public cert |
Default for Argo CD on a single cluster |
| SOPS + age | Files encrypted with an age key; decrypted at render time | You use Flux (built in), or want to edit encrypted YAML in place; with Argo CD it needs the KSOPS plugin |
| External Secrets Operator | Pulls from 1Password, Bitwarden, Vault, cloud KMS | You already keep secrets in a password manager or vault; good later step |
Step 6.1: Install the controller from Git¶
In the platform repo, infrastructure/sealed-secrets/kustomization.yaml:
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: sealed-secrets
helmCharts:
- name: sealed-secrets
repo: https://bitnami-labs.github.io/sealed-secrets
version: 2.x.y # pin the latest
releaseName: sealed-secrets
namespace: sealed-secrets
includeCRDs: true
valuesInline:
fullnameOverride: sealed-secrets-controller
Push, wait for the sealed-secrets app to go Healthy, then install the CLI (brew install kubeseal, or the binary from the GitHub releases page) and save yourself some typing:
alias kubeseal='kubeseal --controller-namespace sealed-secrets --controller-name sealed-secrets-controller'
Step 6.2: Back up the sealing key (do this today)¶
If you rebuild the cluster without this key, every sealed secret in Git becomes unreadable.
kubectl -n sealed-secrets get secret \
-l sealedsecrets.bitnami.com/sealed-secrets-key -o yaml > sealed-secrets-key.backup.yaml
# Store it in your password manager or offline. NEVER commit it.
# On a rebuilt cluster: kubectl apply -f sealed-secrets-key.backup.yaml BEFORE the controller starts (or restart it after)
The controller rotates in a new key every 30 days by default and keeps the old ones, so re-run the backup occasionally.
Step 6.3: Seal a secret (in the project repo)¶
Project owners seal their own secrets and commit them to their own repo. They need the cluster's public cert, not cluster access (see the first bullet below).
kubectl create secret generic blog-secrets -n blog \
--from-literal=DB_PASSWORD='s3cr3t' \
--from-literal=API_KEY='abc123' \
--dry-run=client -o yaml \
| kubeseal --format yaml > deploy/overlays/prod/sealed-secret.yaml
git add deploy/overlays/prod/sealed-secret.yaml
git commit -m "blog: add secrets" && git push
By default a sealed secret is bound to its name and namespace: copying it to blog-staging won't decrypt. Seal per environment, which is what you want.
Useful extras:
- Seal offline / in CI:
kubeseal --fetch-cert > pub-cert.pem, commit the cert to the platform repo (it's public), then anyone with read access can runkubeseal --cert pub-cert.pem ...without cluster access. - Update one key:
echo -n newvalue | kubeseal --raw --namespace blog --name blog-secrets --from-file=/dev/stdin, then paste intospec.encryptedData. - Existing secrets in the cluster:
kubectl get secret blog-secrets -n blog -o yaml | kubeseal --format yaml > ...(stripresourceVersion/uidfirst or use thekubectl neatplugin).
Step 6.4: Guard rails against leaks¶
# .gitignore
*.backup.yaml
*.dec.yaml
*-unsealed.yaml
.env
Put this .gitignore in the platform repo and every project repo. Add a secret scanner such as gitleaks as a pre-commit hook or CI check. If a plain secret is ever pushed, rotate the credential; rewriting history isn't enough.
SOPS path (if you prefer it): generate a key with age-keygen -o age.key, add a .sops.yaml with creation_rules: [{path_regex: .*\.sops\.yaml$, encrypted_regex: ^(data|stringData)$, age: <public key>}], encrypt with sops -e -i secret.sops.yaml, store age.key as a Secret for Flux (decryption.provider: sops) or for the KSOPS sidecar in Argo CD's repo-server.
Step 6.5: Seal the repo credential into the platform repo¶
GitLab: seal the
gitlab-credsSecret from G2 the same way, with its four GitLab values.
The credential template from Step 3.5 exists only in the cluster. Seal it so a rebuild restores it from Git:
kubectl -n argocd create secret generic github-creds \
--from-literal=type=git \
--from-literal=url=https://github.com/YOUR_USER \
--from-literal=githubAppID=<APP_ID> \
--from-literal=githubAppInstallationID=<INSTALLATION_ID> \
--from-file=githubAppPrivateKey=$HOME/.secrets/argocd-home.pem \
--dry-run=client -o yaml \
| kubectl label -f - --local -o yaml argocd.argoproj.io/secret-type=repo-creds \
| kubeseal --format yaml > argocd/repo-creds.sealed.yaml
# Let the SealedSecret take over the Secret you created by hand in Step 3.5
kubectl -n argocd annotate secret github-creds sealedsecrets.bitnami.com/managed=true
Uncomment repo-creds.sealed.yaml in argocd/kustomization.yaml, commit and push. On a fresh cluster you still create the plain Secret once by hand (Step 3.5), because Argo CD needs it to read the platform repo in the first place.
7. Migrate existing workloads without downtime¶
Argo CD adopts existing objects when Git has the same kind, name and namespace. It patches them in place instead of recreating them. Migrate one namespace at a time, least critical first. Register each project with autoPrune: false until its diff is clean.
Step 7.1: Inventory what's running¶
kubectl get ns
helm list -A # anything installed with Helm
kubectl get helmcharts.helm.cattle.io -A # anything installed via k3s HelmChart CRs
ls /var/lib/rancher/k3s/server/manifests/ # on the server node: k3s auto-deploy files
# Per namespace, what you actually need to capture:
kubectl -n blog get deploy,sts,ds,svc,ingress,ingressroute,cm,secret,pvc,cronjob
For each namespace, decide which repo it goes to: the app's existing source repo (add deploy/), or a new <name>-deploy repo. Note how it was deployed (plain YAML, helm install, or a k3s HelmChart), since each has its own path below.
Step 7.2: Create the project repo and export clean manifests¶
cd ~/src/blog # existing app repo, or: gh repo create YOUR_USER/blog-deploy --private --clone
mkdir -p deploy/base deploy/overlays/prod
kubectl krew install neat # one-time; or strip server fields by hand
for kind in deploy svc ingress cm pvc; do
kubectl -n blog get $kind -o yaml | kubectl neat > deploy/base/$kind.yaml
done
If you still have the original YAML files, start from those; they're cleaner. Then tidy up:
- Remove
namespace:fields; the overlay sets it. - Delete the
kube-root-ca.crtConfigMap, ServiceAccount token secrets, and any Namespace object (the platform repo owns it). - Convert real Secrets to SealedSecrets (Step 6.3).
- Write
deploy/base/kustomization.yamlanddeploy/overlays/prod/kustomization.yamlas in section 5.
Things that must not change during adoption
- Deployment
spec.selectoris immutable. Keep the exact labels, which is why Step 5.1 usesincludeSelectors: false. - PVC spec (storageClassName, size, accessModes) is immutable. Copy it exactly so the existing volume and data are kept.
- Service
clusterIP: leave it out of Git.
Step 7.3: Protect data before the first sync¶
k3s's local-path provisioner creates volumes with reclaim policy Delete. If a PVC were ever pruned, the data on disk would go with it. Before migrating anything stateful:
kubectl get pv -o custom-columns=NAME:.metadata.name,CLAIM:.spec.claimRef.name,POLICY:.spec.persistentVolumeReclaimPolicy
kubectl patch pv <pv-name> -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}'
And tell Argo CD never to prune PVCs, in the project repo's base:
# deploy/base/pvc.yaml
metadata:
name: blog-data
annotations:
argocd.argoproj.io/sync-options: Prune=false,Delete=false
Also take a normal backup of the volume directory (/var/lib/rancher/k3s/storage/ on the node) for anything you care about.
Step 7.4: Register the project with pruning off¶
In the platform repo, run the register script (Step 8.3) or create projects/blog/ by hand. Set autoPrune: false in config.yaml for now, so a file you haven't exported yet can't delete a live object.
Two cautions for existing namespaces:
namespace.yamladopts the existing namespace. Copy any labels or annotations it already has, so Argo CD doesn't strip them.- A
ResourceQuotaon CPU/memory rejects new pods that lack requests/limits. The LimitRange in Step 4.3 supplies defaults, but check that your current usage fits under the quota (kubectl describe quota -n blogafter sync). Or leavequota.yamlout of the kustomization until the project is settled.
Step 7.5: Diff, then sync¶
# In the project repo
kubectl kustomize deploy/overlays/prod | kubectl diff -f - # local preview, should be near-empty
git add deploy && git commit -m "Add GitOps manifests" && git push
# In the platform repo
git add projects/blog && git commit -m "Register blog (pruning off)" && git push
argocd app get project-blog # AppProject, namespace, quota
argocd app get blog # appears within ~3 minutes; or: argocd app get projects --refresh
argocd app diff blog # what Argo would change
Iterate in the project repo until argocd app diff blog shows only harmless changes (the tracking annotation, defaulted fields). Auto-sync then applies it. Watch that pods don't restart unexpectedly:
kubectl -n blog get pods -w
If a field keeps showing as different because a controller sets it (HPA replicas, a mutating webhook), add an ignoreDifferences entry rather than fighting it (section 10).
Step 7.6: Projects installed with Helm or k3s HelmChart¶
Installed with helm install:
helm -n media get values media > /tmp/media-values.yaml # your custom values only
helm -n media list # note chart name and version
Put those into the project repo's deploy/overlays/prod/kustomization.yaml as a helmCharts entry with the same releaseName and chart version, so rendered object names match. Check the chart creates nothing cluster-scoped (Step 5.4). After Argo CD shows Synced/Healthy, remove Helm's bookkeeping so nobody runs helm upgrade out of habit:
# Do NOT run helm uninstall: that deletes the workload.
kubectl -n media delete secret -l owner=helm,name=media
Installed via a k3s HelmChart (manifest in /var/lib/rancher/k3s/server/manifests/): deleting a HelmChart object makes k3s's helm-controller uninstall the release. Plan these as a short, controlled redeploy:
- Set PVs to Retain (Step 7.3) and back up the manifest file.
- Add the equivalent
helmChartsentry, with the same release name, to the project repo. - Move the file out of the auto-deploy directory, and let Argo CD bring the app back up.
- To re-bind a retained volume, clear its old claim with
kubectl patch pv <pv-name> --type json -p '[{"op":"remove","path":"/spec/claimRef"}]'and setspec.volumeName: <pv-name>on the PVC in Git.
How k3s reacts to a removed auto-deploy file has changed between versions, so test this on a non-critical app first.
Step 7.7: Finish each migration¶
Once a project shows Synced/Healthy:
- Set
autoPrune: truein itsprojects/<name>/config.yamland merge. - Check the orphaned-resources warnings (enabled in its AppProject) for leftovers in the namespace, and either add them to the project repo or delete them.
- When all projects are done, stop using cluster-admin kubeconfigs day to day. Git is now the way you change things.
8. Onboard a new project repo¶
A new project takes two PRs: one creating the project repo's deploy/ folder, and one registering it in the platform repo. No Argo CD config changes, no kubectl.
Step 8.1: Scaffold deploy/ in the project repo¶
GitLab: the scripts are the same; the
glabcommands for Steps 8.1 to 8.3 are in G5.
Save this as scripts/new-app.sh in the platform repo, and run it from the root of the project repo:
#!/usr/bin/env bash
# Run from the root of a PROJECT repo. Writes deploy/base and deploy/overlays/prod.
set -euo pipefail
APP=${1:?usage: new-app.sh <name> <image:tag> [port]}
IMAGE=${2:?image required, e.g. ghcr.io/me/app:1.0.0}
PORT=${3:-8080}
DOMAIN=${DOMAIN:-home.example.com}
B=deploy/base O=deploy/overlays/prod
mkdir -p "$B" "$O"
cat > "$B/kustomization.yaml" <<EOF
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources: [deployment.yaml, service.yaml, ingress.yaml]
labels:
- pairs: { app.kubernetes.io/name: $APP }
includeSelectors: false
EOF
cat > "$B/deployment.yaml" <<EOF
apiVersion: apps/v1
kind: Deployment
metadata: { name: $APP }
spec:
replicas: 1
selector: { matchLabels: { app: $APP } }
template:
metadata: { labels: { app: $APP } }
spec:
containers:
- name: $APP
image: ${IMAGE%:*}
ports: [{ containerPort: $PORT }]
resources:
requests: { cpu: 25m, memory: 64Mi }
limits: { memory: 256Mi }
EOF
cat > "$B/service.yaml" <<EOF
apiVersion: v1
kind: Service
metadata: { name: $APP }
spec:
selector: { app: $APP }
ports: [{ port: 80, targetPort: $PORT }]
EOF
cat > "$B/ingress.yaml" <<EOF
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata: { name: $APP }
spec:
ingressClassName: traefik
rules:
- host: $APP.$DOMAIN
http:
paths:
- path: /
pathType: Prefix
backend: { service: { name: $APP, port: { number: 80 } } }
EOF
cat > "$O/kustomization.yaml" <<EOF
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: $APP
resources: [../../base]
images:
- name: ${IMAGE%:*}
newTag: "${IMAGE##*:}"
EOF
kubectl kustomize "$O" >/dev/null && echo "deploy/ scaffolded for $APP and renders OK"
gh repo create YOUR_USER/whoami-deploy --private --clone && cd whoami-deploy
~/src/k3s-platform/scripts/new-app.sh whoami traefik/whoami:v1.10.1 80
git add deploy && git commit -m "Add deploy manifests" && git push
Step 8.2: Give Argo CD read access to the new repo¶
GitLab: nothing to do for projects inside the group covered by the group deploy token; see G5.
- GitHub App installed on All repositories: nothing to do.
- Only select repositories: GitHub, Settings, Applications, your Argo CD app, Configure, then add the new repo.
- Fine-grained token: edit the token and add the repo.
Check access with argocd repo add https://github.com/YOUR_USER/whoami-deploy.git; it inherits the credential template and should report Successful.
Step 8.3: Register it in the platform repo¶
Save this as scripts/register-project.sh in the platform repo:
#!/usr/bin/env bash
# Run from the root of the PLATFORM repo. Writes projects/<name>/.
set -euo pipefail
NAME=${1:?usage: register-project.sh <name> <repo-url> [source-path]}
REPO=${2:?repo url, e.g. https://github.com/YOUR_USER/blog.git}
SRC=${3:-deploy/overlays/prod}
D=projects/$NAME
mkdir -p "$D"
cat > "$D/config.yaml" <<EOF
# Read by the 'projects' ApplicationSet; one entry per environment
- project: $NAME
name: $NAME
namespace: $NAME
repoURL: $REPO
sourcePath: $SRC
targetRevision: main
autoPrune: true
EOF
cat > "$D/namespace.yaml" <<EOF
apiVersion: v1
kind: Namespace
metadata:
name: $NAME
labels:
pod-security.kubernetes.io/enforce: baseline
EOF
cat > "$D/quota.yaml" <<EOF
apiVersion: v1
kind: ResourceQuota
metadata: { name: quota, namespace: $NAME }
spec:
hard: { requests.cpu: "2", requests.memory: 4Gi, limits.memory: 8Gi, persistentvolumeclaims: "5" }
---
apiVersion: v1
kind: LimitRange
metadata: { name: defaults, namespace: $NAME }
spec:
limits:
- type: Container
defaultRequest: { cpu: 50m, memory: 64Mi }
default: { memory: 256Mi }
EOF
cat > "$D/appproject.yaml" <<EOF
apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata: { name: $NAME, namespace: argocd }
spec:
description: $NAME project
sourceRepos: [$REPO]
destinations:
- { server: https://kubernetes.default.svc, namespace: $NAME }
clusterResourceWhitelist: []
namespaceResourceBlacklist:
- { group: '', kind: ResourceQuota }
- { group: '', kind: LimitRange }
orphanedResources: { warn: true }
roles:
- name: developer
policies:
- p, proj:$NAME:developer, applications, get, $NAME/*, allow
- p, proj:$NAME:developer, applications, sync, $NAME/*, allow
- p, proj:$NAME:developer, logs, get, $NAME/*, allow
groups: [$NAME-devs]
EOF
cat > "$D/kustomization.yaml" <<EOF
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources: [namespace.yaml, appproject.yaml, quota.yaml]
EOF
kubectl kustomize "$D" >/dev/null && echo "$D written and renders OK"
cd ~/src/k3s-platform
scripts/register-project.sh whoami https://github.com/YOUR_USER/whoami-deploy.git
git checkout -b register-whoami && git add projects/whoami && git commit -m "Register whoami"
gh pr create --fill # review, merge
Within about 3 minutes Argo CD creates project-whoami (AppProject, namespace, quota) and then the whoami Application. Check with argocd app get whoami.
Step 8.4: The onboarding checklist¶
GitLab: for webhooks (item 7), use a project webhook with a secret token; see G4.
- Create the project repo, or use the app's existing repo, and scaffold
deploy/(Step 8.1). - Adjust resources and probes. Add a
pvc.yamlwith thePrune=false,Delete=falseannotation if it stores data. - Seal its secrets into
deploy/overlays/prod/sealed-secret.yaml(Step 6.3) and list the file in the overlay'sresources. - Add the CI validation workflow to the project repo (Step 8.5).
- Give Argo CD read access to the repo (Step 8.2).
- Register it in the platform repo (Step 8.3), tune its quota, and merge.
- Optional: add a webhook in the project repo (Settings, Webhooks,
https://argocd.home.example.com/api/webhook, JSON, push events) for instant syncs instead of ~3-minute polling. This only works if GitHub can reach your Argo CD. - Optional: grant people access via the AppProject role or
policy.csv(Step 4.6).
Removing a project: delete projects/<name>/ from the platform repo and merge. Because of preserveResourcesOnDeletion: true, the Application and AppProject disappear but the namespace and workload keep running until you kubectl delete ns <name>. Then archive the project repo.
Step 8.5: Validate PRs in each project repo¶
GitLab: use a shared
.gitlab-ci.ymljob pulled in withinclude; see G6.
Put this in each project repo as .github/workflows/validate-deploy.yaml. It only runs when deploy/ changes:
name: validate-deploy
on:
pull_request:
paths: ['deploy/**']
jobs:
render:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: azure/setup-helm@v4
- name: Install kubeconform
run: |
curl -sL https://github.com/yannh/kubeconform/releases/latest/download/kubeconform-linux-amd64.tar.gz | tar xz
sudo mv kubeconform /usr/local/bin/
- name: Render and validate every overlay
run: |
set -e
for d in deploy/overlays/*; do
echo "::group::$d"
kubectl kustomize --enable-helm "$d" | kubeconform -strict -ignore-missing-schemas -summary
# Fail early on objects the AppProject would reject
if kubectl kustomize --enable-helm "$d" | grep -qE '^kind: (Namespace|ClusterRole|ClusterRoleBinding|CustomResourceDefinition)$'; then
echo "Cluster-scoped object found in $d; these belong in the platform repo"; exit 1
fi
echo "::endgroup::"
done
To avoid copying this into every repo, make it a reusable workflow in the platform repo (trigger on: workflow_call) and call it with uses: YOUR_USER/k3s-platform/.github/workflows/validate-deploy.yaml@main. If the platform repo is private, allow other repos to use it under its Settings, Actions, General, Access. The platform repo needs the same kind of check for infrastructure/* and projects/*.
9. k3s specifics¶
k3s ships a few components that it manages itself. The rule: don't put k3s-owned objects in Git unless you first disable them in k3s, otherwise k3s and Argo CD will overwrite each other.
| k3s component | Owned by | GitOps approach |
|---|---|---|
Traefik (HelmChart in kube-system) |
k3s | Customize with a HelmChartConfig in Git, or disable and own it fully |
| CoreDNS, metrics-server, local-path-provisioner | k3s | Leave alone; customize via k3s config files |
| ServiceLB (klipper) | k3s | Works as-is for type: LoadBalancer |
/var/lib/rancher/k3s/server/manifests/ auto-deploy |
k3s | Stop adding files here; move your own ones into Git (Step 7.6) |
HelmChart CRD (helm.cattle.io/v1) |
k3s helm-controller | Usable from Git, but prefer Kustomize helmCharts |
Traefik: option A, keep the bundled one (simplest)¶
Customize it with a HelmChartConfig, which k3s merges into its own Traefik install. Put it in the platform repo's infrastructure/traefik-config/:
# infrastructure/traefik-config/helmchartconfig.yaml
apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
name: traefik
namespace: kube-system # must be kube-system, overrides the app's destination namespace
spec:
valuesContent: |-
logs:
access:
enabled: true
# Add values for the Traefik chart version your k3s ships;
# check it with: kubectl -n kube-system get helmchart traefik -o yaml
# infrastructure/traefik-config/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- helmchartconfig.yaml
- middlewares.yaml # optional: shared Traefik Middlewares (auth, headers, redirects)
Traefik values keys change between major chart versions, so check them against the version k3s bundles after each k3s upgrade.
Traefik: option B, own it in Git (full control)¶
Start k3s with Traefik disabled, then install it like any other chart:
# /etc/rancher/k3s/config.yaml on each server node, then: systemctl restart k3s
disable:
- traefik
Add infrastructure/traefik/kustomization.yaml with a helmCharts entry for https://traefik.github.io/charts. You choose the version and Renovate can bump it, at the cost of one more thing to maintain. Do this during a maintenance window, since ingress is down between disabling and Argo CD reinstalling it.
Ingress vs IngressRoute: standard Ingress objects (as in the scaffold) keep apps portable. Use Traefik's IngressRoute CRD (traefik.io/v1alpha1) only when you need Traefik-specific features; it works fine with Argo CD.
local-path storage¶
- Volumes are directories on one node (
/var/lib/rancher/k3s/storage/). A pod with a local-path PVC is pinned to that node; on a multi-node cluster, expect that. - Default reclaim policy is Delete. Combine
Prune=false,Delete=falseon PVCs (Step 7.4) with a Retain storage class for anything precious:
# infrastructure/storage/local-path-retain.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: local-path-retain
provisioner: rancher.io/local-path
reclaimPolicy: Retain
volumeBindingMode: WaitForFirstConsumer
- Git doesn't back up data. GitOps restores configuration; you still need a volume backup (restic/rsync of the storage dir, Velero, or Longhorn snapshots if you move to Longhorn).
- Size requests on local-path are not enforced, and PVC specs are immutable. Pick a sensible size up front.
HelmChart CRD from Git¶
You can commit helm.cattle.io/v1 HelmChart objects and let Argo CD apply them. It works, but Argo CD then sees only the HelmChart object, not the Deployments it produces, so health, diffs and drift detection go blind. Use it only as a stopgap while migrating, then convert to Kustomize helmCharts.
Small-cluster tuning¶
- Argo CD runs on arm64 (Raspberry Pi) without changes.
- On memory-tight nodes, keep
dexdisabled, don't enable HA Redis, and raisetimeout.reconciliation(for example to300s). Add a GitHub webhook tohttps://argocd.home.example.com/api/webhookif you want instant syncs without fast polling (GitLab: G4). - Your kubeconfig on the server is
/etc/rancher/k3s/k3s.yaml; copy it to your workstation and changeserver:to the node's address.
10. Day-2 operations¶
The daily loop becomes: change Git, merge, watch Argo CD converge. These are the habits and settings that keep that loop safe.
Drift¶
With selfHeal: true, any kubectl edit is reverted within seconds. That's the point: the cluster can't silently diverge from Git.
argocd app list # OutOfSync = drift or an unsynced commit
argocd app diff blog # exactly what differs
argocd app get blog --refresh # force a re-read of Git
Expected differences (fields controllers set) should be ignored, not fought. Per app in the ApplicationSet template, or globally in values.yaml under configs.cm:
# in the ApplicationSet template.spec
ignoreDifferences:
- group: apps
kind: Deployment
jsonPointers: [/spec/replicas] # when an HPA owns replicas
Pausing one app for debugging: the ApplicationSet rewrites Application specs, so first let per-app sync settings stick by adding this to each ApplicationSet's spec:
ignoreApplicationDifferences:
- jsonPointers: [/spec/syncPolicy]
Then argocd app set blog --sync-policy none pauses it, and argocd app set blog --sync-policy automated --self-heal --auto-prune resumes it. Make the real fix in Git, then resume.
Rollbacks¶
Roll back in Git, so Git stays the truth. For an app change, revert in the project repo; for a platform change (quota, AppProject, Argo CD itself), revert in the platform repo:
git log --oneline -- deploy/ # in the project repo: find the bad commit
git revert <sha> && git push # Argo CD syncs the previous state
argocd app history blog shows what was deployed when. argocd app rollback blog <id> exists but only works with auto-sync off, and Git then disagrees with the cluster; use it only as an emergency brake, then revert in Git and re-enable sync.
What a rollback does not undo: database migrations and data in PVCs. Keep schema changes backward compatible, or back up first.
Image updates¶
GitLab: CI write-back needs a project access token or write deploy key, and Renovate runs as your own scheduled pipeline; see G7 and G8. Images in GitLab's registry need a pull secret: G10.
Pick one approach per project. They can coexist across projects.
| Approach | Best for | How |
|---|---|---|
| Renovate bot | Third-party images and Helm charts (most homelab apps) | Opens PRs bumping newTag, helmCharts.version, chart targetRevision; you merge or auto-merge patches |
| CI write-back | Your own apps with their own build pipeline | App's CI builds the image, then commits the new tag to deploy/ in the same repo |
| Argo CD Image Updater | Fully automatic tracking of a registry | Add-on that watches registries and writes new tags back to Git; see its docs for the current config format |
Renovate (GitHub App, free). Install it on the platform repo and every project repo, and add a renovate.json at each repo root (or keep one shared preset in the platform repo and use "extends": ["github>YOUR_USER/k3s-platform"] in the others):
{
"$schema": "https://docs.renovatebot.com/renovate-schema.json",
"extends": ["config:recommended"],
"kubernetes": { "managerFilePatterns": ["/(deploy|infrastructure|projects)/.+\\.yaml$/"] },
"argocd": { "managerFilePatterns": ["/argocd/.+\\.yaml$/"] },
"packageRules": [
{ "matchUpdateTypes": ["patch", "digest"], "automerge": true },
{ "matchUpdateTypes": ["major"], "dependencyDashboardApproval": true }
]
}
Renovate's kustomize manager picks up images: and helmCharts: in kustomization.yaml automatically. Older Renovate versions call managerFilePatterns fileMatch.
CI write-back in the app's own repo, after pushing ghcr.io/YOUR_USER/blog:$TAG. Since deploy/ lives in the same repo, the workflow can push with its built-in token (grant permissions: contents: write):
cd deploy/overlays/prod
kustomize edit set image ghcr.io/YOUR_USER/blog:${TAG}
git commit -am "deploy: blog ${TAG} [skip ci]" && git push # or open a PR for prod
Pushes made with the built-in GITHUB_TOKEN don't trigger new workflow runs, and [skip ci] covers other CI systems, so there's no build loop. If the manifests live in a separate blog-deploy repo, use a fine-grained token scoped to that one repo's contents.
Notifications¶
Argo CD's notifications controller ships with the Helm chart. Configure it in argocd/install/values.yaml so it's GitOps-managed too. Discord via webhook, as an example (Slack, Telegram, ntfy, email and others are built in):
notifications:
enabled: true
secret:
create: false # supply argocd-notifications-secret as a SealedSecret with key discord-url
notifiers:
service.webhook.discord: |
url: $discord-url
headers:
- name: Content-Type
value: application/json
templates:
template.app-sync-failed: |
webhook:
discord:
method: POST
body: |
{"content": ":x: {{.app.metadata.name}} sync failed: {{.app.status.operationState.message}}"}
template.app-degraded: |
webhook:
discord:
method: POST
body: |
{"content": ":warning: {{.app.metadata.name}} is Degraded"}
triggers:
trigger.on-sync-failed: |
- when: app.status.operationState.phase in ['Error', 'Failed']
send: [app-sync-failed]
trigger.on-health-degraded: |
- when: app.status.health.status == 'Degraded'
send: [app-degraded]
subscriptions:
- recipients: [discord]
triggers: [on-sync-failed, on-health-degraded]
Test it by pushing a deliberately broken manifest to a throwaway app folder.
Upgrades¶
- Argo CD: bump
targetRevisioninargocd/argocd.yamlvia PR. Read the release notes for major versions first. - Charts and images: Renovate PRs.
- k3s itself: outside Argo CD's scope. Use the system-upgrade-controller (its
Planobjects can live ininfrastructure/) or upgrade nodes manually. Recheck the Traefik HelmChartConfig afterwards.
Disaster recovery: rebuilding from Git¶
If a node dies, your recovery is short because everything except data is in Git (the platform repo plus each project repo):
- Install k3s (same config file, same disabled components).
kubectl apply -f sealed-secrets-key.backup.yaml(from your password manager).helm install argocd ...exactly as in Step 3.3, and create the repo credential Secret by hand (Step 3.5).kubectl apply -f bootstrap/root.yaml.- Restore PVC data from your volume backups.
Rehearse this once on a spare VM. It's the best test that your repo really is the source of truth.
11. Troubleshooting and next steps¶
| Symptom | Likely cause | Fix |
|---|---|---|
| App shows "repository not accessible" / "authentication required" | Repo not covered by the GitHub App installation or token, or the URL doesn't match the credential prefix | Add the repo to the installation (Step 8.2). URLs must be https://github.com/YOUR_USER/... exactly, not SSH. GitLab: the project must sit inside the group that owns the deploy token, and the URL must end in .git (G2) |
root shows ComparisonError |
A file listed in argocd/kustomization.yaml doesn't exist yet |
Add the file or comment out the line (Step 3.6) |
| App error "AppProject blog not found" | project-blog hasn't synced yet, or the folder name and project: in config.yaml differ |
Wait one reconcile; check argocd app get project-blog; make the names match |
| "application destination ... is not permitted in project" | Namespace in config.yaml isn't in the AppProject's destinations |
Add it to projects/<name>/appproject.yaml |
| "resource Namespace (or ClusterRole, CRD) is not permitted in project" | The project repo contains a cluster-scoped object | Remove it; namespaces come from the platform repo, cluster-wide objects go in infrastructure/ |
| New project doesn't appear | config.yaml path wrong, YAML not a list, or a missing key (with missingkey=error) |
kubectl -n argocd logs deploy/argocd-applicationset-controller; argocd app get projects --refresh |
| Pods rejected with "must specify limits/requests" | ResourceQuota active without defaults | Keep the LimitRange from Step 4.3, or set requests/limits in the Deployment |
| Sync fails: "field is immutable" on a Deployment | Labels changed the selector | Restore the original matchLabels; use includeSelectors: false |
helmCharts ignored / "must specify --enable-helm" |
Build option missing | Check kustomize.buildOptions in argocd-cm (Step 3.2) |
| App flips between Synced and OutOfSync | Something else edits the same field (HPA, webhook, k3s) | ignoreDifferences for that field, or remove the competing manager |
| SealedSecret stuck, no Secret created | Wrong namespace/name when sealing, or sealed with another cluster's cert | kubectl -n sealed-secrets logs deploy/sealed-secrets-controller; reseal |
| Changes in a project repo take minutes to appear | Polling every timeout.reconciliation |
Add the repo webhook (Step 8.4) or argocd app get <name> --refresh |
Your first week, in order¶
- [ ] Create the platform repo and bootstrap Argo CD (section 3)
- [ ] Set up the GitHub App credential template (Step 3.5)
- [ ] Add the platform AppProject and the three ApplicationSets (section 4)
- [ ] Install Sealed Secrets, back up its key, and seal the repo credential (section 6)
- [ ] Set local-path PVs to Retain and add the Prune=false annotations (Step 7.3)
- [ ] Migrate the least critical namespace into its own repo, then the rest one by one (section 7)
- [ ] Flip
autoPruneto true per project and clean up orphans (Step 7.7) - [ ] Add CI validation to the platform repo and each project repo, and install Renovate (Step 8.5, section 10)
- [ ] Turn on notifications (section 10)
- [ ] Rehearse a rebuild on a spare VM (section 10)
Where to go later¶
- SSO: enable Dex with GitHub OAuth (GitLab: G9), map GitHub teams or GitLab subgroups to the
<project>-devsgroups, and disable the admin account withadmin.enabled: "false". - External Secrets Operator if you keep secrets in 1Password, Bitwarden or Vault. Each project then references secrets by name instead of committing ciphertext.
- Policy: Kyverno in
infrastructure/to enforce resource limits, pinned image tags and allowed registries across all project repos. - More clusters: register them with
argocd cluster addand add aclusterfield to eachconfig.yamlentry, used as the destination. - Shared tooling: move
new-app.sh, the validation workflow andrenovate.jsoninto a template repo, and create new project repos from it withgh repo create --template.
Appendix: GitLab steps¶
GitHub is the default throughout this guide. If your repos are on GitLab (gitlab.com or self-managed), everything else stays the same: Argo CD, Kustomize, Sealed Secrets, the ApplicationSets and the k3s handling. Use the steps below wherever the guide points here.
Placeholders: gitlab.com/YOUR_GROUP is the GitLab group that holds k3s-platform and your project repos (subgroups work too). For self-managed GitLab, swap in your own host, for example gitlab.home.example.com.
| Guide step | GitHub (default) | GitLab |
|---|---|---|
| 3.1 Create repos | gh repo create |
glab repo create (G1) |
| 3.5 Repo credential | GitHub App credential template | Group deploy token credential template (G2) |
| 4.6 SSO groups | GitHub teams via Dex | GitLab groups via Dex or OIDC (G9) |
| 4.7 Auto-discovery | SCM Provider, github: |
SCM Provider, gitlab: (G3) |
| 8.2 Repo access on onboarding | Add repo to the App installation | Nothing to do: the group token covers new projects (G5) |
| 8.4 Webhooks | Repo webhook | Project webhook with a secret token (G4) |
| 8.5 CI validation | GitHub Actions | .gitlab-ci.yml, shared via include (G6) |
| 10 CI image write-back | Built-in GITHUB_TOKEN |
Project access token or write deploy key (G7) |
| 10 Renovate | Hosted GitHub App | Self-run Renovate pipeline (G8) |
| Private images | GHCR | GitLab Container Registry pull secret (G10) |
G1. Create the repos¶
GitLab calls repos "projects". Install the glab CLI (brew install glab, or the binary from GitLab's releases), then run glab auth login.
glab repo create YOUR_GROUP/k3s-platform --private
git clone https://gitlab.com/YOUR_GROUP/k3s-platform.git && cd k3s-platform
mkdir -p bootstrap argocd/{install,appsets} infrastructure projects scripts
Use https://gitlab.com/YOUR_GROUP/<name>.git URLs, with the .git suffix, everywhere the guide says https://github.com/YOUR_USER/<name>.git. That covers root.yaml, argocd.yaml, the ApplicationSets, AppProjects and config.yaml. Without the suffix, GitLab answers with a redirect and Argo CD logs warnings.
G2. One credential for all repos in a group¶
Replaces Step 3.5. A group deploy token gives read-only access to every project in the group and its subgroups, including ones created later. It's available on all tiers, including gitlab.com Free.
- In GitLab, open the group (
YOUR_GROUP), then Settings, Repository, Deploy tokens, Add token. - Name it
argocd. Leave the expiry empty, or set one and add a reminder. Tick read_repository only. (Also tick read_registry if you'll use it for image pulls, as in G10, though a separate token is cleaner.) - Copy the generated username (like
gitlab+deploy-token-1234567) and the token. GitLab shows the token only once. - Create the credential template:
kubectl -n argocd create secret generic gitlab-creds \
--from-literal=type=git \
--from-literal=url=https://gitlab.com/YOUR_GROUP \
--from-literal=username=gitlab+deploy-token-1234567 \
--from-literal=password=<deploy-token>
kubectl -n argocd label secret gitlab-creds argocd.argoproj.io/secret-type=repo-creds
argocd repocreds list # shows https://gitlab.com/YOUR_GROUP
Alternatives:
- Group access token (scope
read_repository, role Reporter). This is a bot user that shows up in audit logs. It needs Premium on gitlab.com but is free on self-managed. It always expires, so add a renewal reminder. - Self-managed GitLab with a private CA: add the CA to Argo CD with
argocd cert add-tls gitlab.home.example.com --from ca.pem, or setconfigs.tls.certificatesinvalues.yaml.
To seal it into the platform repo (Step 6.5), use the same kubectl create secret ... --dry-run=client -o yaml | kubectl label ... | kubeseal pipeline with the four GitLab literals above and the name gitlab-creds. In argocd/kustomization.yaml, the file then holds the GitLab credential instead of the GitHub one.
G3. Auto-discover projects in a GitLab group¶
Replaces the generator in Step 4.7. The same trade-offs apply: one shared AppProject, and anyone who can create a project in the group can deploy.
generators:
- scmProvider:
cloneProtocol: https # match the HTTPS credential template
gitlab:
group: YOUR_GROUP # numeric group ID also works
includeSubgroups: true
# api: https://gitlab.home.example.com/ # self-managed only
tokenRef: { secretName: gitlab-scm-token, key: token }
filters:
- labelMatch: k3s-app # GitLab project topic (Settings, General, Topics)
pathsExist: [deploy/overlays/prod]
template:
metadata:
name: '{{ .repository }}'
spec:
project: apps
source:
repoURL: '{{ .url }}'
targetRevision: '{{ .branch }}'
path: deploy/overlays/prod
destination:
server: https://kubernetes.default.svc
namespace: '{{ .repository }}'
syncPolicy:
automated: { prune: true, selfHeal: true }
syncOptions: [CreateNamespace=true]
The SCM provider calls the GitLab API, which a deploy token can't do. Create a group access token (Premium on gitlab.com) or a personal access token of a bot user, with scope read_api, role Reporter. Store it as gitlab-scm-token in argocd (sealed, as in section 6).
G4. Webhooks for instant syncs¶
Replaces the GitHub webhook in Step 8.4. GitLab signs webhooks with a shared secret token, so set one in Argo CD first:
# argocd/install/values.yaml
configs:
secret:
gitlabSecret: <random string> # or supply argocd-secret's webhook.gitlab.secret via a SealedSecret
Then, in each project (and the platform repo), go to Settings, Webhooks, Add new webhook:
- URL:
https://argocd.home.example.com/api/webhook - Secret token: the same random string
- Trigger: Push events (all branches, or
main) - SSL verification: on, if Argo CD has a public certificate
Group-level webhooks (one webhook for every project in the group) need Premium. On Free, add one per project, or rely on the ~3-minute polling. As with GitHub, GitLab must be able to reach your Argo CD URL. Self-managed GitLab on the same LAN can also reach the in-cluster URL; you may need to allow it under Admin, Settings, Network, Outbound requests.
G5. Onboarding a project on GitLab¶
Replaces the GitHub commands in Steps 8.1 to 8.3. The scripts themselves don't change.
# 8.1 Create the project repo and scaffold deploy/
glab repo create YOUR_GROUP/whoami-deploy --private
git clone https://gitlab.com/YOUR_GROUP/whoami-deploy.git && cd whoami-deploy
~/src/k3s-platform/scripts/new-app.sh whoami traefik/whoami:v1.10.1 80
git add deploy && git commit -m "Add deploy manifests" && git push
# 8.2 Repo access: nothing to do if the project is inside YOUR_GROUP (the group deploy token covers it).
argocd repo add https://gitlab.com/YOUR_GROUP/whoami-deploy.git # optional check; should report Successful
# 8.3 Register it in the platform repo via a merge request
cd ~/src/k3s-platform
scripts/register-project.sh whoami https://gitlab.com/YOUR_GROUP/whoami-deploy.git
git checkout -b register-whoami && git add projects/whoami && git commit -m "Register whoami"
git push -u origin register-whoami
glab mr create --fill --yes
If a project lives outside YOUR_GROUP, the credential template doesn't match its URL. Add a deploy token on that project and register it as an individual repo with argocd repo add <url> --username ... --password ..., or move the project into the group.
G6. Validate merge requests with GitLab CI¶
Replaces Step 8.5. Keep the job in the platform repo as ci/validate-deploy.yml:
# k3s-platform/ci/validate-deploy.yml
validate-deploy:
image:
name: alpine/k8s:1.31.4 # pin a current tag from Docker Hub; ships kubectl, kustomize and helm
entrypoint: [""]
rules:
- if: $CI_PIPELINE_SOURCE == "merge_request_event"
changes: [deploy/**/*]
script:
- wget -qO- https://github.com/yannh/kubeconform/releases/latest/download/kubeconform-linux-amd64.tar.gz | tar xz -C /usr/local/bin
- |
set -e
for d in deploy/overlays/*; do
echo "== $d"
kubectl kustomize --enable-helm "$d" | kubeconform -strict -ignore-missing-schemas -summary
if kubectl kustomize --enable-helm "$d" | grep -qE '^kind: (Namespace|ClusterRole|ClusterRoleBinding|CustomResourceDefinition)$'; then
echo "Cluster-scoped object found in $d; these belong in the platform repo"; exit 1
fi
done
Each project repo pulls it in with a one-line .gitlab-ci.yml (add your normal build stages alongside):
include:
- project: YOUR_GROUP/k3s-platform
ref: main
file: ci/validate-deploy.yml
Cross-project include works when the user running the pipeline can read k3s-platform, which is true for members of the group. For the platform repo itself, add a similar job looping over infrastructure/* and projects/*.
G7. CI image write-back¶
Replaces the CI write-back in section 10. Unlike GitHub's GITHUB_TOKEN, GitLab's CI_JOB_TOKEN can't push to the repo by default, so give the pipeline a dedicated token:
- In the app's project: Settings, Access tokens, add a project access token. Role: Developer (Maintainer if
mainis protected against Developer pushes). Scope:write_repository. - Settings, CI/CD, Variables: add it as
DEPLOY_PUSH_TOKEN, Masked and Protected.
Project access tokens are free on self-managed GitLab but need Premium on gitlab.com. On gitlab.com Free, use a deploy key with write access (Settings, Repository, Deploy keys, tick "Grant write permissions") loaded through an SSH-key CI variable, or a bot user's personal access token.
# .gitlab-ci.yml in the app's project, after the job that pushes the image
deploy-tag:
stage: deploy
image: alpine/k8s:1.31.4 # same pinned image as G6
rules:
- if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH && $CI_COMMIT_MESSAGE !~ /\[skip ci\]/
script:
- git config user.name "GitLab CI"
- git config user.email "[email protected]"
- cd deploy/overlays/prod
- kustomize edit set image $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA
- git commit -am "deploy: $CI_PROJECT_NAME $CI_COMMIT_SHORT_SHA [skip ci]"
- git push "https://ci:${DEPLOY_PUSH_TOKEN}@${CI_SERVER_HOST}/${CI_PROJECT_PATH}.git" HEAD:$CI_DEFAULT_BRANCH -o ci.skip
-o ci.skip and [skip ci] stop the push from starting another pipeline. If deploy/ lives in a separate -deploy project, create the token there instead, and clone that project before running kustomize edit.
G8. Renovate on GitLab¶
There's no hosted Renovate app for GitLab, so you run it yourself on a schedule.
- Create a project such as
YOUR_GROUP/renovate-runner. - Create a bot token with scopes
api,read_userandwrite_repository: a group access token (Premium on gitlab.com), or a personal access token of a bot user. Add it as CI/CD variableRENOVATE_TOKEN(masked). Also add a read-only GitHub personal access token asGITHUB_COM_TOKEN, so Renovate can read changelogs and avoid rate limits. - On gitlab.com, add this
.gitlab-ci.yml:
include:
- project: renovate-bot/renovate-runner
file: /templates/renovate.gitlab-ci.yml
variables:
RENOVATE_EXTRA_FLAGS: "--autodiscover=true --autodiscover-filter=YOUR_GROUP/*"
On self-managed GitLab, mirror or copy that template, and set RENOVATE_ENDPOINT: https://gitlab.home.example.com/api/v4 and RENOVATE_PLATFORM: gitlab.
- Under CI/CD, Pipeline schedules, run it hourly or nightly.
The renovate.json from section 10 works unchanged. For a shared preset, put it in the platform repo and use "extends": ["gitlab>YOUR_GROUP/k3s-platform"] in each project.
G9. SSO and RBAC groups¶
For Step 4.6, Dex has a GitLab connector. In argocd/install/values.yaml, set dex.enabled: true and add the connector:
configs:
cm:
url: https://argocd.home.example.com
dex.config: |
connectors:
- type: gitlab
id: gitlab
name: GitLab
config:
baseURL: https://gitlab.com # or your self-managed URL
clientID: $dex.gitlab.clientID # from argocd-secret
clientSecret: $dex.gitlab.clientSecret
groups: [YOUR_GROUP] # only members of these groups can log in
Create the OAuth application in the group: Settings, Applications. Use the redirect URI https://argocd.home.example.com/api/dex/callback and scopes read_user and openid. GitLab groups arrive as group names, so subgroups such as YOUR_GROUP/blog-devs can be used directly in each AppProject's groups: list and in policy.csv.
G10. Pulling images from the GitLab Container Registry¶
If your images live in registry.gitlab.com/YOUR_GROUP/<project>, each namespace needs a pull secret. Create a group deploy token with scope read_registry only, then seal one pull secret per project into its repo:
kubectl create secret docker-registry gitlab-registry -n blog \
--docker-server=registry.gitlab.com \
--docker-username=gitlab+deploy-token-7654321 \
--docker-password=<read_registry token> \
--dry-run=client -o yaml \
| kubeseal --format yaml > deploy/overlays/prod/gitlab-registry.sealed.yaml
Add the file to the overlay's resources, and reference it from the Deployment:
spec:
template:
spec:
imagePullSecrets:
- name: gitlab-registry
Alternatively, configure it once per node in k3s's /etc/rancher/k3s/registries.yaml (an auth entry for registry.gitlab.com). That's simpler, but it lives outside Git and applies to every namespace.