Scope: Architecture, authentication, key management, client/server configuration, certificate-based auth, hardening, and troubleshooting for OpenSSH 8.x/9.x.

This is the conceptual/reference doc — architecture, algorithm choices, certificate design. For working command references and day-to-day notes, see the companion docs:

• openssh-client-config.md — ~/.ssh/config patterns, agent setup
• openssh-key-management.md — generating/deploying/rotating keys
• openssh-server-hardening.md — sshd_config walkthrough + deprecated-advice notes
• openssh-port-forwarding.md — -D/-L/-R examples, sshuttle
• openssh-file-transfer.md — scp, sshfs, tar-over-SSH, remote commands
• openssh-troubleshooting.md — recurring issues and fixes

Table of Contents¶

1. What OpenSSH Is
2. Protocol Architecture
3. Authentication Methods
4. Key Management
5. Client Configuration (~/.ssh/config)
6. Server Configuration (sshd_config)
7. Certificate-Based Authentication
8. Port Forwarding & Tunneling
9. Connection Multiplexing
10. Hardening Checklist
11. Fail2ban / Rate Limiting Integration
12. Logging & Auditing
13. Algorithm Reference Tables
14. Troubleshooting
15. Common Pitfalls
16. Further Reading

1. What OpenSSH Is¶

OpenSSH is the dominant open-source implementation of the SSH (Secure Shell) protocol, developed by the OpenBSD project. It provides:

• ssh — remote shell / command execution client
• sshd — the server daemon
• scp / sftp — file transfer (scp is legacy; prefer sftp)
• ssh-keygen — key and certificate generation
• ssh-agent / ssh-add — in-memory key management
• ssh-copy-id — public key deployment helper

It replaced Telnet, rlogin, and rsh, all of which transmitted credentials and data in plaintext. Everything in SSH — authentication, commands, file transfers, forwarded traffic — is encrypted and integrity-checked.

2. Protocol Architecture¶

SSH-2 (the only protocol version in modern use — SSH-1 is deprecated and insecure) is layered into three sub-protocols defined by RFC 4251–4254:

Connection sequence:

1. TCP handshake to port 22 (or configured port).
2. Protocol/version banner exchange.
3. Key exchange (KEX) — negotiates a shared secret via Diffie-Hellman variants (see §13); this also derives session keys for symmetric encryption.
4. Server host key verification — client checks the server's host key against ~/.ssh/known_hosts. This is what "Trust On First Connect" warnings refer to.
5. User authentication — one or more auth methods attempted in the order configured.
6. Channel(s) opened over the now-encrypted transport — a shell, a command, a forwarded port, or an sftp subsystem.

Every subsequent packet is encrypted with the negotiated cipher and authenticated with the negotiated MAC (or an AEAD cipher, which combines both — see aes256-gcm and chacha20-poly1305).

3. Authentication Methods¶

Configured server-side via AuthenticationMethods and individual *Authentication directives, attempted client-side in the order listed in PreferredAuthentications (client) or as permitted by the server.

Public key auth, mechanically:

1. Client sends a signature over the session identifier, made with its private key.
2. Server checks whether the corresponding public key is listed in the target user's ~/.ssh/authorized_keys (or matches a trusted CA — certificates).
3. Server verifies the signature. No secret ever crosses the wire.

4. Key Management¶

4.1 Key types¶

4.2 Generating keys¶

ssh-keygen -t ed25519 -C "amercer@lab-$(date +%Y%m%d)" -f ~/.ssh/id_ed25519_lab

# RSA fallback for legacy targets only
ssh-keygen -t rsa -b 4096 -C "amercer@legacy-host" -f ~/.ssh/id_rsa_legacy

Always set a passphrase on private keys at rest. Combine with ssh-agent so you're only prompted once per session rather than per connection.

4.3 ssh-agent¶

eval "$(ssh-agent -s)"
ssh-add ~/.ssh/id_ed25519_lab
ssh-add -l                 # list loaded keys
ssh-add -t 3600 ~/.ssh/id_ed25519_lab   # auto-expire after 1h

Agent forwarding (ssh -A / ForwardAgent yes) is a real risk: a root user on the intermediate host can use your forwarded agent socket to authenticate elsewhere as you, for as long as the connection is open. Prefer ProxyJump (§5.3), which never exposes your agent socket to the intermediate host, over agent forwarding.

4.4 Deploying public keys¶

ssh-copy-id -i ~/.ssh/id_ed25519_lab.pub amercer@lab
# or manually:
cat ~/.ssh/id_ed25519_lab.pub | ssh amercer@lab 'mkdir -p ~/.ssh && chmod 700 ~/.ssh && cat >> ~/.ssh/authorized_keys && chmod 600 ~/.ssh/authorized_keys'

authorized_keys entries can be restricted per-key:

command="/usr/local/bin/backup-only.sh",no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty ssh-ed25519 AAAA... backup-key

This is how you hand out narrowly-scoped automation keys (e.g. a CI runner that can only trigger one script) without giving a full shell.

5. Client Configuration (~/.ssh/config)¶

Per-host overrides live in ~/.ssh/config (mode 600). Example, structured for a homelab with a jump host:

# Global defaults
Host *
    AddKeysToAgent yes
    HashKnownHosts yes
    ServerAliveInterval 60
    ServerAliveCountMax 3
    IdentitiesOnly yes

# Jump host
Host bastion
    HostName bastion.umoswg.net
    User amercer
    Port 22
    IdentityFile ~/.ssh/id_ed25519_bastion

# Internal host reached via the bastion
Host lab
    HostName 10.0.10.5
    User amercer
    IdentityFile ~/.ssh/id_ed25519_lab
    ProxyJump bastion
    ForwardAgent no

# Wildcard for a whole internal subnet
Host 10.0.10.*
    User amercer
    ProxyJump bastion
    IdentityFile ~/.ssh/id_ed25519_lab

5.1 Key directives¶

5.2 ProxyJump vs agent forwarding¶

ssh -J bastion lab

is functionally equivalent to nesting connections through the bastion, but the bastion never sees your private key or your agent socket — it only relays encrypted bytes. This is the correct way to reach hosts behind a jump box; avoid ForwardAgent yes unless you specifically trust the intermediate host with impersonation rights.

6. Server Configuration (sshd_config)¶

See the hardened config delivered separately. Key directive groups:

6.1 Authentication restriction¶

PermitRootLogin no
PasswordAuthentication no
KbdInteractiveAuthentication no
AuthenticationMethods publickey
MaxAuthTries 3

6.2 Access scoping¶

AllowGroups ssh-users
# or
AllowUsers amercer@10.0.0.0/24 deploy@203.0.113.10

Match blocks allow conditional overrides — a common pattern is restricting sftp-only accounts:

Match Group sftp-only
    ChrootDirectory /srv/sftp/%u
    ForceCommand internal-sftp
    AllowTcpForwarding no
    X11Forwarding no

6.3 Crypto policy¶

Ciphers, KexAlgorithms, MACs, HostKeyAlgorithms, PubkeyAcceptedAlgorithms — restrict to modern AEAD ciphers and -etm (encrypt-then-MAC) MACs only. See §13 for current recommendations.

6.4 Validating and applying changes¶

sudo sshd -t              # syntax check — do this before every reload
sudo sshd -T              # dump the fully-resolved effective config
sudo systemctl reload sshd

Never restart blind over your only session — reload re-reads config without dropping existing connections; keep a second session open regardless.

7. Certificate-Based Authentication¶

Raw public key distribution doesn't scale — revoking access means editing authorized_keys on every host. SSH certificates solve this the same way TLS certs do: a trusted CA signs short-lived credentials.

7.1 Generate a CA¶

ssh-keygen -t ed25519 -f ssh_user_ca -C "umoswg user CA"
ssh-keygen -t ed25519 -f ssh_host_ca -C "umoswg host CA"

7.2 Sign a user key (issued to a person, time-limited)¶

ssh-keygen -s ssh_user_ca -I "amercer-20260819" \
  -n amercer,root \
  -V +12h \
  -O force-command="none" \
  ~/.ssh/id_ed25519_lab.pub

This produces id_ed25519_lab-cert.pub. -n lists valid principals (usernames the cert may log in as); -V sets an expiry window — short-lived certs remove the need for manual revocation in most cases.

7.3 Sign a host key (so clients trust hosts by CA, not TOFU)¶

ssh-keygen -s ssh_host_ca -I "lab-host-cert" -h -n lab.umoswg.net,10.0.10.5 -V +52w /etc/ssh/ssh_host_ed25519_key.pub

7.4 Server trusts the user CA¶

# sshd_config
TrustedUserCAKeys /etc/ssh/ssh_user_ca.pub

7.5 Clients trust the host CA¶

# ~/.ssh/known_hosts
@cert-authority *.umoswg.net ssh-ed25519 AAAA...hostca...

Result: no more known_hosts prompts per new host, and no more distributing individual public keys per user per host — access is granted/revoked centrally by issuing or expiring certificates.

8. Port Forwarding & Tunneling¶

Server-side, restrict what's permitted:

AllowTcpForwarding local   # local only, no remote/dynamic
GatewayPorts no            # remote forwards bind to loopback only, not 0.0.0.0
PermitTunnel no            # disable layer-2/3 tun/tap tunneling entirely

9. Connection Multiplexing¶

Reuses a single authenticated TCP connection for multiple sessions — avoids repeated handshake/auth latency:

Host *
    ControlMaster auto
    ControlPath ~/.ssh/sockets/%r@%h-%p
    ControlPersist 10m

mkdir -p ~/.ssh/sockets && chmod 700 ~/.ssh/sockets

Subsequent ssh/scp/sftp calls to the same host reuse the existing tunnel — noticeably faster for scripted/repeated access (e.g. CI, config management, or a CLI tool making many short-lived connections).

10. Hardening Checklist¶

• [ ] PermitRootLogin no
• [ ] PasswordAuthentication no, KbdInteractiveAuthentication no (unless intentionally using OTP via PAM)
• [ ] Modern-only Ciphers/KexAlgorithms/MACs (drop CBC ciphers, drop non-ETM MACs, drop diffie-hellman-group1/14-sha1)
• [ ] ed25519 host keys; remove DSA/ECDSA host key files
• [ ] AllowGroups/AllowUsers scoping — don't rely on "everyone with an account" implicitly having SSH access
• [ ] MaxAuthTries 3, LoginGraceTime short (15–30s)
• [ ] X11Forwarding no unless actually needed
• [ ] AllowAgentForwarding no by default; use ProxyJump instead
• [ ] ClientAliveInterval/ClientAliveCountMax set to reap dead sessions
• [ ] LogLevel VERBOSE (logs the key fingerprint used for each login — essential for audit trails)
• [ ] fail2ban or equivalent in front of the daemon (see §11)
• [ ] Consider a non-default port only as noise reduction, never as a security control — it's obscurity, not defense
• [ ] Certificates (§7) instead of raw key sprawl once you're managing more than a handful of hosts/users

11. Fail2ban / Rate Limiting Integration¶

LogLevel VERBOSE (or at least INFO) is required for fail2ban's sshd jail to have anything to match against.

# /etc/fail2ban/jail.local
[sshd]
enabled = true
port = 22
filter = sshd
logpath = /var/log/auth.log
maxretry = 3
findtime = 10m
bantime = 1h
bantime.increment = true

Complement, don't replace, with sshd_config's own MaxStartups (limits concurrent unauthenticated connections — mitigates connection-flood DoS independent of fail2ban's log-based banning).

12. Logging & Auditing¶

• Auth events land in journalctl -u sshd / /var/log/auth.log (Debian/Ubuntu) or /var/log/secure (RHEL family).
• sshd -T dumps the fully-resolved effective configuration — invaluable when Match blocks make the applied config non-obvious.
• Key fingerprint per login: LogLevel VERBOSE logs which specific key authenticated a session — critical for tracing access after a key is suspected compromised.
• For centralized logging (relevant to an ELK/Vector pipeline): ship auth.log/journal sshd unit logs via Vector or Filebeat; grep/alert on Failed publickey, Accepted publickey, and Invalid user patterns.

13. Algorithm Reference Tables¶

Key exchange (KEX) — prefer top-down:

Ciphers — AEAD only where possible:

MACs (only relevant for non-AEAD ciphers):

Host/public key algorithms:

14. Troubleshooting¶

Verbose client debugging (stack up to -vvv for full KEX/auth detail):

ssh -vvv amercer@lab

Common diagnostics:

15. Common Pitfalls¶

• Editing sshd_config and restarting without testing first. Always sshd -t, always keep a second session open.
• Relying on a non-standard port as a security measure. It reduces log noise from opportunistic scanners; it does nothing against a targeted attacker.
• ForwardAgent yes by default in ~/.ssh/config with Host *. Scope it explicitly per-host, or avoid it entirely in favor of ProxyJump.
• Reusing one key everywhere. Compromise of one host/service compromises every relationship that key participates in. Prefer per-purpose keys or certificates with scoped principals.
• Forgetting chmod. sshd silently ignores authorized_keys if directory/file permissions are too permissive (~/.ssh must not be group/world-writable).
• Long-lived, unscoped certificates. Defeats the point of moving to certificate auth — keep -V windows short and principals narrow.

16. Further Reading¶

• man 5 sshd_config / man 5 ssh_config — authoritative, versioned to your installed OpenSSH
• man 1 ssh-keygen — certificate signing options (-s, -I, -n, -V, -O)
• RFC 4251–4254 — SSH-2 protocol architecture, authentication, transport, connection layers
• OpenSSH release notes (https://www.openssh.com/releasenotes.html) — track algorithm deprecations across upgrades