Comprehensive Network Troubleshooting Guide¶

Table of Contents¶

DNS Tools
Connectivity Tools
Packet Capture Tools
Real-World Scenarios
Troubleshooting Workflows

DNS Tools¶

dig (Domain Information Groper)¶

Installation:

# Debian/Ubuntu
sudo apt-get install dnsutils

# CentOS/RHEL
sudo yum install bind-utils

# macOS
# Already installed

Basic Usage:

# Simple DNS lookup
dig example.com

# Query specific record type
dig example.com A          # IPv4 address
dig example.com AAAA       # IPv6 address
dig example.com MX         # Mail servers
dig example.com NS         # Name servers
dig example.com TXT        # Text records
dig example.com SOA        # Start of Authority
dig example.com CNAME      # Canonical name

# Short answer only
dig example.com +short

# Query specific DNS server
dig @8.8.8.8 example.com
dig @1.1.1.1 example.com

# Reverse DNS lookup
dig -x 8.8.8.8

# Trace DNS resolution path
dig example.com +trace

# Show only answer section
dig example.com +noall +answer

# Check DNSSEC
dig example.com +dnssec

Advanced Usage:

# Query all record types
dig example.com ANY

# Disable recursion (query authoritative server directly)
dig example.com +norecurse

# Show query time
dig example.com +stats

# Batch queries from file
dig -f domains.txt

# Multiple queries
dig example.com A example.com MX

# TCP instead of UDP
dig example.com +tcp

# Debug mode (verbose output)
dig example.com +trace +additional

Common Use Cases:

# Check if domain resolves
dig google.com +short

# Find authoritative nameservers
dig google.com NS +short

# Check MX records for email troubleshooting
dig gmail.com MX +short

# Verify DNS propagation
dig @8.8.8.8 example.com
dig @1.1.1.1 example.com

# Check TTL values
dig example.com +noall +answer +ttlid

# Troubleshoot slow DNS
dig example.com +stats | grep "Query time"

dog (Modern DNS client)¶

Installation:

# Using cargo (Rust)
cargo install dog

# Debian/Ubuntu
sudo apt-get install dog

# macOS
brew install dog

Basic Usage:

# Simple lookup
dog example.com

# Specific record type
dog example.com A
dog example.com MX
dog example.com TXT

# Query specific DNS server
dog example.com @8.8.8.8
dog example.com @1.1.1.1

# JSON output
dog example.com --json

# Short output
dog example.com --short

# Color output (default)
dog example.com

Advanced Features:

# Query over HTTPS (DoH)
dog example.com @https://cloudflare-dns.com/dns-query

# Query over TLS (DoT)
dog example.com @tls://1.1.1.1

# Multiple queries
dog example.com A AAAA MX

# Show query time
dog example.com --time

# Trace DNS resolution
dog example.com --trace

Comparison with dig:

Feature	dig	dog
Modern syntax	❌	✅
Colored output	❌	✅
DoH support	❌	✅
DoT support	❌	✅
JSON output	❌	✅
Learning curve	Steeper	Easier

delv (Domain Entity Lookup & Validation)¶

Installation:

# Debian/Ubuntu
sudo apt-get install bind9-utils

# CentOS/RHEL
sudo yum install bind-utils

Basic Usage:

# Simple lookup with DNSSEC validation
delv example.com

# Query specific record type
delv example.com A
delv example.com MX

# Query specific server
delv @8.8.8.8 example.com

# Disable DNSSEC validation
delv +novalidate example.com

# Trace resolution
delv +trace example.com

DNSSEC-Specific:

# Check DNSSEC signatures
delv example.com +multiline

# Show trust anchor
delv +root=.

# Validate entire chain
delv example.com +vtrace

# Check specific key
delv example.com DNSKEY

Use Cases:

# Verify DNSSEC is working
delv cloudflare.com
# Look for "fully validated" in output

# Debug DNSSEC failures
delv example.com +vtrace +multiline

# Check if domain uses DNSSEC
delv example.com DS

Connectivity Tools¶

ping¶

Basic Usage:

# Simple ping
ping google.com

# Ping IPv6
ping6 google.com

# Limit to N packets
ping -c 4 google.com

# Set interval (seconds)
ping -i 0.5 google.com

# Set packet size
ping -s 1000 google.com

# Flood ping (requires root)
sudo ping -f google.com

# Set TTL
ping -t 64 google.com

# Numeric output only (no DNS)
ping -n 8.8.8.8

# Verbose output
ping -v google.com

Advanced Usage:

# Ping with timestamp
ping google.com | while read line; do echo "$(date): $line"; done

# Ping until success
ping google.com -c 1 -W 1
while [ $? -ne 0 ]; do
    sleep 1
    ping google.com -c 1 -W 1
done

# Ping multiple hosts
for host in google.com cloudflare.com; do
    ping -c 1 $host
done

# Continuous ping with statistics every N packets
ping -c 100 -i 0.2 google.com

# Adaptive ping (adjust to RTT)
ping -A google.com

# Set source IP
ping -I eth0 google.com

Interpreting Output:

PING google.com (142.250.80.46) 56(84) bytes of data.
64 bytes from lga34s34-in-f14.1e100.net (142.250.80.46): icmp_seq=1 ttl=117 time=12.3 ms

--- google.com ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3005ms
rtt min/avg/max/mdev = 12.3/13.5/15.2/1.1 ms

Key Metrics:
- ttl: Time to live (hops remaining)
- time: Round-trip time
- packet loss: Percentage of lost packets
- mdev: Standard deviation (jitter)

nping (Nmap's packet generation tool)¶

Installation:

# Debian/Ubuntu
sudo apt-get install nmap

# CentOS/RHEL
sudo yum install nmap

# macOS
brew install nmap

Basic Usage:

# TCP ping to port 80
sudo nping --tcp -p 80 example.com

# UDP ping
sudo nping --udp -p 53 8.8.8.8

# ICMP ping (like ping)
sudo nping --icmp example.com

# ARP ping (local network)
sudo nping --arp 192.168.1.1

# Count of packets
sudo nping -c 4 --tcp -p 443 example.com

Advanced Features:

# Set TCP flags
sudo nping --tcp -p 80 --flags SYN example.com
sudo nping --tcp -p 80 --flags ACK example.com

# Set packet rate
sudo nping --tcp -p 80 --rate 10 example.com

# Custom TTL
sudo nping --tcp -p 80 --ttl 64 example.com

# Source port
sudo nping --tcp -p 80 --source-port 12345 example.com

# Delay between packets
sudo nping --tcp -p 80 --delay 500ms example.com

# Echo mode (measure packet loss)
sudo nping --tcp -p 80 --echo-client "public" echo.nmap.org

Use Cases:

# Test if firewall blocks SYN packets
sudo nping --tcp -p 80 --flags SYN example.com

# Check if port is open (SYN/ACK response)
sudo nping --tcp -p 443 example.com -c 1

# Test UDP connectivity
sudo nping --udp -p 53 8.8.8.8

# Measure packet loss and latency
sudo nping --tcp -p 80 -c 100 example.com

# Test MTU/fragmentation
sudo nping --icmp --data-length 1472 example.com

mtr (My Traceroute)¶

Installation:

# Debian/Ubuntu
sudo apt-get install mtr

# CentOS/RHEL
sudo yum install mtr

# macOS
brew install mtr

Basic Usage:

# Interactive mode
mtr google.com

# Report mode (run N cycles and exit)
mtr --report --report-cycles 10 google.com

# No DNS resolution
mtr -n google.com

# TCP mode (instead of ICMP)
mtr --tcp google.com

# UDP mode
mtr --udp google.com

# Specify port
mtr --tcp --port 443 google.com

# Wide report
mtr --report-wide google.com

Advanced Usage:

# CSV output
mtr --csv google.com

# JSON output
mtr --json google.com

# XML output
mtr --xml google.com

# Continuous with interval
mtr --interval 1 google.com

# Set packet size
mtr --psize 1000 google.com

# Set max TTL
mtr --max-ttl 20 google.com

# Both directions (requires cooperation)
mtr --both google.com

Reading mtr Output:

                             My traceroute  [v0.95]
host (192.168.1.100)                          2024-02-15T20:00:00+0000
Keys:  Help   Display mode   Restart statistics   Order of fields   quit
                                       Packets               Pings
 Host                                Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. 192.168.1.1                       0.0%    10    1.2   1.3   1.1   1.8   0.2
 2. 10.0.0.1                          0.0%    10    8.5   8.7   8.3   9.2   0.3
 3. 100.64.0.1                       10.0%    10   12.3  12.5  12.1  13.5   0.4
 4. 142.250.80.46                     0.0%    10   15.2  15.4  15.0  16.1   0.3

Key Columns:
- Loss%: Packet loss at this hop
- Snt: Packets sent
- Last: Last RTT
- Avg: Average RTT
- Best: Best RTT
- Wrst: Worst RTT
- StDev: Standard deviation (jitter)

Use Cases:

# Find where packet loss occurs
mtr --report --report-cycles 100 example.com

# Trace to specific port
mtr --tcp --port 443 example.com

# Save report to file
mtr --report --report-cycles 50 example.com > mtr-report.txt

# Monitor continuously
mtr --interval 1 --curses google.com

# Quick 10-cycle report
mtr -r -c 10 example.com

Packet Capture Tools¶

tcpdump¶

Installation:

# Debian/Ubuntu
sudo apt-get install tcpdump

# CentOS/RHEL
sudo yum install tcpdump

# macOS
# Already installed

Basic Usage:

# Capture on default interface
sudo tcpdump

# Capture on specific interface
sudo tcpdump -i eth0
sudo tcpdump -i any  # All interfaces

# Limit number of packets
sudo tcpdump -c 100

# Save to file
sudo tcpdump -w capture.pcap

# Read from file
tcpdump -r capture.pcap

# Verbose output
sudo tcpdump -v
sudo tcpdump -vv
sudo tcpdump -vvv

Filtering:

# Host filter
sudo tcpdump host 192.168.1.1
sudo tcpdump src 192.168.1.1
sudo tcpdump dst 192.168.1.1

# Network filter
sudo tcpdump net 192.168.1.0/24

# Port filter
sudo tcpdump port 80
sudo tcpdump src port 80
sudo tcpdump dst port 443

# Protocol filter
sudo tcpdump tcp
sudo tcpdump udp
sudo tcpdump icmp

# Combine filters
sudo tcpdump 'host 192.168.1.1 and port 80'
sudo tcpdump 'tcp and port 443'
sudo tcpdump 'not port 22'

Advanced Filters:

# HTTP traffic
sudo tcpdump -A -s 0 'tcp port 80 and (((ip[2:2] - ((ip[0]&0xf)<<2)) - ((tcp[12]&0xf0)>>2)) != 0)'

# DNS queries
sudo tcpdump -i any -s 0 port 53

# SYN packets
sudo tcpdump 'tcp[tcpflags] & tcp-syn != 0'

# ACK packets
sudo tcpdump 'tcp[tcpflags] & tcp-ack != 0'

# Packets with data
sudo tcpdump 'tcp[((tcp[12:1] & 0xf0) >> 2):4] = 0x47455420'  # GET requests

# Packets larger than 1000 bytes
sudo tcpdump 'greater 1000'

# Broadcast/multicast
sudo tcpdump 'ether broadcast or ether multicast'

Useful Options:

# Show ASCII content
sudo tcpdump -A port 80

# Show hex and ASCII
sudo tcpdump -X port 80

# Don't resolve hostnames
sudo tcpdump -n

# Don't resolve port names
sudo tcpdump -nn

# Increase snapshot length
sudo tcpdump -s 65535

# Add timestamps
sudo tcpdump -tttt

# Rotate capture files
sudo tcpdump -w capture.pcap -C 100  # 100MB per file
sudo tcpdump -w capture.pcap -G 3600  # Rotate every hour

Real-World Examples:

# Capture HTTP traffic to file
sudo tcpdump -i eth0 -s 65535 -w http.pcap 'port 80'

# Monitor DNS queries in real-time
sudo tcpdump -i any -n port 53

# Capture SSH traffic
sudo tcpdump -i eth0 'port 22'

# Debug HTTPS handshake
sudo tcpdump -i eth0 -nn -X 'port 443'

# Monitor specific conversation
sudo tcpdump -i eth0 'host 192.168.1.100 and host 192.168.1.200'

# Capture only SYN packets (connection attempts)
sudo tcpdump -i eth0 'tcp[tcpflags] & tcp-syn != 0 and tcp[tcpflags] & tcp-ack == 0'

# Monitor packet sizes
sudo tcpdump -i eth0 -n -l | awk '{print $NF}' | cut -d: -f1

Wireshark¶

Installation:

# Debian/Ubuntu
sudo apt-get install wireshark

# CentOS/RHEL
sudo yum install wireshark wireshark-gnome

# macOS
brew install --cask wireshark

Capture Filters (Before Capturing):

host 192.168.1.1              # Specific host
net 192.168.1.0/24           # Network range
port 80                       # Specific port
tcp port 443                  # TCP on port 443
not port 22                   # Exclude SSH
host 192.168.1.1 and port 80 # Combined

Display Filters (After Capturing):

# Protocol filters
tcp
udp
icmp
dns
http
tls
arp

# IP filters
ip.addr == 192.168.1.1
ip.src == 192.168.1.1
ip.dst == 192.168.1.1

# Port filters
tcp.port == 80
tcp.srcport == 80
tcp.dstport == 443

# HTTP filters
http.request
http.response
http.request.method == "GET"
http.request.uri contains "api"
http.response.code == 200

# DNS filters
dns.qry.name == "example.com"
dns.flags.response == 1

# TCP filters
tcp.flags.syn == 1
tcp.flags.reset == 1
tcp.analysis.retransmission
tcp.analysis.duplicate_ack

# Combine filters
tcp.port == 80 and ip.addr == 192.168.1.1
http and ip.src == 192.168.1.1

Advanced Display Filters:

# Show only errors
tcp.analysis.flags
http.response.code >= 400

# SSL/TLS handshake
ssl.handshake

# Slow responses (> 1 second)
http.time > 1

# Large packets
frame.len > 1000

# Follow TCP stream
tcp.stream eq 1

# Time-based
frame.time >= "2024-02-15 10:00:00"

# String search
frame contains "password"

Wireshark Features:

Follow Stream (Right-click → Follow → TCP/UDP/HTTP Stream)
- See complete conversation
- Useful for debugging protocols
Statistics
- Statistics → Conversations (see all connections)
- Statistics → Protocol Hierarchy (see protocol distribution)
- Statistics → I/O Graph (visualize traffic over time)
Expert Info
- Analyze → Expert Information
- Shows warnings, errors, notes
Packet Details
- Expand protocols to see fields
- Right-click → Apply as Filter
Time Analysis
- View → Time Display Format
- Set time references (Right-click → Set Time Reference)

Export Options:

File → Export Objects → HTTP  # Extract HTTP files
File → Export Packet Dissections → CSV
File → Export Specified Packets

Termshark (Terminal UI for Wireshark)¶

Installation:

# Using Go
go install github.com/gcla/termshark/v2/cmd/termshark@latest

# Debian/Ubuntu
sudo apt-get install termshark

# Arch Linux
yay -S termshark

# macOS
brew install termshark

Basic Usage:

# Capture live
sudo termshark -i eth0

# Read pcap file
termshark -r capture.pcap

# With display filter
termshark -r capture.pcap -Y 'http'

# Save filtered packets
termshark -r capture.pcap -Y 'tcp.port==80' -w filtered.pcap

Interface Navigation:

Keys:
  q     - Quit
  /     - Filter
  ?     - Help
  Tab   - Switch panes
  Enter - Select packet
  |     - Follow stream
  Space - Mark packet

Filters:
  Type filter in bottom bar
  Same syntax as Wireshark

Views:
  1 - Packet list
  2 - Packet details
  3 - Packet bytes

Termshark Features:

# Filter HTTP traffic
sudo termshark -i eth0 -Y 'http'

# Monitor DNS
sudo termshark -i any -Y 'dns'

# Follow TCP stream (press |)
# After selecting a packet

# Search (press /)
# Enter display filter

# Copy packets
# Mark with Space, then export

Advantages over Wireshark:

✅ Works over SSH
✅ Low resource usage
✅ Same filters as Wireshark
✅ Can run on servers

Advantages of Wireshark:

✅ Better for deep analysis
✅ Export capabilities
✅ Graphs and statistics
✅ Protocol dissectors

Real-World Scenarios¶

Scenario 1: Website Not Loading¶

Symptoms: User reports "example.com won't load"

Troubleshooting Steps:

# 1. Check DNS resolution
dig example.com +short
# or
dog example.com

# 2. Check if IP is reachable
ping -c 4 <IP_from_dig>

# 3. Check specific port
nping --tcp -p 80 example.com -c 4
nping --tcp -p 443 example.com -c 4

# 4. Trace route to see where problem is
mtr --report --report-cycles 10 example.com

# 5. If DNS is the issue, try different servers
dig @8.8.8.8 example.com
dig @1.1.1.1 example.com

# 6. Capture traffic to see what's happening
sudo tcpdump -i any -nn 'host example.com' -c 100

Scenario 2: Slow Website Performance¶

# 1. Measure DNS resolution time
dig example.com +stats | grep "Query time"

# 2. Check for packet loss and latency
mtr --report --report-cycles 50 example.com

# 3. Identify slow hop
mtr --tcp --port 443 example.com

# 4. Capture packets and analyze timing
sudo tcpdump -i eth0 -w slow.pcap 'host example.com'
# Analyze in Wireshark: Statistics → I/O Graph

# 5. Check TCP retransmissions
sudo tcpdump -i eth0 'tcp[tcpflags] & (tcp-syn|tcp-fin) != 0'

Scenario 3: Intermittent Connectivity¶

# 1. Continuous ping to detect drops
ping -i 0.2 google.com | ts '[%Y-%m-%d %H:%M:%S]'

# 2. MTR to identify problematic hop
mtr --interval 1 --report-cycles 300 example.com > mtr-long.txt

# 3. Capture during outage
sudo tcpdump -i eth0 -w intermittent.pcap

# 4. Analyze for patterns in Wireshark
# Look for: retransmissions, resets, timeouts

Scenario 4: DNS Issues¶

# 1. Check if DNS server responds
dig @8.8.8.8 google.com

# 2. Compare different DNS servers
dig @8.8.8.8 example.com +short
dig @1.1.1.1 example.com +short
dig @208.67.222.222 example.com +short  # OpenDNS

# 3. Trace DNS resolution
dig example.com +trace

# 4. Check DNSSEC validation
delv example.com

# 5. Monitor DNS queries
sudo tcpdump -i any -n port 53 -vv

# 6. Capture DNS traffic
sudo tcpdump -i any -w dns.pcap 'port 53'

Scenario 5: Port Blocked by Firewall¶

# 1. Test connectivity to port
nping --tcp -p 22 example.com
nping --tcp -p 80 example.com
nping --tcp -p 443 example.com

# 2. Check if SYN packets are making it through
sudo nping --tcp -p 80 --flags SYN example.com -c 5

# 3. Compare different ports
for port in 22 80 443 3306; do
    echo "Testing port $port"
    sudo nping --tcp -p $port example.com -c 1
done

# 4. Capture to see firewall drops
sudo tcpdump -i eth0 'tcp and port 80' -w firewall-test.pcap

Scenario 6: Email Delivery Issues¶

# 1. Check MX records
dig example.com MX +short

# 2. Test connectivity to mail server
ping mail.example.com
nping --tcp -p 25 mail.example.com
nping --tcp -p 465 mail.example.com  # SMTP with SSL
nping --tcp -p 587 mail.example.com  # SMTP with TLS

# 3. Check SPF records
dig example.com TXT | grep spf

# 4. Check DKIM
dig default._domainkey.example.com TXT

# 5. Check DMARC
dig _dmarc.example.com TXT

# 6. Capture SMTP conversation
sudo tcpdump -i eth0 -A 'port 25'

Troubleshooting Workflows¶

Complete Network Diagnosis Workflow¶

#!/bin/bash
# network-diagnose.sh - Complete network troubleshooting

TARGET=$1

echo "=== Network Diagnosis for $TARGET ==="

echo "
1. DNS Resolution"
dig $TARGET +short
dog $TARGET

echo "
2. Connectivity Test"
ping -c 4 $TARGET

echo "
3. Traceroute Analysis"
mtr --report --report-cycles 10 $TARGET

echo "
4. Port Scan (Common Ports)"
for port in 80 443 22 25; do
    timeout 2 bash -c "echo > /dev/tcp/$TARGET/$port" 2>/dev/null && echo "Port $port: OPEN" || echo "Port $port: CLOSED"
done

echo "
5. Packet Capture (10 seconds)"
sudo timeout 10 tcpdump -i any -c 50 "host $TARGET" -w ${TARGET}-capture.pcap 2>/dev/null
echo "Capture saved to ${TARGET}-capture.pcap"

echo "
=== Diagnosis Complete ==="

DNS Troubleshooting Workflow¶

#!/bin/bash
# dns-diagnose.sh

DOMAIN=$1

echo "=== DNS Diagnosis for $DOMAIN ==="

echo "
1. A Record"
dig $DOMAIN A +short

echo "
2. AAAA Record (IPv6)"
dig $DOMAIN AAAA +short

echo "
3. MX Records"
dig $DOMAIN MX +short

echo "
4. NS Records"
dig $DOMAIN NS +short

echo "
5. TXT Records"
dig $DOMAIN TXT +short

echo "
6. DNS Propagation Check"
for server in 8.8.8.8 1.1.1.1 208.67.222.222; do
    echo "DNS Server: $server"
    dig @$server $DOMAIN A +short
done

echo "
7. DNSSEC Validation"
delv $DOMAIN | grep validated

echo "
8. Full Trace"
dig $DOMAIN +trace +additional | tail -20

echo "
=== DNS Diagnosis Complete ==="

Packet Loss Investigation¶

#!/bin/bash
# packet-loss.sh

TARGET=$1
DURATION=${2:-300}  # Default 5 minutes

echo "=== Packet Loss Investigation for $TARGET ==="
echo "Duration: $DURATION seconds"

# Run MTR
echo "Running MTR analysis..."
mtr --report --report-cycles $((DURATION / 3)) $TARGET > mtr-${TARGET}.txt

# Continuous ping
echo "Running continuous ping..."
ping -i 0.5 -c $((DURATION * 2)) $TARGET > ping-${TARGET}.txt

# Analyze results
echo "
=== Results ==="
echo "MTR Report:"
cat mtr-${TARGET}.txt

echo "
Ping Statistics:"
tail -5 ping-${TARGET}.txt

echo "
Packet Loss Summary:"
grep -o "[0-9]*% packet loss" ping-${TARGET}.txt

Quick Reference¶

DNS Lookup Cheat Sheet¶

# Quick DNS lookup
dig example.com +short

# All records
dig example.com ANY

# Specific server
dig @8.8.8.8 example.com

# Reverse lookup
dig -x 8.8.8.8

# Trace
dig example.com +trace

# Modern alternative
dog example.com

Connectivity Cheat Sheet¶

# Basic ping
ping -c 4 example.com

# TCP ping
sudo nping --tcp -p 80 example.com

# Traceroute
mtr --report -c 10 example.com

# Port test
nc -zv example.com 80

Packet Capture Cheat Sheet¶

# Capture to file
sudo tcpdump -i eth0 -w capture.pcap

# Filter by host
sudo tcpdump host 192.168.1.1

# Filter by port
sudo tcpdump port 80

# Read file
tcpdump -r capture.pcap

# Terminal UI
sudo termshark -i eth0

Best Practices¶

Always start simple - ping before packet capture
Use the right tool - don't use tcpdump when ping would work
Save captures - always save pcap files for later analysis
Document findings - keep notes on what you tried
Compare baselines - know what "normal" looks like
Check multiple DNS servers - don't trust just one
Use filters - focus on relevant traffic only
Time your tests - network issues can be time-dependent

This guide covers the essential tools and techniques for network troubleshooting!