Cluster Introspection

kubectl get services                # List all services 
kubectl get pods                    # List all pods
kubectl get nodes -w                # Watch nodes continuously
kubectl version                     # Get version information
kubectl cluster-info                # Get cluster information
kubectl config view                 # Get the configuration
kubectl describe node <node>        # Output information about a node

Pod and Container Introspection

kubectl get pods                         # List the current pods
kubectl describe pod <name>              # Describe pod <name>
kubectl get rc                           # List the replication controllers
kubectl get rc --namespace="<namespace>" # List the replication controllers in <namespace>
kubectl describe rc <name>               # Describe replication controller <name>
kubectl get svc                          # List the services
kubectl describe svc <name>              # Describe service <name>

Interacting with Pods

kubectl run <name> --image=<image-name>
# Launch a pod called <name>
# using image <image-name>
 
kubectl create -f <manifest.yaml>
# Create a service described
# in <manifest.yaml>

kubectl scale --replicas=<count> rc <name>
# Scale replication controller
# <name> to <count> instances
 
kubectl expose rc <name> --port=<external> --target-port=<internal>
# Map port <external> to
# port <internal> on replication
# controller <name>

Stopping Kubernetes

kubectl delete pod <name>                                         # Delete pod <name>
kubectl delete rc <name>                                          # Delete replication controller <name>
kubectl delete svc <name>                                         # Delete service <name>
kubectl drain <n> --delete-local-data --force --ignore-daemonsets # Stop all pods on <n>
kubectl delete node <name>                                        # Remove <node> from the cluster

Debugging

kubectl exec <service> <command> [-c <$container>]
# execute <command> on <service>, optionally 
# selecting container <$container>
 
kubectl logs -f <name> [-c <$container>]
# Get logs from service <name>, optionally
# selecting container <$container>
 
watch -n 2 cat /var/log/kublet.log
# Watch the Kublet logs
kubectl top node
# Show metrics for nodes
kubectl top pod
# Show metrics for pods

Administration

kubeadm init    
# Initialize your master node
kubeadm join --token <token> <master-ip>:<master-port> 
# Join a node to your Kubernetes cluster
kubectl create namespace <namespace>  
# Create namespace <name>
kubectl taint nodes --all node-role.kubernetes.io/master-
# Allow Kubernetes master nodes to run pods
kubeadm reset 
# Reset current state
kubectl get secrets 
# List all secrets

Troubleshooting Networking Issues

IP forwarding

allows forwarding of the traffic coming from one interface to be routed to another interface.

is necessary for Linux kernel to route traffic from containers to the outside world.

otherwise Pod to service connection times out, tcpdump could show that lots of repeated SYN packets are sent, but no ACK is received.

# check that  ipv4 forwarding is enabled
sysctl net.ipv4.ip_forward
# 0 means that forwarding is disabled
net.ipv4.ip_forward = 0
# this will turn things back on a live server
sysctl -w net.ipv4.ip_forward=1
# on Centos this will make the setting apply after reboot
echo net.ipv4.ip_forward=1 >> /etc/sysconf.d/10-ipv4-forwarding-on.conf

Bridge-netfilter

enables iptables rules to work on Linux bridges just like the ones set up by Docker and Kubernetes.

is necessary for the Linux kernel to be able to perform address translation in packets going to and from hosted containers.

otherwise network requests to services outside the Pod network will start timing out with destination host unreachable or connection refused errors.

# check that bridge netfilter is enabled
sysctl net.bridge.bridge-nf-call-iptables
# 0 means that bridging is disabled
net.bridge.bridge-nf-call-iptables = 0
# Note some distributions may have this compiled with kernel,
# check with cat /lib/modules/$(uname -r)/modules.builtin | grep netfilter
modprobe br_netfilter
# turn the iptables setting on
sysctl -w net.bridge.bridge-nf-call-iptables=1
echo net.bridge.bridge-nf-call-iptables=1 >> /etc/sysconf.d/10-bridge-nf-call-iptables.conf

Firewall rules block overlay network traffic

one of most common on-premise Kubernetes networking setups leverages a VxLAN overlay network, where IP packets are encapsulated in UDP and sent over port 8472.

there is 100% packet loss between pod IPs either with lost packets or destination host unreachable.

better to use the same protocol to transfer the data, as firewall rules can be protocol specific, e.g. could be blocking UDP traffic.

#  on the server side
iperf -s -p 8472 -u
# on the client side 
iperf -c 172.28.128.103 -u -p 8472 -b 1K
# to fix, update the firewall rule to stop blocking the traffic

AWS source/destination check

AWS performs source destination check by default. This means that AWS checks if the packets going to the instance have the target address as one of the instance IPs.

many Kubernetes networking backends use target and source IP addresses that are different from the instance IP addresses to create Pod overlay networks.

when problem arise, pod to service connection times out, tcpdump could show that lots of repeated SYN packets are sent, without a corresponding ACK anywhere in sight.

Pod CIDR conflicts

Kubernetes sets up special overlay network for container to container communication.

with isolated pod network, containers can get unique IPs and avoid port conflicts on a cluster.

problem arise when Pod network subnets start conflicting with host networks, pod to pod communication is disrupted with routing problems.

# start with a quick look at the allocated pod IP addresses
kubectl get pods -o wide
# compare host IP range with the kubernetes subnets specified in the apiserver
ip addr list

Kubernetes imposes the following fundamental requirements on any networking implementation (barring any intentional network segmentation policies):

• all containers can communicate with all other containers without NAT
• all nodes can communicate with all containers (and vice-versa) without NAT
• the IP that a container sees itself as is the same IP that others see it as