September 27th, 2017
Symptom:
- Service/Pod could create success, but could not connect to pod.
- Could not connect to another pod in another node (even in the same node)
- All kubectl status works well
- Your docker is newer than 1.13 (it works well if your docker version is 1.12)
It will happen on “kubeadm” but not happen in “minikube”.
Diagnosis:
Check iptable rule.
sudo iptables-save
-A INPUT -j KUBE-FIREWALL
-A FORWARD -j DOCKER-ISOLATION
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -
j ACCEPT
-A FORWARD -i docker0 ! -o docker0 -j DROP
-A FORWARD -i docker0 -o docker0 -j DROP
-A OUTPUT -j KUBE-FIREWALL
-A DOCKER-ISOLATION -j RETURN
As you could observe “A FORWARD -i docker0 ! -o docker0 -j DROP”
Root cause:
Refer to moby issue 40182 (still not resolve until kubernetes 1.8)
- Docker 1.13 changed the default iptables forwarding policy to DROP
- So all forward traffic will be blocked by the default iptable forward rule.
Solution:
- Downgrade to docker v1.12.x
- Add iptable forward rule to all (not suggest)
sudo iptables -P FORWARD ACCEPT
- Start every container with
docker --iptables=false(not easy when you use kubernetes)
More related problem?
Refer great slide “All The Troubles You Get Into When Setting up a Production-ready Kubernetes Cluster” by Jimmy Lu
Reference: