Symptom:

  • Service/Pod could create success, but could not connect to pod.
  • Could not connect to another pod in another node (even in the same node)
  • All kubectl status works well
  • Your docker is newer than 1.13 (it works well if your docker version is 1.12)

It will happen on “kubeadm” but not happen in “minikube”.

Diagnosis:

Check iptable rule.

sudo iptables-save

-A INPUT -j KUBE-FIREWALL
-A FORWARD -j DOCKER-ISOLATION
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -
j ACCEPT
-A FORWARD -i docker0 ! -o docker0 -j DROP
-A FORWARD -i docker0 -o docker0 -j DROP
-A OUTPUT -j KUBE-FIREWALL
-A DOCKER-ISOLATION -j RETURN

As you could observe “A FORWARD -i docker0 ! -o docker0 -j DROP

Root cause:

Refer to moby issue 40182 (still not resolve until kubernetes 1.8)

Solution:

  1. Downgrade to docker v1.12.x
  2. Add iptable forward rule to all (not suggest)
    • sudo iptables -P FORWARD ACCEPT
  3. Start every container with docker --iptables=false (not easy when you use kubernetes)

Refer great slide “All The Troubles You Get Into When Setting up a Production-ready Kubernetes Cluster” by Jimmy Lu

Reference:


Evan

Attitude is everything