Problem with two containers communicating with each others on the same host

[ahoiroman]

Hi,

I am using this module to configure iptables to work with docker.

The context

I got two docker-containers:

• Container Adding tests and documentation. #1 is hosting an open source ticketing system
• Container use firewall class for package and firewalld fix, CentOS7 #2 is hosting a mailserver

The mailserver is up and running and I can connect to it using my local PC and a default mailclient.

d53cb727de95   tvial/docker-mailserver  "supervisord -c /etc…"   4 days ago    Up 3 minutes   0.0.0.0:25->25/tcp, :::25->25/tcp, 110/tcp, 0.0.0.0:143->143/tcp, :::143->143/tcp, 0.0.0.0:587->587/tcp, :::587->587/tcp, 465/tcp, 995/tcp, 0.0.0.0:993->993/tcp, :::993->993/tcp, 4190/tcp   mail

Using nmap on my local computer I can see that the port is open wide:

PORT    STATE SERVICE
993/tcp open  imaps

iptables got these rules:

-P INPUT ACCEPT
-P FORWARD DROP
-P OUTPUT ACCEPT
-N f2b-sshd
-N DOCKER
-N DOCKER-ISOLATION-STAGE-1
-N DOCKER-ISOLATION-STAGE-2
-N DOCKER-USER
-A INPUT -p tcp -m multiport --dports 22 -j f2b-sshd
-A FORWARD -j DOCKER-USER
-A FORWARD -j DOCKER-ISOLATION-STAGE-1
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A FORWARD -o br-fe03c792b7c6 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o br-fe03c792b7c6 -j DOCKER
-A FORWARD -i br-fe03c792b7c6 ! -o br-fe03c792b7c6 -j ACCEPT
-A FORWARD -i br-fe03c792b7c6 -o br-fe03c792b7c6 -j ACCEPT
-A FORWARD -o br-8d0b3e4c5cd8 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o br-8d0b3e4c5cd8 -j DOCKER
-A FORWARD -i br-8d0b3e4c5cd8 ! -o br-8d0b3e4c5cd8 -j ACCEPT
-A FORWARD -i br-8d0b3e4c5cd8 -o br-8d0b3e4c5cd8 -j ACCEPT
-A FORWARD -o br-12fecdadfb26 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o br-12fecdadfb26 -j DOCKER
-A FORWARD -i br-12fecdadfb26 ! -o br-12fecdadfb26 -j ACCEPT
-A FORWARD -i br-12fecdadfb26 -o br-12fecdadfb26 -j ACCEPT
-A f2b-sshd -s 95.236.224.201/32 -j REJECT --reject-with icmp-port-unreachable
-A f2b-sshd -j RETURN
-A DOCKER -d 172.22.0.2/32 ! -i br-fe03c792b7c6 -o br-fe03c792b7c6 -p tcp -m tcp --dport 993 -j ACCEPT
-A DOCKER -d 172.22.0.2/32 ! -i br-fe03c792b7c6 -o br-fe03c792b7c6 -p tcp -m tcp --dport 587 -j ACCEPT
-A DOCKER -d 172.22.0.2/32 ! -i br-fe03c792b7c6 -o br-fe03c792b7c6 -p tcp -m tcp --dport 143 -j ACCEPT
-A DOCKER -d 172.22.0.2/32 ! -i br-fe03c792b7c6 -o br-fe03c792b7c6 -p tcp -m tcp --dport 25 -j ACCEPT
-A DOCKER -d 192.168.64.4/32 ! -i br-8d0b3e4c5cd8 -o br-8d0b3e4c5cd8 -p tcp -m tcp --dport 8080 -j ACCEPT
-A DOCKER -d 192.168.64.4/32 ! -i br-8d0b3e4c5cd8 -o br-8d0b3e4c5cd8 -p tcp -m tcp --dport 443 -j ACCEPT
-A DOCKER -d 192.168.64.4/32 ! -i br-8d0b3e4c5cd8 -o br-8d0b3e4c5cd8 -p tcp -m tcp --dport 80 -j ACCEPT
-A DOCKER-ISOLATION-STAGE-1 -i docker0 ! -o docker0 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -i br-fe03c792b7c6 ! -o br-fe03c792b7c6 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -i br-8d0b3e4c5cd8 ! -o br-8d0b3e4c5cd8 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -i br-12fecdadfb26 ! -o br-12fecdadfb26 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -j RETURN
-A DOCKER-ISOLATION-STAGE-2 -o docker0 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -o br-fe03c792b7c6 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -o br-8d0b3e4c5cd8 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -o br-12fecdadfb26 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -j RETURN
-A DOCKER-USER -j RETURN

The problem

As I am using puppet to manage this server, I created a module that manages iptables for me. As docker is terrible to use this way, I had to write some rules to ignore those docker-related rules.

This is the result of my puppetized iptables:

-P INPUT DROP
-P FORWARD DROP
-P OUTPUT DROP
-N f2b-sshd
-N DOCKER
-N DOCKER-ISOLATION-STAGE-1
-N DOCKER-ISOLATION-STAGE-2
-N DOCKER-USER
-N DROP_INPUT
-N DROP_OUTPUT
-N DROP_FORWARD
-A INPUT -i lo -m comment --comment "000 allow incoming on loopback" -j ACCEPT
-A INPUT -i lo -m comment --comment "000 allow incoming on loopback IPv6" -j ACCEPT
-A INPUT -p tcp -m tcp --tcp-flags FIN,SYN FIN,SYN -m comment --comment "005 drop bogus fin,syn" -j DROP_INPUT
-A INPUT -p tcp -m tcp --tcp-flags SYN,RST SYN,RST -m comment --comment "006 drop bogus syn,rst" -j DROP_INPUT
-A INPUT -m state --state RELATED,ESTABLISHED -m comment --comment "007 allow incoming established, related" -j ACCEPT
-A INPUT -m state --state RELATED,ESTABLISHED -m comment --comment "007 allow incoming established, related IPv6" -j ACCEPT
-A INPUT -p icmp -m comment --comment "008 allow incoming icmp echo-requests" -j ACCEPT
-A INPUT -p ipv6-icmp -m comment --comment "008 allow incoming icmp echo-requests IPv6" -j ACCEPT
-A INPUT -p tcp -m multiport --dports 22 -m comment --comment "020 allow incoming ssh" -j ACCEPT
-A INPUT -p tcp -m multiport --dports 22 -m comment --comment "020 allow incoming ssh IPv6" -j ACCEPT
-A INPUT -m comment --comment "999 drop all incoming" -j DROP_INPUT
-A INPUT -m comment --comment "999 drop all incoming IPv6" -j DROP_INPUT
-A FORWARD -j DOCKER-USER
-A FORWARD -j DOCKER-ISOLATION-STAGE-1
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A FORWARD -o br-fe03c792b7c6 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o br-fe03c792b7c6 -j DOCKER
-A FORWARD -i br-fe03c792b7c6 ! -o br-fe03c792b7c6 -j ACCEPT
-A FORWARD -i br-fe03c792b7c6 -o br-fe03c792b7c6 -j ACCEPT
-A FORWARD -o br-8d0b3e4c5cd8 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o br-8d0b3e4c5cd8 -j DOCKER
-A FORWARD -i br-8d0b3e4c5cd8 ! -o br-8d0b3e4c5cd8 -j ACCEPT
-A FORWARD -i br-8d0b3e4c5cd8 -o br-8d0b3e4c5cd8 -j ACCEPT
-A FORWARD -o br-12fecdadfb26 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o br-12fecdadfb26 -j DOCKER
-A FORWARD -i br-12fecdadfb26 ! -o br-12fecdadfb26 -j ACCEPT
-A FORWARD -i br-12fecdadfb26 -o br-12fecdadfb26 -j ACCEPT
-A FORWARD -m comment --comment "999 drop all forwarding" -j DROP_FORWARD
-A FORWARD -m comment --comment "999 drop all forwarding IPv6" -j DROP_FORWARD
-A OUTPUT -o lo -m comment --comment "000 allow outgoing on loopback" -j ACCEPT
-A OUTPUT -o lo -m comment --comment "000 allow outgoing on loopback IPv6" -j ACCEPT
-A OUTPUT -m state --state RELATED,ESTABLISHED -m comment --comment "005 allow outgoing established, related" -j ACCEPT
-A OUTPUT -m state --state RELATED,ESTABLISHED -m comment --comment "005 allow outgoing established, related IPv6" -j ACCEPT
-A OUTPUT -m state --state NEW -m comment --comment "006 allow new outgoing" -j ACCEPT
-A OUTPUT -m state --state NEW -m comment --comment "006 allow new outgoing IPv6" -j ACCEPT
-A OUTPUT -m comment --comment "999 drop all outgoing" -j DROP_OUTPUT
-A OUTPUT -m comment --comment "999 drop all outgoing IPv6" -j DROP_OUTPUT
-A f2b-sshd -s 95.236.224.201/32 -j REJECT --reject-with icmp-port-unreachable
-A f2b-sshd -j RETURN
-A DOCKER -d 172.22.0.2/32 ! -i br-fe03c792b7c6 -o br-fe03c792b7c6 -p tcp -m tcp --dport 993 -j ACCEPT
-A DOCKER -d 172.22.0.2/32 ! -i br-fe03c792b7c6 -o br-fe03c792b7c6 -p tcp -m tcp --dport 587 -j ACCEPT
-A DOCKER -d 172.22.0.2/32 ! -i br-fe03c792b7c6 -o br-fe03c792b7c6 -p tcp -m tcp --dport 143 -j ACCEPT
-A DOCKER -d 172.22.0.2/32 ! -i br-fe03c792b7c6 -o br-fe03c792b7c6 -p tcp -m tcp --dport 25 -j ACCEPT
-A DOCKER -d 192.168.64.4/32 ! -i br-8d0b3e4c5cd8 -o br-8d0b3e4c5cd8 -p tcp -m tcp --dport 8080 -j ACCEPT
-A DOCKER -d 192.168.64.4/32 ! -i br-8d0b3e4c5cd8 -o br-8d0b3e4c5cd8 -p tcp -m tcp --dport 443 -j ACCEPT
-A DOCKER -d 192.168.64.4/32 ! -i br-8d0b3e4c5cd8 -o br-8d0b3e4c5cd8 -p tcp -m tcp --dport 80 -j ACCEPT
-A DOCKER-ISOLATION-STAGE-1 -i docker0 ! -o docker0 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -i br-fe03c792b7c6 ! -o br-fe03c792b7c6 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -i br-8d0b3e4c5cd8 ! -o br-8d0b3e4c5cd8 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -i br-12fecdadfb26 ! -o br-12fecdadfb26 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -j RETURN
-A DOCKER-ISOLATION-STAGE-2 -o docker0 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -o br-fe03c792b7c6 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -o br-8d0b3e4c5cd8 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -o br-12fecdadfb26 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -j RETURN
-A DOCKER-USER -j RETURN
-A DROP_INPUT -m limit --limit 10/min -m comment --comment "000 log dropped INPUT IPv4" -j LOG --log-prefix "iptables DROP_INPUT: " --log-level 7
-A DROP_INPUT -m comment --comment "001 drop INPUT IPv4" -j DROP
-A DROP_OUTPUT -m limit --limit 10/min -m comment --comment "000 log dropped OUTPUT IPv4" -j LOG --log-prefix "iptables DROP_OUTPUT: " --log-level 7
-A DROP_OUTPUT -m comment --comment "001 drop OUTPUT IPv4" -j DROP
-A DROP_FORWARD -m limit --limit 10/min -m comment --comment "000 log dropped FORWARD IPv4" -j LOG --log-prefix "iptables DROP_FORWARD: " --log-level 7
-A DROP_FORWARD -m comment --comment "001 drop FORWARD IPv4" -j DROP

This is working, too - for my local computer. But having these rules enabled, Container #1 can no longer connect to Container #1 (host unreachable) and I do not see the reason.