K8s-Guestbook-IP

[peterlamar]

As a tenant, when I run the guestbook example I can easily reach the guestbook itself so that the platform is easy to introduce to new tenants

Additional Info: Currently, this needs /etc/host modifications and other workarounds. It should be possible to enable a good experience

[davidwalter0]

Some thoughts.

Does this mean from an external host [ over a public ip ] or from within the k8s cluster?

If we assume that the k8s cluster is using flannel, we can separate the problem into two different pieces.
For this example assume that flannel is a private subnet of 172.24.0.0/16 managed by flannel & k8s
nodes:
node0: 10.1.12.10
node1: 10.1.12.11
node2: 10.1.12.12
run etcd on node0

cloudinit configuration for etcd2

#cloud-config

coreos:
  etcd2:
    advertise-client-urls: http://10.1.12.10:2379
    listen-client-urls: http://0.0.0.0:2379,http://0.0.0.0:4001
    listen-peer-urls: http://10.1.12.10:2380
    proxy: off

  units:
    - name: etcd2.service
      command: restart
      enable: true

flannel drop in for cloudinit [systemd]

  drop-ins:
    - name: 50-network-config.conf
      content: |
        [Service]
        ExecStartPre=-/usr/bin/etcdctl --peers=10.1.12.10:2379 set /coreos.com/network/config '{ "Network": "172.24.0.0/16" }'
        [Install]
        WantedBy=multi-user.target

coreos-cloudinit --from-file etcd.yaml

$ for node in $(etcdctl ls --recursive /coreos.com/network); do echo ${node} $(etcdctl get ${node} 2>/dev/null); done
/coreos.com/network/config { "Network": "172.24.0.0/16" }
/coreos.com/network/subnets
/coreos.com/network/subnets/172.24.5.0-24 {"PublicIP":"10.1.12.11"}
/coreos.com/network/subnets/172.24.92.0-24 {"PublicIP":"10.1.12.10"}
/coreos.com/network/subnets/172.24.95.0-24 {"PublicIP":"10.1.12.12"}

These are directly routeable via the host routing table

$ ip r
default via 10.1.12.1 dev eth0  proto dhcp  src 10.1.12.10  metric 1024 
10.1.12.0/24 dev eth0  proto kernel  scope link  src 10.1.12.10 
10.1.12.1 dev eth0  proto dhcp  scope link  src 10.1.12.10  metric 1024 
172.24.0.0/16 dev flannel0  proto kernel  scope link  src 172.24.92.0 
172.24.92.0/24 dev docker0  proto kernel  scope link  src 172.24.92.1 

So flannel fixes container to container and host to containers connectivity via ip.

[davidwalter0]

kubectl exec redis-master-xwjsb env

{
"HOME": "/",
"HOSTNAME": "guestbook-di4c6",
"REDIS_MASTER_PORT": "tcp://172.24.254.79:6379",
"REDIS_MASTER_PORT_6379_TCP": "tcp://172.24.254.79:6379",
"REDIS_MASTER_PORT_6379_TCP_ADDR": "172.24.254.79",
"REDIS_MASTER_PORT_6379_TCP_PORT": "6379",
"REDIS_MASTER_PORT_6379_TCP_PROTO": "tcp",
"REDIS_MASTER_SERVICE_HOST": "172.24.254.79",
"REDIS_MASTER_SERVICE_PORT": "6379",
"REDIS_SLAVE_PORT": "tcp://172.24.254.80:6379",
"REDIS_SLAVE_PORT_6379_TCP": "tcp://172.24.254.80:6379",
"REDIS_SLAVE_PORT_6379_TCP_ADDR": "172.24.254.80",
"REDIS_SLAVE_PORT_6379_TCP_PORT": "6379",
"REDIS_SLAVE_PORT_6379_TCP_PROTO": "tcp",
"REDIS_SLAVE_SERVICE_HOST": "172.24.254.80",
"REDIS_SLAVE_SERVICE_PORT": "6379",
}

All of the internal redirects that are dynamically inserted are
defaulting to ip addresses only visible to the private subnet for the
flannel members.

curl can connect to the services and members on their service ip / ports from a
member host.

From a remote host [ with port redirection and a exposing ports to a
public ip ] the guest book can be accessed, but it gives a waiting on
database connections message on the front end and the back end [info
link] gives an error.

The internal lookup for redis it appears isn't using the environment
variable for the ip address to resolve the service, rather a name
'redis-master' so for internal application discovery it seems like dns
needs to be enabled.

PANIC: dial tcp: lookup redis-master: no such host
goroutine 5 [running]:
github.com/codegangsta/negroni.func·003()
    /gopath/src/github.com/codegangsta/negroni/recovery.go:34 +0x10d

So it looks like for this step the guest book is assuming that the cluster has
dns enabled.

[davidwalter0]

https://github.com/kubernetes/kubernetes/blob/v1.0.3/cluster/addons/dns/README.md

It appears that there was a missing parameter in the yaml for dns.

      - name: kube2sky
        image: gcr.io/google_containers/kube2sky:1.11
        resources:
          limits:
            cpu: 100m
            memory: 50Mi
        args:
        # command = "/kube2sky"
        - -domain=k8s.local
        - -kube_master_url=http://10.1.12.10:8080

Without the kube master defined there were errors in the log.

similar to this issue:

kubernetes/kubernetes#12534

    - -kube_master_url=http://10.1.12.10:8080

But the example nslookup in a busybox cluster member pod fails.

kubectl exec busybox -- nslookup kubernetes

Edit
After clearing the cluster and restarting with dns, the cluster config appears to be working with full dns discovery

core@coreos-alpha-00 ~ $ kubectl exec -it busybox -- cat /etc/resolv.conf 
nameserver 172.24.254.53
nameserver 10.1.12.1
search default.svc.k8s.local svc.k8s.local k8s.local novalocal
options ndots:5

core@coreos-alpha-00 ~ $ kubectl exec -it busybox -- nslookup redis-master
Server:    172.24.254.53
Address 1: 172.24.254.53

Name:      redis-master
Address 1: 172.24.254.79

core@coreos-alpha-00 ~ $ kubectl exec -it busybox -- nslookup redis-slave
Server:    172.24.254.53
Address 1: 172.24.254.53

Name:      redis-slave
Address 1: 172.24.254.80

Though it seems that some number of times the lookup fails, possibly due to etcd entry timeout configurations

A failed lookup

curl http://208.90.61.54:3000/info
PANIC: dial tcp: lookup redis-master: no such host
goroutine 11 [running]:
github.com/codegangsta/negroni.func·003()

A successful lookup

$ curl http://208.90.61.54:3000/info
# Server
redis_version:3.0.3
os:Linux 4.1.6-coreos-r1 x86_64
arch_bits:64
. . . 
run_id:bcfd2fe00bb87d17d91d5d384f8a49d7506627db
tcp_port:6379

# Clients
connected_clients:6

# Memory
used_memory_human:1.62M

[altvnk]

About service discovery. It depends which deployment you are used to preform testing. If you used this automation, then service discovery (kube-dns addon) must work OOB, if not - it's a bug.

[davidwalter0]

This was on a non-standard install to coreos instances with coreos-cloudinit.

It seems that this crosses over with #12 Automated IP Assignment - Design in that the problem described says there's a problem with DNS. The guestbook works with kubedns/skydns and cluster discovery.

If this is working OOB, should this be closed?