- Install etcd server
- Install docker
- Install flanneld and put config in etcd
- Install Kubernetes package
- Configure Kubernetes master
- Configure Kubernetes minion
- Install kube-dns service discovery and DNS resolution pod
- DNS for Service Discovery
- Kubernetes UI
- Logging Service for containers
Each addon is enabled by default but can be disabled by changing the options within group_vars/all.yml
All of the addons depend on the DNS addon for Service Discovery.
-
Install terraform
# change 0.6.9 to the desired version wget -q -O terraform.zip https://dl.bintray.com/mitchellh/terraform/terraform_0.6.9_linux_amd64.zip unzip terraform.zip -d /usr/local/bin -
Install pip package (CentOS/RedHat)
# yum install python-devel python-pip -
Install the OpenStack command-line clients
When following the instructions in this section, replace PROJECT with the lowercase name of the client to install, such as nova. Repeat for each client. The following values are valid:
- barbican - Key Manager Service API
- ceilometer - Telemetry API
- cinder - Block Storage API and extensions
- glance - Image service API
- heat - Orchestration API
- magnum - Containers service API
- manila - Shared file systems API
- mistral - Workflow service API
- murano - Application catalog API
- neutron - Networking API
- nova - Compute API and extensions
- sahara - Data Processing API
- swift - Object Storage API
- trove - Database service API
- tuskar - Deployment service API
- openstack - Common OpenStack client supporting multiple services
# How to install with pip:
pip install python-PROJECTclient
(Replace PROJECT with the lowercase name of the client)
# How to update with pip:
pip install --upgrade python-PROJECTclient
(Replace PROJECT with the lowercase name of the client)
# To remove the client, run the pip uninstall command:
pip uninstall python-PROJECTclient
(Replace PROJECT with the lowercase name of the client)
Additional OpenStack CLI information here
-
Download Openstack RC file from Openstack Project Web Interface(Access & Security --> API Access)
source openrc.sh # prompted for your password for the Openstack Project -
Provide configurations for Openstack (From GitHub branch https://github.com/CiscoCloud/kubernetes-ansible/tree/master/terraform)
cp terraform/openstack.sample.tf openstack.tf cp terraform/terraform.tfvars terraform.tfvars # edit the terraform.tfvars file by providing the following - auth_url (found within *openrc.sh) - tenant_id (found within *openrc.sh) - tenant_name (found within *openrc.sh) - location of ssh public key and a unique name - VM Flavor for Master node (*nova flavor-list) - VM Flavor for Worker node (*nova flavor-list) - Network ID (*nova net-list) - OS Image Name (*nova image-list) - Number of worker nodes - size (GB) of storage for kubernetes master to use *nova - example if you using NOVA Compute API and extensions *openrc.sh - Openstack Project Web Interface
Note: You must use image with pre-installed cloud-init.
-
Provision Environment
terraform get (Get modules) terraform plan (Checking configuration) terraform apply (Apply ) -
Verify SSH access to the hosts
ansible -m ping all
If you are running on a cloud provider make sure that firewall configuration permits traffic between nodes. If you used terraform to provision environment then a security group has already been created for you.
Port list on roles: TBD
There is 3 main roles:
- etcd server
- kubernetes master
- kubernetes node (minion)
You can safely combine etcd and kubernetes master on one host, eventually you can run kubernetes minion on that host also.
For this setup you will need 1 host that would be kubernetes master and 2 or more hosts as minions.
At least 2 minion nodes are needed to use flannel or any other networking for kubernetes.
If you already have prepared hosts you can provide simple ansible inventory (sample is in root of project).
Use Getting started with Ansible if you are not familiar with ansible.
Verify that ansible can reach your hosts.
ansible -m ping all
Validate the global configurations found in group_vars/all.yml and update as needed.
To run ansible on hosts you prepared run:
ansible-playbook -i inventory setup.yml
If you used Terraform to provision your hosts, a plugin is provided that dynamically extracts the inventory from .tfstate file.
ansible-playbook setup.yml
The same plugin can be used to either print out a lists of hosts or to add those hosts to your local /etc/hosts file so that you can reference the hosts by name.
The following command will append the hosts to your /etc/hosts file.
./plugins/inventory/terraform.py --hostfile >> /etc/hosts
Install Serverspec environment :
bundle install --path vendor/bundle
Run Serverspec test for all nodes and specs in parallel (using 8 threads), print short summary in JSON format and provide 0 exit code for succeed validation of Ansible playbooks :
bundle exec rake -m -j 8
Run Serverspec tests for different plays in parallel :
bundle exec rake spec:play:All -m -j 8
bundle exec rake spec:play:Master -m -j 8
bundle exec rake spec:play:Node -m -j 8
Show all available Rake-tasks :
bundle exec rake -T
To use different RSpec output formats (json is default one) :
FORMAT=documentation bundle exec rake spec:play:All -m -j 8
FORMAT=json bundle exec rake spec:play:All -m -j 8
FORMAT=progress bundle exec rake spec:play:All -m -j 8
When using FORMAT=json (default) the output will contain tests summary only :
{
"succeed": true,
"example_count": 490,
"failure_count": 0
}Detailed results could be found inside serverspec_results.json file at project root directory :
[
{
"name": "docker::k-master-01",
"exit_code": 0,
"output": {
"version": "3.4.0",
"examples": [
{
"description": "should be installed",
"full_description": "docker : Main | Package \"docker\" should be installed",
"status": "passed",
"file_path": "./roles/docker/spec/main_spec.rb",
"line_number": 5,
"run_time": 3.202775,
"pending_message": null
},
{
"description": "should be enabled",
"full_description": "docker : Main | Service \"docker\" should be enabled",
"status": "passed",
"file_path": "./roles/docker/spec/main_spec.rb",
"line_number": 9,
"run_time": 0.443939,
"pending_message": null
}
],
"summary": {
"duration": 4.07774,
"example_count": 3,
"failure_count": 0,
"pending_count": 0
},
"summary_line": "3 examples, 0 failures"
}
},
{
"name": "flannel::k-master-01",
"exit_code": 0,
"output": {
"version": "3.4.0",
"examples": [
{
"description": "should be installed",
"full_description": "flannel : Main | Service | Package \"flannel\" should be installed",
"status": "passed",
"file_path": "./roles/flannel/spec/main_spec.rb",
"line_number": 6,
"run_time": 3.253822,
"pending_message": null
}
],
"summary": {
"duration": 6.399068,
"example_count": 10,
"failure_count": 0,
"pending_count": 0
},
"summary_line": "10 examples, 0 failures"
}
}
]-
Check if all nodes are ready
kubectl get nodes -o wide -
Check if all Pods, Replication Controllers and Services are running
kubectl get rc,svc,po --all-namespaces -o wide -
Check status of Kubernetes processes
sudo systemctl status etcd kube-apiserver kube-controller-manager kube-scheduler -l -
View logs of Kubernetes processes
sudo journalctl -u etcd sudo journalctl -u kube-apiserver sudo journalctl -u kube-controller-manager sudo journalctl -u kube-scheduler -
Verify DNS working
# https://github.com/GoogleCloudPlatform/kubernetes/tree/v1.0.1/cluster/addons/dns # busybox.yaml apiVersion: v1 kind: Pod metadata: name: busybox namespace: default spec: containers: - image: busybox command: - sleep - "3600" imagePullPolicy: IfNotPresent name: busybox restartPolicy: Always kubectl create -f busybox.yaml kubectl get pods busybox kubectl exec busybox -- nslookup kubernetes -
Verify NAT settings
sudo iptables -t nat -L -n -v
-
Check status of Kubernetes processes
sudo systemctl status kubelet kube-proxy flanneld docker -l -
View logs of Kubernetes processes
sudo journalctl -u kubelet sudo journalctl -u kube-proxy sudo journalctl -u flanneld sudo journalctl -u docker -
Verify NAT settings
sudo iptables -t nat -L -n -v