+
+
+
+ 
+ 
+
+
+
+
+
+
+
+
+
+
+
+
+
+ +`/etc/resolver/consul` +``` +nameserver 10.9.99.10 +port 8600 +``` +Now names such as `nomad.service.consul` and `fabio.service.consul` will work + +`vagrant up --provision-with basetools,docker,docsify,consul` + +```log +Bringing machine 'user.local.dev' up with 'virtualbox' provider... +==> user.local.dev: Checking if box 'ubuntu/xenial64' version '20190918.0.0' is up to date... +==> user.local.dev: [vagrant-hostsupdater] Checking for host entries +==> user.local.dev: [vagrant-hostsupdater] found entry for: 10.9.99.10 user.local.dev +==> user.local.dev: [vagrant-hostsupdater] found entry for: 10.9.99.10 user.local.dev +==> user.local.dev: Running provisioner: consul (shell)... + user.local.dev: Running: /var/folders/7j/gsrjvmds05n53ddg28krf4_80001p9/T/vagrant-shell20200110-35654-11zwf6z.sh + user.local.dev: Reading package lists... + user.local.dev: Building dependency tree... + user.local.dev: Reading state information... + user.local.dev: unzip is already the newest version (6.0-20ubuntu1). + user.local.dev: curl is already the newest version (7.47.0-1ubuntu2.14). + user.local.dev: jq is already the newest version (1.5+dfsg-1ubuntu0.1). + user.local.dev: 0 upgraded, 0 newly installed, 0 to remove and 4 not upgraded. + user.local.dev: primary_datacenter = "dc1" + user.local.dev: client_addr = "10.9.99.10 127.0.0.1 ::1" + user.local.dev: bind_addr = "0.0.0.0" + user.local.dev: data_dir = "/var/lib/consul" + user.local.dev: datacenter = "dc1" + user.local.dev: disable_host_node_id = true + user.local.dev: disable_update_check = true + user.local.dev: leave_on_terminate = true + user.local.dev: log_level = "INFO" + user.local.dev: ports = { + user.local.dev: grpc = 8502 + user.local.dev: dns = 8600 + user.local.dev: https = -1 + user.local.dev: } + user.local.dev: protocol = 3 + user.local.dev: raft_protocol = 3 + user.local.dev: recursors = [ + user.local.dev: "8.8.8.8", + user.local.dev: "8.8.4.4", + user.local.dev: ] + user.local.dev: server_name = "consul.service.consul" + user.local.dev: ui = true + user.local.dev: ++++ Consul already installed at /usr/local/bin/consul + user.local.dev: ++++ Consul v1.6.2 + user.local.dev: Protocol 2 spoken by default, understands 2 to 3 (agent will automatically use protocol >2 when speaking to compatible agents) + user.local.dev: ==> Starting Consul agent... + user.local.dev: Version: 'v1.6.2' + user.local.dev: Node ID: '3e943a0a-d73e-5797-cb3e-f3dc2e6df832' + user.local.dev: Node name: 'user' + user.local.dev: Datacenter: 'dc1' (Segment: '
+https://doc.traefik.io/traefik/v2.8/providers/nomad/ + + + +We are thrilled to announce the full integration of the new Nomad built-in Service Discovery with Traefik Proxy. This is a first-of-its-kind ingress integration that simplifies ingress in HashiCorp Nomad. Utilizing Nomad directly with Traefik Proxy has never been so easy! + +In early May, Hashicorp announced Nomad Version 1.3. Among other updates, it also includes a nice list of improvements on usability and developer experience. Before this release, when using service discovery with Nomad, Traefik Proxy users had to use Hashicorp Consul and Nomad side-by-side in order to benefit from Traefik Proxy’s famous automatic configuration. Now, Nomad has a simple and straightforward way to use service discovery built-in. This improves direct usability a lot! Not only in simple test environments but also on the edge. + +`http://localhost:8080/` and `http://localhost:8181` + + + + + +## Traefik Nomad Job template +[filename](nomad/jobs/traefik.nomad ':include :type=code hcl') + +`vagrant up --provision-with basetools,docker,docsify,consul,nomad` + +The new native Service Discovery in Nomad really does work seamlessly. With this integration, delivering load balancing, dynamic routing configuration, and ingress traffic routing become easier than ever. Check out the Traefik Proxy 2.8 Release Candidate and the Nomad 1.3 release notes. + +`curl -H "Host: whoami.nomad.localhost" http://localhost:8080 -v` + +```log +* Trying 127.0.0.1:8080... +* Connected to localhost (127.0.0.1) port 8080 (#0) +> GET / HTTP/1.1 +> Host: whoami.nomad.localhost +> User-Agent: curl/7.79.1 +> Accept: */* +> +* Mark bundle as not supporting multiuse +< HTTP/1.1 200 OK +< Content-Length: 365 +< Content-Type: text/plain; charset=utf-8 +< Date: Thu, 16 Jun 2022 02:08:56 GMT +< +Hostname: 86bb7e3d366a +IP: 127.0.0.1 +IP: 172.18.0.5 +RemoteAddr: 172.18.0.1:51192 +GET / HTTP/1.1 +Host: whoami.nomad.localhost +User-Agent: curl/7.79.1 +Accept: */* +Accept-Encoding: gzip +X-Forwarded-For: 172.17.0.1 +X-Forwarded-Host: whoami.nomad.localhost +X-Forwarded-Port: 80 +X-Forwarded-Proto: http +X-Forwarded-Server: 5d7dc64220c8 +X-Real-Ip: 172.17.0.1 + +* Connection #0 to host localhost left intact +``` + +## Traefik Whoami Nomad Job template +[filename](nomad/jobs/traefik-whoami.nomad ':include :type=code hcl') + +## Fabio Load Balancer for Nomad +https://github.com/fabiolb/fabio
+https://fabiolb.net + +Fabio is an HTTP and TCP reverse proxy that configures itself with data from Consul. + +Traditional load balancers and reverse proxies need to be configured with a config file. The configuration contains the hostnames and paths the proxy is forwarding to upstream services. This process can be automated with tools like consul-template that generate config files and trigger a reload. + +Fabio works differently since it updates its routing table directly from the data stored in Consul as soon as there is a change and without restart or reloading. + +When you register a service in Consul all you need to add is a tag that announces the paths the upstream service accepts, e.g. urlprefix-/user or urlprefix-/order and fabio will do the rest. + +`http://localhost:9999/` and `http://localhost:9998` + + + +`vagrant up --provision-with basetools,docker,docsify,consul,nomad` + +Fabio runs as a Nomad job, see `nomad/nomad/jobs/fabio.nomad` + +Some routes are added via Consul, see `consul/consul.sh` + +## Fabio Nomad Job template +[filename](nomad/jobs/fabio.nomad ':include :type=code hcl') + +## Fabio Properties file +[filename](nomad/jobs/fabio.properties ':include :type=code config') \ No newline at end of file diff --git a/nomad/images/fabio.png b/nomad/images/fabio.png new file mode 100644 index 0000000..78e726c Binary files /dev/null and b/nomad/images/fabio.png differ diff --git a/nomad/images/nomad-logo.png b/nomad/images/nomad-logo.png new file mode 100644 index 0000000..3f0bd7e Binary files /dev/null and b/nomad/images/nomad-logo.png differ diff --git a/nomad/images/nomad.png b/nomad/images/nomad.png new file mode 100644 index 0000000..1b27e01 Binary files /dev/null and b/nomad/images/nomad.png differ diff --git a/nomad/images/traefik-dashboard.png b/nomad/images/traefik-dashboard.png new file mode 100644 index 0000000..1a6f40e Binary files /dev/null and b/nomad/images/traefik-dashboard.png differ diff --git a/nomad/images/traefik-logo.png b/nomad/images/traefik-logo.png new file mode 100644 index 0000000..0b8b8ce Binary files /dev/null and b/nomad/images/traefik-logo.png differ diff --git a/nomad/images/traefik-proxy.png b/nomad/images/traefik-proxy.png new file mode 100644 index 0000000..6bda3eb Binary files /dev/null and b/nomad/images/traefik-proxy.png differ diff --git a/nomad/nomad.sh b/nomad/nomad.sh new file mode 100644 index 0000000..c78dde9 --- /dev/null +++ b/nomad/nomad.sh @@ -0,0 +1,196 @@ +#!/bin/bash + +function nomad-install() { + + if pgrep -x "consul" >/dev/null + then + echo "Consul is running" + else + echo -e '\e[38;5;198m'"++++ Ensure Consul is running.." + sudo bash /vagrant/consul/consul.sh + fi + + arch=$(lscpu | grep "Architecture" | awk '{print $NF}') + if [[ $arch == x86_64* ]]; then + ARCH="amd64" + elif [[ $arch == aarch64 ]]; then + ARCH="arm64" + fi + echo -e '\e[38;5;198m'"CPU is $ARCH" + + sudo DEBIAN_FRONTEND=noninteractive apt-get --assume-yes install -qq curl unzip jq < /dev/null > /dev/null + yes | sudo docker system prune -a + yes | sudo docker system prune --volumes + mkdir -p /etc/nomad + cat <
https://developer.hashicorp.com/vault/docs/configuration/telemetry -In hashicorp/vault.sh we enabled Telemetry in the Vault config file see `hashicorp/vault.sh` +In vault/vault.sh we enabled Telemetry in the Vault config file see `vault/vault.sh` ```hcl # https://developer.hashicorp.com/vault/docs/configuration/telemetry @@ -136,7 +136,7 @@ https://developer.hashicorp.com/nomad/docs/operations/monitoring-nomad
https://developer.hashicorp.com/nomad/tutorials/manage-clusters/prometheus-metrics -In hashicorp/nomad.sh we enabled Telemetry in the Nomad config file see `hashicorp/nomad.sh` +In nomad/nomad.sh we enabled Telemetry in the Nomad config file see `nomad/nomad.sh` ```hcl # https://developer.hashicorp.com/nomad/docs/configuration/telemetry @@ -185,7 +185,7 @@ https://lvinsf.medium.com/monitor-consul-using-prometheus-and-grafana-1f2354cc00 https://grafana.com/grafana/dashboards/13396-consul-server-monitoring/
https://developer.hashicorp.com/consul/docs/agent/telemetry -In hashicorp/consul.sh we enabled Telemetry in the Consul config file see `hashicorp/consul.sh` +In consul/consul.sh we enabled Telemetry in the Consul config file see `consul/consul.sh` ```hcl # https://lvinsf.medium.com/monitor-consul-using-prometheus-and-grafana-1f2354cc002f diff --git a/sentinel/README.md b/sentinel/README.md new file mode 100644 index 0000000..c9991d3 --- /dev/null +++ b/sentinel/README.md @@ -0,0 +1,103 @@ +# Sentinel + +https://docs.hashicorp.com/sentinel/ +https://github.com/hashicorp/tfe-policies-example +https://docs.hashicorp.com/sentinel/language/ + +Sentinel is a language and framework for policy built to be embedded in existing software to enable fine-grained, logic-based policy decisions. A policy describes under what circumstances certain behaviors are allowed. Sentinel is an enterprise-only feature of HashiCorp Consul, Nomad, Terraform, and Vault. + +`vagrant up --provision-with basetools,docsify,sentinel` + +```log +Bringing machine 'user.local.dev' up with 'virtualbox' provider... +==> user.local.dev: Checking if box 'ubuntu/bionic64' version '20191218.0.0' is up to date... +==> user.local.dev: [vagrant-hostsupdater] Checking for host entries +==> user.local.dev: [vagrant-hostsupdater] found entry for: 10.9.99.10 user.local.dev +==> user.local.dev: Running provisioner: sentinel (shell)... + user.local.dev: Running: /var/folders/7j/gsrjvmds05n53ddg28krf4_80001p9/T/vagrant-shell20200310-40084-1bbypjm.sh + user.local.dev: Reading package lists... + user.local.dev: Building dependency tree... + user.local.dev: + user.local.dev: Reading state information... + user.local.dev: unzip is already the newest version (6.0-21ubuntu1). + user.local.dev: jq is already the newest version (1.5+dfsg-2). + user.local.dev: curl is already the newest version (7.58.0-2ubuntu3.8). + user.local.dev: 0 upgraded, 0 newly installed, 0 to remove and 6 not upgraded. + user.local.dev: ++++ Sentinel Simulator v0.9.2 already installed at /usr/local/bin/sentinel + user.local.dev: hour = 4 + user.local.dev: main = rule { hour >= 0 and hour < 12 } + user.local.dev: ++++ cat /tmp/policy.sentinel + user.local.dev: hour = 4 + user.local.dev: main = rule { hour >= 0 and hour < 12 } + user.local.dev: ++++ sentinel apply /tmp/policy.sentinel + user.local.dev: Pass + user.local.dev: ++++ Let's test some more advanced Sentinel Policies + user.local.dev: ++++ https://github.com/hashicorp/tfe-policies-example + user.local.dev: ++++ https://docs.hashicorp.com/sentinel/language/ + user.local.dev: ++++ sentinel test aws-block-allow-all-cidr.sentinel + user.local.dev: PASS - aws-block-allow-all-cidr.sentinel + user.local.dev: PASS - test/aws-block-allow-all-cidr/empty.json + user.local.dev: PASS - test/aws-block-allow-all-cidr/fail.json + user.local.dev: PASS - test/aws-block-allow-all-cidr/pass.json + user.local.dev: ERROR - test/aws-block-allow-all-cidr/plan.json + user.local.dev: + user.local.dev: ++++ sentinel apply -config ./test/aws-block-allow-all-cidr/pass.json aws-block-allow-all-cidr.sentinel + user.local.dev: Pass + user.local.dev: ++++ sentinel apply -config ./test/aws-block-allow-all-cidr/fail.json aws-block-allow-all-cidr.sentinel + user.local.dev: Fail + user.local.dev: + user.local.dev: Execution trace. The information below will show the values of all + user.local.dev: the rules evaluated and their intermediate boolean expressions. Note that + user.local.dev: some boolean expressions may be missing if short-circuit logic was taken. + user.local.dev: FALSE - aws-block-allow-all-cidr.sentinel:69:1 - Rule "main" + user.local.dev: TRUE - aws-block-allow-all-cidr.sentinel:70:2 - ingress_cidr_blocks + user.local.dev: TRUE - aws-block-allow-all-cidr.sentinel:50:2 - all get_resources("aws_security_group") as sg { + user.local.dev: all sg.applied.ingress as ingress { + user.local.dev: all disallowed_cidr_blocks as block { + user.local.dev: ingress.cidr_blocks not contains block + user.local.dev: } + user.local.dev: } + user.local.dev: } + user.local.dev: FALSE - aws-block-allow-all-cidr.sentinel:71:2 - egress_cidr_blocks + user.local.dev: FALSE - aws-block-allow-all-cidr.sentinel:60:2 - all get_resources("aws_security_group") as sg { + user.local.dev: all sg.applied.egress as egress { + user.local.dev: all disallowed_cidr_blocks as block { + user.local.dev: egress.cidr_blocks not contains block + user.local.dev: } + user.local.dev: } + user.local.dev: } + user.local.dev: + user.local.dev: FALSE - aws-block-allow-all-cidr.sentinel:59:1 - Rule "egress_cidr_blocks" + user.local.dev: + user.local.dev: TRUE - aws-block-allow-all-cidr.sentinel:49:1 - Rule "ingress_cidr_blocks" + user.local.dev: + user.local.dev: ++++ sentinel test aws-alb-redirect.sentinel + user.local.dev: PASS - aws-alb-redirect.sentinel + user.local.dev: PASS - test/aws-alb-redirect/empty.json + user.local.dev: PASS - test/aws-alb-redirect/fail.json + user.local.dev: PASS - test/aws-alb-redirect/pass.json + user.local.dev: ERROR - test/aws-alb-redirect/plan.json + user.local.dev: + user.local.dev: ++++ sentinel apply -config ./test/aws-alb-redirect/fail.json aws-alb-redirect.sentinel + user.local.dev: Fail + user.local.dev: + user.local.dev: Execution trace. The information below will show the values of all + user.local.dev: the rules evaluated and their intermediate boolean expressions. Note that + user.local.dev: some boolean expressions may be missing if short-circuit logic was taken. + user.local.dev: FALSE - aws-alb-redirect.sentinel:69:1 - Rule "main" + user.local.dev: FALSE - aws-alb-redirect.sentinel:70:2 - default_action + user.local.dev: FALSE - aws-alb-redirect.sentinel:49:2 - all get_resources("aws_lb_listener") as ln { + user.local.dev: all ln.applied.default_action as action { + user.local.dev: + user.local.dev: all action.redirect as rdir { + user.local.dev: + user.local.dev: rdir.status_code == redirect_status_code + user.local.dev: } + user.local.dev: } + user.local.dev: } + user.local.dev: + user.local.dev: FALSE - aws-alb-redirect.sentinel:48:1 - Rule "default_action" + user.local.dev: + user.local.dev: ++++ sentinel apply -config ./test/aws-alb-redirect/pass.json aws-alb-redirect.sentinel + user.local.dev: Pass +``` diff --git a/sentinel/sentinel.sh b/sentinel/sentinel.sh new file mode 100644 index 0000000..740da33 --- /dev/null +++ b/sentinel/sentinel.sh @@ -0,0 +1,52 @@ +#!/bin/bash + +function sentinel-install() { + arch=$(lscpu | grep "Architecture" | awk '{print $NF}') + if [[ $arch == x86_64* ]]; then + ARCH="amd64" + elif [[ $arch == aarch64 ]]; then + ARCH="arm64" + fi + echo -e '\e[38;5;198m'"CPU is $ARCH" + + sudo DEBIAN_FRONTEND=noninteractive apt-get --assume-yes install -qq curl unzip jq < /dev/null > /dev/null + if [ -f /usr/local/bin/sentinel ]; then + echo -e '\e[38;5;198m'"++++ `/usr/local/bin/sentinel version` already installed at /usr/local/bin/sentinel" + else + LATEST_URL=$(curl --silent https://releases.hashicorp.com/index.json | jq '{sentinel}' | egrep "linux.*$ARCH" | sort -rh | head -1 | awk -F[\"] '{print $4}') + wget -q $LATEST_URL -O /tmp/sentinel.zip + mkdir -p /usr/local/bin + (cd /usr/local/bin && unzip /tmp/sentinel.zip) + + echo -e '\e[38;5;198m'"++++ Installed: `/usr/local/bin/sentinel version`" + fi + # add basic configuration settings for Vault to /etc/vault/config.hcl file + cat <
+https://www.hashicorp.com/resources/what-is-terraform-cloud
+__Authentication__
+https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/guides/service_principal_client_secret#creating-a-service-principal
+https://registry.terraform.io/providers/hashicorp/aws/latest/docs#environment-variables
+https://registry.terraform.io/providers/hashicorp/google/latest/docs/guides/provider_reference.html#authentication-configuration + +Terraform Cloud is a SaaS that we support—that instead, when you run Terraform you still could run it on your local machine, but now it saves and retrieves the state file from Terraform Cloud—which is running over here. + +This simplifies a lot of things. First of all, this is pretty much invisible. It still exists, but we manage it for you. Second of all, we could perform a lot more security on this access. You can see who is accessing your state file, control who accesses the state file, and more. + +In addition to that, Terraform Cloud will also version and back up your state file so that you could go back in time and see what your infrastructure looked like in the past—or if something went wrong, you could restore a past version. This is something that's really tricky with a local file because this is a normal file on your computer—you would have to be responsible for this yourself. In Terraform Cloud's case, you could still talk directly to the various cloud providers. That's how Terraform Cloud works today. That's the major benefit that remote state brings for you. + +[](https://www.youtube.com/watch?v=ihAKcn9SE_M) + +## Terraform Cloud collaboration/governance + +On top of remote state, there are a number of other features in Terraform Cloud in other tiers that enable things like centralized runs, plan approvals, and more. This changes this behavior, so that instead of talking directly to the cloud providers it talks instead to Terraform Cloud. + +Here, instead of talking directly to the cloud providers, what would happen is all your requests to plan and apply would go through Terraform Cloud. Then from here—would then go to the cloud providers. As I said, this is optional. You could use the state storage and talk directly to the cloud providers or you could add this on and use this in the middle. + +The benefit is you have a full history here of all the runs that have ever happened. Terraform ensures that only one run happens at a time—and you can get approvals. So, if Alice submits a plan to change infrastructure, Bob has to approve it before it goes through. + +You can see how having a SaaS around Terraform can simplify and hide a lot of internal details that are difficult to do with Terraform alone. Broadly, the theme around this is collaboration. + +Terraform on your computer—by itself—is a great, powerful tool. But it makes it really hard as soon as you're working with a team or with many people. You can do it. There are ways to coordinate this, but we're introducing Terraform Cloud to make this easy and automatic, and idiomatic in terms of how it should work across all Terraform users. + +This makes it clean to have access control here, access control here, history—and you still keep the same Terraform workflow. It's still terraform plan, terraform apply, just like you would here. It will automatically use Terraform Cloud in the backend. + +## Hashiqube Multi-Cloud + +I use Terraform Cloud to build and test the Terraform changes for Hashiqube's Multi-Cloud Terraform Registry Module + +You can read more about Hashiqube Multi-Cloud here: +[__Hashiqube Multi Cloud__](/multi-cloud/#terraform-hashicorp-hashiqube) + +https://registry.terraform.io/modules/star3am/hashiqube/hashicorp/latest + +## Hashiqube Variables + +| Key | Value | Category | +|-----|-------|----------| +| aws_region | ap-southeast-2 | terraform | +| deploy_to_aws | true | terraform | +| deploy_to_azure | true | terraform | +| deploy_to_gcp | true | terraform | +| gcp_project | SENSITIVE | terraform | +| gcp_region | australia-southeast1 | terraform | +| ssh_public_key | SENSITIVE | terraform | +| ARM_CLIENT_ID | SENSITIVE | env | +| ARM_CLIENT_SECRET | SENSITIVE | env | +| ARM_SUBSCRIPTION_ID | SENSITIVE | env | +| ARM_TENANT_ID | SENSITIVE | env | +| AWS_ACCESS_KEY_ID | SENSITIVE | env | +| AWS_SECRET_ACCESS_KEY | SENSITIVE | env | +| AWS_REGION | ap-southeast-2 | env | +| GOOGLE_CREDENTIALS | SENSITIVE | env | + +:bulb: Google credentials was generated by passing the Google Authentication JSON file to this command + +`cat ~/.gcp/credentials.json | jq -c` + +### Terraform Cloud run + + +## Terraform Enterprise +https://www.terraform.io/docs/enterprise/index.html + +Terraform Enterprise is our self-hosted distribution of Terraform Cloud. It offers enterprises a private instance of the Terraform Cloud application, with no resource limits and with additional enterprise-grade architectural features like audit logging and SAML single sign-on. + +Terraform Cloud is an application that helps teams use Terraform together. It manages Terraform runs in a consistent and reliable environment, and includes easy access to shared state and secret data, access controls for approving changes to infrastructure, a private registry for sharing Terraform modules, detailed policy controls for governing the contents of Terraform configurations, and more. + +For independent teams and small to medium-sized businesses, Terraform Cloud is also available as a hosted service at https://app.terraform.io. + +__Make sure you get a Terraform Licence file and place it in hashicorp directory e.g hashicorp/ptfe-license.rli__ + +When you run `vagrant up --provision-with terraform-enterprise` system logs and docker logs will be followed, the output will be in read, don't worry. This is for status output, __the installation takes a while__. The output will end when Terraform Enterprise is ready. + +Once done, you will see __++++ To finish the installation go to http://10.9.99.10:8800__ + + + + +`vagrant up --provision-with terraform-enterprise` + +```log +Bringing machine 'user.local.dev' up with 'virtualbox' provider... +==> user.local.dev: Checking if box 'ubuntu/xenial64' version '20190918.0.0' is up to date... +==> user.local.dev: [vagrant-hostsupdater] Checking for host entries +==> user.local.dev: [vagrant-hostsupdater] found entry for: 10.9.99.10 user.local.dev +==> user.local.dev: [vagrant-hostsupdater] found entry for: 10.9.99.10 user.local.dev +==> user.local.dev: [vagrant-hostsupdater] found entry for: 10.9.99.10 consul-user.local.dev +==> user.local.dev: [vagrant-hostsupdater] found entry for: 10.9.99.10 vault-user.local.dev +==> user.local.dev: [vagrant-hostsupdater] found entry for: 10.9.99.10 nomad-user.local.dev +==> user.local.dev: Running provisioner: terraform-enterprise (shell)... + user.local.dev: Running: /var/folders/7j/gsrjvmds05n53ddg28krf4_80001p9/T/vagrant-shell20191118-33309-16vz6hz.sh + ... + user.local.dev: Installing replicated-operator service + user.local.dev: Starting replicated-operator service + user.local.dev: + user.local.dev: Operator installation successful + user.local.dev: To continue the installation, visit the following URL in your browser: + user.local.dev: + user.local.dev: http://10.9.99.10:8800 +``` diff --git a/terraform/images/hashiqube-multi-cloud-terraform-cloud-plan.png b/terraform/images/hashiqube-multi-cloud-terraform-cloud-plan.png new file mode 100644 index 0000000..98f45b4 Binary files /dev/null and b/terraform/images/hashiqube-multi-cloud-terraform-cloud-plan.png differ diff --git a/terraform/images/terraform-enterprise.png b/terraform/images/terraform-enterprise.png new file mode 100644 index 0000000..dc535b4 Binary files /dev/null and b/terraform/images/terraform-enterprise.png differ diff --git a/terraform/images/terraform-enterprise_logged_in.png b/terraform/images/terraform-enterprise_logged_in.png new file mode 100644 index 0000000..a6966e7 Binary files /dev/null and b/terraform/images/terraform-enterprise_logged_in.png differ diff --git a/terraform/images/terraform-logo.png b/terraform/images/terraform-logo.png new file mode 100644 index 0000000..3e1def6 Binary files /dev/null and b/terraform/images/terraform-logo.png differ diff --git a/terraform/terraform.sh b/terraform/terraform.sh new file mode 100644 index 0000000..00aa850 --- /dev/null +++ b/terraform/terraform.sh @@ -0,0 +1,38 @@ +#!/bin/bash + +function terraform-install() { + + arch=$(lscpu | grep "Architecture" | awk '{print $NF}') + if [[ $arch == x86_64* ]]; then + ARCH="amd64" + elif [[ $arch == aarch64 ]]; then + ARCH="arm64" + fi + echo -e '\e[38;5;198m'"CPU is $ARCH" + + sudo DEBIAN_FRONTEND=noninteractive apt-get --assume-yes install -qq curl unzip jq < /dev/null > /dev/null + if [ -f /usr/local/bin/terraform ]; then + echo -e '\e[38;5;198m'"++++ `/usr/local/bin/terraform version` already installed at /usr/local/bin/terraform" + else + LATEST_URL=$(curl -sL https://releases.hashicorp.com/terraform/index.json | jq -r '.versions[].builds[].url' | sort -t. -k 1,1n -k 2,2n -k 3,3n -k 4,4n | egrep -v 'rc|beta' | egrep "linux.*$ARCH" | sort -V | tail -n1) + wget -q $LATEST_URL -O /tmp/terraform.zip + mkdir -p /usr/local/bin + (cd /usr/local/bin && unzip /tmp/terraform.zip) + echo -e '\e[38;5;198m'"++++ Installed: `/usr/local/bin/terraform version`" + fi + pip3 install --upgrade awscli-local + export PATH=$HOME/.local/bin:$PATH + sudo rm -rf awscliv2.zip + # https://aws.amazon.com/blogs/developer/aws-cli-v2-now-available-for-linux-arm/ aarch64 + curl -s "https://awscli.amazonaws.com/awscli-exe-linux-${arch}.zip" -o "awscliv2.zip" + sudo unzip -q awscliv2.zip + yes | sudo ./aws/install --update + echo -e '\e[38;5;198m'"++++ aws --version" + aws --version + + # ensure localstack is running + echo -e '\e[38;5;198m'"++++ To Terraform plan, and apply using Localstack, run the following command: vagrant up --provision-with localstack" + echo -e '\e[38;5;198m'"++++ See Localstack folder for Terraform files" +} + +terraform-install diff --git a/vagrant/README.md b/vagrant/README.md new file mode 100644 index 0000000..55a55b7 --- /dev/null +++ b/vagrant/README.md @@ -0,0 +1,83 @@ +# Vagrant + +https://www.vagrantup.com/ + + + +HashiCorp Vagrant provides the same, easy workflow regardless of your role as a developer, operator, or designer. It leverages a declarative configuration file which describes all your software requirements, packages, operating system configuration, users, and more. + +### The Vagrantfile + +`Vagrantfile` + +[filename](../Vagrantfile.txt ':include :type=code ruby') + +`vagrant up --provision` +```log +Bringing machine 'user.local.dev' up with 'virtualbox' provider... +==> user.local.dev: Checking if box 'ubuntu/xenial64' version '20190918.0.0' is up to date... +==> user.local.dev: [vagrant-hostsupdater] Checking for host entries +==> user.local.dev: [vagrant-hostsupdater] found entry for: 10.9.99.10 user.local.dev +==> user.local.dev: [vagrant-hostsupdater] found entry for: 10.9.99.10 user.local.dev +==> user.local.dev: Running provisioner: bootstrap (shell)... + user.local.dev: Running: inline script + user.local.dev: BEGIN BOOTSTRAP 2020-01-10 00:44:49 + user.local.dev: running vagrant as user + user.local.dev: Get:1 https://deb.nodesource.com/node_10.x xenial InRelease [4,584 B] + ... + user.local.dev: END BOOTSTRAP 2020-01-10 00:45:53 +==> user.local.dev: Running provisioner: docker (shell)... + user.local.dev: Running: /var/folders/7j/gsrjvmds05n53ddg28krf4_80001p9/T/vagrant-shell20200110-35289-lj8d6b.sh + ... + user.local.dev: ++++ open http://localhost:8889 in your browser + user.local.dev: ++++ you can also run below to get apache2 version from the docker container + user.local.dev: ++++ vagrant ssh -c "docker exec -it apache2 /bin/bash -c 'apache2 -t -v'" +==> user.local.dev: Running provisioner: terraform (shell)... + user.local.dev: Running: /var/folders/7j/gsrjvmds05n53ddg28krf4_80001p9/T/vagrant-shell20200110-35289-gf77w9.sh + ... + user.local.dev: ++++ Terraform v0.12.18 already installed at /usr/local/bin/terraform +==> user.local.dev: Running provisioner: vault (shell)... + user.local.dev: Running: /var/folders/7j/gsrjvmds05n53ddg28krf4_80001p9/T/vagrant-shell20200110-35289-igtj7e.sh + ... + user.local.dev: ++++ Vault already installed and running + user.local.dev: ++++ Vault http://localhost:8200/ui and enter the following codes displayed below + ... +==> user.local.dev: Running provisioner: consul (shell)... + user.local.dev: Running: /var/folders/7j/gsrjvmds05n53ddg28krf4_80001p9/T/vagrant-shell20200110-35289-u3hjac.sh + user.local.dev: Reading package lists... + ... + user.local.dev: ++++ Adding Consul KV data for Fabio Load Balancer Routes + user.local.dev: Success! Data written to: fabio/config/vault + user.local.dev: Success! Data written to: fabio/config/nomad + user.local.dev: Success! Data written to: fabio/config/consul + user.local.dev: ++++ Consul http://localhost:8500 +==> user.local.dev: Running provisioner: nomad (shell)... + user.local.dev: Running: /var/folders/7j/gsrjvmds05n53ddg28krf4_80001p9/T/vagrant-shell20200110-35289-1s3k8i2.sh + ... + user.local.dev: ++++ Nomad already installed at /usr/local/bin/nomad + user.local.dev: ++++ Nomad v0.10.2 (0d2d6e3dc5a171c21f8f31fa117c8a765eb4fc02) + user.local.dev: ++++ cni-plugins already installed + user.local.dev: ==> Loaded configuration from /etc/nomad/server.conf + user.local.dev: ==> Starting Nomad agent... + ... +==> user.local.dev: Running provisioner: packer (shell)... + user.local.dev: Running: /var/folders/7j/gsrjvmds05n53ddg28krf4_80001p9/T/vagrant-shell20200110-35289-18twg6l.sh + ... +==> user.local.dev: Running provisioner: sentinel (shell)... + user.local.dev: Running: /var/folders/7j/gsrjvmds05n53ddg28krf4_80001p9/T/vagrant-shell20200110-35289-18qv6vf.sh + ... + user.local.dev: ++++ Sentinel Simulator v0.9.2 already installed at /usr/local/bin/sentinel + user.local.dev: hour = 4 + user.local.dev: main = rule { hour >= 0 and hour < 12 } + user.local.dev: ++++ cat /tmp/policy.sentinel + user.local.dev: hour = 4 + user.local.dev: main = rule { hour >= 0 and hour < 12 } + user.local.dev: ++++ sentinel apply /tmp/policy.sentinel + user.local.dev: Pass +==> user.local.dev: Running provisioner: localstack (shell)... + ... +==> user.local.dev: Running provisioner: docsify (shell)... + user.local.dev: Running: /var/folders/7j/gsrjvmds05n53ddg28krf4_80001p9/T/vagrant-shell20200110-35289-1du0q9e.sh + ... + user.local.dev: ++++ Docsify: http://localhost:3333/ +``` diff --git a/vagrant/images/vagrant-logo.png b/vagrant/images/vagrant-logo.png new file mode 100644 index 0000000..5d28aee Binary files /dev/null and b/vagrant/images/vagrant-logo.png differ diff --git a/vault/README.md b/vault/README.md new file mode 100644 index 0000000..47836fc --- /dev/null +++ b/vault/README.md @@ -0,0 +1,91 @@ +# Vault + +https://www.vaultproject.io/ + + + +Manage Secrets and Protect Sensitive Data. +Secure, store and tightly control access to tokens, passwords, certificates, encryption keys for protecting secrets and other sensitive data using a UI, CLI, or HTTP API. + +[](https://www.youtube.com/watch?v=VYfl-DpZ5wM) + +`vagrant up --provision-with basetools,docsify,vault` + +```log +Bringing machine 'user.local.dev' up with 'virtualbox' provider... +==> user.local.dev: Checking if box 'ubuntu/xenial64' version '20190918.0.0' is up to date... +==> user.local.dev: [vagrant-hostsupdater] Checking for host entries +==> user.local.dev: [vagrant-hostsupdater] found entry for: 10.9.99.10 user.local.dev +==> user.local.dev: [vagrant-hostsupdater] found entry for: 10.9.99.10 user.local.dev +==> user.local.dev: Running provisioner: vault (shell)... + user.local.dev: Running: /var/folders/7j/gsrjvmds05n53ddg28krf4_80001p9/T/vagrant-shell20200110-35357-1112dsr.sh + user.local.dev: Reading package lists... + user.local.dev: Building dependency tree... + user.local.dev: + user.local.dev: Reading state information... + user.local.dev: unzip is already the newest version (6.0-20ubuntu1). + user.local.dev: curl is already the newest version (7.47.0-1ubuntu2.14). + user.local.dev: jq is already the newest version (1.5+dfsg-1ubuntu0.1). + user.local.dev: 0 upgraded, 0 newly installed, 0 to remove and 4 not upgraded. + user.local.dev: sed: -e expression #1, char 34: unknown option to `s' + user.local.dev: ++++ Vault already installed and running + user.local.dev: ++++ Vault http://localhost:8200/ui and enter the following codes displayed below + user.local.dev: ++++ Auto unseal vault + user.local.dev: Key Value + user.local.dev: --- ---- + user.local.dev: - + user.local.dev: Seal Type shamir + user.local.dev: Initialize + user.local.dev: d true + user.local.dev: Sealed false + user.local.dev: Total Shares 5 + user.local.dev: Threshold 3 + user.local.dev: Version 1.3.1 + user.local.dev: Cluster Name vault + user.local.dev: Cluster ID 11fa4aed + user.local.dev: -dc06-2d64-5429-7fadc5d8473a + user.local.dev: HA Enabled false + user.local.dev: Key Value + user.local.dev: --- ----- + user.local.dev: Seal Type shamir + user.local.dev: Initialized + user.local.dev: true + user.local.dev: Sealed false + user.local.dev: Total + user.local.dev: Shares 5 + user.local.dev: Threshold 3 + user.local.dev: Version 1.3.1 + user.local.dev: Cluster Name vault + user.local.dev: Cluster ID 11fa4aed-dc06-2d6 + user.local.dev: Unseal Key 1: XsVFkqDcG7JCXaAYHEUcg1VrKE6uO7Zs90FV9XqL7S1X + user.local.dev: Unseal Key 2: eUNVAQbFxbGTkQ0rdT1RRp1E/hdgMVmOXCTyddsYOzOV + user.local.dev: Unseal Key 3: eaIbXrTA+VA/g7/Tm1iCdfzajjRSx6k1xfIUHvd/IiKp + user.local.dev: Unseal Key 4: 7lcRnPqLaQiopY3NFCcRAfUHc9shxHTqmUXjzsxAQdbr + user.local.dev: Unseal Key 5: l9GpctLEhzOS1O9K2qk09B3vFU85PUC1s8KWHKNYplj8 + user.local.dev: + user.local.dev: Initial Root Token: s.rrftkbzQ8XBKVTijFyxaRWkH + user.local.dev: + user.local.dev: Vault initialized with 5 key shares and a key threshold of 3. Please securely + user.local.dev: distribute the key shares printed above. When the Vault is re-sealed, + user.local.dev: restarted, or stopped, you must supply at least 3 of these keys to unseal it + user.local.dev: before it can start servicing requests. + user.local.dev: + user.local.dev: Vault does not store the generated master key. Without at least 3 key to + user.local.dev: reconstruct the master key, Vault will remain permanently sealed! + user.local.dev: + user.local.dev: It is possible to generate new unseal keys, provided you have a quorum of + user.local.dev: existing unseal keys shares. See "vault operator rekey" for more information. +``` + + +## Vault Vagrant Provisioner + +`vault.sh` + +[filename](vault.sh ':include :type=code') + +## Monitoring Hashicorp Vault + +We use Prometheus and Grafana to Monitor Vault + +See: [__Monitoring Hashicorp Vault__](prometheus-grafana/README?id=monitoring-hashicorp-vault) diff --git a/vault/images/vault-logo.png b/vault/images/vault-logo.png new file mode 100644 index 0000000..3bf422b Binary files /dev/null and b/vault/images/vault-logo.png differ diff --git a/vault/images/vault.png b/vault/images/vault.png new file mode 100644 index 0000000..538d8d2 Binary files /dev/null and b/vault/images/vault.png differ diff --git a/vault/images/waypoint-logo.png b/vault/images/waypoint-logo.png new file mode 100644 index 0000000..0d40a32 Binary files /dev/null and b/vault/images/waypoint-logo.png differ diff --git a/vault/vault.sh b/vault/vault.sh new file mode 100644 index 0000000..33f7740 --- /dev/null +++ b/vault/vault.sh @@ -0,0 +1,328 @@ +#!/bin/bash + +# https://computingforgeeks.com/install-and-configure-vault-server-linux/ +# https://www.vaultproject.io/ + +# Terraform Enterprise should not be running, creates conflict since it has it's own vault +ps aux | grep -q "replicated" | grep -v grep +if [ $? -eq 0 ]; then + service replicated stop + service replicated-ui stop + service replicated-operator stop + docker stop replicated-premkit + docker stop replicated-statsd + docker rm -f replicated replicated-ui replicated-operator replicated-premkit replicated-statsd retraced-api retraced-processor retraced-cron retraced-nsqd retraced-postgres + docker images | grep "quay\.io/replicated" | awk '{print $3}' | xargs sudo docker rmi -f + docker images | grep "registry\.replicated\.com/library/retraced" | awk '{print $3}' | xargs sudo docker rmi -f +fi + +arch=$(lscpu | grep "Architecture" | awk '{print $NF}') +if [[ $arch == x86_64* ]]; then + ARCH="amd64" +elif [[ $arch == aarch64 ]]; then + ARCH="arm64" +fi +echo -e '\e[38;5;198m'"CPU is $ARCH" + +# apt-get remove -y replicated replicated-ui replicated-operator +# apt-get purge -y replicated replicated-ui replicated-operator +# rm -rf /var/lib/replicated* /etc/replicated* /etc/init/replicated* /etc/init.d/replicated* /etc/default/replicated* /var/log/upstart/replicated* /etc/systemd/system/replicated* +sudo DEBIAN_FRONTEND=noninteractive apt-get --assume-yes install -qq curl unzip jq < /dev/null > /dev/null +# only do if vault is not found +if [ ! -f /usr/local/bin/vault ]; then + + echo -e '\e[38;5;198m'"++++ " + echo -e '\e[38;5;198m'"++++ Vault not installed, installing.." + echo -e '\e[38;5;198m'"++++ " + + LATEST_URL=$(curl -sL https://releases.hashicorp.com/vault/index.json | jq -r '.versions[].builds[].url' | sort -t. -k 1,1n -k 2,2n -k 3,3n -k 4,4n | egrep -v 'rc|ent|beta' | egrep "linux.*$ARCH" | sort -V | tail -n 1) + wget -q $LATEST_URL -O /tmp/vault.zip + + mkdir -p /usr/local/bin + (cd /usr/local/bin && unzip /tmp/vault.zip) + echo -e '\e[38;5;198m'"++++ " + echo -e '\e[38;5;198m'"++++ Installed `/usr/local/bin/vault --version`" + echo -e '\e[38;5;198m'"++++ " + + # create /var/log/nomad.log + sudo touch /var/log/nomad.log + + # enable command autocompletion + vault -autocomplete-install + complete -C /usr/local/bin/vault vault + + # create Vault data directories + sudo mkdir /etc/vault + sudo mkdir -p /var/lib/vault/data + + # create user named vault + sudo useradd --system --home /etc/vault --shell /bin/false vault + sudo chown -R vault:vault /etc/vault /var/lib/vault/ + + # create a Vault service file at /etc/systemd/system/vault.service + cat <
+
+
+ 
+
+
+
+
+
+
+
+
+
diff --git a/waypoint/waypoint/custom-examples/kubernetes-trex-nodejs/views/partials/header.ejs b/waypoint/waypoint/custom-examples/kubernetes-trex-nodejs/views/partials/header.ejs
new file mode 100755
index 0000000..a2f42d5
--- /dev/null
+++ b/waypoint/waypoint/custom-examples/kubernetes-trex-nodejs/views/partials/header.ejs
@@ -0,0 +1,2 @@
+
+
+
+
+
+
+ 
+
+
+ PRESS SPACEBAR TO START.
+The files are located in /vagrant/waypoint/waypoint/custom-examples/kubernetes-trex-nodejs, and this file is views/pages/index.ejs
+
+ Try making a change to this text locally and run waypoint up again to see it.
+
+ Read the documentation for more about Waypoint. +
+
+
+
+
+
+
+
+
+
+
+
+
+
diff --git a/waypoint/waypoint/custom-examples/nomad-trex-nodejs/views/partials/header.ejs b/waypoint/waypoint/custom-examples/nomad-trex-nodejs/views/partials/header.ejs
new file mode 100755
index 0000000..a2f42d5
--- /dev/null
+++ b/waypoint/waypoint/custom-examples/nomad-trex-nodejs/views/partials/header.ejs
@@ -0,0 +1,2 @@
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ PRESS SPACEBAR TO START.
+The files are located in /vagrant/waypoint/waypoint/custom-examples/nomad-trex-nodejs, and this file is views/pages/index.ejs
+
+ Try making a change to this text locally and run waypoint up again to see it.
+
+ Read the documentation for more about Waypoint. +
+
+
+
+
+
+
+
+
+
+
+
+
+
diff --git a/waypoint/waypoint/custom-examples/trex-nodejs/views/partials/header.ejs b/waypoint/waypoint/custom-examples/trex-nodejs/views/partials/header.ejs
new file mode 100755
index 0000000..a2f42d5
--- /dev/null
+++ b/waypoint/waypoint/custom-examples/trex-nodejs/views/partials/header.ejs
@@ -0,0 +1,2 @@
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ PRESS SPACEBAR TO START.
+The files are located in /vagrant/waypoint/waypoint/custom-examples/trex-nodejs, and this file is views/pages/index.ejs
+
+ Try making a change to this text locally and run waypoint up again to see it.
+
+ Read the documentation for more about Waypoint. +
+