[ci.jenkins.io] Move controller (VM) to AWS

[dduportal]

This issue tracks the work to migrate ci.jenkins.io controller to the AWS Sponsored.

Current resources:

• User facing documentation: https://github.com/jenkins-infra/documentation/blob/main/ci.adoc
• Terraform definition of the current VM: https://github.com/jenkins-infra/azure/blob/main/ci.jenkins.io.tf
  • Utilizes a Terraform module: https://github.com/jenkins-infra/shared-tools/tree/main/terraform/modules/azure-jenkinsinfra-controller
  • With a dedicated subnet: https://github.com/jenkins-infra/azure-net/blob/5e16478469344f8eced5224e553a5f660794b9dd/vnets.tf#L204-L209 (there is a former subnet for the controller in the public_vnet for it but it is not used anymore)
  • 2 IPs (public for HTTP/HTTPS/JNLP and private for SSH) - https://github.com/jenkins-infra/azure/blob/e81b9697e4a9cedebf10e119b6a5a112e09b651f/ci.jenkins.io.tf#L57-L60
• VM is provisioned with puppet: https://github.com/jenkins-infra/jenkins-infra/blob/bb9e757b20dcec1da678948ef8b453d6d2cac928/manifests/site.pp#L77-L85

Target for the new VM:

• Same size (vCPUs and memory)
• Same disk topology (rootfs and data with jenkins home in it)
• For network: see [ci.jenkins.io] Define virtual networking for AWS #4320
  • We'll need a aws.ci.jenkins.io DNS A record (pointed by the CNAME ci.jenkins.io once migrated) to the public IPv4 of the controller, and a AAAA record with the public IPv6.
  • If need be, we'll define a private DNS A record aws.ci.jenkins.io so agents can reach the controller through private subnets
  • Inbound:
    • Anywhere HTTP + HTTPS on both IPv4 and IPv6
    • SSH from the (Azure private VPN) only. We'll set up the VPN routing of users to override routing like we did for the pkg.origin.jenkins.io, usage.jio and census.jio VMs.
    • Private subnets HTTP, HTTPS, JNLP
  • Outbound (IPv4 only):
    • HTTP+ HTTPS + HKP to everywhere
    • SSH to GitHub public IPs and to private subnets

[dduportal]

Discussed with @smerle33:

• [ci.jenkins.io] Define virtual networking for AWS #4320 reached a first milestone allowing starting work on the issue here
• Stephane prefers more the EKS topic than the VM topic: assigning VM to @dduportal
• Scheduling for current milestone

[dduportal]

Update:

• We need a DNS zone in AWS, delegated from our Azure DNS, to ensure we keep the DNS records up to date when changing IPs

• We want full in and out IPv6 support

• Controller must be in a public subnet, with both Network ACL and security groups restrictions.

• Big PR network + DNS + reports in feat: set up foundations for the VPC network terraform-aws-sponsorship#36

• Initial DNS delegation to validate it works in feat(DNS) delegate 'aws.ci.jenkins.io' subdomain to AWS route53 azure-net#317 (need an update with the new DNS zone name servers => automation incoming)

• Initial LDAP allow list update (need an update with the new DNS zone name servers => automation incoming) in feat(publick8s) add the new aws.ci.jio controller outbound IP to allowed list for LDAP kubernetes-management#5891

[dduportal]

Update:

• VM is now live (ref. feat(aws.ci.jenkins.io) create the first VM for controller terraform-aws-sponsorship#37)
• SSH access is good (required a few VPN routing changes, see attached issues above)
• VM packages upgraded, we can proceed to the new Puppet config (initial deployment) - feat(ci.jenkins.io) set up initial aws.ci.jenkins.io VM client data jenkins-infra#3730

[dduportal]

Update:

• Initial Puppet provisioning was successful. VM rebooted and access with nominative account is ok
  • Container jenkins is up and running
• Next steps:
  • Update LDAP allowed IPs to add the new CI IP
  • Update Network ACLs to allow inbound HTTP/HTTPS from internet
  • Update Puppet setup to support vhosts for both the internal hostname (aws.ci.jenkins.io) and the top-level CNAME (ci.jenkins.io)

[dduportal]

Update: the new controller runs but need its domain name fixed to ensure we can access Jenkins in HTTPS without certificate error. The Jenkins UI message about "broken reverse proxy" is expected (as we don't use "ci.jenkins.io" in the browser for access until we've finished the migration).

It requires:

• Adding support for multiple hostnames for TLS in Puppet
• Fixing the puppet execution error when trying to install certbot:

Jan 20 16:22:30 ip-10-0-1-240 puppet-agent[431383]: Starting Puppet client version 6.28.0
Jan 20 16:22:40 ip-10-0-1-240 puppet-agent[431395]: (/Stage[main]/Profile::Letsencrypt/Exec[Install certbot-apache plugin]/returns) Collecting certbot-apache==1.32.0
Jan 20 16:22:40 ip-10-0-1-240 puppet-agent[431395]: (/Stage[main]/Profile::Letsencrypt/Exec[Install certbot-apache plugin]/returns)   Using cached certbot_apache-1.32.0-py3-none-any.whl.metadata (1.3 kB)
Jan 20 16:22:40 ip-10-0-1-240 puppet-agent[431395]: (/Stage[main]/Profile::Letsencrypt/Exec[Install certbot-apache plugin]/returns) Requirement already satisfied: acme>=1.32.0 in /usr/local/lib/python3.10/dist-packages (from certbot-apache==1.32.0) (1.32.0)
Jan 20 16:22:40 ip-10-0-1-240 puppet-agent[431395]: (/Stage[main]/Profile::Letsencrypt/Exec[Install certbot-apache plugin]/returns) Requirement already satisfied: certbot>=1.32.0 in /usr/local/lib/python3.10/dist-packages (from certbot-apache==1.32.0) (1.32.0)
Jan 20 16:22:40 ip-10-0-1-240 puppet-agent[431395]: (/Stage[main]/Profile::Letsencrypt/Exec[Install certbot-apache plugin]/returns) Collecting python-augeas (from certbot-apache==1.32.0)
Jan 20 16:22:40 ip-10-0-1-240 puppet-agent[431395]: (/Stage[main]/Profile::Letsencrypt/Exec[Install certbot-apache plugin]/returns)   Using cached python-augeas-1.1.0.tar.gz (93 kB)
Jan 20 16:22:40 ip-10-0-1-240 puppet-agent[431395]: (/Stage[main]/Profile::Letsencrypt/Exec[Install certbot-apache plugin]/returns)   Preparing metadata (setup.py): started
Jan 20 16:22:40 ip-10-0-1-240 puppet-agent[431395]: (/Stage[main]/Profile::Letsencrypt/Exec[Install certbot-apache plugin]/returns)   Preparing metadata (setup.py): finished with status 'error'
Jan 20 16:22:40 ip-10-0-1-240 puppet-agent[431395]: (/Stage[main]/Profile::Letsencrypt/Exec[Install certbot-apache plugin]/returns)   error: subprocess-exited-with-error
Jan 20 16:22:40 ip-10-0-1-240 puppet-agent[431395]: (/Stage[main]/Profile::Letsencrypt/Exec[Install certbot-apache plugin]/returns)
Jan 20 16:22:40 ip-10-0-1-240 puppet-agent[431395]: (/Stage[main]/Profile::Letsencrypt/Exec[Install certbot-apache plugin]/returns)   × python setup.py egg_info did not run successfully.
Jan 20 16:22:40 ip-10-0-1-240 puppet-agent[431395]: (/Stage[main]/Profile::Letsencrypt/Exec[Install certbot-apache plugin]/returns)   │ exit code: 1
Jan 20 16:22:40 ip-10-0-1-240 puppet-agent[431395]: (/Stage[main]/Profile::Letsencrypt/Exec[Install certbot-apache plugin]/returns)   ╰─> [56 lines of output]
Jan 20 16:22:40 ip-10-0-1-240 puppet-agent[431395]: (/Stage[main]/Profile::Letsencrypt/Exec[Install certbot-apache plugin]/returns)       /usr/local/lib/python3.10/dist-packages/setuptools/__init__.py:94: _DeprecatedInstaller: setuptools.installer and fetch_build_eggs are deprecated.
Jan 20 16:22:40 ip-10-0-1-240 puppet-agent[431395]: (/Stage[main]/Profile::Letsencrypt/Exec[Install certbot-apache plugin]/returns)       !!
Jan 20 16:22:40 ip-10-0-1-240 puppet-agent[431395]: (/Stage[main]/Profile::Letsencrypt/Exec[Install certbot-apache plugin]/returns)
Jan 20 16:22:40 ip-10-0-1-240 puppet-agent[431395]: (/Stage[main]/Profile::Letsencrypt/Exec[Install certbot-apache plugin]/returns)               ********************************************************************************
Jan 20 16:22:40 ip-10-0-1-240 puppet-agent[431395]: (/Stage[main]/Profile::Letsencrypt/Exec[Install certbot-apache plugin]/returns)               Requirements should be satisfied by a PEP 517 installer.
Jan 20 16:22:40 ip-10-0-1-240 puppet-agent[431395]: (/Stage[main]/Profile::Letsencrypt/Exec[Install certbot-apache plugin]/returns)               If you are using pip, you can try `pip install --use-pep517`.
Jan 20 16:22:40 ip-10-0-1-240 puppet-agent[431395]: (/Stage[main]/Profile::Letsencrypt/Exec[Install certbot-apache plugin]/returns)               ********************************************************************************
Jan 20 16:22:40 ip-10-0-1-240 puppet-agent[431395]: (/Stage[main]/Profile::Letsencrypt/Exec[Install certbot-apache plugin]/returns)
Jan 20 16:22:40 ip-10-0-1-240 puppet-agent[431395]: (/Stage[main]/Profile::Letsencrypt/Exec[Install certbot-apache plugin]/returns)       !!
Jan 20 16:22:40 ip-10-0-1-240 puppet-agent[431395]: (/Stage[main]/Profile::Letsencrypt/Exec[Install certbot-apache plugin]/returns)         dist.fetch_build_eggs(dist.setup_requires)
Jan 20 16:22:40 ip-10-0-1-240 puppet-agent[431395]: (/Stage[main]/Profile::Letsencrypt/Exec[Install certbot-apache plugin]/returns)       /usr/local/lib/python3.10/dist-packages/setuptools/_distutils/dist.py:261: UserWarning: Unknown distribution option: 'test_suite'
Jan 20 16:22:40 ip-10-0-1-240 puppet-agent[431395]: (/Stage[main]/Profile::Letsencrypt/Exec[Install certbot-apache plugin]/returns)         warnings.warn(msg)
Jan 20 16:22:40 ip-10-0-1-240 puppet-agent[431395]: (/Stage[main]/Profile::Letsencrypt/Exec[Install certbot-apache plugin]/returns)       running egg_info
Jan 20 16:22:40 ip-10-0-1-240 puppet-agent[431395]: (/Stage[main]/Profile::Letsencrypt/Exec[Install certbot-apache plugin]/returns)       creating /tmp/pip-pip-egg-info-othcbt7_/python_augeas.egg-info
Jan 20 16:22:40 ip-10-0-1-240 puppet-agent[431395]: (/Stage[main]/Profile::Letsencrypt/Exec[Install certbot-apache plugin]/returns)       writing /tmp/pip-pip-egg-info-othcbt7_/python_augeas.egg-info/PKG-INFO
Jan 20 16:22:40 ip-10-0-1-240 puppet-agent[431395]: (/Stage[main]/Profile::Letsencrypt/Exec[Install certbot-apache plugin]/returns)       writing dependency_links to /tmp/pip-pip-egg-info-othcbt7_/python_augeas.egg-info/dependency_links.txt
Jan 20 16:22:40 ip-10-0-1-240 puppet-agent[431395]: (/Stage[main]/Profile::Letsencrypt/Exec[Install certbot-apache plugin]/returns)       writing requirements to /tmp/pip-pip-egg-info-othcbt7_/python_augeas.egg-info/requires.txt
Jan 20 16:22:40 ip-10-0-1-240 puppet-agent[431395]: (/Stage[main]/Profile::Letsencrypt/Exec[Install certbot-apache plugin]/returns)       writing top-level names to /tmp/pip-pip-egg-info-othcbt7_/python_augeas.egg-info/top_level.txt
Jan 20 16:22:40 ip-10-0-1-240 puppet-agent[431395]: (/Stage[main]/Profile::Letsencrypt/Exec[Install certbot-apache plugin]/returns)       writing manifest file '/tmp/pip-pip-egg-info-othcbt7_/python_augeas.egg-info/SOURCES.txt'
Jan 20 16:22:40 ip-10-0-1-240 puppet-agent[431395]: (/Stage[main]/Profile::Letsencrypt/Exec[Install certbot-apache plugin]/returns)       reading manifest file '/tmp/pip-pip-egg-info-othcbt7_/python_augeas.egg-info/SOURCES.txt'
Jan 20 16:22:40 ip-10-0-1-240 puppet-agent[431395]: (/Stage[main]/Profile::Letsencrypt/Exec[Install certbot-apache plugin]/returns)       reading manifest template 'MANIFEST.in'
Jan 20 16:22:40 ip-10-0-1-240 puppet-agent[431395]: (/Stage[main]/Profile::Letsencrypt/Exec[Install certbot-apache plugin]/returns)       warning: no files found matching 'test/Makefile'
Jan 20 16:22:40 ip-10-0-1-240 puppet-agent[431395]: (/Stage[main]/Profile::Letsencrypt/Exec[Install certbot-apache plugin]/returns)       adding license file 'COPYING'
Jan 20 16:22:40 ip-10-0-1-240 puppet-agent[431395]: (/Stage[main]/Profile::Letsencrypt/Exec[Install certbot-apache plugin]/returns)       adding license file 'AUTHORS'
Jan 20 16:22:40 ip-10-0-1-240 puppet-agent[431395]: (/Stage[main]/Profile::Letsencrypt/Exec[Install certbot-apache plugin]/returns)       Traceback (most recent call last):
Jan 20 16:22:40 ip-10-0-1-240 puppet-agent[431395]: (/Stage[main]/Profile::Letsencrypt/Exec[Install certbot-apache plugin]/returns)         File "<string>", line 2, in <module>
Jan 20 16:22:40 ip-10-0-1-240 puppet-agent[431395]: (/Stage[main]/Profile::Letsencrypt/Exec[Install certbot-apache plugin]/returns)         File "<pip-setuptools-caller>", line 34, in <module>
Jan 20 16:22:40 ip-10-0-1-240 puppet-agent[431395]: (/Stage[main]/Profile::Letsencrypt/Exec[Install certbot-apache plugin]/returns)         File "/tmp/pip-install-gg7cjsmj/python-augeas_9f27429373514ce2adb086818733c3bf/setup.py", line 16, in <module>
Jan 20 16:22:40 ip-10-0-1-240 puppet-agent[431395]: (/Stage[main]/Profile::Letsencrypt/Exec[Install certbot-apache plugin]/returns)           setup(name=name,
Jan 20 16:22:40 ip-10-0-1-240 puppet-agent[431395]: (/Stage[main]/Profile::Letsencrypt/Exec[Install certbot-apache plugin]/returns)         File "/usr/local/lib/python3.10/dist-packages/setuptools/__init__.py", line 117, in setup
Jan 20 16:22:40 ip-10-0-1-240 puppet-agent[431395]: (/Stage[main]/Profile::Letsencrypt/Exec[Install certbot-apache plugin]/returns)           return distutils.core.setup(**attrs)
Jan 20 16:22:40 ip-10-0-1-240 puppet-agent[431395]: (/Stage[main]/Profile::Letsencrypt/Exec[Install certbot-apache plugin]/returns)         File "/usr/local/lib/python3.10/dist-packages/setuptools/_distutils/core.py", line 183, in setup
Jan 20 16:22:40 ip-10-0-1-240 puppet-agent[431395]: (/Stage[main]/Profile::Letsencrypt/Exec[Install certbot-apache plugin]/returns)           return run_commands(dist)
Jan 20 16:22:40 ip-10-0-1-240 puppet-agent[431395]: (/Stage[main]/Profile::Letsencrypt/Exec[Install certbot-apache plugin]/returns)         File "/usr/local/lib/python3.10/dist-packages/setuptools/_distutils/core.py", line 199, in run_commands
Jan 20 16:22:40 ip-10-0-1-240 puppet-agent[431395]: (/Stage[main]/Profile::Letsencrypt/Exec[Install certbot-apache plugin]/returns)           dist.run_commands()
Jan 20 16:22:40 ip-10-0-1-240 puppet-agent[431395]: (/Stage[main]/Profile::Letsencrypt/Exec[Install certbot-apache plugin]/returns)         File "/usr/local/lib/python3.10/dist-packages/setuptools/_distutils/dist.py", line 954, in run_commands
Jan 20 16:22:40 ip-10-0-1-240 puppet-agent[431395]: (/Stage[main]/Profile::Letsencrypt/Exec[Install certbot-apache plugin]/returns)           self.run_command(cmd)
Jan 20 16:22:40 ip-10-0-1-240 puppet-agent[431395]: (/Stage[main]/Profile::Letsencrypt/Exec[Install certbot-apache plugin]/returns)         File "/usr/local/lib/python3.10/dist-packages/setuptools/dist.py", line 994, in run_command
Jan 20 16:22:40 ip-10-0-1-240 puppet-agent[431395]: (/Stage[main]/Profile::Letsencrypt/Exec[Install certbot-apache plugin]/returns)           super().run_command(command)
Jan 20 16:22:40 ip-10-0-1-240 puppet-agent[431395]: (/Stage[main]/Profile::Letsencrypt/Exec[Install certbot-apache plugin]/returns)         File "/usr/local/lib/python3.10/dist-packages/setuptools/_distutils/dist.py", line 973, in run_command
Jan 20 16:22:40 ip-10-0-1-240 puppet-agent[431395]: (/Stage[main]/Profile::Letsencrypt/Exec[Install certbot-apache plugin]/returns)           cmd_obj.run()
Jan 20 16:22:40 ip-10-0-1-240 puppet-agent[431395]: (/Stage[main]/Profile::Letsencrypt/Exec[Install certbot-apache plugin]/returns)         File "/usr/local/lib/python3.10/dist-packages/setuptools/command/egg_info.py", line 312, in run
Jan 20 16:22:40 ip-10-0-1-240 puppet-agent[431395]: (/Stage[main]/Profile::Letsencrypt/Exec[Install certbot-apache plugin]/returns)           self.find_sources()
Jan 20 16:22:40 ip-10-0-1-240 puppet-agent[431395]: (/Stage[main]/Profile::Letsencrypt/Exec[Install certbot-apache plugin]/returns)         File "/usr/local/lib/python3.10/dist-packages/setuptools/command/egg_info.py", line 320, in find_sources
Jan 20 16:22:40 ip-10-0-1-240 puppet-agent[431395]: (/Stage[main]/Profile::Letsencrypt/Exec[Install certbot-apache plugin]/returns)           mm.run()
Jan 20 16:22:40 ip-10-0-1-240 puppet-agent[431395]: (/Stage[main]/Profile::Letsencrypt/Exec[Install certbot-apache plugin]/returns)         File "/usr/local/lib/python3.10/dist-packages/setuptools/command/egg_info.py", line 548, in run
Jan 20 16:22:40 ip-10-0-1-240 puppet-agent[431395]: (/Stage[main]/Profile::Letsencrypt/Exec[Install certbot-apache plugin]/returns)           self.prune_file_list()
Jan 20 16:22:40 ip-10-0-1-240 puppet-agent[431395]: (/Stage[main]/Profile::Letsencrypt/Exec[Install certbot-apache plugin]/returns)         File "/usr/local/lib/python3.10/dist-packages/setuptools/command/sdist.py", line 162, in prune_file_list
Jan 20 16:22:40 ip-10-0-1-240 puppet-agent[431395]: (/Stage[main]/Profile::Letsencrypt/Exec[Install certbot-apache plugin]/returns)           super().prune_file_list()
Jan 20 16:22:40 ip-10-0-1-240 puppet-agent[431395]: (/Stage[main]/Profile::Letsencrypt/Exec[Install certbot-apache plugin]/returns)         File "/usr/local/lib/python3.10/dist-packages/setuptools/_distutils/command/sdist.py", line 380, in prune_file_list
Jan 20 16:22:40 ip-10-0-1-240 puppet-agent[431395]: (/Stage[main]/Profile::Letsencrypt/Exec[Install certbot-apache plugin]/returns)           base_dir = self.distribution.get_fullname()
Jan 20 16:22:40 ip-10-0-1-240 puppet-agent[431395]: (/Stage[main]/Profile::Letsencrypt/Exec[Install certbot-apache plugin]/returns)         File "/usr/local/lib/python3.10/dist-packages/setuptools/_core_metadata.py", line 267, in get_fullname
Jan 20 16:22:40 ip-10-0-1-240 puppet-agent[431395]: (/Stage[main]/Profile::Letsencrypt/Exec[Install certbot-apache plugin]/returns)           return _distribution_fullname(self.get_name(), self.get_version())
Jan 20 16:22:40 ip-10-0-1-240 puppet-agent[431395]: (/Stage[main]/Profile::Letsencrypt/Exec[Install certbot-apache plugin]/returns)         File "/usr/local/lib/python3.10/dist-packages/setuptools/_core_metadata.py", line 285, in _distribution_fullname
Jan 20 16:22:40 ip-10-0-1-240 puppet-agent[431395]: (/Stage[main]/Profile::Letsencrypt/Exec[Install certbot-apache plugin]/returns)           canonicalize_version(version, strip_trailing_zero=False),
Jan 20 16:22:40 ip-10-0-1-240 puppet-agent[431395]: (/Stage[main]/Profile::Letsencrypt/Exec[Install certbot-apache plugin]/returns)       TypeError: canonicalize_version() got an unexpected keyword argument 'strip_trailing_zero'
Jan 20 16:22:40 ip-10-0-1-240 puppet-agent[431395]: (/Stage[main]/Profile::Letsencrypt/Exec[Install certbot-apache plugin]/returns)       [end of output]
Jan 20 16:22:40 ip-10-0-1-240 puppet-agent[431395]: (/Stage[main]/Profile::Letsencrypt/Exec[Install certbot-apache plugin]/returns)
Jan 20 16:22:40 ip-10-0-1-240 puppet-agent[431395]: (/Stage[main]/Profile::Letsencrypt/Exec[Install certbot-apache plugin]/returns)   note: This error originates from a subprocess, and is likely not a problem with pip.
Jan 20 16:22:40 ip-10-0-1-240 puppet-agent[431395]: (/Stage[main]/Profile::Letsencrypt/Exec[Install certbot-apache plugin]/returns) error: metadata-generation-failed
Jan 20 16:22:40 ip-10-0-1-240 puppet-agent[431395]: (/Stage[main]/Profile::Letsencrypt/Exec[Install certbot-apache plugin]/returns) × Encountered error while generating package metadata.
Jan 20 16:22:40 ip-10-0-1-240 puppet-agent[431395]: (/Stage[main]/Profile::Letsencrypt/Exec[Install certbot-apache plugin]/returns) ╰─> See above for output.
Jan 20 16:22:40 ip-10-0-1-240 puppet-agent[431395]: (/Stage[main]/Profile::Letsencrypt/Exec[Install certbot-apache plugin]/returns) note: This is an issue with the package mentioned above, not pip.
Jan 20 16:22:40 ip-10-0-1-240 puppet-agent[431395]: (/Stage[main]/Profile::Letsencrypt/Exec[Install certbot-apache plugin]/returns) hint: See above for details.
Jan 20 16:22:40 ip-10-0-1-240 puppet-agent[431395]: '/usr/bin/python3.10 -m pip install --upgrade certbot-apache==1.32.0' returned 1 instead of one of [0]
Jan 20 16:22:40 ip-10-0-1-240 puppet-agent[431395]: (/Stage[main]/Profile::Letsencrypt/Exec[Install certbot-apache plugin]/returns) change from 'notrun' to ['0'] failed: '/usr/bin/python3.10 -m pip install --upgrade certbot-apache==1.32.0' returned 1 instead of one of [0] (corrective)