[INFRA-3100] Migrate updates.jenkins.io to another Cloud

[jenkins-infra-bot]

Why

Read the EPIC (aws cost decrease: #2646)

What

• updates.jenkins.io serves the JSON of the update center, but it is not behind a CDN (need to be updated regularly) causing a lot of outbound data transfer (2.4 Mb for a JSON download, esitmation around 300 Gb of data sent daily, estimation around 10 Tb per month which costs some $$$.

• The VM pkg.origin.jenkins.io is managing multiple services and separting them could be a great help.

How

Different paths can be taken here: to be discussed in infra meeting + validated by the board as it's an important service.

• Migrating the service to Oracle Cloud:
  • the first 10 Tb of outbound data is free, and if the worst case, 50 Tb of outbound would cost less than 3k $ per month - https://www.oracle.com/be/cloud/networking/networking-pricing.html
  • ARM machines: cheaper and better performance for a simple webserver + SSH + data
  • Block storage: close to archives.jenkins.io
  • Risk: Oracle sponsoring program is only 1 year, and is not easily secured for paiment as other account

• Migrating to Azure
  • Easily in AKS: easier management
  • Safer paiment process
  • Risk: cost of outbound to be evaluated

---

Originally reported by dduportal, imported from: Migrate updates.ci.jenkins.io to another Cloud

• status: Open
• priority: Major
• resolution: Unresolved
• imported: 2022/01/10

[note]

• Check that Claim that updates.jenkins.io certificate is sometimes expiring #3017 is not happening once migration is finished.
• The error curl: (16) Error in the HTTP2 framing layer should not happen anymore with the latest Apache versions

[dduportal]

Requires setting up the Oracle Terraform project, like #2682

Todo list:

• Specify a new infra (VM + data storage of the same size as the actual pkg.origin.jenkins.io machine + any other Oracle infra requirements). Implies checking the pricing to get an overall idea
• Once VM is created, add it to puppet management: new node, with the role / profiles associated to "update.jenkins.io"
• Rsync the data one time
• Update the jenkins-infra/update-center2 's associated job in trusted.ci to update both "pkg.origin.jenkins.io" VM and the new one (⚠️ inline pipeline, don't search for a Jenkinsfile as code)
• Validate the new instance (including checking with Daniel, Tim and other contributors)
• Communicate about the upcoming change
• As proposed by Stephane, use a round-robin DNS record value to split traffic between old/new during some time and check for error

[dduportal]

Blocked by #2973

[smerle33]

actual machine size is :
32Gb RAM
8cpu
1,2Tb data disk (372Gb free)

about half of the power is used currently (checked with the local SAR probe)

[smerle33]

Infra to specify :

• 1 VM (similar to archive.jenkins.io)
• 1 network ("mirrors") + 1 subnet ("20210630-1531")
• 1 volume 1,2Tb
• 1 set of security groups to restrict network access
• 1 ssh key pair

[smerle33]

VM specifications :

• 4vCPU/16Gb RAM : half of actual machine
• proposal to use ARM like archive.jenkins.io --> 4ocpu (VM Type Standard A1 Flex)
• Image Ubuntu 20.04 (FULL [non minimal]) --> upgrade from ubuntu 18.04 for actual machine
  • option for 22.04 but need testing for puppet agent (+ openssl v3)

[dduportal]

Update:

• We are ready for production. proposed date: Monday 18 November 2024 at 09:00 UTC
  • feat(blog) add a post for the new UC in production jenkins.io#7688 => https://www.jenkins.io/blog/2024/11/16/new-update-center/
  • feat(dns-records) switch Update Center to the new Azure implementation azure-net#325 => DNS change.
    • ⚠️ If the new UC breaks, we have to revert this change to go back to the old machine (still being kept up to date)
• In ~1 month (ideally 4 weeks in production with no rollbacks or major breakages) we'll decommision the service updates.jenkins.io on the old VM pkg. Distinct issue to create
  • Remove DNS pointing to the machine related to UC (aws.updates.jenkins.io)
  • Remove Puppet code setting up the UC site
  • Remove updates from update_center2 and crawler in trusted.ci (keep credential as also used by pkg updates in UC2)
  • Remove (manually as Puppet doesn't unless configured with "absent" stuff) Apache vhost
• Optimizations along the way:
  • Add archives.jenkins.io as a mirror fallback (see distinct issue)
  • Once old service is decommisioned: optimize UC2 calls

[dduportal]

Update after 30 hours of production:

• We saw hiccups on the Nginx reverse proxy after ~ 26 hours. The PR fix(publick8s/updates.jio) disable ingrerss rewrite logs as it is resources hungry kubernetes-management#5903 seems to have fixed the aforementioned hiccups (verbose rewrite were still enabled causing unexpected overload on the reverse proxy).
  • Symptoms on the datadog monitors was a time to first byte and/or TLS handshakes really slow, which denotes of a slow reverse-proxy
  • We are now watching if these symptoms come back or not
• Opened [Update Center] Add archives.jenkins.io as a mirror fallback for JSON metadatas #4401 to add archives.jio as fallback
• Opened [Update Center] "eventually consistent" Azure Shared File storage for HTTPD htdocs (aka. corrupted files some time to time) #4402 to track the work related to inconsistent behavior of the htdocs shared directory due to Apache + Azure File Share + azcopy

[dduportal]

Update:

• HTTP/404 errors on tool installer metadata were reported in 404 downloading some tool installer JSONs (geo dependant) #4403. Problem has been fixed

[dduportal]

Update:

• [Update Center] "eventually consistent" Azure Shared File storage for HTTPD htdocs (aka. corrupted files some time to time) #4402 (SMB + azcopy unreliable data storage has been done (switch to NFS + rsync) and looks like a good long term solution.
• There has been a major outage before the switch to NFS with the SMB CSI mount stealing all the VM CPUs leading to major issues. It looks like it was also caused by an Azure Storage account latency increase (1s to 2s per req !!). Incident closed and should not happen (in the way how NFS CSI works).
  • Wrong update-center returned for 2.452.4 #4427
  • HTTP errors on multiple sites #4428
• Opened [Update Center] HTTP/404 on /current/updates/*.json* links #4432 (minor)
• We still need to work on [Update Center] Add archives.jenkins.io as a mirror fallback for JSON metadatas #4401 has we had Cloudflare outages today