Outage about 10 hours ago

[AArnott]

All my builds failed about 10 hours ago, and failed on re-run. It appears to have aligned with the 1.9.0 release, which may have been in progress.

I'd like to ensure future builds don't fail on an unpredictable schedule like that. How can I change this script in my pipeline so that I'm not vulnerable to such timing breaks?

    if ($env:AGENT_OS -eq 'Windows_NT') {
      iex (iwr "https://raw.githubusercontent.com/cargo-bins/cargo-binstall/main/install-from-binstall-release.ps1").Content
    } else {
      curl -L --proto '=https' --tlsv1.2 -sSf https://raw.githubusercontent.com/cargo-bins/cargo-binstall/main/install-from-binstall-release.sh | bash
    }

Following are each of the errors with which my builds failed last night:

Windows agents

"C:\Program Files\PowerShell\7\pwsh.exe" -NoLogo -NoProfile -NonInteractive -ExecutionPolicy Unrestricted -Command ". 'c:\vsts-agent\_work\_temp\04cf9bd5-f808-4b3d-b6e9-f6da84965252.ps1'"
DEBUG:    3+  >>>> $tmpdir = $Env:TEMP

DEBUG:    4+  >>>> $base_url = "[https://github.com/cargo-bins/cargo-binstall/releases/latest/download/cargo-binstall-"](https://github.com/cargo-bins/cargo-binstall/releases/latest/download/cargo-binstall-%22)

DEBUG:    5+  >>>> $proc_arch = [Environment]::GetEnvironmentVariable("PROCESSOR_ARCHITECTURE", [EnvironmentVariableTarget]::Machine)

DEBUG:    6+ if ( >>>> $proc_arch -eq "AMD64") {

DEBUG:    7+ 	 >>>> $arch = "x86_64"

DEBUG:   14+  >>>> $url = "$base_url$arch-pc-windows-msvc.zip"

DEBUG:   15+  >>>> Invoke-WebRequest $url -OutFile $tmpdir\cargo-binstall.zip

Invoke-WebRequest : Not Found
At line:15 char:1
+ Invoke-WebRequest $url -OutFile $tmpdir\cargo-binstall.zip
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo          : InvalidOperation: (Method: GET, Reques…ept-Encoding: br
}:HttpRequestMessage) [Invoke-WebRequest], HttpResponseException
+ FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeWebRequestCommand

Linux agents

++ mktemp -d

• cd /tmp/tmp.3mtb9EGEng
• base_url=https://github.com/cargo-bins/cargo-binstall/releases/latest/download/cargo-binstall-
  ++ uname -s
• os=Linux
• '[' Linux == Darwin ']'
• '[' Linux == Linux ']'
  ++ uname -m
• machine=x86_64
• '[' x86_64 == armv7l ']'
• target=x86_64-unknown-linux-musl
• '[' x86_64 == armv7 ']'
• url=https://github.com/cargo-bins/cargo-binstall/releases/latest/download/cargo-binstall-x86_64-unknown-linux-musl.tgz
• curl -L --proto =https --tlsv1.2 -sSf https://github.com/cargo-bins/cargo-binstall/releases/latest/download/cargo-binstall-x86_64-unknown-linux-musl.tgz
• tar -xvzf -
  curl: (22) The requested URL returned error: 404

gzip: stdin: unexpected end of file
tar: Child returned status 1
tar: Error is not recoverable: exiting now

MacOS agents

++ mktemp -d

• cd /var/folders/x_/qywhbr356z3dr69_l_fbmfx40000gp/T/tmp.ucjFsNuiQd
• base_url=https://github.com/cargo-bins/cargo-binstall/releases/latest/download/cargo-binstall-
  ++ uname -s
• os=Darwin
• '[' Darwin == Darwin ']'
• url=https://github.com/cargo-bins/cargo-binstall/releases/latest/download/cargo-binstall-universal-apple-darwin.zip
• curl -LO --proto =https --tlsv1.2 -sSf https://github.com/cargo-bins/cargo-binstall/releases/latest/download/cargo-binstall-universal-apple-darwin.zip
  curl: (22) The requested URL returned error: 404

[NobodyXu]

I think this is a duplicate of #1864 , it should already be fixed.

[AArnott]

The underlying issue may be a dupe (and thanks for the link, as I didn't know about it). But I already knew the outage was fixed.

This issue focuses on what I can change in my pipeline to avoid such outages in the future. It seems to be due to using releases/latest in the URI, when new releases are created but not yet fully stocked with files to be downloaded. I'd rather pin all the tools I build with to specific versions and control when they are upgraded so that I reduce risk of failures due to new releases.

[passcod]

We could probably change the scripts to look for a CARGO_BINSTALL_VERSION (default to latest if not present or empty) and use that in the download URLs.

[NobodyXu]

seems to be due to using releases/latest in the URI, when new releases are created but not yet fully stocked with files to be downloaded.

The new release for cargo-binstall is marked as draft by default and requires me to set it to latest manually, I admit previously I didn't realize a race condition could happen and just do it whenever I saw it.)l

However, the outrage you've seen now is probably not related to this.

Just before the 1.9.0 release, I updated CI to use release-plz for library crates, and it creates a github release for each library crate and sets them to latest, hence why it's broken.

I proposed a new feature in release-plz that should fix this release-plz/release-plz#1583

I'd rather pin all the tools I build with to specific versions and control when they are upgraded so that I reduce risk of failures due to new releases.

Yes we should do that as well, starting by supporting installing specific version in installation script and action.

I will welcome contribution for this!