Teleport upload to DigitalOcean spaces (S3)

[jamshidyerzakov]

Hi there!

Today I updated the Teleport server from 9.1.2 to 10.3.4. After the update there are errors related to session upload to s3 bucket:

Jan 01 19:20:34 teleport-app teleport[806512]: 2023-01-01T19:20:34Z WARN [UPLOAD]    Upload failed. error:[
Jan 01 19:20:34 teleport-app teleport[806512]: ERROR REPORT:
Jan 01 19:20:34 teleport-app teleport[806512]: Original Error: *trace.ConnectionProblemError timeout waiting for stream status update
Jan 01 19:20:34 teleport-app teleport[806512]: Stack Trace:
Jan 01 19:20:34 teleport-app teleport[806512]:         /go/src/github.com/gravitational/teleport/lib/events/filesessions/fileasync.go:471 github.com/gravitational/teleport/lib/events/filesessions.(*Uploader).upload
Jan 01 19:20:34 teleport-app teleport[806512]:         /go/src/github.com/gravitational/teleport/lib/events/filesessions/fileasync.go:402 github.com/gravitational/teleport/lib/events/filesessions.(*Uploader).startUpload.func1
Jan 01 19:20:34 teleport-app teleport[806512]:         /opt/go/src/runtime/asm_amd64.s:1571 runtime.goexit
Jan 01 19:20:34 teleport-app teleport[806512]: User Message: timeout waiting for stream status update] filesessions/fileasync.go:403

Could not understand the timeout problem.

Can anybody help with this? Thank you in advance!

[jamshidyerzakov]

my /etc/teleport.yaml:

     ...
     region: eu-central
     audit_sessions_uri: "s3://******/records?region=eu-central&endpoint=fra1.digitaloceanspaces.com&insecure=false&disablesse=true&acl=public-read-write&use_fips_endpoint=false"
     ...

[webvictim]

@zmb3 Did we alter the way S3 uploads work between 9.1.2 and 10.3.4? I'm wondering whether this worked by accident in the past.

[zmb3]

We sure did, but this timeout is in lib/events/filesessions not in the S3 uploader so I'm not sure I can explain that.

Most notable change is to complete empty updloads (33622b1#diff-f928b1b5054b11b47c6447e60675c9b8b06300c0b8f63752705378c3e1ad8596)

Is it possible that DO Spaces does not support the abort upload API?