[Bug]: 索引数据导出报 copyObject error

[18965050]

Current Behavior

执行 milvus-backup create 命令一段时间后(大概十几分钟后), 报如下错误:

[2024/10/10 09:38:14.554 +08:00] [ERROR] [storage/minio_chunk_manager.go:447] ["copyObject error"] [srcObjectKey=file/insert_log/452466374303636288/452466374303636289/452466374301282722/106/452466374301282742] [
dstObjectKey=file/my_backup/binlogs/insert_log/452466374303636288/452466374303636289/452466374301282722/452466374301282722/106/452466374301282742] [error="Put "http://10.110.7.33:32000/backup-bucket/file/my_bac
kup/binlogs/insert_log/452466374303636288/452466374303636289/452466374301282722/452466374301282722/106/452466374301282742": net/http: timeout awaiting response headers"] [stack="github.com/zilliztech/milvus-bac
kup/core/storage.(*MinioChunkManager).Copy\n\t/home/runner/work/milvus-backup/milvus-backup/core/storage/minio_chunk_manager.go:447\ngithub.com/zilliztech/milvus-backup/core/storage.(*Copier).copyRemote\n\t/home
/runner/work/milvus-backup/milvus-backup/core/storage/copier.go:261\ngithub.com/zilliztech/milvus-backup/core/storage.(*Copier).Copy\n\t/home/runner/work/milvus-backup/milvus-backup/core/storage/copier.go:242\ng
ithub.com/zilliztech/milvus-backup/core.(*BackupContext).copySegment.func2\n\t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_create_backup.go:859\ngithub.com/zilliztech/milvus-backup/internal/uti
l/retry.Do\n\t/home/runner/work/milvus-backup/milvus-backup/internal/util/retry/retry.go:37\ngithub.com/zilliztech/milvus-backup/core.(*BackupContext).copySegment\n\t/home/runner/work/milvus-backup/milvus-backup
/core/backup_impl_create_backup.go:857\ngithub.com/zilliztech/milvus-backup/core.(*BackupContext).copySegments.func1\n\t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_create_backup.go:800\ngithub
.com/zilliztech/milvus-backup/internal/common.(*WorkerPool).work.func1\n\t/home/runner/work/milvus-backup/milvus-backup/internal/common/workerpool.go:70\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/home/run
ner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75"]

导致备份索引的meta数据未导出来, 从而不能进行索引restore操作

我看了下srcObjectKey, 确实不存在这个文件, 不能拷贝到dstObjectKey中

Expected Behavior

期望此命令执行成功

Steps To Reproduce

1. minio中建立 backup-bucket

2. mivlus-backup configs配置如下:
   # milvus proxy address, compatible to milvus.yaml
milvus:
  address: 10.110.7.33
  port: 30530
  authorizationEnabled: true
  # tls mode values [0, 1, 2]
  # 0 is close, 1 is one-way authentication, 2 is two-way authentication.
  tlsMode: 0
  user: "root"
  password: "Milvus"

# Related configuration of minio, which is responsible for data persistence for Milvus.
minio:
  # cloudProvider: "minio" # deprecated use storageType instead
  storageType: "minio" # support storage type: local, minio, s3, aws, gcp, ali(aliyun), azure

  address: 10.110.7.33
  port: 32000   # Port of MinIO/S3
  accessKeyID: minioadmin  # accessKeyID of MinIO/S3
  secretAccessKey: minioadmin # MinIO/S3 encryption string
  useSSL: false # Access to MinIO/S3 with SSL
  useIAM: false
  iamEndpoint: ""

  bucketName: "milvus-bucket" # Milvus Bucket name in MinIO/S3, make it the same as your milvus instance
  rootPath: "file" # Milvus storage root path in MinIO/S3, make it the same as your milvus instance

  # only for azure
  backupAccessKeyID: minioadmin  # accessKeyID of MinIO/S3
  backupSecretAccessKey: minioadmin # MinIO/S3 encryption string

  backupBucketName: "backup-bucket" # Bucket name to store backup data. Backup data will store to backupBucketName/backupRootPath
  backupRootPath: "file" # Rootpath to store backup data. Backup data will store to backupBucketName/backupRootPath

backup:
  maxSegmentGroupSize: 2G

  parallelism:
    # collection level parallelism to backup
    backupCollection: 4
    # thread pool to copy data. reduce it if blocks your storage's network bandwidth
    copydata: 128
    # Collection level parallelism to restore
    restoreCollection: 2

  # keep temporary files during restore, only use to debug 
  keepTempFiles: false

3. 执行命令 ./milvus-backup create -c xxx -n my_backup

Environment

milvus版本: v2.3.5
milvus-backup版本: 0.4.5 (使用版本 0.4.24此问题依然存在)

Anything else?

No response

[wayblink]

@18965050 可能是配置的不对 运行一下 ./milvus-backup check

[18965050]

./milvus-backup check 是检测连通情况, 运行是正常的.

后来我写代码对此业务进行了压缩处理, 并检查压缩是否完成. 压缩完成后再执行此命令依然报相同的错误.

今天我再试了下, 居然成功了

我怀疑是etcd中有些segment信息未在minio存在导致, 不知是否如此