[Bug]: When running ci cases, there will be restore timeout issues #473

zhuwenxing · 2024-12-10T07:10:16Z

Current Behavior

[2024/12/09 09:31:43.908 +00:00] [ERROR] [core/backup_impl_restore_backup.go:380] ["executeRestoreCollectionTask failed"] [TargetDBName=db_LLijFN3I] [TargetCollectionName=restore_backup_hJrFzhlC_bak] [error="import job does not exist, jobID=454487866216294669, dbID=1: importing data failed"] [errorVerbose="import job does not exist, jobID=454487866216294669, dbID=1: importing data failed\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus-sdk-go/v2/client.handleRespStatus\n  | \t/go/pkg/mod/github.com/wayblink/milvus-sdk-go/[email protected]/client/collection.go:39\n  | github.com/milvus-io/milvus-sdk-go/v2/client.(*GrpcClient).GetBulkInsertState\n  | \t/go/pkg/mod/github.com/wayblink/milvus-sdk-go/[email protected]/client/insert.go:434\n  | github.com/zilliztech/milvus-backup/core.(*MilvusClient).GetBulkInsertState\n  | \t/home/jenkins/agent/workspace/core/milvus_sdk_wrapper.go:128\n  | github.com/zilliztech/milvus-backup/core.(*BackupContext).watchBulkInsertState\n  | \t/home/jenkins/agent/workspace/core/backup_impl_restore_backup.go:843\n  | github.com/zilliztech/milvus-backup/core.(*BackupContext).executeBulkInsert\n  | \t/home/jenkins/agent/workspace/core/backup_impl_restore_backup.go:827\n  | github.com/zilliztech/milvus-backup/core.(*BackupContext).executeRestoreCollectionTask.func3\n  | \t/home/jenkins/agent/workspace/core/backup_impl_restore_backup.go:642\n  | github.com/zilliztech/milvus-backup/core.(*BackupContext).executeRestoreCollectionTask.func7\n  | \t/home/jenkins/agent/workspace/core/backup_impl_restore_backup.go:722\n  | github.com/zilliztech/milvus-backup/internal/common.(*WorkerPool).work.func1\n  | \t/home/jenkins/agent/workspace/internal/common/workerpool.go:70\n  | golang.org/x/sync/errgroup.(*Group).Go.func1\n  | \t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:78\n  | runtime.goexit\n  | \t/usr/local/go/src/runtime/asm_amd64.s:1695\nWraps: (2) import job does not exist, jobID=454487866216294669, dbID=1: importing data failed\nError types: (1) *withstack.withStack (2) *errutil.leafError"] [stack="github.com/zilliztech/milvus-backup/core.(*BackupContext).executeRestoreBackupTask.func1\n\t/home/jenkins/agent/workspace/core/backup_impl_restore_backup.go:380\ngithub.com/zilliztech/milvus-backup/internal/common.(*WorkerPool).work.func1\n\t/home/jenkins/agent/workspace/internal/common/workerpool.go:70\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:78"]
[2024/12/09 09:31:43.908 +00:00] [ERROR] [core/backup_impl_restore_backup.go:819] ["fail to bulk insert"] [db=db_LLijFN3I] [collectionName=restore_backup_hJrFzhlC_bak] [partitionName=_default] [files="[backup/backup_PRkhrubZ/binlogs/insert_log/454487866216261085/454487866216261086/454487866216261116/,]"] [error="rpc error: code = Canceled desc = context canceled"] [stack="github.com/zilliztech/milvus-backup/core.(*BackupContext).executeBulkInsert\n\t/home/jenkins/agent/workspace/core/backup_impl_restore_backup.go:819\ngithub.com/zilliztech/milvus-backup/core.(*BackupContext).executeRestoreCollectionTask.func3\n\t/home/jenkins/agent/workspace/core/backup_impl_restore_backup.go:642\ngithub.com/zilliztech/milvus-backup/core.(*BackupContext).executeRestoreCollectionTask.func7\n\t/home/jenkins/agent/workspace/core/backup_impl_restore_backup.go:722\ngithub.com/zilliztech/milvus-backup/internal/common.(*WorkerPool).work.func1\n\t/home/jenkins/agent/workspace/internal/common/workerpool.go:70\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:78"]
[2024/12/09 09:31:43.909 +00:00] [ERROR] [core/backup_impl_restore_backup.go:722] ["fail to bulk insert to partition"] [backup_db_name=db_LLijFN3I] [backup_collection_name=db_LLijFN3I] [target_db_name=db_LLijFN3I] [target_collection_name=restore_backup_hJrFzhlC_bak] [skipDiskQuotaCheck=false] [maxShardNum=0] [partition=_default] [error="rpc error: code = Canceled desc = context canceled"] [stack="github.com/zilliztech/milvus-backup/core.(*BackupContext).executeRestoreCollectionTask.func7\n\t/home/jenkins/agent/workspace/core/backup_impl_restore_backup.go:722\ngithub.com/zilliztech/milvus-backup/internal/common.(*WorkerPool).work.func1\n\t/home/jenkins/agent/workspace/internal/common/workerpool.go:70\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:78"]
[2024/12/09 09:31:43.969 +00


[2024-12-09T09:53:30.053Z] =========================== short test summary info ============================

[2024-12-09T09:53:30.053Z] FAILED testcases/test_restore_backup.py::TestRestoreBackup::test_milvus_restore_back_with_new_feature_support[True-True-False] - AssertionError

[2024-12-09T09:53:30.053Z] FAILED testcases/test_restore_backup.py::TestRestoreBackup::test_milvus_restore_back_with_new_feature_support[True-False-False] - AssertionError

[2024-12-09T09:53:30.053Z] FAILED testcases/test_restore_backup.py::TestRestoreBackup::test_milvus_restore_back_with_new_feature_support[False-True-False] - AssertionError

[2024-12-09T09:53:30.054Z] FAILED testcases/test_restore_backup.py::TestRestoreBackup::test_milvus_restore_back_with_multi_partition[all-3-True-True-True-3000] - Failed: Timeout >240.0s

[2024-12-09T09:53:30.054Z] FAILED testcases/test_restore_backup.py::TestRestoreBackup::test_milvus_restore_back_with_new_feature_support[False-False-False] - Failed: Timeout >240.0s

[2024-12-09T09:53:30.054Z] FAILED testcases/test_restore_backup.py::TestRestoreBackup::test_milvus_restore_back_with_db_support - Failed: Timeout >240.0s

[2024-12-09T09:53:30.054Z] FAILED testcases/test_restore_backup.py::TestRestoreBackup::test_milvus_restore_with_db_collections[True-True] - Failed: Timeout >240.0s

[2024-12-09T09:53:30.054Z] FAILED testcases/test_restore_backup.py::TestRestoreBackup::test_milvus_restore_with_db_collections[True-False] - Failed: Timeout >240.0s

[2024-12-09T09:53:30.054Z] FAILED testcases/test_restore_backup.py::TestRestoreBackup::test_milvus_restore_with_db_collections[False-True] - Failed: Timeout >240.0s

[2024-12-09T09:53:30.054Z] FAILED testcases/test_restore_backup.py::TestRestoreBackup::test_milvus_restore_with_db_collections[False-False] - Failed: Timeout >240.0s

[2024-12-09T09:53:30.054Z] FAILED testcases/test_restore_backup.py::TestRestoreBackup::test_milvus_restore_back_with_array_datatype[True-True] - Failed: Timeout >240.0s

[2024-12-09T09:53:30.054Z] FAILED testcases/test_restore_backup.py::TestRestoreBackup::test_milvus_restore_back_with_array_datatype[True-False] - Failed: Timeout >240.0s

[2024-12-09T09:53:30.054Z] FAILED testcases/test_restore_backup.py::TestRestoreBackup::test_milvus_restore_back_with_array_datatype[False-True] - Failed: Timeout >240.0s

[2024-12-09T09:53:30.054Z] FAILED testcases/test_restore_backup.py::TestRestoreBackup::test_milvus_restore_back_with_array_datatype[False-False] - Failed: Timeout >240.0s

[2024-12-09T09:53:30.054Z] FAILED testcases/test_restore_backup.py::TestRestoreBackup::test_milvus_restore_back_with_delete - Failed: Timeout >240.0s

[2024-12-09T09:53:30.054Z] FAILED testcases/test_restore_backup.py::TestRestoreBackup::test_milvus_restore_back_with_upsert - Failed: Timeout >240.0s

Expected Behavior

No response

Steps To Reproduce

No response

Environment

failed job: https://qa-jenkins.milvus.io/blue/organizations/jenkins/milvus_backup_api_test/detail/milvus_backup_api_test/82/pipeline


milvus info
4am
chaos-testing


[2024-12-09T09:53:38.140Z] + kubectl get pods -o wide

[2024-12-09T09:53:38.141Z] + grep backup-test-82

[2024-12-09T09:53:38.397Z] backup-test-82-etcd-0                                             1/1     Running            0                74m     10.104.33.198   4am-node36   <none>           <none>

[2024-12-09T09:53:38.397Z] backup-test-82-etcd-1                                             1/1     Running            0                74m     10.104.25.28    4am-node30   <none>           <none>

[2024-12-09T09:53:38.397Z] backup-test-82-etcd-2                                             1/1     Running            0                74m     10.104.23.178   4am-node27   <none>           <none>

[2024-12-09T09:53:38.397Z] backup-test-82-milvus-standalone-7948d8697-j76x2                  1/1     Running            2 (73m ago)      74m     10.104.21.120   4am-node24   <none>           <none>

[2024-12-09T09:53:38.397Z] backup-test-82-minio-86d47c8cbc-qfc94                             1/1     Running            0                74m     10.104.21.123   4am-node24   <none>           <none>

[2024-12-09T09:53:38.397Z] backup-test-82-pulsarv3-bookie-0                                  1/1     Running            0                74m     10.104.20.241   4am-node22   <none>           <none>

[2024-12-09T09:53:38.397Z] backup-test-82-pulsarv3-bookie-1                                  1/1     Running            0                74m     10.104.33.200   4am-node36   <none>           <none>

[2024-12-09T09:53:38.397Z] backup-test-82-pulsarv3-bookie-2                                  1/1     Running            0                74m     10.104.23.179   4am-node27   <none>           <none>

[2024-12-09T09:53:38.397Z] backup-test-82-pulsarv3-bookie-init-nnzrb                         0/1     Completed          0                74m     10.104.13.154   4am-node16   <none>           <none>

[2024-12-09T09:53:38.397Z] backup-test-82-pulsarv3-broker-0                                  1/1     Running            0                74m     10.104.13.153   4am-node16   <none>           <none>

[2024-12-09T09:53:38.397Z] backup-test-82-pulsarv3-broker-1                                  1/1     Running            0                74m     10.104.6.170    4am-node13   <none>           <none>

[2024-12-09T09:53:38.397Z] backup-test-82-pulsarv3-proxy-0                                   1/1     Running            0                74m     10.104.14.203   4am-node18   <none>           <none>

[2024-12-09T09:53:38.397Z] backup-test-82-pulsarv3-proxy-1                                   1/1     Running            0                74m     10.104.9.83     4am-node14   <none>           <none>

[2024-12-09T09:53:38.397Z] backup-test-82-pulsarv3-pulsar-init-zh99w                         0/1     Completed          0                74m     10.104.13.151   4am-node16   <none>           <none>

[2024-12-09T09:53:38.397Z] backup-test-82-pulsarv3-recovery-0                                1/1     Running            0                74m     10.104.14.204   4am-node18   <none>           <none>

[2024-12-09T09:53:38.397Z] backup-test-82-pulsarv3-zookeeper-0                               1/1     Running            0                74m     10.104.33.196   4am-node36   <none>           <none>

[2024-12-09T09:53:38.397Z] backup-test-82-pulsarv3-zookeeper-1                               1/1     Running            0                74m     10.104.25.27    4am-node30   <none>           <none>

[2024-12-09T09:53:38.397Z] backup-test-82-pulsarv3-zookeeper-2                               1/1     Running            0                74m     10.104.20.242   4am-node22   <none>           <none>

Anything else?

backup.log

The text was updated successfully, but these errors were encountered:

zhuwenxing · 2024-12-10T07:14:43Z

/assign @czs007

please take a look.there are some bulk insert issue.

sre-ci-robot · 2024-12-10T07:14:46Z

@zhuwenxing: GitHub didn't allow me to assign the following users: czs007.

Note that only zilliztech members, repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time.
For more information please see the contributor guide

In response to this:

/assign @czs007

please take a look.there are some bulk insert issue.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

zhuwenxing · 2024-12-11T03:28:50Z

When there are multiple partitions, restore timeout (240s) will consistently reproduce

failed ci job: https://qa-jenkins.milvus.io/blue/organizations/jenkins/milvus_backup_api_test/detail/milvus_backup_api_test/92/pipeline

cluster: 4am
namespase: chaos-testing
pod info:

milvus

[2024-12-11T02:49:48.203Z] + kubectl get pods -o wide

[2024-12-11T02:49:48.204Z] + grep backup-test-92

[2024-12-11T02:49:48.204Z] backup-test-92-etcd-0                                      1/1     Running            0                  11m     10.104.26.39    4am-node32   <none>           <none>

[2024-12-11T02:49:48.204Z] backup-test-92-etcd-1                                      1/1     Running            0                  11m     10.104.25.167   4am-node30   <none>           <none>

[2024-12-11T02:49:48.204Z] backup-test-92-etcd-2                                      1/1     Running            0                  11m     10.104.23.118   4am-node27   <none>           <none>

[2024-12-11T02:49:48.204Z] backup-test-92-kafka-0                                     1/1     Running            2 (11m ago)        11m     10.104.26.37    4am-node32   <none>           <none>

[2024-12-11T02:49:48.204Z] backup-test-92-kafka-1                                     1/1     Running            1 (11m ago)        11m     10.104.25.169   4am-node30   <none>           <none>

[2024-12-11T02:49:48.204Z] backup-test-92-kafka-2                                     1/1     Running            1 (11m ago)        11m     10.104.23.120   4am-node27   <none>           <none>

[2024-12-11T02:49:48.204Z] backup-test-92-milvus-standalone-ddb56c8bf-4gff6           1/1     Running            2 (11m ago)        11m     10.104.16.195   4am-node21   <none>           <none>

[2024-12-11T02:49:48.204Z] backup-test-92-minio-6d55c6fcf-2g24d                       1/1     Running            0                  11m     10.104.26.36    4am-node32   <none>           <none>

[2024-12-11T02:49:48.204Z] backup-test-92-zookeeper-0                                 1/1     Running            0                  11m     10.104.26.38    4am-node32   <none>           <none>

[2024-12-11T02:49:48.204Z] backup-test-92-zookeeper-1                                 1/1     Running            0                  11m     10.104.25.166   4am-node30   <none>           <none>

[2024-12-11T02:49:48.204Z] backup-test-92-zookeeper-2                                 1/1     Running            0                  11m     10.104.27.156   4am-node31   <none>           <none>

backp

milvus-backup-92-57cbbc7758-4zhqd   1/1     Running   0          2m1s

zhuwenxing · 2024-12-11T03:57:13Z

/assign @bigsheeper

PTAL

sre-ci-robot · 2024-12-11T03:57:17Z

@zhuwenxing: GitHub didn't allow me to assign the following users: bigsheeper.

Note that only zilliztech members, repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time.
For more information please see the contributor guide

In response to this:

/assign @bigsheeper

PTAL

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

zhuwenxing · 2024-12-11T07:26:17Z

/assign @huanghaoyuanhhy

sre-ci-robot · 2024-12-11T07:26:20Z

@zhuwenxing: GitHub didn't allow me to assign the following users: huanghaoyuanhhy.

Note that only zilliztech members, repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time.
For more information please see the contributor guide

In response to this:

/assign @huanghaoyuanhhy

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

huanghaoyuanhhy · 2024-12-12T02:57:53Z

It has been fixed in this commit.

zhuwenxing · 2024-12-12T09:56:19Z

The restore performance is degraded. There are 3 collections, each collection has 6 partitions, and each collection has 3000 num entities of data. A restore of this order of magnitude takes more than 6 minutes.

[2024-12-12T09:24:38.524Z] [2024-12-12 09:24:37 - DEBUG - urllib3.connectionpool]: Starting new HTTP connection (1): 10.255.199.4:8080 (connectionpool.py:246)

[2024-12-12T09:24:56.556Z] [2024-12-12 09:24:53 - DEBUG - urllib3.connectionpool]: http://10.255.199.4:8080/ "POST /api/v1/create HTTP/1.1" 200 262 (connectionpool.py:475)

[2024-12-12T09:24:56.556Z] [2024-12-12 09:24:53 - DEBUG - ci_test]: 

[2024-12-12T09:24:56.556Z] method: post, 

[2024-12-12T09:24:56.556Z] url: http://10.255.199.4:8080/api/v1/create, 

[2024-12-12T09:24:56.556Z] cost time: 15.768121242523193, 

[2024-12-12T09:24:56.556Z] header: {'Content-Type': 'application/json', 'RequestId': 'e87d85b4-b86a-11ef-95e1-ea1b5643dcfe'}, 

[2024-12-12T09:24:56.556Z] payload: {"async": false, "backup_name": "backup_KgIWN0Jo", "collection_names": ["restore_backup_8SR6wA2j", "restore_backup_3dBoErUB", "restore_backup_7g6g4L3T"]}, 

[2024-12-12T09:24:56.556Z] response: {"requestId":"f196a54f-b86a-11ef-b7f1-eee64936f590","msg":"success","data":{"id":"f196a54f-b86a-11ef-b7f1-eee64936f590","state_code":2,"start_time":1733995493106,"end_time":1733995493588,"progress":0,"name":"backup_KgIWN0Jo","size":0,"milvus_version":"a514f83"}} (milvus_backup.py:24)

[2024-12-12T09:24:56.556Z] [2024-12-12 09:24:53 - INFO - ci_test]: create backup response: {'requestId': 'f196a54f-b86a-11ef-b7f1-eee64936f590', 'msg': 'success', 'data': {'id': 'f196a54f-b86a-11ef-b7f1-eee64936f590', 'state_code': 2, 'start_time': 1733995493106, 'end_time': 1733995493588, 'progress': 0, 'name': 'backup_KgIWN0Jo', 'size': 0, 'milvus_version': 'a514f83'}} (test_restore_backup.py:191)

[2024-12-12T09:24:56.556Z] [2024-12-12 09:24:53 - DEBUG - urllib3.connectionpool]: Starting new HTTP connection (1): 10.255.199.4:8080 (connectionpool.py:246)

[2024-12-12T09:24:56.556Z] [2024-12-12 09:24:53 - DEBUG - urllib3.connectionpool]: http://10.255.199.4:8080/ "GET /api/v1/get_backup?backup_name=backup_KgIWN0Jo HTTP/1.1" 200 None (connectionpool.py:475)

[2024-12-12T09:24:56.556Z] [2024-12-12 09:24:53 - DEBUG - ci_test]: 

[2024-12-12T09:24:56.556Z] method: get, 

[2024-12-12T09:24:56.556Z] url: http://10.255.199.4:8080/api/v1/get_backup, 

[2024-12-12T09:24:56.556Z] cost time: 0.013997316360473633, 

[2024-12-12T09:24:56.556Z] header: {'Content-Type': 'application/json', 'RequestId': 'f1e3afa2-b86a-11ef-95e1-ea1b5643dcfe'}, 

[2024-12-12T09:24:56.556Z] payload: null, 

[2024-12-12T09:24:56.556Z] response: {"requestId":"f1e599fa-b86a-11ef-b7f1-eee64936f590","msg":"success","data":{"id":"f196a54f-b86a-11ef-b7f1-eee64936f590","state_code":2,"start_time":1733995493106,"end_time":1733995493588,"progress":100,"name":"backup_KgIWN0Jo","collection_backups":[{"id":"f196a54f-b86a-11ef-b7f1-eee64936f590","state_code":0,"start_time":1733995493,"end_time":1733995493,"progress":0,"collection_id":454556425285273127,"db_name":"default","collection_name":"restore_backup_8SR6wA2j","schema":{"name":"restore_backup_8SR6wA2j","fields":[{"fieldID":100,"name":"int64","is_primary_key":true,"data_type":5,"autoID":true},{"fieldID":101,"name":"float","data_type":10},{"fieldID":102,"name":"varchar","data_type":21,"type_params":[{"key":"max_length","value":"65535"}]},{"fieldID":103,"name":"json","data_type":23},{"fieldID":104,"name":"binary_vector","data_type":100,"type_params":[{"key":"dim","value":"128"}]}]},"shards_num":2,"backup_timestamp":454556507777794049,"size":130598,"has_index":false,"load_state":"NotLoad","backup_physical_timestamp":1733995493,"channel_checkpoints":{"by-dev-rootcoord-dml_0_454556425285273127v0":"CitieS1kZXYtcm9vdGNvb3JkLWRtbF8wXzQ1NDU1NjQyNTI4NTI3MzEyN3YwEggPAAAAAAAAABo7ZGF0YW5vZGUtMS1ieS1kZXYtcm9vdGNvb3JkLWRtbF8wXzQ1NDU1NjQyNTI4NTI3MzEyN3YwLXRydWUggYDAw8SXuqcG","by-dev-rootcoord-dml_1_454556425285273127v1":"CitieS1kZXYtcm9vdGNvb3JkLWRtbF8xXzQ1NDU1NjQyNTI4NTI3MzEyN3YxEggPAAAAAAAAABo7ZGF0YW5vZGUtMS1ieS1kZXYtcm9vdGNvb3JkLWRtbF8xXzQ1NDU1NjQyNTI4NTI3MzEyN3YxLXRydWUggYDAw8SXuqcG"}},{"id":"f196a54f-b86a-11ef-b7f1-eee64936f590","state_code":0,"start_time":1733995493,"end_time":1733995493,"progress":0,"collection_id":454556425285705593,"db_name":"default","collection_name":"restore_backup_7g6g4L3T","schema":{"name":"restore_backup_7g6g4L3T","fields":[{"fieldID":100,"name":"int64","is_primary_key":true,"data_type":5,"autoID":true},{"fieldID":101,"name":"float","data_type":10},{"fieldID":102,"name":"varchar","data_type":21,"type_params":[{"key":"max_length","value":"65535"}]},{"fieldID":103,"name":"json","data_type":23},{"fieldID":104,"name":"float_vector","data_type":101,"type_params":[{"key":"dim","value":"128"}]}]},"shards_num":2,"backup_timestamp":454556510084661249,"size":1642684,"has_index":false,"load_state":"NotLoad","backup_physical_timestamp":1733995493,"channel_checkpoints":{"by-dev-rootcoord-dml_4_454556425285705593v0":"CitieS1kZXYtcm9vdGNvb3JkLWRtbF80XzQ1NDU1NjQyNTI4NTcwNTU5M3YwEggRAAAAAAAAABo7ZGF0YW5vZGUtMS1ieS1kZXYtcm9vdGNvb3JkLWRtbF80XzQ1NDU1NjQyNTI4NTcwNTU5M3YwLXRydWUggYDAj82XuqcG","by-dev-rootcoord-dml_5_454556425285705593v1":"CitieS1kZXYtcm9vdGNvb3JkLWRtbF81XzQ1NDU1NjQyNTI4NTcwNTU5M3YxEggRAAAAAAAAABo7ZGF0YW5vZGUtMS1ieS1kZXYtcm9vdGNvb3JkLWRtbF81XzQ1NDU1NjQyNTI4NTcwNTU5M3YxLXRydWUggYDAj82XuqcG"}},{"id":"f196a54f-b86a-11ef-b7f1-eee64936f590","state_code":0,"start_time":1733995493,"end_time":1733995493,"progress":0,"collection_id":454556425285687961,"db_name":"default","collection_name":"restore_backup_3dBoErUB","schema":{"name":"restore_backup_3dBoErUB","fields":[{"fieldID":100,"name":"int64","is_primary_key":true,"data_type":5,"autoID":true},{"fieldID":101,"name":"float","data_type":10},{"fieldID":102,"name":"varchar","data_type":21,"type_params":[{"key":"max_length","value":"65535"}]},{"fieldID":103,"name":"json","data_type":23},{"fieldID":104,"name":"float_vector","data_type":101,"type_params":[{"key":"dim","value":"128"}]}]},"shards_num":2,"backup_timestamp":454556509075406849,"size":1642697,"has_index":false,"load_state":"NotLoad","backup_physical_timestamp":1733995493,"channel_checkpoints":{"by-dev-rootcoord-dml_2_454556425285687961v0":"CitieS1kZXYtcm9vdGNvb3JkLWRtbF8yXzQ1NDU1NjQyNTI4NTY4Nzk2MXYwEggRAAAAAAAAABo7ZGF0YW5vZGUtMS1ieS1kZXYtcm9vdGNvb3JkLWRtbF8yXzQ1NDU1NjQyNTI4NTY4Nzk2MXYwLXRydWUggYCgrsmXuqcG","by-dev-rootcoord-dml_3_454556425285687961v1":"CitieS1kZXYtcm9vdGNvb3JkLWRtbF8zXzQ1NDU1NjQyNTI4NTY4Nzk2MXYxEggRAAAAAAAAABo7ZGF0YW5vZGUtMS1ieS1kZXYtcm9vdGNvb3JkLWRtbF8zXzQ1NDU1NjQyNTI4NTY4Nzk2MXYxLXRydWUggYCgrsmXuqcG"}}],"size":3415979,"milvus_version":"a514f83"}} (milvus_backup.py:24)

[2024-12-12T09:24:56.556Z] [2024-12-12 09:24:53 - DEBUG - urllib3.connectionpool]: Starting new HTTP connection (1): 10.255.199.4:8080 (connectionpool.py:246)

[2024-12-12T09:31:04.615Z] [2024-12-12 09:30:56 - DEBUG - urllib3.connectionpool]: http://10.255.199.4:8080/ "POST /api/v1/restore HTTP/1.1" 200 957 (connectionpool.py:475)

[2024-12-12T09:31:04.615Z] [2024-12-12 09:30:56 - DEBUG - ci_test]: 

[2024-12-12T09:31:04.615Z] method: post, 

[2024-12-12T09:31:04.615Z] url: http://10.255.199.4:8080/api/v1/restore, 

[2024-12-12T09:31:04.615Z] cost time: 363.18878197669983, 

[2024-12-12T09:31:04.615Z] header: {'Content-Type': 'application/json', 'RequestId': 'f1e5f208-b86a-11ef-95e1-ea1b5643dcfe'}, 

[2024-12-12T09:31:04.615Z] payload: {"async": false, "backup_name": "backup_KgIWN0Jo", "collection_suffix": "_bak", "collection_names": ["restore_backup_8SR6wA2j", "restore_backup_3dBoErUB", "restore_backup_7g6g4L3T"]}, 

[2024-12-12T09:31:04.615Z] response: {"requestId":"f1e78f53-b86a-11ef-b7f1-eee64936f590","msg":"success","data":{"id":"restore_2024_12_12_09_24_53_630802497","state_code":0,"start_time":1733995493,"end_time":1733995856,"collection_restore_tasks":[{"id":"f1e8459d-b86a-11ef-b7f1-eee64936f590","state_code":2,"start_time":1733995493,"target_collection_name":"restore_backup_8SR6wA2j_bak","restored_size":130598,"to_restore_size":130598,"progress":0,"target_db_name":"default"},{"id":"f1e8e319-b86a-11ef-b7f1-eee64936f590","state_code":2,"start_time":1733995493,"target_collection_name":"restore_backup_7g6g4L3T_bak","restored_size":1642684,"to_restore_size":1642684,"progress":0,"target_db_name":"default"},{"id":"f1e96beb-b86a-11ef-b7f1-eee64936f590","state_code":2,"start_time":1733995493,"target_collection_name":"restore_backup_3dBoErUB_bak","restored_size":1642697,"to_restore_size":1642697,"progress":0,"target_db_name":"default"}],"restored_size":0,"to_restore_size":3415979,"progress":0}} (milvus_backup.py:24)

[2024-12-12T09:31:04.616Z] [2024-12-12 09:30:56 - INFO - ci_test]: restore_backup: {'requestId': 'f1e78f53-b86a-11ef-b7f1-eee64936f590', 'msg': 'success', 'data': {'id': 'restore_2024_12_12_09_24_53_630802497', 'state_code': 0, 'start_time': 1733995493, 'end_time': 1733995856, 'collection_restore_tasks': [{'id': 'f1e8459d-b86a-11ef-b7f1-eee64936f590', 'state_code': 2, 'start_time': 1733995493, 'target_collection_name': 'restore_backup_8SR6wA2j_bak', 'restored_size': 130598, 'to_restore_size': 130598, 'progress': 0, 'target_db_name': 'default'}, {'id': 'f1e8e319-b86a-11ef-b7f1-eee64936f590', 'state_code': 2, 'start_time': 1733995493, 'target_collection_name': 'restore_backup_7g6g4L3T_bak', 'restored_size': 1642684, 'to_restore_size': 1642684, 'progress': 0, 'target_db_name': 'default'}, {'id': 'f1e96beb-b86a-11ef-b7f1-eee64936f590', 'state_code': 2, 'start_time': 1733995493, 'target_collection_name': 'restore_backup_3dBoErUB_bak', 'restored_size': 1642697, 'to_restore_size': 1642697, 'progress': 0, 'target_db_name': 'default'}], 'restored_size': 0, 'to_restore_size': 3415979, 'progress': 0}} (test_restore_backup.py:215)

[2024-12-12T09:31:04.616Z] [2024-12-12 09:30:56 - DEBUG - urllib3.connectionpool]: Starting new HTTP connection (1): 10.255.199.4:8080 (connectionpool.py:246)

[2024-12-12T09:31:04.616Z] [2024-12-12 09:30:56 - DEBUG - urllib3.connectionpool]: http://10.255.199.4:8080/ "GET /api/v1/get_restore?id=restore_2024_12_12_09_24_53_630802497 HTTP/1.1" 200 965 (connectionpool.py:475)

test job: https://qa-jenkins.milvus.io/blue/organizations/jenkins/milvus_backup_api_test/detail/milvus_backup_api_test/100/pipeline

huanghaoyuanhhy · 2024-12-13T06:13:57Z

/assign

zhuwenxing changed the title ~~[Bug]: When running ci cases, there will be backup and restore timeout issues, but if running a single case separately, it can be successful~~ [Bug]: When running ci cases, there will be restore timeout issues Dec 11, 2024

sre-ci-robot assigned huanghaoyuanhhy Dec 13, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Bug]: When running ci cases, there will be restore timeout issues #473

[Bug]: When running ci cases, there will be restore timeout issues #473

zhuwenxing commented Dec 10, 2024 •

edited

Loading

zhuwenxing commented Dec 10, 2024

sre-ci-robot commented Dec 10, 2024

zhuwenxing commented Dec 11, 2024

zhuwenxing commented Dec 11, 2024

sre-ci-robot commented Dec 11, 2024

zhuwenxing commented Dec 11, 2024

sre-ci-robot commented Dec 11, 2024

huanghaoyuanhhy commented Dec 12, 2024

zhuwenxing commented Dec 12, 2024

huanghaoyuanhhy commented Dec 13, 2024

[Bug]: When running ci cases, there will be restore timeout issues #473

[Bug]: When running ci cases, there will be restore timeout issues #473

Comments

zhuwenxing commented Dec 10, 2024 • edited Loading

Current Behavior

Expected Behavior

Steps To Reproduce

Environment

Anything else?

zhuwenxing commented Dec 10, 2024

sre-ci-robot commented Dec 10, 2024

zhuwenxing commented Dec 11, 2024

zhuwenxing commented Dec 11, 2024

sre-ci-robot commented Dec 11, 2024

zhuwenxing commented Dec 11, 2024

sre-ci-robot commented Dec 11, 2024

huanghaoyuanhhy commented Dec 12, 2024

zhuwenxing commented Dec 12, 2024

huanghaoyuanhhy commented Dec 13, 2024

zhuwenxing commented Dec 10, 2024 •

edited

Loading