Recovery milvus after minio down

[kietheros]

My MinIO went down, and after recovering all components, Milvus is unable to load some log files from MinIO. I checked in MinIO, and these files do not exist. Could you guide me on how to configure Milvus to ignore the missing files and continue to load collections?

Milvus cluster v2.3.21 with external kafka, minio, etcd

milvus-datanode-1 | [2024/11/04 08:37:06.847 +00:00] [ERROR] [retry/retry.go:46] ["retry func failed"] ["retry time"=0] [error="NoSuchKey(key=files/stats_log/449576166950553184/453397315907946999/453397315909285711/100/453397315899218214)"] [stack="github.com/milvus-io/milvus/pkg/util/retry.Do\n\t/workspace/source/pkg/util/retry/retry.go:46\ngithub.com/milvus-io/milvus/internal/storage.(*RemoteChunkManager).Read\n\t/workspace/source/internal/storage/remote_chunk_manager.go:166\ngithub.com/milvus-io/milvus/internal/storage.(*RemoteChunkManager).MultiRead\n\t/workspace/source/internal/storage/remote_chunk_manager.go:222\ngithub.com/milvus-io/milvus/internal/datanode.(*ChannelMeta).loadStats\n\t/workspace/source/internal/datanode/channel_meta.go:430\ngithub.com/milvus-io/milvus/internal/datanode.(*ChannelMeta).initPKstats\n\t/workspace/source/internal/datanode/channel_meta.go:472\ngithub.com/milvus-io/milvus/internal/datanode.(*ChannelMeta).InitPKstats\n\t/workspace/source/internal/datanode/channel_meta.go:331\ngithub.com/milvus-io/milvus/internal/datanode.(*ChannelMeta).addSegment\n\t/workspace/source/internal/datanode/channel_meta.go:242\ngithub.com/milvus-io/milvus/internal/datanode.getChannelWithEtcdTickler.func1\n\t/workspace/source/internal/datanode/data_sync_service.go:249\ngithub.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n\t/workspace/source/pkg/util/conc/pool.go:81\ngithub.com/panjf2000/ants/v2.(*goWorker).run.func1\n\t/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:67"]

[yhmo]

Similar to this issue: #32228

If only the stat log is missed, maybe you can manually compact to regenerate it:

collection = Collection(collection_name)
collection.compact()

[yhmo]

The second way is using the birdwatch tool: https://github.com/milvus-io/birdwatcher. By this tool, you can do something hack to milvus.

1. Download the tool from here: https://github.com/milvus-io/birdwatcher/releases
2. Make sure the 2379 port of the etcd container has been exposed
3. In terminal, start the birdwatcher, enter the command line mode
4. Connect the etcd by command: connect --etcd [ip]:[port] --rootPath xxx
5. Remove the missed log path from etcd by this command: remove binlog --collectionID 449576166950553184 --partitionID 453397315907946999 --segmentID 453397315909285711 --fieldID 100 --logType statslog run

[kietheros]

Thank you very much! I’ll try your solution now.

[xiaofan-luan]

1. Don't know if you minio lose some files, how many files is reporting missing on your error log? Are you using standalone minio?
2. if there is not much, then you can use birdwatcher to remove all the missing stats logs. But if it is then probably we need more info about why this is happening. for example, does any files still exist on minio?

[kietheros]

@xiaofan-luan This is my development environment with a MinIO cluster and Kafka cluster. I powered off two servers for testing, and there are three missing files which printed in the DataNode logs:

files/stats_log/449576166950553184/453397315907946999/453397315909285711/100/453397315899218214
files/stats_log/449576166950553184/453397315907946999/453397315909285711/100/453397315899218392
files/stats_log/449576166950686518/453397315907946949/453397315909348208/100/453397315899013659

In this command
remove binlog --collectionID 449576166950553184 --partitionID 453397315907946999 --segmentID 453397315909285711 --fieldID 100 --logType statslog run

After fieldID 100, do I need to set file name 453397315899218214, 453397315899218392, 453397315899013659 for removing ?

[kietheros]

@yhmo @xiaofan-luan I tried using two methods:

collection = Collection(collection_name)
collection.compact()

remove binlog --collectionID 449576166950553184 --partitionID 453397315907946999 --segmentID 453397315909285711 --fieldID 100 --logType statslog run

Milvus still not working, collection loading hang at 32%, after 30 min, loaded at 42%
I have attached full milvus logs.
node-1.tar.gz
node-2.tar.gz
node-3.tar.gz

[xiaofan-luan]

I20241106 09:46:00.823225 279 Utils.cpp:711] [SEGCORE][LoadFieldDatasFromRemote][milvus] failed to load data from remote: Error in GetObjectSize[errcode:404, exception:, errmessage:No response body., params:params, bucket=milvus, object=files/insert_log/449576166950686518/453397315902656797/453397315903827344/106/453397315899004273

there is still a lot of logs can not be found.

[xiaofan-luan]

I don't know what is going on on your minio cluster, but it seems there are many object lost

here is another one
[2024/11/06 09:49:38.060 +00:00] [WARN] [segments/segment_loader.go:265] ["load segment failed when load data into memory"] [traceID=8f634119844965ff0598133bd455dce4] [collectionID=449576166950686518] [segmentType=Growing] [partitionID=453397315907946949] [segmentID=453397315909348208] [error="Error in GetObjectSize[errcode:502, exception:, errmessage:No response body., params:params, bucket=milvus, object=files/insert_log/449576166950686518/453397315907946949/453397315909348208/0/453397315899013486]"]
[2024/11/06 09:49:38.060 +00:00] [ERROR] [funcutil/parallel.go:88] [loadSegmentFunc] [error="Error in GetObjectSize[errcode:502, exception:, errmessage:No response body., params:params, bucket=milvus, object=files/insert_log/449576166950686518/453397315907946949/453397315909348208/0/453397315899013486]"] [idx=2] [stack="github.com/milvus-io/milvus/pkg/util/funcutil.ProcessFuncParallel.func3\n\t/workspace/source/pkg/util/funcutil/parallel.go:88"]

[kietheros]

@yhmo Thank you, I will try again. Could you also let me know which command to use for this file
files/insert_log/449576166950686518/453397315905412923/453397315899008945/1/453397315899008954

[yhmo]

If a insertlog/statslog file is missing, milvus will run into trouble because the query nodes firstly read the paths from etcd and use the paths to read files from s3/minio. In fact it is a disaster.
We don't have a fast way to clean all the missing file paths from etcd. The birdwatcher tool needs to be used carefully.

Yes, if an insert log file is missed, the vectors in it will be lost.

[kietheros]

@yhmo I accept data loss as long as Milvus starts working again. Could you confirm the steps I need to follow?

1. Remove stats log

remove binlog ... --logType statslog --run

Restart Milvus, then compact and load the collection.

2. Remove insert log

I’m not sure what logType should be set to here: binlog, deltalog, or statslog. Are there any additional steps I need to take for this?

3. Birdwatcher is running in interactive mode

Is there a way to run "birdwatcher remove binlog ... --logType statslog --run" directly from the command line?

[yhmo]

• An insert log is missed, you will get this error log:
  [WARN] [segments/segment_loader.go:322] ["load segment failed when load data into memory"] [traceID=8888e6d8d52f9fc91d9111092d33b4f5] [collectionID=453787358847565895] [segmentType=Sealed] [requestSegments="[453787358847965923]"] [preparedSegments="[453787358847965923]"] [partitionID=453787358847565896] [segmentID=453787358847965923] [segmentType=L1] [error="At LoadSegment: Error in GetObjectSize[errcode:404, exception:, errmessage:No response body., params:params, bucket=a-bucket, object=files/insert_log/453787358847565895/453787358847565896/453787358847965923/101/453787358847965926]"]

Then you can use birdwatcher to remove its path from etcd:
remove binlog --collectionID 453787358847565895 --partitionID 453787358847565896 --segmentID 453787358847965923 --fieldID 101 --logType binlog --run

• A delta log is missed, you will get error when loading the collection:
  [WARN] [storage/remote_chunk_manager.go:207] ["failed to read object"] [path=files/delta_log/453787183301263392/-1/453787183301463412/453787183301463413]

Then you can use birdwatcher to remove its path from etcd:
remove binlog --collectionID 453787183301263392 --partitionID -1 --segmentID 453787183301463412 --logType deltalog --run

[kietheros]

@yhmo @xiaofan-luan Thank you all for your dedicated support. My MinIO lost quite a few files, so I decided to clear data and reinstall Milvus.