How to deal with data file corrupted situation

[Reidddddd]

One of an insert_log stored in S3 is corrupted, and couldn't recover (all of its blocks are gone)

I observed that the compaction keeps compacting the corrupted insert_log, failed and re-downloaded looping endless..

Want to check what's the proper action to deal with these kinds of situation

[yhmo]

The path of the corrupted file is still recorded in etcd, so milvus tries to compact it.

The is a developer tool birdwatch: https://github.com/milvus-io/birdwatcher. By this tool, you can do something hack to milvus.

1. Download the tool from here: https://github.com/milvus-io/birdwatcher/releases
2. Make sure the 2379 port of the etcd container has been exposed
3. In the terminal, start the birdwatcher, enter the command line mode
4. Connect the etcd by command: connect --etcd [ip]:[port] --rootPath xxx
5. Remove the missed log path from etcd by this command: remove binlog --collectionID xxxxxx --partitionID xxxxxx --segmentID xxxxxx --fieldID xxx --logType statslog run You can find the collectionID/partitionID/segmentID/fieldID/logType from the full path of the corrupted file.

[Reidddddd]

i‘m still confused: If only deleting meta in etcd, how does it help make every role back to normal state.

Sound like still needs manual interference.

Manual deleting S3 file is operatable, but how to deal with like sealed segments on QueryNode side, also memory data in every coords..

[yhmo]

After you delete the path from etcd, the file in minio becomes a garage, and occupies disk space. In fact, it is ok you don't manually delete it from minio, milvus won't read it since the etcd doesn't have its path.

[Reidddddd]

「but how to deal with like sealed segments on QueryNode side, also memory data in every coords..」

what about these

[xiaofan-luan]

once you cleaned etcd data, restart the cluster could work. because all the memory meta is load from etcd
Btw how could a file on S3 corrupted? any ideas?

[Reidddddd]

Underlying is ozone, i'm informed that there are missing blocks and couldn't be recovered

[Reidddddd]

Thanks @yhmo @xiaofan-luan

[yhmo]

Similar with this discussion: #37411

@Reidddddd
Sorry, the command should be:
remove binlog --collectionID 449576166950553184 --partitionID 453397315907946999 --segmentID 453397315909285711 --fieldID 100 --logType statslog --run

I forgot to add "--" to "run".
After this command, we will see:

[Reidddddd]

Skimmed the discussion, can i still use logType binlog or delatlog when either of them is unreadable and needs to be cleaned up..

[yhmo]

Yes, in the same way.
For example, a delta log missed, you will get error when loading the collection:
[WARN] [storage/remote_chunk_manager.go:207] ["failed to read object"] [path=files/delta_log/453787183301263392/-1/453787183301463412/453787183301463413]

Then you can use birdwatcher to remove its path from etcd:
remove binlog --collectionID 453787183301263392 --partitionID -1 --segmentID 453787183301463412 --logType deltalog --run