Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

tikv crash during volumerestore [restore for 5577] #5819

Open
ti-chi-bot opened this issue Oct 25, 2024 · 1 comment
Open

tikv crash during volumerestore [restore for 5577] #5819

ti-chi-bot opened this issue Oct 25, 2024 · 1 comment
Assignees
@BornChanger
Labels
area/ebs-br type/bug Something isn't working

Comments

@ti-chi-bot
Copy link
Member

Bug Report

What version of Kubernetes are you using?

What version of TiDB Operator are you using?

What storage classes exist in the Kubernetes cluster and what are used for PD/TiKV pods?

What's the status of the TiDB cluster pods?

What did you do?

  1. do volumebackup
  2. do volumerestore

What did you expect to see?

  1. volumerestore success

What did you see instead?

  1. volumerestore failed
  2. Confirmed corruption is likely in rocksdb layer, rather than EBS snapshots. Corrupt log file was generated (and last fsynced as per manifest), 15 seconds before ebs snapshots started.
    There is some unknown write path which does partial writes to wal file, without updating the manifest.
@ti-chi-bot ti-chi-bot added area/ebs-br type/bug Something isn't working labels Oct 25, 2024
@ti-chi-bot
Copy link
Member Author

This issue is restored for #5577, it originally created at 2024-03-15 04:16:51.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@BornChanger BornChanger

Labels
area/ebs-br type/bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@ti-chi-bot @BornChanger