Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Initial Meilisearch Support #54

Merged
merged 54 commits into from
Sep 20, 2023
Merged

Initial Meilisearch Support #54

merged 54 commits into from
Sep 20, 2023

Conversation

majst01
Copy link
Contributor

@majst01 majst01 commented Aug 24, 2023

implements #47

TODO:

  • create a dump before the upgrade with a copy of the previous binary. This solves the burden to have a high backup frequency in order to not loose data during a upgrade
  • put dumps directly into the backup volume, start meilisearch with --dump-dir pointing to the extra backup volume to save additional copy and duplicate data in the datadir

@majst01
Copy link
Contributor Author

majst01 commented Sep 7, 2023

Integration fails now because initially test timeout was set to 10m, raised it to 20m to make it pass, but a better approach would be to make a matrix job which iterates over database and restore/upgrade

Copy link
Contributor

@Gerrit91 Gerrit91 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for taking this effort, I know it's cumbersome. Looking forward to get this running!

cmd/internal/database/meilisearch/meilisearch.go Outdated Show resolved Hide resolved
cmd/internal/database/meilisearch/meilisearch.go Outdated Show resolved Hide resolved
cmd/internal/database/meilisearch/meilisearch.go Outdated Show resolved Hide resolved
cmd/internal/database/meilisearch/meilisearch.go Outdated Show resolved Hide resolved
cmd/internal/database/meilisearch/meilisearch.go Outdated Show resolved Hide resolved
cmd/internal/database/meilisearch/meilisearch.go Outdated Show resolved Hide resolved
cmd/internal/database/meilisearch/meilisearch.go Outdated Show resolved Hide resolved
cmd/internal/database/meilisearch/meilisearch.go Outdated Show resolved Hide resolved
cmd/internal/database/meilisearch/meilisearch.go Outdated Show resolved Hide resolved
integration/meilisearch_test.go Outdated Show resolved Hide resolved
@vknabel vknabel self-requested a review September 14, 2023 12:15
@Gerrit91
Copy link
Contributor

Gerrit91 commented Sep 18, 2023

During @majst01 vacation, I took over this PR. 😅

I applied the following changes now:

  • Copying the old meilisearch binary after a backup and use this older version during the upgrade procedure, which now goes "start with old binary" -> "create a dump" -> "import with current binary" -> "done" (similar to Postgres upgrade)
  • Stabilized the tests (integration tests take ~7m now, which I think is ok)
    • Longer probe timeouts as the wait process in the database pod was killed by K8s all the time, switching into exponential crash loop, which caused delays in the tests
    • Started the "helper databases" on Port 1 in order to prevent the wait process to assume that Meilisearch is already running during the restoration / upgrade
  • Proper usage of db-path and dump-dir to avoid a lot of path handling in the implementation
  • Moved example generation out of the tests into the Makefile, as well as added a start-meilisearch target for dev purposes
  • Rollback after failed recovery attempt in order not to start with empty database

Still need to think about all the error handling one more time.

README.md Outdated Show resolved Hide resolved
return fmt.Errorf("could not clean backup directory: %w", err)
}

if err := os.MkdirAll(constants.BackupDir, 0777); err != nil {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this too broad? Would 755 or something similar be enough?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probably yes, but there are many other places where same permissions are used. Maybe we should create a separate PR and address this.

cmd/internal/database/meilisearch/meilisearch.go Outdated Show resolved Hide resolved
@Gerrit91 Gerrit91 merged commit 3983ac2 into master Sep 20, 2023
2 checks passed
@Gerrit91 Gerrit91 deleted the meilisearch-support branch September 20, 2023 13:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants