Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Incremental Store LSI #243

Open
wants to merge 59 commits into
base: main
Choose a base branch
from
Open

Incremental Store LSI #243

wants to merge 59 commits into from

Conversation

DanEngelbrecht
Copy link
Owner

@DanEngelbrecht DanEngelbrecht commented Feb 24, 2023 •
edited
Loading

  • ADDED New store index strategy allowing incremental download/upload of store index files for remote stores. Options to control and enable new strategy are --max-store-index-size and --store-index-cache-path.
  • ADDED New options for cp command: --max-store-index-size
  • ADDED New options for create-version-store-index command: --store-index-cache-path
  • ADDED New options for downsync command: --store-index-cache-path
  • ADDED New options for get command: --store-index-cache-path
  • ADDED New options for print-version-usage command: --store-index-cache-path
  • ADDED New options for prune-store command: --store-index-cache-path
  • ADDED New options for put command: --store-index-cache-path and --max-store-index-size
  • ADDED New options for upsync command: --store-index-cache-path and --max-store-index-size
  • ADDED New options for validate-version command: --store-index-cache-path
  • FIXED Test case usage of temporary folders
  • FIXED Fixed help for --storage-uri option
  • FIXED Fixed help for --s3-endpoint-resolver-uri option

@DanEngelbrecht DanEngelbrecht force-pushed the de/incremental-store-lsi branch from f1575f5 to 89019ec Compare February 24, 2023 23:49
@DanEngelbrecht DanEngelbrecht force-pushed the de/incremental-store-lsi branch 2 times, most recently from b04f913 to 71569d8 Compare March 5, 2023 23:15
@DanEngelbrecht DanEngelbrecht force-pushed the de/incremental-store-lsi branch from f4d721c to f10bfc1 Compare April 11, 2023 22:54
@DanEngelbrecht DanEngelbrecht changed the title WIP Incremental Store LSI Apr 12, 2023
@DanEngelbrecht DanEngelbrecht force-pushed the de/incremental-store-lsi branch 2 times, most recently from 89f7079 to d60a0b9 Compare April 16, 2023 21:18
@DanEngelbrecht
Copy link
Owner Author

Is cleaning up the local lsi cache not working properly?

@DanEngelbrecht DanEngelbrecht force-pushed the de/incremental-store-lsi branch from c954c70 to ca262bd Compare April 22, 2023 19:01
@DanEngelbrecht DanEngelbrecht marked this pull request as ready for review April 23, 2023 07:32
@DanEngelbrecht DanEngelbrecht force-pushed the de/incremental-store-lsi branch from 912e08f to f626925 Compare April 23, 2023 08:29
@DanEngelbrecht
use testify2 for all command tests
afd065e
@DanEngelbrecht
turn all asserts the right way
9d23faa
@DanEngelbrecht
only look for .lsi files when scanning for store index
9af4427
@DanEngelbrecht
add MaxStoreIndexSizeOption, default to legacy behaviour but support …
b67a9bb 
…reading new style stores
@DanEngelbrecht
a little less spam in log when reading/writing/deleting with retry
1f85eae
@DanEngelbrecht
reduce log spam
f2a5337
@DanEngelbrecht
remove lock in fsBlobStore and use rename to avoid partial files
1a6f106
@DanEngelbrecht
WIP
0415e1d 
# Conflicts:
#	commands/go.mod
#	remotestore/go.mod
#	remotestore/go.sum
@DanEngelbrecht
fixed PutStoreLSI sometimes deleting unmerged source lsis
ae034e7
@DanEngelbrecht
safer putstorelsi retry
78141f8
@DanEngelbrecht
add control over worker count when doing put/get store lsis
576031f
@DanEngelbrecht
propagate max index store size
0c032ec
@DanEngelbrecht
exists(exists * merged lsicount) -> getobjects(1)
d0360a0
@DanEngelbrecht
handle lsi cache store as input parameter with path
279bf1e
@DanEngelbrecht
enable lsi-cache for put/upsync
0178c3b
@DanEngelbrecht
don't use legacy lsi reading if a lsi cache path is provided
4ff2612
@DanEngelbrecht
use explicit store-index-cache-path option for lsi cache path
edcccf1
@DanEngelbrecht
lsi cache store path
a01823f
@DanEngelbrecht
fix upsynch store index cache option
0703ddb 
tests
@DanEngelbrecht
os.ModeDir -> os.ModePerm
05303e6
@DanEngelbrecht
simple test folder
d439916
@DanEngelbrecht
use t.TempDir() for all tests
38ac48e
@DanEngelbrecht
changelog
de1d2f1 
# Conflicts:
#	CHANGELOG.md
@DanEngelbrecht
remove maxStoreIndexSize option from create-version-store-index command
b987456
@DanEngelbrecht DanEngelbrecht force-pushed the de/incremental-store-lsi branch from 258af02 to b987456 Compare April 24, 2023 10:10
@DanEngelbrecht
Copy link
Owner Author

TODO

  • Split up index created at "init" if it goes beyond the size limit.
  • Add size limit argument to "init" command
  • More places we should split indexes?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@DanEngelbrecht