updatae #79
Merged
updatae #79
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
davedavemckay
added a commit
that referenced
this pull request
Nov 21, 2024
* added api option to CLI * adding swift helper functions to bm and allowing api choice in lsst_backup.py * made swift version of object_list * main done * tested download_file_swift * print api name * . * fix cli * . * fix find_metadata args * . * removed large file * initiate Dask before checking metadata * print Dask details * added timing * parallel metadata search * download keys for swift api * explicit dask dataframe calls * npartitions * test boto3 * genuinely excited * pass api to process_files * undo test change * . * partition by worker count * . * 21 seconds to go * n_workers*100 partitions * . * nightly - looking at line 692 * adding swift functions * added swift test script for large file uplaod * . * fixed paths * . * create SwiftUploadObjects * add SwiftUploadObject to bm * . * try service.upload * . * . * did it upload? * prepend remote path * use generator * use auth_version 1 * export within Python * typo * bm version CI * updatae (#79) * dask can serialise swift!! * Create bm_version.yml * push branch * . * doublequotes * update in bm folder * test gh actions * bm version: updated to -dev- * proper dev version naming * bm version: updated to bm_version-dev-git_hash * remove file * bm version: updated to bm_version-dev-git_hash * finished versioning actions * bm version: updated to bm_version-dev-git_hash * remove option to switch off checksumming * no_checksum option deprecated * always use separate object for zip-contents metadata (as S3 and Swift have differenve value size limitations) * . * . * remove exit * prepend object name * print result * print result * . * . * try local path in segmented_upload * segmented upload may just work * still segment, but give original path for each segment * prepend all S3 host URLs with https:// * put_container before list contents * print put results * print from generator * if not None * try None * prepend s3_host iwth https:// * print remote path * use remote path * was giving bucket_name in as source * https * . * segment upload working in swift_tests.py * separate container * print all results * swap bucket and segment bucket * https * print results * time test * try remote path * large file path in SwiftUploadObjects * pass object_name to SwiftUploadObject * use only environment variables for creds * bm version: updated to bm_version-dev-git_hash * test environment from bm * bm version: updated to bm_version-dev-git_hash * test with only env variables * ST_KEY *not* ST_SECRET * bm version: updated to bm_version-dev-git_hash * default api is now swift! * bm version: updated to bm_version-dev-git_hash * all done except Swfit segment upload * segment upload done - need to print success/fail from results * update zip_and_upload * pass api * update remove_bucket * update list_buckets.py * updating DAGs to use env vars * try content_type None * update bucket_contents to use new env vars * try checksum string as etag * remove file_data_size line * send mem_check info to stderr * done. --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.