Skip to content

Releases: AfricasVoices/Pipeline-Infrastructure

0.0.11

13 Aug 14:30
Compare
Choose a tag to compare
  • Automatically handle duplicate files on Drive (#38).
  • Improve logging of Drive uploads (#37).

0.0.10

14 Jul 12:27
Compare
Choose a tag to compare
  • Adds a cache to FirestoreUuidTable, which improves the performance of batch data/uuid lookups and reduces Firestore costs. The API is unchanged, just update to this version to benefit (#36).
  • Removes tools/migrate_phone_number_uuid_table_to_firestore.py and associated functions. Not marking as a breaking change because attempting to use this has resulted in an AssertionError for the last few versions, so all that's changed is the error (#35).

0.0.9

24 Jun 11:24
7f07ec4
Compare
Choose a tag to compare
  • Prints the correct page numbers in logs for drive_upload_client (#33).

0.0.8

09 Jun 10:33
Compare
Choose a tag to compare
  • Adds drive_client_wrapper.update_or_create_batch, for batch uploading multiple files to a directory (#32).

0.0.7

03 Jun 12:19
Compare
Choose a tag to compare
  • Automatically retries Drive upload requests on socket.timeout and HttpError 503 with binary exponentiated backoff (#28, #27).
  • Fixes the Google Cloud uploader's automatic retry on timeout checking the wrong chunk size (#30).
  • Adds google_cloud_utils.list_blobs (#31).

0.0.6

04 May 13:34
dfc0a90
Compare
Choose a tag to compare

Automatically retries g-cloud uploads that fail due to connection or timeout errors #25

0.0.5

20 Apr 12:10
Compare
Choose a tag to compare

Automatically retries Drive uploads that fail with HttpError 500 (#23)

0.0.4

15 Jan 14:28
Compare
Choose a tag to compare

IMPORTANT: Prevents an issue where the FirestoreUuidTable could overwrite existing phone number <-> uuid mappings with a new uuid. All users should upgrade immediately.

  • Applies an emergency fix to the FirestoreUuidTable to prevent it overwriting existing uuids. In instances where it would previously have overwritten a uuid, it now crashes instead #21). A more performant, more reliable fix will come in a future release.
  • Adds de_identify_csv.py, which takes a CSV containing a column of phone numbers and replaces the phone numbers with uuids (#20).

0.0.3

16 Jul 13:47
Compare
Choose a tag to compare
  • Added a tool for migrating phone number <-> uuid tables from CoreDataModules to Firestore (id_infrastructure/migrate_phone_number_uuid_table_to_firestore.py) (#12).
  • Added functions for uploading files directly to blobs and downloading blobs directly to files (storage.google_cloud.googcle_cloud_utils.{download_blob_to_file, upload_file_to_blob}) (#16)

0.0.2

10 May 07:39
Compare
Choose a tag to compare
  • Convert all print/Python logging calls to use core_data_modules.Logging (#11)