Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

(Chore): Migrate to poetry #141

Open
wants to merge 66 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 63 commits
Commits
Show all changes
66 commits
Select commit Hold shift + click to select a range
0c74d5f
(Chore): Migrate to poetry
BinamB Oct 9, 2024
d9272d6
Error handling
BinamB Oct 10, 2024
d7e735d
fix logger
BinamB Oct 10, 2024
8b8bdb6
Add more logs about etag
BinamB Oct 18, 2024
c27df4d
AWS replicate add second profile
BinamB Oct 21, 2024
d990825
revert
BinamB Oct 22, 2024
72f49d1
Add support for multi account
BinamB Oct 22, 2024
5cba3fc
change thread size
BinamB Oct 22, 2024
9f20daf
etag logs11
BinamB Oct 24, 2024
0def6f4
log
BinamB Oct 24, 2024
423bd10
more logs
BinamB Oct 24, 2024
3273d6c
more logs
BinamB Oct 24, 2024
f8d8b2e
etag re-calculation
BinamB Oct 25, 2024
fdd6f95
urllib2
BinamB Oct 25, 2024
046e79f
sig digest
BinamB Oct 25, 2024
5c41ec4
back to urllib
BinamB Oct 25, 2024
93da37e
Add back the requirements file
BinamB Oct 28, 2024
48e5d4b
more logs
BinamB Oct 28, 2024
7775539
hexdigest
BinamB Oct 28, 2024
0fd6d6b
part level md5 does not match
BinamB Oct 28, 2024
2f0f820
Remove md5 calculations
BinamB Oct 29, 2024
6e5d10d
upadte requirements
BinamB Oct 31, 2024
c30a4e6
update requirements
BinamB Nov 1, 2024
f8016f4
poetry lock
BinamB Nov 1, 2024
7f63af6
Add log
BinamB Nov 1, 2024
29f1e33
add logs
BinamB Nov 4, 2024
b4fad60
add error logs
BinamB Nov 4, 2024
9ebca74
change account profile
BinamB Nov 15, 2024
674a12e
remve proxy
BinamB Nov 15, 2024
2fe9460
remove session
BinamB Nov 15, 2024
937698d
add back pofile
BinamB Nov 15, 2024
5e348dc
change profile
BinamB Nov 15, 2024
ca8da7a
change profile
BinamB Nov 15, 2024
679a476
revert open account profile
BinamB Nov 18, 2024
c58ad24
change profile
BinamB Nov 18, 2024
397927c
revert
BinamB Nov 18, 2024
e3dac00
revert
BinamB Nov 19, 2024
3309989
add single upload
BinamB Nov 19, 2024
d82d4b8
fix single part handler
BinamB Nov 20, 2024
5260da3
fix handler
BinamB Nov 20, 2024
07ae027
add logs
BinamB Nov 20, 2024
0573864
fix loop
BinamB Nov 20, 2024
ac62c62
add default profile
BinamB Nov 20, 2024
be03314
fix profile_name
BinamB Nov 20, 2024
10cba4a
fix size check
BinamB Nov 20, 2024
d2c6847
comment out size
BinamB Nov 20, 2024
7d3fb26
size
BinamB Nov 20, 2024
d8dcbf5
fix things
BinamB Nov 20, 2024
71efa75
fix tries
BinamB Nov 20, 2024
1974040
decouple
BinamB Nov 21, 2024
45d886e
fix
BinamB Nov 21, 2024
c99478c
remove single part upload stuff
BinamB Nov 25, 2024
6a8c564
add acl
BinamB Nov 25, 2024
16c4ef4
add logs
BinamB Nov 26, 2024
8164872
fix file_name
BinamB Nov 26, 2024
8522d64
or
BinamB Nov 26, 2024
1f41685
comment out log bucket
BinamB Dec 2, 2024
0cceceb
change file directory
BinamB Dec 4, 2024
3ae835f
check only for GDC files
BinamB Dec 5, 2024
8feafa5
Add validation stuff
BinamB Dec 12, 2024
78736e3
fix get filename
BinamB Dec 13, 2024
162375f
Fix imports
BinamB Dec 13, 2024
f663e6c
Update Dockerfile josn args and update to key=value
MaribelleHGomez Dec 18, 2024
2fe488b
Update google_replicate.py
MaribelleHGomez Dec 18, 2024
f9172cc
resolve comments
BinamB Jan 14, 2025
ec8c69e
Pinning urllib3 to fix dependency issue
jacob50231 Jan 23, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Binary file added .DS_Store
Binary file not shown.
2 changes: 1 addition & 1 deletion .pre-commit-config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,6 @@ repos:
- id: no-commit-to-branch
args: [--branch, develop, --branch, master, --pattern, release/.*]
- repo: https://github.com/psf/black
rev: 20.8b1
rev: 22.3.0
hooks:
- id: black
13 changes: 7 additions & 6 deletions Dockerfile
Original file line number Diff line number Diff line change
@@ -1,21 +1,22 @@
FROM quay.io/cdis/python:3.7-slim-buster
FROM quay.io/cdis/python:python3.9-buster-2.0.0

RUN pip install --upgrade pip poetry
RUN apt-get update && apt-get install -y git jq curl bash snapd groff python3-pip zip

RUN pip install --upgrade pip
RUN pip install awscli

# Installing gcloud package (includes gsutil)
RUN curl https://dl.google.com/dl/cloudsdk/release/google-cloud-sdk.tar.gz > /tmp/google-cloud-sdk.tar.gz
RUN mkdir -p /usr/local/gcloud \
&& tar -C /usr/local/gcloud -xvf /tmp/google-cloud-sdk.tar.gz \
&& /usr/local/gcloud/google-cloud-sdk/install.sh
ENV PATH $PATH:/usr/local/gcloud/google-cloud-sdk/bin
ENV PATH=$PATH:/usr/local/gcloud/google-cloud-sdk/bin


COPY . /dcf-dataservice
WORKDIR /dcf-dataservice

RUN pip install -r requirements.txt
RUN poetry config virtualenvs.create false \
&& poetry install -vv --no-root --no-dev --no-interaction \
&& poetry show -v

CMD /bin/bash
CMD ["/bin/bash"]
6 changes: 3 additions & 3 deletions dataflow_pipeline.py
Original file line number Diff line number Diff line change
Expand Up @@ -11,13 +11,13 @@
from apache_beam.options.pipeline_options import SetupOptions
import json

from scripts.google_replicate import google_copy_wrapper
from scripts.utils import (
from dcfdataservice.google_replicate import google_copy_wrapper
from dcfdataservice.utils import (
get_ignored_files,
build_object_dataset_gs,
prepare_txt_manifest_google_dataflow,
)
from scripts.settings import IGNORED_FILES, PROJECT_ACL
from dcfdataservice.settings import IGNORED_FILES, PROJECT_ACL

try:
unicode # pylint: disable=unicode-builtin
Expand Down
Binary file added dcfdataservice/.DS_Store
Binary file not shown.
File renamed without changes.
Loading
Loading