Skip to content
This repository has been archived by the owner on Mar 11, 2020. It is now read-only.

Adding verbose option enable on curl. #77

Open
wants to merge 2 commits into
base: master
Choose a base branch
from
Open

Adding verbose option enable on curl. #77

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
2f0573a
Adding verbose option enable on curl.
davidsiefert May 18, 2016
b6bbb00
Adding support for skipping over images that fail to fetch..
davidsiefert May 18, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 27 additions & 2 deletions migrator.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,7 @@ initialize_migrator() {

# set default to require curl to perform ssl certificate validation
USE_INSECURE_CURL=${USE_INSECURE_CURL:-false}
VERBOSE_CURL=${VERBOSE_CURL:-false}

# set default to require https
USE_HTTP=${USE_HTTP:-false}
Expand Down Expand Up @@ -106,6 +107,19 @@ verify_ready() {
fi
fi

# enable verbose logging for curl requests
if [ "${VERBOSE_CURL}" != "true" ] && [ "${VERBOSE_CURL}" != "false" ]
then
catch_error "${BOLD}VERBOSE_CURL${CLEAR} environment variable (${VERBOSE_CURL}) invalid; must be either ${BOLD}true${CLEAR} or ${BOLD}false${CLEAR}"
else
# set VERBOSE_CURL environment variable to appropriate value
if [ "${VERBOSE_CURL}" = "true" ]
then
V1_OPTIONS="$V1_OPTIONS -v"
V2_OPTIONS="$V2_OPTIONS -v"
fi
fi

if [ "${V1_USE_HTTP}" = "true" ]
then
V1_PROTO="http"
Expand Down Expand Up @@ -177,6 +191,17 @@ catch_error() {
exit 1
}

# generic error catching
skipping() {
echo -e "\n${NOTICE} ${@}"
if [ "${DOCKER_HUB}" = "true" ]
then
echo -e "${NOTICE} Could not query ${1} from Docker Hub, skipping..."
else
echo -e "${NOTICE} Could not query ${1} from v1, skipping..."
fi
}

# catch push/pull error
catch_push_pull_error() {
# set environment variables to handle arguments
Expand Down Expand Up @@ -367,7 +392,7 @@ query_source_images() {
for i in ${REPO_LIST}
do
# get tags for repo
IMAGE_TAGS=$(curl ${INSECURE_CURL} -sf -H "Authorization: JWT ${TOKEN}" https://hub.docker.com/v2/repositories/${NAMESPACE}/${i}/tags/?page_size=100000 | jq -r '.results|.[]|.name') || catch_error "curl => API failure"
IMAGE_TAGS=$(curl ${INSECURE_CURL} -sf -H "Authorization: JWT ${TOKEN}" https://hub.docker.com/v2/repositories/${NAMESPACE}/${i}/tags/?page_size=100000 | jq -r '.results|.[]|.name') || echo "Hey man, this didn't work: ${NAMESPACE}/${i}, ya dig?"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This no longer catches the error; would be good if this could be optional to skip or fail like the other comment. It would be nice if the error was descriptive, switching this:
echo "Hey man, this didn't work: ${NAMESPACE}/${i}, ya dig?"

to something more specific to the tune of curl to the API failed to retrieve tags. with the namespace and repo. I definitely like that it tells you what failed though.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ditto above, this was just me playing with trying to get it to skip on errors... don't want this in my PR.


# build a list of images from tags
for j in ${IMAGE_TAGS}
Expand Down Expand Up @@ -398,7 +423,7 @@ query_source_images() {
for i in ${REPO_LIST}
do
# get list of tags for image i
IMAGE_TAGS=$(curl ${V1_OPTIONS} -sf ${V1_PROTO}://${AUTH_CREDS}@${V1_REGISTRY}/v1/repositories/${i}/tags | jq -r 'keys | .[]') || catch_error "curl => API failure"
IMAGE_TAGS=$(curl ${V1_OPTIONS} -sf ${V1_PROTO}://${AUTH_CREDS}@${V1_REGISTRY}/v1/repositories/${i}/tags | jq -r 'keys | .[]') || skipping "curl => API failure" "${i}"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like the idea of allowing skipping but it would be nice if there was a way to disable skipping of repos since if you're performing a migration, skipping means you're missing data.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is in the second commit which I do not want in my PR. Github is trigger happy and taking both my commits. How can I avoid this? Maybe just revert my commit (and/or commit it to a separate branch) so it doesn't get included here?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah yes, I see you've made a commit to your master branch. I'd create a feature branch from your master and then revert your last commit once you've created and checked out the new feature branch.

git branch <my descriptive name>
git reset --hard HEAD~1

You should be able to see in git log that it only includes the first commit then.

Then you can push that branch and you'll need to submit a new PR for the new feature branch.


# loop through tags to create list of full image names w/tags
for j in ${IMAGE_TAGS}
Expand Down