migrator not able to pull from v1 registry

[AXavez]

I am attempting to migrate images from a 0.8.1 registry running on a rhel 6.5 server with docker engine 1.4 to a v2 registry running on a rhel 6.7 server with docker engine 1.7.1. I am able to pull the list of images but when the script tries to pull the images I get an error response from the daemon : invalid registry endpoint <v1_registry>/v0/:unable to ping registry endpoint.
I get the same error when trying to pull from an individual command. I noticed the registry adds https in front of the registry name even though I have added the --insecure-registry flag on both machines.
Here is my command:

  sudo docker run -it -v /var/run/docker.sock:/var/run/docker.sock -e V1_REGISTRY=<v1_registry> -e V2_REGISTRY=<v2_registry> -e ERROR_ACTION=prompt -e NO_LOGIN=true -e USE_HTTP=true docker/migrator

Any help will be greatly appreciated.

[mbentley]

Yes, it adds https by default but if your registry runs without TLS, you can use the following flags:
USE_HTTP - both v1 and v2 use http only (equivalent of having both of the below flags in use)

...or if just one of them uses http:
V1_USE_HTTP - only v1 uses http
V2_USE_HTTP - only v2 uses http

[AXavez]

I already have that flag set. Any ideas as to why I'm still getting that error?

[mbentley]

For the actual values of:
-e V1_REGISTRY=<v1_registry> and -e V2_REGISTRY=<v2_registry>

Are you using localhost or anything with http:// in it?

Also, at what point are you getting the error?

[AXavez]

No, I am using server names with port 5000 specified. e.g myServername:5000

I am getting the error when the migrator tries to pull the images from the V1 registry. I should note that I am running the migrator on the second host machine that has registry v2, Here is a snippet of the error:

[OK] End full list of images from <v1_registry>

[!!] Press any key to begin migration process [ctrl+c to abort]

[INFO] Pulling all images from <v1_registry> to your local system
[INFO] Pulling <v1_registry>/base/centos:centos6 (1 of 9)

[mbentley]

And by "I get the same error when trying to pull from an individual command.", you're saying that if you do a:
docker pull <v1_registry>/base/centos:centos6
from the machine that you're running the v2 on (also where you're running migrator), it fails?

[AXavez]

Yep, you got it.

[mbentley]

OK, that's definitely going to be an issue. Anything in the docker daemon logs from that node when you're trying to pull? Like watching the daemon logs while attempting to pull (same node):
journalctl -fu docker.service

Might actually need debug logs to get enough info though.

Also, RHEL 6 is notoriously terrible for running docker because there isn't proper kernel support to do so.

[AXavez]

I noticed when I tried to watch the log concurrently that it would not update during the 'trouble part' in the script. Seeing the same behavior when running the command individually.

[mbentley]

That's probably where enabling debugging on the daemon logs would be useful. It's been a while since I've used RHEL 6 but I believe it should just be updating systemd to add the debug flag to the ExecStart line:
https://docs.docker.com/engine/admin/systemd

[AXavez]

Just to clarify am I getting the debug logs from the v1 registry? Or from the migrator container?

[mbentley]

This would be the debug logs from the node from where you're running the docker pull, not the node running the v1 registry.

[AXavez]

Here are the debug logs. I noticed a warning: client and server don't have the same version (client 1.6.2, server:1.7.1) which is strange because my client has 1.4.1.

time="2017-03-08T10:52:25.491250741-05:00" level=debug msg="attach: stdin: begin"
time="2017-03-08T10:52:25.491738950-05:00" level=debug msg="attach: stdout: begin"
time="2017-03-08T10:52:25.491979472-05:00" level=debug msg="attach: stderr: begin"
time="2017-03-08T10:52:25.493359779-05:00" level=debug msg="Calling POST /containers/{name:.*}/start"
time="2017-03-08T10:52:25.493447033-05:00" level=info msg="POST /v1.19/containers/59146c1f90b2c3371e3f82cce35c7995c4e293e4f1c829bceac014fb1a75f482/start"
time="2017-03-08T10:52:25.493625641-05:00" level=debug msg="activateDeviceIfNeeded(59146c1f90b2c3371e3f82cce35c7995c4e293e4f1c829bceac014fb1a75f482)"
time="2017-03-08T10:52:26.149023312-05:00" level=debug msg="Calling POST /containers/{name:.*}/resize"
time="2017-03-08T10:52:26.149311393-05:00" level=info msg="POST /v1.19/containers/59146c1f90b2c3371e3f82cce35c7995c4e293e4f1c829bceac014fb1a75f482/resize?h=33&w=175"
time="2017-03-08T10:52:26.335720419-05:00" level=debug msg="Calling GET /info"
time="2017-03-08T10:52:26.336073055-05:00" level=info msg="GET /v1.18/info"
time="2017-03-08T10:52:26.336220651-05:00" level=debug msg="Warning: client and server don't have the same version (client: 1.6.2, server: 1.7.1)"
time="2017-03-08T10:52:29.865983783-05:00" level=debug msg="Calling POST /images/create"
time="2017-03-08T10:52:29.866207976-05:00" level=info msg="POST /v1.18/images/create?fromImage=myV1Registry%3A5000%2Fbase%2Fcentos%3Acentos6"
time="2017-03-08T10:52:29.866294559-05:00" level=debug msg="Warning: client and server don't have the same version (client: 1.6.2, server: 1.7.1)"
time="2017-03-08T10:52:29.916966279-05:00" level=debug msg="pulling image from host \"myV1Registry:5000\" with remote name \"base/centos\""
time="2017-03-08T10:52:29.917436755-05:00" level=debug msg="pinging registry endpoint https://myV1Registry:5000/v0/"
time="2017-03-08T10:52:29.917602366-05:00" level=debug msg="attempting v2 ping for registry endpoint https://myV1Registry:5000/v2/"
time="2017-03-08T10:52:29.917725290-05:00" level=debug msg="hostDir: /etc/docker/certs.d/myV1Registry:5000"
time="2017-03-08T10:52:32.122754861-05:00" level=debug msg="attempting v1 ping for registry endpoint https://myV1Registry:5000/v1/"
time="2017-03-08T10:52:32.122983271-05:00" level=debug msg="hostDir: /etc/docker/certs.d/myV1Registry:5000"
time="2017-03-08T10:52:34.282202123-05:00" level=error msg="Handler for POST /images/create returned error: invalid registry endpoint https://myV1Registry:5000/v0/: unable to ping registry endpoint https://myV1Registry:5000/v0/\nv2 ping attempt failed with error: Get https://myV1Registry:5000/v2/: EOF\n v1 ping attempt failed with error: Get https://myV1Registry:5000/v1/_ping: EOF. If this private registry supports only HTTP or HTTPS with an unknown CA certificate, please add `--insecure-registry myV1Registry:5000` to the daemon's arguments. In the case of HTTPS, if you have access to the registry's CA certificate, no need for the flag; simply place the CA certificate at /etc/docker/certs.d/myV1Registry:5000/ca.crt"
time="2017-03-08T10:52:34.282716956-05:00" level=error msg="HTTP Error" err="invalid registry endpoint https://myV1Registry:5000/v0/: unable to ping registry endpoint https://myV1Registry:5000/v0/\nv2 ping attempt failed with error: Get https://myV1Registry:5000/v2/: EOF\n v1 ping attempt failed with error: Get https://myV1Registry:5000/v1/_ping: EOF. If this private registry supports only HTTP or HTTPS with an unknown CA certificate, please add `--insecure-registry myV1Registry:5000` to the daemon's arguments. In the case of HTTPS, if you have access to the registry's CA certificate, no need for the flag; simply place the CA certificate at /etc/docker/certs.d/myV1Registry:5000/ca.crt" statusCode=500

[mbentley]

The client/server warning is probably because you're running the migrator container; it has a 1.6.2 docker client for backwards compatibility reasons but it isn't a critical warning.

The error I am seeing is complaining about not having the proper --insecure-registry myV1Registry:5000 flag specified. Did you include the port in the setting on the daemon that is performing the pulling?

[AXavez]

Yep, I did.

[mbentley]

Can you provide the output of docker info from the node where you're attempting the pull? The daemon logs indicate that the insecure registry setting isn't set.

[AXavez]

I set the flags in /etc/sysconfig/docker because my system does not use systemd. But, here is the output:

Containers: 3
Images: 16
Storage Driver: devicemapper
 Pool Name: docker-253:0-6029417-pool
 Pool Blocksize: 65.54 kB
 Backing Filesystem: extfs
 Data file: /dev/loop0
 Metadata file: /dev/loop1
 Data Space Used: 505.9 MB
 Data Space Total: 107.4 GB
 Data Space Available: 106.9 GB
 Metadata Space Used: 1.311 MB
 Metadata Space Total: 2.147 GB
 Metadata Space Available: 2.146 GB
 Udev Sync Supported: true
 Deferred Removal Enabled: false
 Data loop file: /var/lib/docker/devicemapper/devicemapper/data
 Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
 Library Version: 1.02.95-RHEL6 (2015-09-08)
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 2.6.32-642.6.2.el6.x86_64
Operating System: <unknown>
CPUs: 8
Total Memory: 15.58 GiB
Name: <hostServerName>
ID: HIXG:C46Z:T2CV:E2UX:DUBM:LVL2:YTRJ:F73D:DBFV:LXH3:X2LP:FKAZ
Debug mode (server): true
File Descriptors: 16
Goroutines: 23
System Time: 2017-03-08T11:27:14.653694003-05:00
EventsListeners: 0
Init SHA1: 4193d771605617409a8d2578127b8fbd5754e969
Init Path: /usr/libexec/docker/dockerinit
Docker Root Dir: /var/lib/docker