private ip address changes upon restart, apps try to use old ip

[colnpanic]

when initially setting up a database for an app the current private ip for the postgres container is written to the ENV file, then if the container is stopped and a new one created it (usually?) has a different private ip.

what i currently do is hand modify each ENV file to point to the new address, then dokku release $APP && dokku deploy $APP. i'd like to automate this so that it automagically finds all apps (potentially looking at /home/dokku/.postgresql/pass_*), tweaks their ENV, and re-deploys if currently running.

another probably better idea is to implement the new-ish (docker >= 0.6.5) '-name postgresql' combined with 'link' on the app run, although i need to look into that a bit more to make sure it does what i think.

the reason i haven't done this yet is that i'm curious how other people handle this, or if it is taken care of somehow that i just don't have set up or am missing.

[jeffutter]

Name and link is the 'best' solution here. However I don't think dokku supports starting the app containers with the link flags. I'll try to ping the dokku devs and see if they have any suggestions.

[jeffutter]

Ok, I think i have a solution here. Can you test out this branch:

https://github.com/jeffutter/dokku-postgresql-plugin/tree/docker-name-link

You will need either delete and re-create your DB or change some things in your $DOKKU_ROOT/$APP/ENV

They need to be changed to look like this:

https://github.com/jeffutter/dokku-postgresql-plugin/blob/docker-name-link/commands#L60-L63

Note that the it uses seperate variables for the user, password and db_name. And then sets DATABASE_URL to literally equal '"postgress://${DATABASE_APP}:${DATABASE_PASS}@${DATABASE_PORT_5432_TCP_ADDR}:${DATABASE_PORT_5432_TCP_PORT}/${DATABASE_DB}"'

This causes that ENV variable to be set from the other variables when the container is launched and thus can pull the ip/port from the -link command.

It's kinda hacky but it works.

[poirier]

This is not quite working for me; the DATABASE_URL env var in my app doesn't get the host or port.

(Printed from my app:)
os.environ['DATABASE_URL']='postgres://app-name:VFRiOWVmOVc2UThiRHUxamp5M0IzU25zNGhNS2JVUThzeUt1UDRNZ0k4OD0K@:/app-name_production'

# dokku config app-name
=== app-name Config Vars
DATABASE_DATABASE:      app-name_production
DATABASE_HOST:          ${DATABASE_PORT_5432_TCP_ADDR}
DATABASE_PASSWORD:      VFRiOWVmOVc2UThiRHUxamp5M0IzU25zNGhNS2JVUThzeUt1UDRNZ0k4OD0K
DATABASE_PORT:          ${DATABASE_PORT_5432_TCP_PORT}
DATABASE_URL:           postgres://app-name:VFRiOWVmOVc2UThiRHUxamp5M0IzU25zNGhNS2JVUThzeUt1UDRNZ0k4OD0K@${DATABASE_PORT_5432_TCP_ADDR}:${DATABASE_PORT_5432_TCP_PORT}/app-name_production
DATABASE_USERNAME:      app-name
...

# dokku version
v0.2.1

root@dokku:/var/lib/dokku/plugins/postgresql# git status
# On branch docker-name-link
nothing to commit (working directory clean)

[jeffutter]

This is odd. DATABASE_PORT_5432_TCP_PORT should be set in the ENV of the container by docker. It seems like none of the ENV vars are being set. Are you using a current build of dokku? Perhaps something changed recently.

Can you check your docker ps output that the postgresql container has the name of the app listed under Names. It should look something like dokku-postgresql, appname/database

[poirier]

Dokku v0.2.1, which has been current for a while now.

Docker ps shows

b276537ba9d8        postgresql/app-name:latest   /usr/bin/start_pgsql   16 hours ago        Up 16 hours         0.0.0.0:49167->5432/tcp   happy_pasteur

dokku run app-name printenv shows only these DATABASE_ variables:

DATABASE_DATABASE=app-name_production
DATABASE_HOST=
DATABASE_PASSWORD=VFRiOWVmOVc2UThiRHUxamp5M0IzU25zNGhNS2JVUThzeUt1UDRNZ0k4OD0K
DATABASE_PORT=
DATABASE_URL=postgres://root:[email protected]:49167/db
DATABASE_USERNAME=app-name

[jeffutter]

Can you check on a couple more things for me.

1.) Does $DOKKU_ROOT/.postgresql/pass_$APP_NAME exist? It should be a file containing the password for the database user for the app
2.) is /var/lib/dokku/plugins/postgresql/docker-args marked executable?

[poirier]

I think the reason this isn't working is that Dokku v0.2.1 doesn't implement docker-args.

[jeffutter]

@poirier Good observation. This is the issue. I thought docker-args had been implemented for a while but I guess it wasn't in v0.2.1

Would you care to try with a more recent version?

[poirier]

I haven't had time to get things working completely again with the master branch of dokku, but I have gotten far enough to see a complete DATABASE_URL in my app and verify the app can access the database.

[jeffutter]

OK, thanks for troubleshooting and reminding me docker-args isn't in a dokku release. I'll ping the dokku guys to see if they will release soon. Until then this will have to remain a separate branch.

[bdentino]

any plans to merge docker-name-link back into master?

[joshco]

+1 Im running into this hassle as well. This seems like a common scenario. The setup should be able to survive a server reboot.
If there is a recommended solution or workaround could you put it in the wiki?

[joshco]

I found a solution. I'm using the virtual network gateway IP address rather than the container's
The way things get mapped, the postgres server is accessible on the gateway IP on port 5432
The gateway address is always the same across reboots and restarts.
I don't know enough about docker/dokku know if there is a piano waiting to fall on my head somewhere, but this is working for me.

To make the postgres auto start on reboot, I added this to my /etc/rc.local:

# make sure the output is logged somewhere
exec 2> /tmp/rc.local.log      # send stderr from rc.local to a log file
exec 1>&2                      # send stdout to the same log file
set -x                         # tell sh to display commands before execution

echo "Starting postgres container"
/usr/local/bin/dokku postgresql:start

[codeincontext]

Latest thoughts on this? Biggest pain with this plugin :(