Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a backup page, due to problems with parallelized restore #17

Open
wants to merge 3 commits into
base: master
Choose a base branch
from
Open

Add a backup page, due to problems with parallelized restore #17

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
dfc7c6e
Add a backup page, due to problems with parallelized restore
Krysztophe Feb 11, 2020
4d878c4
Backup page - fixes
Krysztophe Dec 2, 2020
5234359
Backup page - pg_dump example
Krysztophe Dec 2, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions general/getting_started.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,3 +27,6 @@ See the :doc:`Installation <../opm-core/Installation>` page to install:
* the :ref:`nagios dispatcher script<nagios_and_nagios_dispatcher>` to
retrieve the perfdata
* the :ref:`web user interface <user_interface>`.

When preparing your backup and restoration procedures,
see :doc:`Backups <../opm-core/Backups>`.
22 changes: 22 additions & 0 deletions opm-core/Backups.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
Backups
Krysztophe marked this conversation as resolved.
Show resolved Hide resolved
=======

Usual backup procedures apply. The procedure should include the way to reinstall the extensions on a virgin server.
Krysztophe marked this conversation as resolved.
Show resolved Hide resolved

Restoring logical backups
-------------------------

In a general way, restoring a backup created with pg_dump should not be a problem as the backup contains the necessary `CREATE EXTENSION` orders. The extension binaries must be present.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OPM doesn't have any binaries. If you want to be stick to general recommendation that to apply to OPM, we should mention something like "All extension files (binaries, shared librraries, SQL files...)".

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changed

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Up!

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oops, sorry I totally forgot about that PR. I will come back to it very soon!


You may wish to parallelize the restoration to spare time. In this case, it may fail with a foreign key error, as the extensions install tables with foreign keys at the very beginning of the restore, and pg_restore does not try to respect this order. The trick is to parallelize only the restoration of the biggest tables.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it would be better to say "to save time".

Also, you mention the target PostgreSQL version. It kind of imply that this is a procedure generally used for major upgrade more than a general backup solution. It would be better to be more precise here that as we don't want to recommend pg_dump style backup as the best backup approach. So I would recommend to mention that PITR style backup don't have specific gotchas with OPM, and that's a form of backup that should generally be preferred.

I'm also not a fan of simply saying "The trick is...". Maybe "the best way to work around that problem is..."?

It also wouldn't hurt to explicitly mention here that the service_counters_* tables are the biggest table for OPM.


The following commands assume that the database and the roles were already created :
Krysztophe marked this conversation as resolved.
Show resolved Hide resolved

.. code-block:: bash

pg_restore -e --section=pre-data -d opm opm.dump
pg_restore -e --section=data -l opm.dump|grep -v 'service_counters_' > restore.1.list
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe some better names? Like opm_small_tables.list and opm_big_tables? Feel free to suggest better names.

pg_restore -e --section=data -l opm.dump|grep 'service_counters_' > restore.2.list
pg_restore -e -d opm -L restore.1.list opm.dump
pg_restore -e -d opm -L restore.2.list opm.dump --jobs=4
pg_restore -e --section=post-data -d opm opm.dump --jobs=4