Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a backup page, due to problems with parallelized restore #17

Open
wants to merge 3 commits into
base: master
Choose a base branch
from
Open

Add a backup page, due to problems with parallelized restore #17

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
dfc7c6e
Add a backup page, due to problems with parallelized restore
Krysztophe Feb 11, 2020
4d878c4
Backup page - fixes
Krysztophe Dec 2, 2020
5234359
Backup page - pg_dump example
Krysztophe Dec 2, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions general/getting_started.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,3 +27,6 @@ See the :doc:`Installation <../opm-core/Installation>` page to install:
* the :ref:`nagios dispatcher script<nagios_and_nagios_dispatcher>` to
retrieve the perfdata
* the :ref:`web user interface <user_interface>`.

When preparing your backup and restoration procedures,
see :doc:`Backups <../opm-core/Backups>`.
32 changes: 32 additions & 0 deletions opm-core/Backups.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
Backups & Restore
=================

Backup
------

To create logical backups of the OPM database, use pg_dump as usual.
You can save much time with parallelization, eg:

.. code-block:: bash

pg_dump -Fd -j8 -d opm -f opm.dump


Restoring logical backups
-------------------------

In a general way, restoring a backup created with pg_dump should not be a problem, as the backup contains the necessary `CREATE EXTENSION` orders. The extensions must be installed for the target PostgreSQL version (see `Installation <Installation>`).

You may wish to parallelize the restoration to spare time. In this case, it may fail with a foreign key error, as the extensions install tables with foreign keys at the very beginning of the restore, and pg_restore does not try to respect this order. The trick is to parallelize only the restoration of the biggest tables.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it would be better to say "to save time".

Also, you mention the target PostgreSQL version. It kind of imply that this is a procedure generally used for major upgrade more than a general backup solution. It would be better to be more precise here that as we don't want to recommend pg_dump style backup as the best backup approach. So I would recommend to mention that PITR style backup don't have specific gotchas with OPM, and that's a form of backup that should generally be preferred.

I'm also not a fan of simply saying "The trick is...". Maybe "the best way to work around that problem is..."?

It also wouldn't hurt to explicitly mention here that the service_counters_* tables are the biggest table for OPM.


The following commands assume that the database and the roles were already created,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe "The following commands are an example of such a workaround, and assume that..."?

and the backup was not done in a plain format:

.. code-block:: bash

pg_restore -e --section=pre-data -d opm opm.dump
pg_restore -e --section=data -l opm.dump|grep -v 'service_counters_' > restore.1.list
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe some better names? Like opm_small_tables.list and opm_big_tables? Feel free to suggest better names.

pg_restore -e --section=data -l opm.dump|grep 'service_counters_' > restore.2.list
pg_restore -e -d opm -L restore.1.list opm.dump
pg_restore -e -d opm -L restore.2.list opm.dump --jobs=4
pg_restore -e --section=post-data -d opm opm.dump --jobs=4