Upgraded Borgmatic to version 1.9.0

[tony1661]

Description

• Borgmatic upgraded to version 1.9.0

Full list of changes can be found here.

New documentation can be found here.

Fixes #6195

Checklist

• Build rule all-supported completed successfully
• New installation of package completed successfully
• Package upgrade completed successfully (Manually install the package again)
• Package functionality was tested
• Any needed documentation is updated/created

Type of change

• Bug fix
• New Package
• Package update
• Includes small framework changes
• This change requires a documentation update (e.g. Wiki)

[tony1661]

I am ready to create documentation should this get merged. Just need to know where is the best place to do so.

[hgy59]

@tony1661 I there a reason not to update borgbackup (to v1.4.0)?

it is defined in src/requirements-crossenv.txt

an I guess other python wheels need updates too

[tony1661]

@hgy59 no reason. I have not tested Borg 1.4 personally since Debian ships with 1.2.4.

[hgy59]

I am ready to create documentation should this get merged. Just need to know where is the best place to do so.

package documentation are located in the wiki and linked to https://github.com/SynoCommunity/spksrc/wiki/Package-Documentation-Index (and the side panel).

[hgy59]

@hgy59 no reason. I have not tested Borg 1.4 personally since Debian ships with 1.2.4.

updates done.

[tony1661]

Thanks guys. So is all that's left documentation to get this merged?

[tony1661]

@hgy59 thanks for considering my PR. I've added documentation for Borgmatic

[th0ma7]

LGTM and timely considering other upcoming changes to the python wheel build process.

[tony1661]

I noticed that qoriq-6.2.4 is failing. Is that something to be concerned about?

[mreid-tt]

Can you review the change log? With the two commits for this PR, the change log may not be accurate.

[th0ma7]

I noticed that qoriq-6.2.4 is failing. Is that something to be concerned about?

This is being tracked through #6297 and fix found and being worked on at #6307.

This may in turn delay for a few days the publishing of this PR once merged so qoriq fix can be reviewed and merged as well.

[hgy59]

Can you review the change log? With the two commits for this PR, the change log may not be accurate.

@mreid-tt do you mean the commit comments with change log?
This is never a problem since it can be edited on "squash and merge" (and I always edit the commit history to remove obsolete entries and some empty lines)

[mreid-tt]

I'm not familiar with Borgmatic, so I can't comment on this directly. However, I came across this issue in the Discord channel and asked the user to submit a bug report. If anyone has insights or a way to reproduce the problem, it might be worth investigating further.

Since the last update of Borg, I can't run borgmatic anymore. I'm on DS214play (i686) which I think is 32 bits. When I try to do, for instance, a borgmatic -c borgmatic.yml list I get a Traceback, which ends up in:

  File "/volume1/@appstore/borgbackup/env/lib/python3.11/site-packages/ruamel/yaml/main.py", line 529, in init
    CParser.init(selfx, stream)
  File "_ruamel_yaml.pyx", line 302, in ruamel.yaml.clib.ruamelyaml.CParser.__init
TypeError: a string or stream input is required

I tried to uninstall borg and python3, then reinstall, but I have the same error.

[tony1661]

@mreid-tt I'd ask the user to verify their config is correct since sometimes borgmatic has changes to syntax that breaks old configs

[jfangmann]

I'm not familiar with Borgmatic, so I can't comment on this directly. However, I came across this issue in the Discord channel and asked the user to submit a bug report. If anyone has insights or a way to reproduce the problem, it might be worth investigating further.

Since the last update of Borg, I can't run borgmatic anymore. I'm on DS214play (i686) which I think is 32 bits. When I try to do, for instance, a borgmatic -c borgmatic.yml list I get a Traceback, which ends up in:

  File "/volume1/@appstore/borgbackup/env/lib/python3.11/site-packages/ruamel/yaml/main.py", line 529, in init
    CParser.init(selfx, stream)
  File "_ruamel_yaml.pyx", line 302, in ruamel.yaml.clib.ruamelyaml.CParser.__init
TypeError: a string or stream input is required

I tried to uninstall borg and python3, then reinstall, but I have the same error.

I am facing the same problem on my DS214play since the last update. Based on the suggested solution from @tony1661 I tried to create a sample configuration and diff with my existing one. While generating via borgmatic config generate --destination ./config.yaml I got this error:

Generating a configuration file at: ./config.yaml
Traceback (most recent call last):
  File "/usr/local/bin/borgmatic", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/volume1/@appstore/borgbackup/env/lib/python3.11/site-packages/borgmatic/commands/borgmatic.py", line 940, in main
    list(
  File "/volume1/@appstore/borgbackup/env/lib/python3.11/site-packages/borgmatic/commands/borgmatic.py", line 726, in collect_highlander_action_summary_logs
    borgmatic.actions.config.generate.run_generate(
  File "/volume1/@appstore/borgbackup/env/lib/python3.11/site-packages/borgmatic/actions/config/generate.py", line 25, in run_generate
    borgmatic.config.generate.generate_sample_configuration(
  File "/volume1/@appstore/borgbackup/env/lib/python3.11/site-packages/borgmatic/config/generate.py", line 292, in generate_sample_configuration
    schema = ruamel.yaml.YAML(typ='safe').load(open(schema_filename))
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/volume1/@appstore/borgbackup/env/lib/python3.11/site-packages/ruamel/yaml/main.py", line 451, in load
    return constructor.get_single_data()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/volume1/@appstore/borgbackup/env/lib/python3.11/site-packages/ruamel/yaml/constructor.py", line 114, in get_single_data
    node = self.composer.get_single_node()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "_ruamel_yaml.pyx", line 701, in ruamel.yaml.clib._ruamel_yaml.CParser.get_single_node
  File "_ruamel_yaml.pyx", line 902, in ruamel.yaml.clib._ruamel_yaml.CParser._parse_next_event
  File "_ruamel_yaml.pyx", line 919, in ruamel.yaml.clib._ruamel_yaml.input_handler
TypeError: a string value is expected

There seems to be a similar bug report from a reddit-user here.

Thanks for any help in advance! :)

[tony1661]

@jfangmann can you provide your borgmatic config?

[jfangmann]

Hey @tony1661, thanks for the quick reply! Of course, here it is. I deleted all the comment-lines for better readability. The patterns_from statement is considered experimental but was working fine before the update.

repositories:
    - path: ssh://[...]
      label: borgbase
# Read include/exclude patterns from one or more separate named files,
# one pattern per line. Note that Borg considers this option
# experimental. See the output of "borg help patterns" for more
# details.
patterns_from:
    - /etc/borgmatic/lightroom_patterns
encryption_passphrase: "[...]"
ssh_command: ssh [...]
keep_daily: 7
keep_weekly: 4
keep_monthly: 6
before_backup:
    - echo "Starting a backup."
after_backup:
    - echo "Finished a backup."

[tony1661]

@jfangmann hmm, this looks fine. Are you able to test this config on a Linux device to confirm that the issue persists?

The upgrade was from 1.2.0 to 1.9.0 and I'm not sure if that config is just not compatible with 1.9.0

I have this working on DS218 and DS220 so it's possible it's an issue with the chip the DS215play uses.

Let me know 🙂👍🏻

[jfangmann]

I wasn't able to test the config on another system, yet. But I will do so and let you know. But because the config generation fails, I tested this on my DS as well as a RPi running borgmatic via docker:

RPi output

jfangmann@rpi3:/opt/borgmatic $ docker exec -it borgmatic /bin/bash
db4fa901eee9:/# borgmatic --version
1.9.3
db4fa901eee9:/# borgmatic config generate --destination ./config.yaml
Generating a configuration file at: ./config.yaml

This includes all available configuration options with example values, the few
required options as indicated. Please edit the file to suit your needs.

If you ever need help: https://torsion.org/borgmatic/#issues

summary:
Generate successful
db4fa901eee9:/#

DS output

jfangmann@DiskStation:~$ borgmatic --version
1.9.0
jfangmann@DiskStation:~$ borgmatic config generate --destination ./config.yaml
Generating a configuration file at: ./config.yaml
Traceback (most recent call last):
  File "/usr/local/bin/borgmatic", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/volume1/@appstore/borgbackup/env/lib/python3.11/site-packages/borgmatic/commands/borgmatic.py", line 940, in main
    list(
  File "/volume1/@appstore/borgbackup/env/lib/python3.11/site-packages/borgmatic/commands/borgmatic.py", line 726, in collect_highlander_action_summary_logs
    borgmatic.actions.config.generate.run_generate(
  File "/volume1/@appstore/borgbackup/env/lib/python3.11/site-packages/borgmatic/actions/config/generate.py", line 25, in run_generate
    borgmatic.config.generate.generate_sample_configuration(
  File "/volume1/@appstore/borgbackup/env/lib/python3.11/site-packages/borgmatic/config/generate.py", line 292, in generate_sample_configuration
    schema = ruamel.yaml.YAML(typ='safe').load(open(schema_filename))
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/volume1/@appstore/borgbackup/env/lib/python3.11/site-packages/ruamel/yaml/main.py", line 451, in load
    return constructor.get_single_data()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/volume1/@appstore/borgbackup/env/lib/python3.11/site-packages/ruamel/yaml/constructor.py", line 114, in get_single_data
    node = self.composer.get_single_node()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "_ruamel_yaml.pyx", line 701, in ruamel.yaml.clib._ruamel_yaml.CParser.get_single_node
  File "_ruamel_yaml.pyx", line 902, in ruamel.yaml.clib._ruamel_yaml.CParser._parse_next_event
  File "_ruamel_yaml.pyx", line 919, in ruamel.yaml.clib._ruamel_yaml.input_handler
TypeError: a string value is expected
jfangmann@DiskStation:~$

So, the same error (or at least a similar one) seems to occur, even if my config isn't involved. Hope this helps and I will try to test the config soon.

[jfangmann]

After reviewing the conversation in reddit, I referred to, I noticed the following regarding the python-version on the DS:

jfangmann@DiskStation:~$ python --version
Python 3.8.12
jfangmann@DiskStation:~$ python3.11 --version
Python 3.11.10

Since I'm not familiar with python or what version is required, I have no idea if this is how it meant to be. @tony1661 Might this be a problem, or is it identical on your systems?

[hgy59]

I have executed the command borgmatic config generate --destination ./config.yaml on the following systems without any issue:

• virutalDSM (x64) DSM 7.2.2-72806 Update 2
• DS218 (aarch64) DSM 6.2.4-25556 Update 7
• DS115j (armada370) DSM 7.1.1-42962 Update 7

Unfortunately I have no access to a Diskstation with evansport arch (like DS214play), but I tested DS115j with (probably) the same DSM version as the DS214play.
So I guess it must be an issue with the system and not the package.

[hgy59]

After reviewing the conversation in reddit, I referred to, I noticed the following regarding the python-version on the DS:

jfangmann@DiskStation:~$ python --version
Python 3.8.12
jfangmann@DiskStation:~$ python3.11 --version
Python 3.11.10

Since I'm not familiar with python or what version is required, I have no idea if this is how it meant to be. @tony1661 Might this be a problem, or is it identical on your systems?

@jfangmann the installed package does not use python or python3.11 in the path.
it is installed into a virtual environment and you can inspect the venv configuration with:

$ cat /var/packages/borgbackup/target/env/pyvenv.cfg
home = /var/packages/python311/target/bin
include-system-site-packages = true
version = 3.11.10
executable = /volume1/@appstore/python311/bin/python3.11
command = /var/packages/python311/target/bin/python3 -m venv --system-site-packages /volume1/@appstore/borgbackup/env

[hgy59]

To validate the ruamel.yaml package, I run (successfully) the very basic ruamel example (found at https://yaml.readthedocs.io/en/latest/example/)

$ cat ruyaml_test.py

import sys
from ruamel.yaml import YAML

inp = """\
# example
name:
  # details
  family: Smith   # very common
  given: Alice    # one of the siblings
"""

yaml = YAML()
code = yaml.load(inp)
code['name']['given'] = 'Bob'

yaml.dump(code, sys.stdout)

running this example should provide the expected output:

$ /var/packages/borgbackup/target/env/bin/python ./ruyaml_test.py
# example
name:
  # details
  family: Smith   # very common
  given: Bob      # one of the siblings

If this doesn't work on DS412play either, the ruamel.yaml wheel in the package is corrupted.