question about Tape and Autoloader

[becchett]

Dear all,
my Almalinux server has got one HPE Ultrium LTO8 and Autolader MSL 1/8 G2.
It uses these devices correctly through two special files /dev/sch0 and /dev/nst0.
I did some tests with TAR and MT.

The server has bacula 15.0.2 and bacularis 3.2.1, installed with RPM.

The first thing I wanted to do was to setup the tape drive via the web interface but I have a problem.

After creating the Tape ,and then the autoloader, the web interface does not display either of the two devices. The bacula-sd.conf is modified correctly.

Any ideas ?

Has anyone had this problem?

What is the correct sequence to add and display a tape in bacularis ?

Thank you
Best Regards
Enrico

[ganiuszka]

Hello @becchett,

The process of adding the autoloader to be displayed and managed in the web interface consists of two steps.

1. Adding the devices to Bacula configuration
2. Enabling advanced autochanger management in Bacularis

The 1) can be done on the Bacularis web interface by adding the appropriate Bacula configuration resources to the Bacula Storage and Bacula Director components. They are:

On the Bacula Storage Daemon side they are:

• Device resource for each tape drive
• Autochanger resource for the changer device

On the Director side:

• Storage resource with connection parameters to this Autochanger

The Device resources you can create in Bacularis in the path:

[Main menu] => [Page: Storage] => [Table row with the Storage resource] => [Button: Details] (it is the three bars button) => [Tab: Configure storage daemon] => [SubTab: Device] => [Button: Add Device]

The Autochanger resources you can create in Bacularis in the path:

[Main menu] => [Page: Storage] => [Table row with the Storage resource] => [Button: Details] (it is the three bars button) => [Tab: Configure storage daemon] => [SubTab: Autochanger] => [Button: Add Autochanger]

Here you can see examples how sample configs could look like:

https://www.bacula.org/15.0.x-manuals/en/main/Autochanger_Resource.html

Once you configure Device and Autochanger, you need to restart the Storage Daemon and after you can add on the Director side the Storage resource:

[Main menu] => [Page: Director] => [Tab: Configure director] => [SubTab: Storage] => [Button: Add storage]

Here you can read details about the Storage resource directives:

https://www.bacula.org/15.0.x-manuals/en/main/Configuring_Director.html#SECTION00231400000000000000000

For the device special files, the /dev/nst0 is correct, for the autochanger file, I would propose to use SCSI generic names like /dev/sg0. You can see the name in lsscsi -g command. Also is possible to define udev rules to make the constant symlink to the device.

At this stage, the Devices are ready to use with Bacula and Bacularis. It enables doing many tasks with the autochanger like: label tapes, update slots, mount, umount, release...etc.

In the future we are going to automate this process in Bacularis by adding the tape autochanger wizard.

Once you will try your Device with Bacula, after you may be interesting adding the mentioned advanced autochanger management ( 2) ) in Bacularis. You can read about it and see it here:

https://bacularis.app/doc/brief/configuration.html#autochanger-management

Good luck with configuring this autoloader :-)

Best regards,
Marcin Haba (gani)

[becchett]

dear @ganiuszka
I'm following your instruction. Can you tell me if inside Directory Section I need to add only Autochanger ?
After that Can I add both autochanger and drive inside Bacularis API Panel ?
Thanks a lot
Best Regards
Enrico

[ganiuszka]

Hello @becchett

Yes, it is exactly as you said. On the Director side in the Storage resource we add only the autochanger device.

For the second question, it is correct as well. Once you have the autoloader working with Bacula, then you can add both autochanger and drive to Bacularis API panel.

Best regards,
Marcin Haba (gani)

[becchett]

dear @ganiuszka
I make all steps: Storage, Director and API Panel , after that I restart all services bacula-sd bacula-dir bacula-fd and httpd.

Now inside Storage I see only autochanger :

in bacula-sd.conf there is LTO-8 drive section and with bconsole LTO-8 correctly appeare.

Inside Storage section I see "manage autochanger" but this feature is currently unavailable:

I've already settings API with the same device names

maybe I'm make some other errors

Best Regards
Enrico

[ganiuszka]

Hello @becchett,

Well done. You have almost everything :-)

On the Storage list page the autochanger is fine. Devices are not listed on this page and it is correct.

The Manage autochanger tab is displayed incorrectly. It can be caused that you used /dev/sch0 device in the Bacularis API panel. The mtx-changer script provided by Bacula and Bacularis internally uses the mtx tool to manage autochangers. The mtx tool supports the SCSI generic device names. You need to check in lsscsi -g command what is the SCSI generic device name for your autochanger device (ex. /dev/sg4 or similar) and use this device instead of /dev/sch0 in the Bacularis API panel. You will need to edit the Autochanger in the API panel and change it to the SCSI generic device.

For the rest settings, it looks OK.

Best regards,
Marcin Haba (gani)

[ganiuszka]

Hello @becchett,

One more thing. If you used /dev/sch0 in the Storage Daemon config (in the Autochanger resource in directive ChangerDevice), it needs to be updated to the generic device name as well.

Best regards,
Marcin Haba (gani)

[becchett]

Hi @ganiuszka , Hi all

I'm trying to complete this job because I need to start using a new bacula backup system with
Autoloader and a LTO-8 as soon as possibile.

I've reconfigured everything twice without success !

My scsi devices are seen correctly by linux standard tools:

`[root@infn-vm05 enrico]# lsscsi -g
[0:0:0:0] storage HP P420i 8.00 - /dev/sg0

[0:1:0:0] disk HP LOGICAL VOLUME 8.00 /dev/sda /dev/sg1

[1:0:0:0] disk HP P2000 G3 FC T252 /dev/sdb /dev/sg2

[1:0:1:0] tape HPE Ultrium 8-SCSI N4Q1 /dev/st0 /dev/sg3

[1:0:1:1] mediumx HP 1x8 G2 AUTOLDR 6.00 /dev/sch0 /dev/sg4

...omissis...
`

they run on Autochanger and LTO, for example:

`[root@infn-vm05 enrico]# mtx -f /dev/sg4 status

Storage Changer /dev/sg4:1 Drives, 8 Slots ( 0 Import/Export )

Data Transfer Element 0:Full (Storage Element 7 Loaded):VolumeTag = ADB000L8

  Storage Element 1:Empty
  Storage Element 2:Empty
  Storage Element 3:Empty
  Storage Element 4:Empty
  Storage Element 5:Empty
  Storage Element 6:Empty
  Storage Element 7:Empty
  Storage Element 8:Full :VolumeTag=CLN980L1  `

and

`[root@infn-vm05 enrico]# tar -tvf /dev/st0

-rw-rw---- bacula/apache 1317 2024-09-05 16:12 opt/bacula/etc/bacula-sd.conf`

"tar" doesn't work with /dev/sg3 but it doesn't seem to be a problem.

After I reconfigure all I'm unable to manage autochanger with bacularis:

If you have more time for this case it will be much appreciated !

The following screeshots show configuration of: Storage , Director and Bacularis API:

Is there any error ?
Thank you so much
Best Regards
Enrico

[ganiuszka]

Hello @becchett

Thanks for your detailed report.

I think that you are close to enable support for autochanger in Bacularis. I noticed only one place where is something wrong. I am attaching screenshot what is to change.

Rest looks to be correct.

After changing the /dev/sg4 into MSL in the Device directive in Director Storage resource, please check the Manage autochanger tab once again.

Please let us know how it went. Good luck.

Best regards,
Marcin Haba (gani)

[ganiuszka]

Hello @becchett

One more important thing.

Could you tell me if the address that you use in the current API host is the same as storage address:

infn-vm05.management

If not, it will not work and it will behave as on your screenshots. To check it, please go to the:

[Main menu] => [Page: Security] => [Tab: API hosts] => [Button: 'Edit' your API host]

and in the window, please check if in field:

IP Address/Hostname:

there is:

infn-vm05.management

If not, you will need to change it to infn-vm05.management. This way Bacularis web will know that the Autochanger belongs to this host and will be able to send API requests to infn-vm05.management.

Best regards,
Marcin Haba (gani)

[becchett]

dear @ganiuszka
I didn't read very well the documentation !

Now I've changed the "device" name and I see this :

Bacularis can manage Autochanger !

After that I check the hostname inside API Host and I see this value:

Do I leave it or it must be changed to infn-vm05.management ?

I've got just one server for everything: Storage Directory Web.

I should now be almost ready to perform my first tape backup

Thanks a lot again !
Best Regards
Enrico

[ganiuszka]

Hello @becchett,

Great to hear that it started working. Well done, Enrico 😄

If you have the only one Bacularis instance, then this change with infn-vm05.management is not needed at all. On your previous screenshots I couldn't see if you use one Bacularis API or more, so I suggested to check the address in the API host. But as I said, in case one Bacularis and autochanger connected to the same host everything is fine and you don't need to change it.

Best regards,
Marcin Haba (gani)

[becchett]

dear @ganiuszka

I chose to put everything in a single server to have more performance between directory and storage. I think that in my configuration when the server bacula saves a local directory to tape the data flow remains internal and does not pass through the network or the tcp/ip stack.

To make the first backup I have to configure a fileset, a job , the spool, and the first tape.

What do you recommend for the tape label?

As you saw I have an autoloader with a barcode reader, so I have to add it?

Which type is best between: ANSI, IBM or Bacula?

I don't want to use the spooling area for this backup because Bacula FD SD and DIRector are on the same server , I hope this doesn't cause any problems.

Bye
Enrico

[ganiuszka]

Hello @becchett,

Sure, having everything on the same host has advantages. I don't see any problem with it too.

For the Label Type, I would propose to not set this directive at all. It is specific directive that I have never heard that anybody needed to change it. If not set, the default value is 'Bacula' and it work well with tapes managed by Bacula.

For labeling tapes, it is possible to do in two places:

[Main menu] => [Page: Volumes] => [Button Label volume(s)]

or

[Main menu] => [Page: Storage] => [Button: Details for selected storage] => [Tab: Manage autochanger] => [Select tapes to label in table] => [Actions: Label using barcodes]

Before labeling there is needed to prepare tapes. You will need to manually load each tape to tape drive and then run:

mt -f /dev/nst0 rewind
mt -f /dev/nst0 weof

If your tapes are new, probably you will not need to do it at all. The key here is to write the EOF mark at the beginning of the tape.

Please make sure that this drive at the moment of preparing tapes is unmounted in Bacula Storage daemon. Othewrise you will see a message in mt that the device is busy.

After having all tapes prepares, you can choose label with barcodes option in the label volumes window.

Best regards,
Marcin Haba (gani)

[becchett]

dear @ganiuszka
I would like to ask you for a little more help regarding Bacula's performance in writing to LTO-8.
After completing the configuration, I ran a job that saves the file /ovirt/virtual-machine.qcow.
This file is about 100GB and is written to tape at an average speed of 130MB/s.
If I write the same file to the LTO-8 using tar, I get a speed of about 260MB/s.
In my configuration, both the LTO drive and the storage are Fibre Channel.
Below are some details about the configuration:

Job { Name = "backup-di-ovirt" Description = "" Type = "Backup" Level = "Full" Messages = "Standard" Storage = "MSL" Pool = "LTO-8" Client = "infn-vm05-fd" Fileset = "ovirt-100GB" }

Fileset { Name = "ovirt-100GB" Description = "test" Include { File = "/ovirt/macchina-virtuale.qcow" }

Device { Name = "LTO-8" Description = "HPE Ultrium LTO-8" MediaType = "LTO-8" DeviceType = "Tape" ArchiveDevice = "/dev/nst0" AutomaticMount = yes LabelMedia = yes Autochanger = yes AlertCommand = "/opt/bacula/scripts/tapealert %l" MaximumNetworkBufferSize = 131072 MinimumBlockSize = 0 MaximumBlockSize = 4194304 MaximumFileSize = 1000000000 MaximumConcurrentJobs = 1 SpoolDirectory = "/opt/bacua/spool" MaximumSpoolSize = 10737418240 }

Autochanger { Name = "MSL" Description = "HPE Autochanger MSL 1/8 G2" Device = "LTO-8" ChangerDevice = "/dev/sg4" ChangerCommand = "/opt/bacula/scripts/mtx-changer %c %o %S %a %d" }

Log from Bacula:

Start time: 17-Sep-2024 12:40:44
End time: 17-Sep-2024 12:54:21
Elapsed time: 13 mins 37 secs
Priority: 10
FD Files Written: 1
SD Files Written: 1
FD Bytes Written: 107,374,182,400 (107.3 GB)
SD Bytes Written: 107,374,182,502 (107.3 GB)
Rate: 131424.9 KB/s
Software Compression: None
Comm Line Compression: None
Snapshot/VSS: no
Encryption: no
Accurate: no
Volume name(s): ADB000L8
Volume Session Id: 9

Details from Tar run:

[root@infn-vm05 ovirt]# time tar -b 512 -cvf /dev/nst0 macchina-virtuale.qcow
macchina-virtuale.qcow

real 5m44.185s
user 0m1.666s
sys 1m27.133s

What can I do to improve the writing speed?

Thanks
Bye
Enrico

[ganiuszka]

Hello @becchett

Since a couple of years I have not been using physical tape devices, but I will try to advise something.

I remember that you mentioned, you don't want to use data spooling because your Bacula components are on the same host. Despite this, I would propose to enable it (Spool Data = yes) and also try to increase the MaximumSpoolSize or remove this directive (to enable no limit). This spooling is not only for performance and continues writing, but also for keeping better drives "condition" by reducing stoping, rewinding, moving tape... etc. that could also cause tape "shoe-shine" effect. For the last one I don't know if tapes are still so sensitive on that, but less moves back/forth is always better, anyway.

For MinimumBlockSize and MaximumBlockSize, if your device works in variable block size mode, I would advise to remove these directives. They are mainly for fixed block size tape drives to adjust the Bacula block size to device block size, but as long as it is variable in the drive, they are not needed.

Good luck with improving the performance. Please let us know what you were able to achieve in this matter.

Best regards,
Marcin Haba (gani)

[becchett]

dear @ganiuszka
I've made some tests and currently the best performances, 130MBytes/s, have been achieved with this configuration:

I've never use MinimumBlockSize. Does the difference between Tar and Bacula depend from hardware compression or not ?
How Can I check if Bacula use compression or make "shoe-shine" effect ?

Best Regards
Enrico

[ganiuszka]

Hello @becchett,

OK, thanks for sharing with us your best settings.

In Bacula the software compression is possible to configure in FileSet, but I didn't see any compression set in your FileSet.

For "shoe-shine" it is not something to check directly. It is just effect when tape goes many times back and forth through the the tape drive head and becomes worn. Without continues write with data spooling, the drive can stop/start and do positioning much often.

Best regards,
Marcin Haba (gani)

[ganiuszka]

Aaa, you wrote about hardware compression. This compression for Bacula is transparent, I think.