Reproducing CVMFS issue from May 3rd, 2021

[mboisson]

Notes from the event.

== Description of the problem ==

1. A minor update of glibc was done. This is a process which we have done many times before, and which every sysadmin does without issue quite often when they run a yum update. This is also something which has been done on multiple occasions on whole clusters with NFS root.
2. The libc.so file displayed on some CVMFS clients was incorrect. Running sha256sum on libc-2.30.so multiple times in a row would return a different checksum every time.
3. We confirmed that the CVMFS cache itself had the right content. It was between the cache and the view that was presented to the client.
4. About 10% of the nodes were affected.

== What we did to fix it ==
Two options worked

1. Forcing to restart the CVMFS client, on the affected nodes: sudo cvmfs_config killall
2. Killing the job that kept libc-2.30.so opened.

== What do we do from now ==

1. We have gathered information to open a bug report with CVMFS.
2. We will try to reproduce the issue using our development repository.
3. We will consider other options for more critical-than-average libraries, such as libc-2.30.so. For example, introducing a symlink layer for those.

[mboisson]

Bug report has been filed here: https://sft.its.cern.ch/jira/browse/CVM-2001
Bug has been reproduced.

[mboisson]

Support for Direct I/O is in CVMFS 2.9. To be tested.
https://cvmfs.readthedocs.io/en/stable/cpt-releasenotes.html

[mboisson]

This is marked as fixed in the future release version 2.10.
https://sft.its.cern.ch/jira/browse/CVM-2001