consolidated s390 device configuration #5250
Conversation
|
Hello @steffen-maier! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:
Comment last updated at 2024-10-10 15:27:41 UTC |
|
Hello @steffen-maier, you are targeting this change into Fedora-39, however, Fedora 39 is in final freeze so until we have a bug with approved exception we can't get it in. I would suggest you to switch branch to |
There was a problem hiding this comment.
Thank you! I'll defer to @poncovka, @rvykydal and @vojtechtrefny for more details. I am not sure about the high level matters, such as if this should be all somewhere in blivet or not.
Apart from the failing tests, here's what I found...
There was a problem hiding this comment.
Since chzdev and /lib/s390-tools/zdev-to-rd.znet are used, anaconda.spec should be updated so that anaconda-core requires the s390utils-core package (assuming the zdev-to-rd.znet helper script will be placed there) and also the minimal version of s390utils-core should be specified once it is known.
I have already prepared such change commit but defer pushing until I looked at the unit tests fails, so I don't trigger the git workflows too often and unnecessarily. |
Unfortunately, I'm not sure why that is. I fail to see how my code change introduced that. The only preceding non-import statement I can see is: But that was already there before my changeset. |
Except for the currently inevitable "FIXME", I hope this is what you were looking for: 48f6232 |
Yes, that is still used. However, for the long term we are trying to move towards having the dependencies in anaconda metapackages By the way, I can't find any package with these tools? |
You mean the helper scripts
As Jan stated earlier, they will end up in a future release of |
|
Actually, I can't find even the package |
Good point. I pointed to the upstream repo, which does not have any packaging definitions. Downstream packaging is by means of https://src.fedoraproject.org/rpms/s390utils/blob/rawhide/f/s390utils.spec#_206 already creating s390utils-core. Maybe I should add spec file changes for my upstream changes to my steffen-maier/s390utils#1, which currently only exists of pure downstream contributions. |
|
How can I easily (without containers or other non-trivial setup) run the unit tests locally? Pointers to doc might be sufficient. |
it is a s390x specific package, not available anywhere else (x86_64, ...) |
|
Some of the tests here run only after they're manually confirmed. For tests, tl;dr is:
That said - https://anaconda-installer.readthedocs.io/en/latest/testing.html |
|
As for storaged-project/blivet#1162 (comment), the docs have a section about patching the container used for unit tests, which might help. |
FYI: I had to manually load the ext4 kernel module on my local container host |
|
I'm still working on the unit tests, got the running locally but then got distracted. Maybe I will need help with the unit tests. About the current re-run of the git workflow from https://github.com/rhinstaller/anaconda/actions/runs/6577747989/job/18665300996?pr=5250: The following ones are new and I don't see how my unchanged code could have cause these. Maybe there is a new blivet version being pulled in and the anaconda unit tests were not yet updated for those blivet changes unrelted to this PR?:
|
steffen-maier
had a problem deploying
to
gh-cockpituous
GitHub Actions
Failure
|
This PR is stale because it has been open 60 days with no activity. |
|
@steffen-maier is this still a draft? The dracut changes have been merged in dracut-ng/dracut-ng#73 and the changes are included in dracut-102-3.fc41, which is present in Fedora-41 composes. Blivet changes are tracked in storaged-project/blivet#1259 (the PR is also marked as Draft). On RHEL-10 we merged the anaconda part first and then the blivet part and we have not noticed any problem, so I believe this PR is not blocked. Since this feature has already been merged into RHEL-10 (#5676), removing the "port to RHEL10" label, as well as the "stale" and "blocked" labels, since I believe all prerequisites have been satisfied. |
|
Hi Jan, thanks for your updates. I also had the impression that dependencies are meanwhile fulfilled and I need to update this PR (such as dracut version in code [thanks for looking that up!] and remove the draft state), but I currently don't get to it. |
steffen-maier
had a problem deploying
to
gh-cockpituous
GitHub Actions
Failure
I updated the code to depend on dracut 102-3.
(https://koji.fedoraproject.org/koji/buildinfo?buildID=2490409) I also removed the draft state so this can get merged. |
|
/kickstart-test --testtype smoke |
|
Validation tests failure is expected, depends on the related blivet update. |
|
Quickly retested the changes using an updates.img provided by Radek. Tested against rawhide on a KVM and z/VM system, no issue found. |
|
@steffen-maier it seems that we need to rebase the PR to current |
…802482,#1937049)
Implements the dasd part of referenced bugs.
This allows to delegate s390-specifics to zdev from s390-tools and to
subsequently remove s390-specific code here.
Additionally, the change supports to subsequently solely rely on the zdev
persistent device configuration "database". Create entries on activating
devices. Simply copy the persistent device configuration to sysroot finally
without having to deal with device properties.
The spec file update reflects the new dependency on `chzdev` from the
s390 architecture specific sub-package s390utils-core. Actually, this
commit here only depends on `chzdev` in older versions already packaged
and shipped, so no version comparison necessary here.
Regarding unit test:
As blockdev was already mocked it's not clear how the test would mock
that sanitize_dev_input would actually canonicalize a DASD device bus-ID.
Also, the argument to execWithRedirect() is much more involved than just
the device bus-ID previously. It does not make sense for the unit test
to hard code the full argument of how execWithRedirect() is currently
invoked in the code.
So just check that the new code calls execWithRedirect() exactly once and
ignore the arguments.
It would be a separate fix patch to improve a fake sanitize unit test. I
suppose it would need to start with a DASD device bus-ID that is not yet
canonical/sanitized, e.g. something along the lines of:
DASDDiscoverTask("A100").run()
blockdev.s390.sanitize_dev_input.return_value = "0.0.A100"
blockdev.s390.sanitize_dev_input.assert_called_once_with("A100")
sanitized_input = blockdev.s390.sanitize_dev_input.return_value
execWithRedirect.assert_called_once_with(...)
It's unclear how much value that would have, though.
Signed-off-by: Steffen Maier <[email protected]>
…ot (#1802482,#1937049) Implements a part of the referenced bugs. Depends on ibm-s390-linux/s390-tools@bc4f455 ("zdev/dracut: retain early persistent config over switch root"). The spec file update reflects this new dependency on the new v2.31.0 of the s390 architecture specific sub-package s390utils-core. A new s390-tools zdev dracut module hook retain-zdev.sh copies the zdev persistent configuration from the initrd into /run/zdev.initrd.config. Import that persistent device configuration. It can be used: * to transfer the configuration into the installed system. * by python-blivet ZFCPDiskDevice.dracut_setup_args(), Any s390 device configuration in the installer user interface produces persistent zdev config entries. Instead of treating each s390 device type differently, simply transfer the entire combined resulting zdev persistent config into the installed system. The import above also fixes the problem that installations, which got zfcp paths activated by means of rd.zfcp= dracut cmdline arguments, were missing those paths in the installed system. Since commit 87ab1ab ("Support cio_ignore functionality for zFCP devices (#533492)"), /etc/zfcp.conf replaced /tmp/fcpconfig. Since commit 011ea0a ("Remove linuxrc.s390"), /etc/zfcp.conf only exists if the user specified dracut cmdline parameter rd.zfcp=. https://github.com/ibm-s390-linux/s390-tools/tree/master/zdev/ handles parsing of rd.zfcp= without /etc/zfcp.conf as of commit 06a30ae529a5 ("zdev/dracut: add rd.zfcp cmdline option handling"). https://src.fedoraproject.org/rpms/s390utils.git no longer writes /etc/zfcp.conf during deprecated parsing of rd.zfcp= as of commit ("zfcp: migrate to consolidated persistent device config with zdev") Hence, nothing populates /etc/zfcp.conf during installer boot anymore. So python-blivet has no more initial import input to carry forward. Signed-off-by: Steffen Maier <[email protected]>
Implements the znet part of referenced bugs. Depends on ibm-s390-linux/s390-tools@73f51e4 ("zdev: add helper to convert from zdev config to rd.znet"). The spec file already reflects the new dependency on `zdev-to-rd.znet` in the new v2.31.0 of the s390 architecture specific sub-package s390utils-core. Dracut commit ("feat(znet): use zdev for consolidated device configuration"), replaces the distribution-specific persistent configuration of s390 (channel-attached) network devices with a common consolidated mechanism using chzdev from s390-tools. So there is no more ccw.conf nor s390-specific low-level network config in NetworkManager connections nor ifcfg files. The spec file update reflects this new dependency on the updated dracut module "znet" in a new version of the dracut-network sub-package. Therefore, drop NETTYPE and OPTIONS. Keep SUBCHANNELS nonetheless because it can still serve as a matching key for NM connections. Delegate the generation of rd.znet statements to a helper tool from s390-tools, which gets its low-level config information from the consolidated mechanism using chzdev. There are two different code paths involved: * Root-fs on something (such as iSCSI or NFS) that depends on znet =>_get_dracut_znet_argument_from_connection(). Related earlier commits: commit fa174ab ("Write rd_CCW when root fs is on a network device on s390x (#577193)") and lately replacing former code: commit f85682f ("network module: add support for getting dracut arguments") commit 7cf4d64 ("network module: use network module to get dracut arguments") and finally replacing initscripts ifcfg by NetworkManager connection: commit 840c984 ("network: generate dracut arguments from connections (#1751189)") (Note that this generated rd.znet is independent of the (last) one just inherited from the boot parameters between commit 64fb106 ("Preserve network args on s390x.") and commit a4ba9ae ("Do not pass rd.znet on to installed system unconditionally").) * Configure znet on boot with rd.znet= but without any corresponding ip= and instead use the kickstart command "network" to perform high-level configuration of the network interface created with rd.znet. This creates a non-initramfs NM connection. => pyanaconda.modules.network.initialization.ApplyKickstartTask.run => pyanaconda.modules.network.nm_client.add_connection_from_ksdata => pyanaconda.modules.network.nm_client.create_connections_from_ksdata => get_s390_settings() and _update_wired_connection_with_s390_settings() (In contrast, early initrd network setups get both the low-level s390 config and high-level interface config via nm-initrd-generator, which parses rd.znet= as well as ip=.) Signed-off-by: Steffen Maier <[email protected]>
storaged-project/blivet#1162 (comment) removes DASDDevice.opts. Anticipating that blivet change, update the anaconda unit tests making use of DASDDevice. Signed-off-by: Steffen Maier <[email protected]>
steffen-maier
had a problem deploying
to
gh-cockpituous
GitHub Actions
Failure
|
/kickstart-test --testtype smoke |
steffen-maier
had a problem deploying
to
gh-cockpituous
GitHub Actions
Failure
Consolidate the persistent and dynamic configuration of s390-specific devices in Linux distributions by delegating the configuration to the existing framework
zdevfrom s390-tools.This pull request completes consolidated s390 device configuration in
anaconda.This also fixes a newly discovered bug about missing persistent device configuration if device is enabled outside of anaconda, e.g. using rd.zfcp from dracut.
Some of the commits [see their descriptions] depend on certain commits from:
This PR works together with storaged-project/blivet#1162.
Zdev's job is to perform low-level configuration after which the user gets architecture-independent objects such as block devices, SCSI devices, or network interfaces. Those can and should in turn be configured with existing common code mechanisms. So there's a clear separated layering for configuration duties.
In particular, the s390-specific devices currently are: DASD (traditional disk), ZFCP (scsi), and ZNET representing channel-attached network (QETH incl. OSA and HiperSockets, LCS, CTC). Zdev has a stable command line user interface and abstracts from sysfs and from a persistent configuration representation. Zdev encapsulates configuration details. Systems management code can simply delegate configuration to zdev and thus reduce architecture-specific code.
This improves user experience, serviceability, maintainability, and reduces test effort.
@jstodola @poncovka @sharkcz
Even though this is a draft pull request, I would appreciate review comments.
It's only a draft until we sorted out the dependencies and merge order of pull requests for different related projects. Some thoughts in the integration process are in #5250 (comment).
I developed and tested this locally with updates.img based on code (anaconda+blivet) branched closely enough for the updates to work with RHEL9.1 products.img (current when I started development).
This is a forward port, which applied pretty much automatically to the new base.