feat: add filesystem scrubbing controller

[dsseng]

TODO

• tests for scrub controller (needs ideas how do we avoid waiting a lot)
• report status for monitoring of scheduled scrubs, most recent scrubs date, duration and result.

Fixes #9545

[dsseng]

Docs must mention this feature is experimental and perhaps link to kernel docs about this

[dsseng]

I guess we need to somehow protect from user creating multiple documents referencing the same mountpoint.

[smira]

I wonder if we should have a way to cancel running scrub process?

[dsseng]

We can add a channel to signal this once document is deleted. However I haven't studied whether or not it's okay to abort the process and does it terminate safely

[smira]

as xfs_scrub is experimental, I'd say let's do it the first thing we merge in 1.10

[dsseng]

as xfs_scrub is experimental, I'd say let's do it the first thing we merge in 1.10

Didn't we consider it in 1.9 planning? Or it wasn't expected to use experimental kernel feature?

[dsseng]

Idea: run scrub in a goroutine (still single-threaded to not run two scrub tasks in parallel) and report when it's started so we can see it's running right now from the status. Current status example (and there's no way to tell whether one for /var is running or not yet, as status is updated on completion only):

[dsseng]

Other things to consider:

• add an option to scrub on boot
• check whether or not scrub may be aborted. If not ensure it's not and we must delay reboot/poweroff to ensure this

[smira]

I wonder if we could refactor this to first list all capabilities (via libcap), and remove those we want to drop, so that if we new capabilities are introduced, they are dropped as well. (probably taking it out of machinery back to the controller)

[dsseng]

Yes, I also want to do such a thing on the level of our interface to libcap. Actually systemd does manage such a thing, since in xfs scrubbing service there is a list of capabilities to give, not take

[smira]

I wonder if we should move this under block. (both resources and config)

[smira]

I almost wonder if we should split up this controller into two controllers:

• one which reads FSScrubConfig and outputs FSScrubSchedule resources (builds a schedule, handles jitter, updates schedule, etc.) - this controller can be fully unit-tested
• another which reads FSScrubSchedule, and based on the schedule runs the actual xfs_scrub tasks (or cancels them)

We can test time-based stuff using fake clocks, example

talos/internal/app/machined/pkg/controllers/runtime/cri_image_gc_test.go

Lines 33 to 52 in f756043

	func TestCRIImageGC(t *testing.T) {
	mockImageService := &mockImageService{}
	fakeClock := clock.NewMock()
	suite.Run(t, &CRIImageGCSuite{
	mockImageService: mockImageService,
	fakeClock: fakeClock,
	DefaultSuite: ctest.DefaultSuite{
	AfterSetup: func(suite *ctest.DefaultSuite) {
	suite.Require().NoError(suite.Runtime().RegisterController(&runtimectrl.CRIImageGCController{
	ImageServiceProvider: func() (runtimectrl.ImageServiceProvider, error) {
	return mockImageService, nil
	},
	Clock: fakeClock,
	}))
	},
	},
	})
	}

[dsseng]

Yes, sane. We already have the config controller which populates the structures looking the same as the yaml source, why not handle scheduling there, yes

[smira]

No, no... config controller is good, don't touch it ;)

Schedule controller will be fully testable without mocks.

Scrub controller will need a clock mock and an actual scrub mock, and it can be fully tested once again by manipulating schedules.