Worker tests: use unique pointers when possible #4717

Job	Run time
ci (ubuntu-20.04, 18, Release)	1m 27s
ci (ubuntu-20.04, 18, Debug)	1m 32s
ci (ubuntu-22.04, 20, Release)	1m 24s
ci (ubuntu-22.04, 20, Debug)	1m 26s
ci (ubuntu-24.04, 20, -Db_sanitize=address, Release)	1s
ci (ubuntu-24.04, 20, -Db_sanitize=address, Debug)	1m 16s
ci (ubuntu-24.04, 20, -Db_sanitize=undefined, Release)	1m 26s
ci (ubuntu-24.04, 20, -Db_sanitize=undefined, Debug)	1m 26s
ci (ubuntu-24.04, 20, -Db_sanitize=thread, Release)	1s
ci (ubuntu-24.04, 20, -Db_sanitize=thread, Debug)	1s
ci (macos-12, 18, Release)	1m 34s
ci (macos-12, 18, Debug)	1m 28s
ci (macos-14, 20, Release)	1m 24s
ci (macos-14, 20, Debug)	1m 25s
ci (windows-2022, 20, Release)	1m 9s
ci (windows-2022, 20, Debug)	1m 17s
	18m 17s

Provide feedback