loadable libraries and smart-amp-test as an example #8180

lyakh · 2023-09-08T15:50:28Z

This implements support for loadable libraries in SOF and adds the necessary interface to the smart-amp-test to be built and used as such a module. Still WiP

lgirdwood

Looks like a lot of patches here are ready to be sent as seperate PRs.
Additionally, we need to use the same module build and linking methodolody as Linux i.e. .o file instead of shared libraries.

lgirdwood · 2023-09-11T14:00:50Z

lmdk/cmake/build.cmake

+		"${SOF_BASE}/zephyr/include"
+		"${SOF_BASE}/xtos/include"


@lgirdwood yes, needed for compiler_attributes.h, although might be possible to get rid of it - builds without it at least as a module too

lgirdwood · 2023-09-11T14:03:09Z

lmdk/pic/smart_amp_test/CMakeLists.txt

@@ -0,0 +1,23 @@
+cmake_minimum_required(VERSION 3.20)


This should all go in the module directory for smart amp, same for each module.

lgirdwood · 2023-09-11T14:05:06Z

lmdk/pic/smart_amp_test/smart_amp_test.toml

@@ -0,0 +1,80 @@
+version = [3, 0]


We should really run the pre-processor on the toml files like we do with linker scripts. @kv2019i fyi.

@lgirdwood @aiChaoSONG Hmm, given we always have one platform and one algo bit, not sure we need to go that far. But for sure this is one option for thesofproject/rimage#176 One benefit is that we'll have cpp on all platforms, so this would cover the portability part.

so upgrading to use cpp here would also be easy enough to do and also allow us to pick UUIDs and memory addreses and sizes from headers (instead of coding them more than once).

lgirdwood · 2023-09-11T14:05:30Z

lmdk/pic/smart_amp_test/smart_amp_test.toml

+[adsp]
+name = "mtl"
+image_size = "0x2C0000" # (22) bank * 128KB
+alias_mask = "0xE0000000"
+
+[[adsp.mem_zone]]
+type = "ROM"
+base = "0x1FF80000"
+size = "0x400"
+[[adsp.mem_zone]]
+type = "IMR"
+base = "0xA104A000"
+size = "0x2000"
+[[adsp.mem_zone]]
+type = "SRAM"
+base = "0xa00f0000"
+size = "0x100000"
+
+[[adsp.mem_alias]]
+type = "uncached"
+base = "0x40000000"
+[[adsp.mem_alias]]
+type = "cached"
+base = "0xA0000000"


We can the #include <platform.tml> for above

lgirdwood · 2023-09-11T14:06:15Z

lmdk/pic/smart_amp_test/smart_amp_test.toml

+
+[cse]
+partition_name = "ADSP"
+[[cse.entry]]
+name = "ADSP.man"
+offset = "0x5c"
+length = "0x464"
+[[cse.entry]]
+name = "ADSP.met"
+offset = "0x4c0"
+length = "0x70"
+[[cse.entry]]
+name = "ADSP"
+offset = "0x540"
+length = "0x0"  # calculated by rimage
+
+[css]
+
+[signed_pkg]
+name = "ADSP"
+partition_usage = "0x23"
+[[signed_pkg.module]]
+name = "ADSP.met"
+
+[adsp_file]
+[[adsp_file.comp]]
+base_offset = "0x2000"
+
+[fw_desc.header]
+name = "ADSPFW"
+load_offset = "0x40000"


We can the #include <vendor.tml>

lgirdwood · 2023-09-11T14:13:30Z

src/audio/module_adapter/library/native_system_agent.c

@@ -30,9 +30,6 @@ void *native_system_agent_start(uint32_t *sys_service,
 	native_sys_agent.log_handle = log_handle;

 	void *system_agent_p = &native_sys_agent;
-	uint32_t **sys_service_p = &sys_service;
-
-	*sys_service_p = (uint32_t *)(&native_sys_agent.system_service);


seperate fix ?

lgirdwood · 2023-09-11T14:13:50Z

src/audio/module_adapter/module/generic.c

@@ -222,8 +222,7 @@ int module_prepare(struct processing_module *mod,
 	 * as it has been applied during the procedure - it is safe to
 	 * free it.
 	 */
-	if (md->cfg.data)
-		rfree(md->cfg.data);
+	rfree(md->cfg.data);


seperate patch

lgirdwood · 2023-09-11T14:14:46Z

src/ipc/ipc4/handler.c

@@ -782,7 +782,7 @@ static int ipc4_process_glb_message(struct ipc4_message_request *ipc4)

 static int ipc4_init_module_instance(struct ipc4_message_request *ipc4)
 {
-	struct ipc4_module_init_instance module_init = {};
+	struct ipc4_module_init_instance module_init;


seperate fix.

lgirdwood · 2023-09-11T14:16:06Z

lmdk/cmake/build.cmake

+		"-Wl,-Ttext=0xa06ca000"
+		"-Wl,-Tdata=0xa06cb000"
+		"-Wl,--section-start=.rodata=0xa06cb100"


magic - are we copied here ?

there's a comment there - need a script to calculate these. The approach is rather simple: .text is fixed (per module? to make them unique?) the .data should then be page-aligned. Haven't been able to achieve that in a better way so far

lgirdwood · 2023-09-11T14:18:12Z

src/samples/audio/smart_amp_test_ipc4.c

@@ -39,6 +39,8 @@ struct smart_amp_data {
 	uint32_t out_channels;
 };

+static int keep_bss;


needs a comment.

lgirdwood · 2023-09-22T15:38:29Z

src/samples/audio/smart_amp_test_ipc4.c

@@ -65,7 +74,9 @@ static int smart_amp_init(struct processing_module *mod)

 	if (base_cfg->base_cfg_ext.nb_input_pins != SMART_AMP_NUM_IN_PINS ||
 	    base_cfg->base_cfg_ext.nb_output_pins != SMART_AMP_NUM_OUT_PINS) {
-		comp_err(dev, "smart_amp_init(): Invalid pin configuration");
+		comp_err(dev, "smart_amp_init(): Invalid pin configuration: %d %d",


I would keep changes to bare minimum needed in any module. Chnages to log messages should be another PR.

lgirdwood

It looks like we have some seperate fixes and patches in the PR and it seems like they could upstream now as separate PRs.
@pblaszko pls see if any of @lyakh makefile changes will help your PR to reuse headers.

lgirdwood · 2023-09-29T15:35:12Z

src/audio/module_adapter/module/modules.c

+	/*
+	 * Module unloading isn't implemented yet, it has to be done from a
+	 * dedicated IPC, not when stopping streams.
+	 */


Module unloading come from ref counting users based on running pipelines.

yes, forgot to update the comment, fixing, thanks

lgirdwood · 2023-09-29T15:56:29Z

src/audio/module_adapter/module/modules.c

-	/* At this point module resources are allocated and it is moved to L2 memory. */
-	module_entry_point = lib_manager_allocate_module(dev->drv, config, src_cfg);
-	if (module_entry_point == 0) {
-		comp_err(dev, "modules_init(), lib_manager_allocate_module() failed!");
-		return -EINVAL;
-	}
-	md->module_entry_point = module_entry_point;
-	comp_info(mod->dev, "modules_init() start");
-
-	uint32_t module_id = IPC4_MOD_ID(mod->dev->ipc_config.id);
-	uint32_t instance_id = IPC4_INST_ID(mod->dev->ipc_config.id);
-	uint32_t log_handle = (uint32_t) mod->dev->drv->tctx;
-	/* Connect loadable module interfaces with module adapter entity. */
-	/* Check if native Zephyr lib is loaded */
-	struct sof_man_fw_desc *desc;
-
-	desc = lib_manager_get_library_module_desc(module_id);
-	if (!desc) {
-		comp_err(dev, "modules_init(): Failed to load manifest");
-		return -ENOMEM;
-	}
-	struct sof_man_module *module_entry =
-		    (struct sof_man_module *)((char *)desc + SOF_MAN_MODULE_OFFSET(0));
+	if (!md->module_adapter) {
+		byte_array_t mod_cfg;

-	struct sof_module_api_build_info *mod_buildinfo =
-		(struct sof_module_api_build_info *)
-		(module_entry->segment[SOF_MAN_SEGMENT_TEXT].v_base_addr);
+		mod_cfg.data = (uint8_t *)md->cfg.init_data;
+		/* Intel modules expects DW size here */
+		mod_cfg.size = md->cfg.size >> 2;
+		md->private = mod;

-	void *mod_adp;
+		struct comp_ipc_config *config = &(mod->dev->ipc_config);

-	/* Check if module is FDK*/
-	if (mod_buildinfo->api_version_number.fields.major < SOF_MODULE_API_MAJOR_VERSION) {
-		mod_adp = system_agent_start(md->module_entry_point, module_id,
-					     instance_id, 0, log_handle, &mod_cfg);
-	} else {
-		/* If not start agent for sof loadable */
-		mod->is_native_sof = true;
-		mod_adp = native_system_agent_start(mod->sys_service, md->module_entry_point,
-						    module_id, instance_id, 0, log_handle,
-						    &mod_cfg);


Can you go into more detail what is being modified here.

lgirdwood

See its been updated to newer Zephyr. One other thing we need is to have the module specific infra in the module directory, I dont want a duplication of stuff in module/ and lmdk/

lyakh · 2023-12-05T16:58:12Z

Not all of it should be merged (e.g. no need for Zephyr PR and we need to decide whether we set the smart-amp sample as module hard), but it's now close to the final version, I think, I'd say, almost no hacks.

overruled

marc-hb · 2024-01-17T19:18:06Z

From zephyrproject-rtos/zephyr#67431 (comment)

That code was introduced in zephyrproject-rtos/zephyr@aee2d1a as the only example usage of tristate in the Zephyr code. There is no actual need to do that, but I did not want to remove it either, so I just moved that around.

This is typical: "later cleanup" = "never".

Also no clear reason for 17 commits in a single PR
https://docs.zephyrproject.org/latest/contribute/contributor_expectations.html#defining-smaller-prs

All while we all already have severe crashes: #8747

The opposite of "Continuous" integration.

lyakh · 2024-01-18T06:49:46Z

I'm good to go providing the IADK comment from @jxstelter is resolved alongside any CI issues are resolved. We can come back and align with Zephyr cmake once its merged.

@lgirdwood yes: #8180 (comment)

marc-hb · 2024-01-19T18:59:02Z

The review of the llext+CMake work in Zephyr is going extremely well, multiple approvals including from the maintainer:

cmake: add llext compilation module zephyrproject-rtos/zephyr#67431

@lyakh please download it and give it at least a try before merging this.

At the very least we don't want this PR #8180 to immediately introduce a (logical) conflict with this Zephyr PR 67431 about to be merged.

lyakh · 2024-01-22T10:01:33Z

The review of the llext+CMake work in Zephyr is going extremely well, multiple approvals including from the maintainer:
* [cmake: add llext compilation module zephyrproject-rtos/zephyr#67431](https://github.com/zephyrproject-rtos/zephyr/pull/67431)
@lyakh please download it and give it at least a try before merging this.

At the very least we don't want this PR #8180 to immediately introduce a (logical) conflict with this Zephyr PR 67431 about to be merged.

@marc-hb that PR is now merged, but it doesn't help us here a whole lot. It doesn't implement address calculations for us. So at best we can use the new add_llext_target() for the first stage - compiling sources and generating an intermediate binary, but then we anyway need to add a custom target to rewrite addresses. So I still think this can be done as a second step after this PR is merged

marc-hb · 2024-01-22T18:35:45Z

but then we anyway need to add a custom target to rewrite addresses.

Thanks for looking at it! Any idea how we can solve this in the longer term? (= never?). I remember you said it's SOF-specific but SOF is one of the very first llext users so it's still a pity. @pillo79 any idea, even vague?

One thing this PR seems to miss compared to the way Zephyr does it is CFLAGS management...

pillo79 · 2024-01-23T09:56:42Z

@marc-hb I talked about this briefly yesterday with @lyakh - I am preparing a tiny followup to that PR with a custom post processing step. That should help!

pjdobrowolski · 2024-01-24T11:43:00Z

app/prj.conf

+
+CONFIG_LLEXT=y
+CONFIG_LLEXT_STORAGE_WRITABLE=y
+CONFIG_MODULES=y


If we have 3 different approaches to loading modules, I wouldn't use such generic name. It cause misunderstanding.

@pjdobrowolski it isn't a name that we choose, it's hard-coded in kconfig tools. We want to use tristate in module Kconfig and set it to "m" to build as a module. And for that you need CONFIG_MODULES. And I'd propose to use tristate for any modules - whichever implementation they use, but that's up to respective authors / maintainers.

modules:
The Symbol instance for the modules symbol. Currently hardcoded to
MODULES, which is backwards compatible. Kconfiglib will warn if
'option modules' is set on some other symbol. Tell me if you need proper
'option modules' support.

Can't we add another specific symbol for LLEXT?

Tell me if you need proper 'option modules' support.

Please source quotations. This one comes from here:
https://github.com/ulfalizer/Kconfiglib/blame/061e71f7d78cb0/kconfiglib.py#L671

This comment is 7 years old. Zephyr modules are months old and still in the making. I guess this text should be updated.

The reason @teburd and @lyakh could implement Zephyr modules without making any kconfiglib.py change is because kconfiglib.py has stuck to bug-for-bug compatibility with the original Kconfig implementation in C. https://www.kernel.org/doc/html/latest/kbuild/kconfig-language.html
I suspect the kconfiglib.py test suite covered modules even when Zephyr didn't support them yet.

Diverging kconfiglib.py away from the original Linux kernel implementation could break the kconfiglib.py test suite and make them slightly incompatible with each other, require slightly different documentations, disrupt other Zephyr users besides SOF,...A potentially huge amount of hassle for a small naming gain.

BTW kconfiglib.py is also used outside Zephyr.

I agree calling one of SOF's "3 different approaches" with a generic "CONFIG_MODULES" sucks, but Zephyr seems to have only one approach so it has no reason to disambiguate. This is hopefully just an implementation detail that most users will not pay much attention to: Kconfig has a user interface that shows the help text.

Note we have "HOST" abbreviated to "HST" and register sets called DfPMCCH. This is not an excuse not to care about names but it put things in a bit of perspective :-)

In any case this sounds like a Zephyr topic, not an SOF topic?

Thanks for the context, @marc-hb

Tell me if you need proper 'option modules' support.

Please source quotations. This one comes from here: https://github.com/ulfalizer/Kconfiglib/blame/061e71f7d78cb0/kconfiglib.py#L671

This comment is 7 years old. Zephyr modules are months old and still in the making. I guess this text should be updated.

Just to clarify a bit - it isn't just about the text (documentation), it's in the kconfiglib implementation. And yes, your text below actually alludes to that too.

[snip]

I agree calling one of SOF's "3 different approaches" with a generic "CONFIG_MODULES" sucks, but Zephyr seems to have only one approach so it has no reason to disambiguate. This is hopefully just an implementation detail that most users will not pay much attention to: Kconfig has a user interface that shows the help text.

This isn't an option for one of the 3 different approaches. Other approaches are welcome and encouraged to use it too! IMHO it's a simple and natural way to switch a module between a built-in and modular build, so, any Kconfig options, that want this functionality, are free to use this! E.g. if we modularise our Maths libraries. As for which of the "three approaches" is used - I'd propose to use other means to make those decisions, if we ever have modules, that can be built with more than one option. E.g. indeed with this PR the smart-amp-test module would be buildable in three ways - built-in, as a module using system services or as an LLEXT. Currently The "y" Kconfig option selects whether it's built-in, selecting "m" build it as an LLEXT immediately during the base-firmware build, and to build it as a system-services module you run a separate cmake command, which ignores this Kconfig option completely, so you can build such a module regardless of this Kconfig value.

Thanks @lyakh . I didn't expect system-services modules to ignore the Kconfig of the base firmware (sounds quite risky... even autoconf.h?) but if they do then this entire discussion seems indeed irrelevant.

pjdobrowolski · 2024-01-24T11:50:10Z

scripts/llext_link_helper.py

+# We need to calculate ELF section addresses of an LLEXT module and use them to
+# run the linker. We could just use Python to calculate addresses and pass them
+# back to cmake to have it call the linker. However, there doesn't seem to be a
+# portable way to do that. Therefore we pass the linker path and all the command


Why can't we do it in linker scripts?

Add support for loadable modules, built for dynamic linking with Zephyr's LLEXT API. Signed-off-by: Guennadi Liakhovetski <[email protected]>

lib-manager is module-adapter specific, it doesn't need the component API and shouldn't use struct comp_driver in function arguments. Signed-off-by: Guennadi Liakhovetski <[email protected]>

Module adapter drivers can be loaded and unloaded, using the Zephyr loadable extension API. Add its context to struct struct module_data. Signed-off-by: Guennadi Liakhovetski <[email protected]>

Pointers that we store in a global array for each loaded library, aren't really firmware manifest descriptor pointers, they're pointers to storage buffers, where entire libraries are stored. A SOF_MAN_ELF_TEXT_OFFSET offset has to be added to that address to get to the manifest descriptor. Signed-off-by: Guennadi Liakhovetski <[email protected]>

Regroup code in modules_init() to move declarations and assignments closer to where they're needed. Signed-off-by: Guennadi Liakhovetski <[email protected]>

lib_manager_allocate_module() returns a pointer as an integer, use uintptr_t for it. Signed-off-by: Guennadi Liakhovetski <[email protected]>

The first part modules_new() is only needed when a new module is registered, the rest is needed every time a module is instantiated. Signed-off-by: Guennadi Liakhovetski <[email protected]>

While modules are in use, no need to unload and re-load them, re-initialising audio interfaces is enough. With llext modules this uses the LLEXT reference counting to identify when a module should be unloaded. Signed-off-by: Guennadi Liakhovetski <[email protected]>

Convert the smart-amp-test in its IPC4 version to a loadable LLEXT module. Use an overlay configuration to select between monolithic and modular builds. Signed-off-by: Guennadi Liakhovetski <[email protected]>

lyakh · 2024-03-06T15:07:47Z

@pjdobrowolski have we answered all your concerns? If so please consider removing your request for change

changes done.

lyakh · 2024-03-07T09:10:32Z

Now that we upgraded to a Zephyr version with CONFIG_MODULES support, this is mergeable (again). CI failures: https://sof-ci.01.org/sofpr/PR8180/build3168/devicetest/index.html?model=MTLP_RVP_NOCODEC&testcase=multiple-pause-resume-50 I think is known and unrelated (@kv2019i @marc-hb am I right?), https://sof-ci.01.org/sofpr/PR8180/build3169/devicetest/index.html is suspend-resume failures, style - one warning is a moved code line, another one is __used vs. __attribute__((used)) - this one could be addressed I think, Zephyr does have a __used macro, SOF should be able to use it too.

kv2019i

Some minor comments, but given this has been in the works for a long time and touches a lot of code, I'd lean to merge this as-is and address minor issues in follow-up PRs. @softwarecki I think you'll need to rebase yours after this, but probably easier for you than this big set, so if ok, let's go with this first.

kv2019i · 2024-03-07T10:19:39Z

src/samples/audio/smart_amp_llext/CMakeLists.txt

+	"$$(${SOF_BASE}scripts/calc_elf_addresses.py" -f lib${MODULE}.so -t "A06CA000)"
+	-shared -fPIC
+	-o lib${MODULE}_llext.so $<TARGET_OBJECTS:${MODULE}>
+	COMMAND ${CMAKE_STRIP} -R .xt.* -o lib${MODULE}_out.so lib${MODULE}_llext.so


@lyakh ? To me using same extension as upstream Zephyr seems like a good alignment

lgirdwood reviewed Sep 11, 2023

View reviewed changes

lyakh force-pushed the ldlib-wip branch from d555a77 to 94a3122 Compare September 20, 2023 16:07

lgirdwood reviewed Sep 22, 2023

View reviewed changes

lyakh force-pushed the ldlib-wip branch 2 times, most recently from 2526537 to 2527242 Compare September 29, 2023 12:37

lgirdwood reviewed Sep 29, 2023

View reviewed changes

lyakh force-pushed the ldlib-wip branch 4 times, most recently from c413d69 to 003e828 Compare October 4, 2023 15:26

lgirdwood reviewed Oct 5, 2023

View reviewed changes

lyakh mentioned this pull request Oct 24, 2023

sof_api: lmdk: Extraction of api headers for loadable modules #8365

Merged

lyakh force-pushed the ldlib-wip branch 2 times, most recently from 333d97d to 49ed952 Compare November 21, 2023 14:05

lyakh force-pushed the ldlib-wip branch 3 times, most recently from 021cc8d to 4efcce6 Compare December 5, 2023 16:55

lyakh changed the title ~~[WIP] loadable libraries and smart-amp-test as an example~~ [WIP][DNM] loadable libraries and smart-amp-test as an example Dec 5, 2023

lyakh marked this pull request as ready for review December 5, 2023 16:58

lyakh requested review from marc-hb, dabekjakub, iuliana-prodan, dbaluta, abonislawski, bardliao, marcinszkudlinski, pblaszko, singalsu and ranj063 as code owners December 5, 2023 16:58

lyakh mentioned this pull request Jan 18, 2024

1st part of llext #8180 #8759

Merged

pillo79 mentioned this pull request Jan 23, 2024

llext: additional CMake harmonizations zephyrproject-rtos/zephyr#67997

Merged

pjdobrowolski previously requested changes Jan 24, 2024

View reviewed changes

lyakh force-pushed the ldlib-wip branch from 4914c4f to b5df0e9 Compare January 26, 2024 16:20

lyakh added 9 commits March 6, 2024 11:42

lib-manager: add support for llext

1b2c9e3

Add support for loadable modules, built for dynamic linking with Zephyr's LLEXT API. Signed-off-by: Guennadi Liakhovetski <[email protected]>

lib-manager: migrate to module adapter context

a861ae4

lib-manager is module-adapter specific, it doesn't need the component API and shouldn't use struct comp_driver in function arguments. Signed-off-by: Guennadi Liakhovetski <[email protected]>

module-adapter: add Zephyr extension context

ec0038e

Module adapter drivers can be loaded and unloaded, using the Zephyr loadable extension API. Add its context to struct struct module_data. Signed-off-by: Guennadi Liakhovetski <[email protected]>

module: (cosmetic) regroup some declarations and assignments

387bd4f

Regroup code in modules_init() to move declarations and assignments closer to where they're needed. Signed-off-by: Guennadi Liakhovetski <[email protected]>

lib-manager: use an appropriate type for a pointer integer

a2fd06b

lib_manager_allocate_module() returns a pointer as an integer, use uintptr_t for it. Signed-off-by: Guennadi Liakhovetski <[email protected]>

modules: split modules_init() into two parts

19dee61

The first part modules_new() is only needed when a new module is registered, the rest is needed every time a module is instantiated. Signed-off-by: Guennadi Liakhovetski <[email protected]>

smart-amp-test: make a loadable module

c067427

Convert the smart-amp-test in its IPC4 version to a loadable LLEXT module. Use an overlay configuration to select between monolithic and modular builds. Signed-off-by: Guennadi Liakhovetski <[email protected]>

lyakh force-pushed the ldlib-wip branch from 4a67c4e to c067427 Compare March 6, 2024 14:54

lyakh requested a review from softwarecki March 7, 2024 09:11

kv2019i approved these changes Mar 7, 2024

View reviewed changes

lgirdwood approved these changes Mar 8, 2024

View reviewed changes

kv2019i merged commit 980b1c6 into thesofproject:main Mar 8, 2024
35 of 44 checks passed

lyakh deleted the ldlib-wip branch March 8, 2024 14:30

softwarecki mentioned this pull request Mar 11, 2024

module: Revert "loading module: fix native module interface register" #8923

Closed

loadable libraries and smart-amp-test as an example #8180

loadable libraries and smart-amp-test as an example #8180

Conversation

lyakh commented Sep 8, 2023

lgirdwood left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lyakh Sep 12, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lgirdwood left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lgirdwood left a comment

Choose a reason for hiding this comment

lyakh commented Dec 5, 2023

marc-hb commented Jan 17, 2024 • edited Loading

lyakh commented Jan 18, 2024

marc-hb commented Jan 19, 2024 • edited Loading

lyakh commented Jan 22, 2024

marc-hb commented Jan 22, 2024

pillo79 commented Jan 23, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lyakh Jan 29, 2024 • edited Loading

Choose a reason for hiding this comment

marc-hb Jan 30, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lyakh commented Mar 6, 2024

lyakh commented Mar 7, 2024

kv2019i left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lyakh Sep 12, 2023 •

edited

Loading

marc-hb commented Jan 17, 2024 •

edited

Loading

marc-hb commented Jan 19, 2024 •

edited

Loading

lyakh Jan 29, 2024 •

edited

Loading

marc-hb Jan 30, 2024 •

edited

Loading