Feature: wasm processor

[gedw99]

Feature description

There are currently built in and js processors.

Wazero could be used as an engine for a WASM processor. It has no cgo and supports various higher level languages. https://github.com/tetratelabs/wazero

https://wazero.io/languages/

there is even a golang based wasm profiler for Wazero now. It was only opened in May 2023. It’s also golang. https://github.com/stealthrocket/wzprof

https://github.com/ConduitIO/conduit/tree/main/pkg/processor Is where this all could be done of course.

Technical due diligence:

The Wazero wasm Engine is mature , easy to introduce into this golang database and has a responsive team behind it. So it’s not a major headache to introduce and support. Technical debt should be reasonably low.

There are a lot of languages that compile to wasm . Wazero team are pretty good at supporting this. I need to check how many and maturity . Rust and golang ( via tinygo ) have been there . C#, Java, python etc are probably still ongoing ..

Provides sandboxing just like js but gives even more control of the access a guest wasm has to the outside world and well formularised based on the wasi / wasm specs. It’s only going to get better too.

Perf. Hard to say but should be faster than goja.

wasm is gaining momentum and so going away

Diverse design / compile time targets. Devs don’t need docker as Wazero supports windows and Mac.

Diverse runtime targets. Desktops, servers, iot, web. Typically a processor will run on a server but with Wazero there is nothing preventing deploying processors anywhere. Processing at the “Edge” would require the transport from conduit server to the Edge path to be secure and auth / authz to be handled. Don’t this thos aspect is even part of Conduit yet. But it’s a rather nice thing. Processing as a Service.

enterprises may prefer wasm as a sandbox compared to js. Hard to say though.

Use case due diligence :

reuse. There are devs writing logic all over the place in various languages . They could have quite a lot of specialised logic in their code. With wasm they can reuse all that logic and just write a wrapper that confirms o the Conduit Processor spec and they have good reuse. I think this is the most compelling functional reason.

They might also want their custom logic to call out to their own micro services. Wasm and its spec based sandboxing can provide that robustly . For sone users this will be critical as thee is not enough knowledge at runtime in the code and they must perform secdobsry check in third party services in order to do their processing . The limitation of what transports can be used ( http, GRPC , etc ) to make a call out from inside a processor ) can be managed by the wasm host . Auth also.

[lovromazgon]

Thanks for opening the issue and posting a thorough explanation of why it's the way to go. We have actually already considered this (see #21) and even did a spike to explore how feasible it would be to add this. We are currently undecided if we should add custom processors using WASM or use the same plugin architecture we use for connectors using gRPC and go-plugin. But we are aware we need to choose something because the current solution with JS processors is not ideal.

There is a plan to revisit processors as a whole and determine how we can make them "better". We will take your issue into account there!

[gedw99]

hey @lovromazgon

thanks for reply..

WASM supports the hashicorp go-plugin based way to hook up. Its a nice way to keep things decoupled.

here it is: https://github.com/knqyf263/go-plugin

examples: https://github.com/knqyf263/go-plugin/network/dependents

a nice simple example using it with Wazero:
https://github.com/Taction/dapr-wasm/blob/master/go.mod

[gedw99]

wazero is non cgo and runs pretty much anywhere. I use it with NATS. I noticed you have NATS streamer in conduit.

anyway if want to discuss, happy to do so.... see my profile:https://github.com/gedw99

[simonl2002]

considering this blocked by "better processors work" #997

[gedw99]

I ended up using this : https://github.com/stealthrocket/timecraft.

mots a was runner and pipelibe system.

So far it’s been great. So if you ever move to wasm check it out..

[lovromazgon]

This is included in v0.9.0, check out the processor SDK on information about building a WASM processor.