Is there a Python SDK?

[aberges-grd]

Webpage says there's a Python SDK + Client, but the documentation doesn't show the SDK, only the client.

If there is a python SDK to extend streampipes I would be greatly interested.

Thanks

[bossenti]

Hey @aberges-grd,

EDIT: sorry I didn't get your question right.

StreamPipes Python is not yet a full SDK for StreamPipes. However, it supports to create StreamPipes functions, which allow you to consume & send data streams from/to StreamPipes via Python. This behaves like a lightweight processing element.
Does this help you? How would you like to extend StreamPipes via Python?

If not, I would be happy to hear your requirements and ideas. We are at an early stage with the Python support and are aiming to extend it further.

[aberges-grd]

hey, thank you for your reply.

Well, I'm interested in creating new "boxes" for data processing and representing ML models that have been previously trained, plus monitoring.

The stack would need MLFlow (has Java API, though we are accustomed to the Python one), Spark (has Scala API so may be integrable in StreamPipes I guess), a library to compute custom model health metrics (it's Python entirely) and representing models (Dockerized) whose predictions may be used as the inputs of other models. We would also be interested in having ML training pipelines. Orchestration is also a need, but StreamPipes has that already.

Streampipes is interesting for us because it had a GUI, streaming capabilities and all the data sources and sinks already present, and the ability to extend and customize it.
Our expertise is with Python and .NET so the need for Java raised some doubts in the team for adoption. Our main concern is the integration of ML libraries that are Python-only.

[bossenti]

This sounds really interesting! This is exactly what we had in mind when we started working on Python support for StreamPipes: to use the StreamPipes as a platform and extend or support analytics & ML use cases.

If you are talking about boxes for processing data and representing ML models, StreamPipes functions might be the right tool for you. A function can consume event streams from StreamPipes, apply completely custom processing logic (written entirely in Python), and produce an output event stream (e.g., predictions from an ML model) that can be reused in StreamPipes [1].

In this context, you might want to take a look at one of our tutorials, which demonstrates how a very simple function could look like [2]
We also provide basic support for ML functions using the online ML library river [3]

Functions in StreamPipes Python are a recently added feature, but something we have on our near-term roadmap for improvement & expansion. So it would be great if you could give us some feedback if you try them out.

[aberges-grd]

Many thanks for your reply, I will have a look at these!