2024/10 GoBPM(S) current state summary

[klajmajk]

Hi guys I would like to summarize my efforts in GoBPM. In the following repos I publish current state of it. Please consider it highly experimental I would say that its purpose mostly was for me to be able to imagine what should be the first steps we should take after we start the proper development phase.

Repositories

• https://github.com/pbinitiative/go-bpms-engine
• https://github.com/pbinitiative/go-bpms-showcase-fe

Pre-development phase

1. Setup some basic guidelines and organizations how we could cooperate on the development.
2. Decide whether changes that would be done to the engine itself would align with the lib-bpmn upstream repo. It totally makes sense to contribute back but I guess it all depends what Martin thinks whether he would be ok with the shift to more complex architecture versus the current approach aimed to in-memory only.
3. Do a benchmark where we could compare some hacked up GoBPM 3 nodes X 3 partitions cluster to similar Zeebe cluster. I would like to do this so that we can prove that the architecture and chosen libs/technologies work well in the clustered setup.

🧑‍💻 Development phase

Groups of requirements for development

1) Persistence

• Introduce interface for holding the state of the engine which enables to implement the persistence with both in-memory any DB of choice
• Handling DB transactions (I completely skipped this one for now with rqlite)

2) Structuring the BPMN evaluation and model so that it adheres to BPMN elements inheritance.

For example there is basically this structure in the activity elements.

Activity #here things like markers and boundary events need to be implmented
│
├── Task #here it depends whether all the tasks would be simply job workers then we could implement those here for all of maby bring some common base class like WorkerHandeledTask
│   ├── User Task
│   ├── Script Task
│   ├── Service Task
│   ├── Manual Task
│   ├── Business Rule Task
│   ├── Send Task
│   └── Receive Task
│
├── Sub-Process
│   ├── Embedded Sub-Process
│   ├── Event Sub-Process
│   └── Transaction
│
└── Call Activity
    ├── Call Task (Calls a Task)
    └── Call Sub-Process (Calls a Sub-Process)

If we structure this properly we should be able to avoid duplication of business logic in the handing code. I think the same principle should be applied both to the model structure itself and to the execution handling as well. I see this important so that we can improve the BPMN coverage in future.

3) Clustering

• we should design how the cluster should be build,
• how should it communicate and
• how should it scale

🎯 Development Priorities

The priorities as I see it right now (this is something we should discuss)

1. Simple non HA deployment
   1. Handling persistence
   2. Improving BPMN Coverage
2. Design the HA architecture and internal API

Please start following PBI on linked in as we are trying to start the community there ;-)
https://www.linkedin.com/company/process-builders-initiative

[richardbizik]

Pre-development phase:
2. I think that we will diverge a lot from the original implementation (and maybe even @nitram509 original vision for the engine) during developement but we can try to keep the bpmn eval engine separate.
3. This would not tell us much for the time being because we will be missing the whole raft protocol implementation and communication inside the cluster (which can slow things down significantly)
4. We should evaluate how hard it would be to extend rqlite raft protocol to our needs or if we will need to create a separate cluster over rqlite instances. (Root cluster will have all the gobpms nodes and child clusters would have rqlite instances as partitions)

Development phase:
I would start the development with clustering in mind. It will be way harder to wire everything in later stage and I think it would be a lot of refactoring.

lib-bpmn-engine is missing feel-expression parser and evaluation which is a big part of the dmn spec. Do we want to start work on that as well or keep the expr
DMN 1.5 spec chapter 9 and 10
Feel expressions are not part of the BPMN spec chapter 8.4.6 but I think a lot of users would be expecting the support.
Would be DMN support part of the gobpms as well?

[thomasek]

Good afternoon,
I believe we first need an alignment with @nitram509 about the future of his library. This decision must be taken together with him whether we contribtue upstream or start growing the fork on our side.
Regarding the priorities: I would prefer discussing this topic in some kind of (online) discussion, but I agree with the priorities from Adam (simple non-HA setup, broader BPMN support, HA setup)
Regarding DMN: I would personally move that feature incl FEEL support to a later phase. First of all, let's focus on the simple non-HA use case and broader support for BPMN elements.
Also, I suggest to include topics such as monitoring and maybe also some basic installation on the TODO list for the first stage - I have been approached by people asking me to try out the Go-BPM and we need to make it easy for them.

[nitram509]

Hi,
regarding the evolution of lib-bpmn-engine, I'm quite open. The only thing I would love to see is the engine can be used in-memory with minimal dependencies. After briefly looking at Adam's repo 'pbinitiative/go-bpms-engine' I could imagine leveraging the interface 'BpmnEnginePersistenceService' and building this into the core of the lib ... with an in-memory storage per Default.
On Adam's point (1), I fully agree, this makes sense and I do support that.

With regards to collaboration and evolution, in terms of enabling a larger team here or future developers, I would be open transfering the lib-bpmn-engine e.g. towards pbinitiative/lib-bpmn-engine (naming up for discussion) and then making my repo an 'archive'.
Which then makes me say: In return, I would like to get a seat in the "Process Builds Initiative Steering Board" ;)

That said, I do appreciate and recommend sharpening not only the roadmap, but also the goals of the initiative as well as refining the overall architecture for the components. E.g. the choice for rsqlite has quite some consequences, which might be right for some use cases, but not for others. Maybe it helps defining personas, so we can better structure documentation, for operators and/or developers.

On (2) structuring ... the BPMN element structs can easily be structured as proposed. In contrast to the business implementation, were I found that a different structure is more helpful, since all tasks are just the definition for what is at runtime called a job. And jobs share quite a lot among the different task types (service, human, etc.)

On (3) Clustering
Indeed the goals need to be defined. Remember the CAP theorem. Whatever we chose, we can't get all three ;)
Since we're not aiming for high performance/throughput (assumption), Consistency and Availability are the more likely choices.
That said: I lack details of rsqlite ... what's their documentation about scaling? == the first PoC obviosly follows that.

On (1) Persistence
One extra crazy idea I had these days ... which is using Delta Tables to abstract storage on S3. Presumingly, this would be less performant than rsqlite, but way easier to setup.

On (4) Roadmap

• I agree, getting the persistence done is key for the storage implementation. It's on my top prio list since quite a while.
• Implementing then some distributed storage + high availability would be a logical next step
  • likely this takes a while to get consistency right (hypothesis)
• after that, I propose to implement a rest server
• then I propose extending the event exporter to improve insights ... it's still rudimentary implemented and not all event types like known from Zeebe are present. Reasoning: in a distributed scenario it is more important to have insights into the processes, else people need to scrape console logs on multiple nodes.
• extending BPMN capabilities might happen in parallel (depends on a per element evaluation) - e.g. see this PR from a contributor looks relatively easy add feature - Inclusive Gateway element nitram509/lib-bpmn-engine#204

On (5) Feel vs. expr
This is a tricky one ... I did not find many FEEL implementations. Implementing this is quite some work, and I don't know anyone using it, besides People from Camunda migrating. Does this really bring some value to the initiative?
It would be really interesting to see adoption and/or expectation from users in a poll or so.
Theoretically, it is quite simple to plugin a go-based Lua or Javascript interpreter, for which plenty of tooling support is already out. FEEL is IMHO still somewhat exotic.

Quite a lengthy answer, I realize.
Looking forward to hear your feedback.

[nitram509]

Talking about architecture, my current understanding is the following...

Please comment, if this matches or not.

Source:
Process Builders Initiative – Architecture Proposal.odp

[klajmajk]

Hi Martin,

thank you for writing this up. I think that what you wrote aligns very well with how I see this. Few months ago I created this pitching deck https://docs.google.com/presentation/d/1qXdpptLRgTgGxDEAA7fd5vzps_59ku8U1DPutW_UR7I/edit?usp=sharing which sums up what the product should be (as I see it).

One of the part is a sketch of the architecture. (It is exactly how you discribed on your model)

We definitely need to go deeper but I hope we can build on this.

P. S. I think your seat in the steering board is pretty much guaranteed.