Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add option to detect and honor backpressure #3

Open
flonix8 opened this issue Mar 26, 2024 · 0 comments
Open

Add option to detect and honor backpressure #3

flonix8 opened this issue Mar 26, 2024 · 0 comments

Comments

@flonix8
Copy link
Contributor

flonix8 commented Mar 26, 2024

Currently, there is no way to communicate backpressure through the pipeline (i.e. the downstream component cannot keep up, so the upstream component waits with its processing).
For perfect reproducibility this feature is needed, however, making the pipeline fully deterministic (provided that all the components are deterministic as well).
It might also be handy at some point to have backpressure between certain compoents.

An ideal place to implement this would be the vision-lib, s.t. we can easily configure it through the stage applications. The backpressure itself could be exerted on the stage application by blocking the publish() call as long as the stream is "full".

Some rough implementation ideas:

  • Do not set the trimming MAXLEN parameter in xadd() in the sender, but rather have the receiver either xdel() or xtrim() the stream after having processed an entry and have the sender track the stream length, and block if the stream length is greater than a defined stream length limit
  • A simpler version could forego streams completely and use a single key with set() for data exchange
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant