Commits on Nov 27, 2024

1. feat(hydroflow_plus)!: mark non-deterministic operators as unsafe and… …

   … introduce timestamped streams
   
   Big PR.
   
   First big change is we introduce a `Timestamped` location. This is a bit of a hybrid between top-level locations and `Tick` locations. The idea is that you choose where timestamps are generated, and then have a guarantee that everything after that will be atomically computed (useful for making sure we add payloads to the log before ack-ing).
   
   The contract is that an operator or module that takes a `Timestamped` input must still be deterministic regardless of the stamps on messages (which are hidden unless you `tick_batch`). But unlike a top-level stream (which has the same constraints), you have the atomicity guarantee. Right now the guarantee is trivial since we have one global tick for everything. But in the future when we want to apply @davidchuyaya's optimizations this will be helpful to know when there are causal dependencies on when data can be sent to others.
   
   Second change is we mark every non-deterministic operator (modulo explicit annotations such as `NoOrder`) with Rust's `unsafe` keyword. This makes it super clear where non-determinism is taking place.
   
   I've used this to put `unsafe` blocks throughout our example code and add `SAFETY` annotations that argue why the non-determinism is safe (or point out that we've explicitly documented / expect non-determinism). I also added `#![warn(unsafe_op_in_unsafe_fn)]` to the examples and the template, since this forces good hygiene of annotating sources of non-determinism even inside a module that is intentionally non-deterministic.
   
   Paxos changes are mostly refactors, and I verified that the performance is the same as before.

   

   shadaj committed Nov 27, 2024
   Configuration menu

   • View commit details
   • Copy full SHA for e5d39ac
   • Browse repository at this point

   Copy the full SHA

   e5d39ac View commit details

   Browse the repository at this point in the history