hydroflow_plus v0.8.0

Reverted

• "feat(hydroflow): Added poll_futures and poll_futures_async operators.", fix #1183
  This reverts commit 997d90a.

  We have been trying to figure out the semantics a bit, and want to give
  it more thought before we commit to maintaining it

  Can un-revert and adjust the semantics later when we use it

Refactor (BREAKING)

• require lifetime on perist*() operators

Commit Statistics

• 2 commits contributed to the release over the course of 20 calendar days.
• 59 days passed between releases.
• 2 commits were understood as conventional.
• 3 unique issues were worked on: #1143, #1216, #1295

Commit Details

view details

• #1143
  • "feat(hydroflow): Added poll_futures and poll_futures_async operators.", fix #1183 (256779a)
• #1216
  • "feat(hydroflow): Added poll_futures and poll_futures_async operators.", fix #1183 (256779a)
• #1295
  • Require lifetime on perist*() operators (67c0e51)

hydroflow_macro v0.8.0

Unchanged from previous release.

Chore

• mark hydroflow_datalog and hydroflow_macro as unchanged for 0.8.0 release

Commit Statistics

• 1 commit contributed to the release.
• 59 days passed between releases.
• 1 commit was understood as conventional.
• 0 issues like '(#ID)' were seen in commit messages

Commit Details

view details

• Uncategorized
  • Mark hydroflow_datalog and hydroflow_macro as unchanged for 0.8.0 release (186310f)

hydroflow_lang v0.8.0

Documentation

• cleanup for clippy::doc_lazy_continuation
• test code snipets, generate output for surface_flows, fix #814
• fix dest_sink tokio links

Bug Fixes

• allow ensure_singleton_referencers_succeed_persist to ignore identity/tee/union operators, helps #1290
  Adds tests for ensure_singleton_referencers_succeed_persist persist
  insertion behavior

  Fixes some cases of #1290, when just identity operators
  (union/tee/identity) are between the singleton referencer and its
  preceding flow state.

  TODO track per-operator instead: #1311

• improve spanning of singleton substitution code, add compile-fail test, fix #1294
  Another small spanning improvement

• add add_state_tick to state API, reset at end of each tick, fix #1298
  Option 2 of #1298

  • Main feature in the title is in src/scheduled/{context,graph}.rs
  • Codegen for some stateful operators (those which can be used as singletons, and some others) is changed to use the new API.
  • Add a test test_cartesian_product_tick_state for #1298
  • Rest of the diff is snapshot changes

  Other 'tick state will need to be cleared, but existing implementation does that when the iterator runs, which is good enough. There is only a problem if a singleton can reference the state before the iterator runs, in that case.

• allow use of generics in demux_enum::<...>() op
  ensures turbofish syntax for the match clauses, which is needed when there are generic parameters in the type

  adds a test as well

Refactor

• improve diagnostics for missing generic params
  Improves diagnostic messages a bit for when no generic params are
  supplied but some are expected. Previously this would span to the entire
  macro invocation.
• improve diagnostics by re-spanning #root
  Inspired by fixing the spans in demux_enum in #1271
  • re-span #root to op_span for better diagnostics
  • use better span func.inputs in demux and demux_enum
  • clippy fixups in source_json, source_stdin
  • fix #1201 (for the most part)

Reverted

• "feat(hydroflow): Added poll_futures and poll_futures_async operators.", fix #1183
  This reverts commit 997d90a.

  We have been trying to figure out the semantics a bit, and want to give
  it more thought before we commit to maintaining it

  Can un-revert and adjust the semantics later when we use it

Style

• fix unnecessary & for clippy

Bug Fixes (BREAKING)

• remove singleton referencer persist::<'static>() insertion
  Also enables singletons for persist() and ensures that only the
  'static lifetime is used

  Singletons are supposed to act like cross_join(). I.e. if we have this
  code:

  stream -> filter(|item| ... #y ...) -> ...

  It should behave equivalently to

  stream -> cj[0];
  y -> cj[1];
  cj = cross_join() -> filter(|(item, y)| ...) -> ...

  This has a very unintuitive replaying behavior, if y receives multiple
  updates:

  1. y receives an item 10
  2. stream receives an item 20
  3. (10, 20) is emitted
  4. y receives an item 11
  5. (11, 20) is emitted
     In this case the item 20 gets emitted twice.

  To emulate this unintuitive behavior, we currently ensure that a
  persist::<'static>() exists before operator that references the
  singleton (filter, in this case). (Note that this is equivalent to
  cross_join::<'static>() and not cross_join::<'tick>())

  However singletons also have had a different mechanism that affects
  this- currently singleton references create a next-stratum constraint,
  that ensures a singleton referencer must be in a later stratum than the
  singleton it is referencing.

  Note that this actually prevents the example situation above from
  happening-- the updates to y will be received all at once at the start
  of the next stratum.

  This means that actually, currently singletons are equivalent to
  something like:

  stream -> cj[0];
  y -> next_stratum() -> last() -> cj[1];
  cj = cross_join() -> filter(|(item, y)| ...) -> ...

  last() is a hypothetical operator that only keeps the most recent item
  output by y. next_stratum() -> last() is equivalent to reduce(|acc, item| *acc = item) (since that comes with a stratum barrier). So
  technically this is a slightly different behavior than just cross_join,
  but it is more intuitive.

  stream -> cj[0];
  y -> reduce(|acc, item| { *acc = item; }) -> cj[1];
  cj = cross_join() -> filter(|(item, y)| ...) -> ...

  Also fixes #1293

Refactor (BREAKING)

• require lifetime on perist*() operators

Commit Statistics

• 13 commits contributed to the release over the course of 48 calendar days.
• 59 days passed between releases.
• 13 commits were understood as conventional.
• 13 unique issues were worked on: #1143, #1216, #1260, #1266, #1271, #1280, #1285, #1295, #1296, #1297, #1300, #1312, #1332

Commit Details

view details

• #1143
  • "feat(hydroflow): Added poll_futures and poll_futures_async operators.", fix #1183 (256779a)
• #1216
  • "feat(hydroflow): Added poll_futures and poll_futures_async operators.", fix #1183 (256779a)
• #1260
  • Test code snipets, generate output for surface_flows, fix #814 (2a4881d)
• #1266
  • Fix dest_sink tokio links (1bc6c61)
• #1271
  • Allow use of generics in demux_enum::<...>() op (b79f1a4)
• #1280
  • Improve diagnostics by re-spanning #root (70a3f9e)
• #1285
  • Cleanup for clippy::doc_lazy_continuation (14067e9)
• #1295
  • Require lifetime on perist*() operators (67c0e51)
• #1296
  • Improve diagnostics for missing generic params (f1442a0)
• #1297
  • Allow ensure_singleton_referencers_succeed_persist to ignore identity/tee/union operators, helps #1290 (f45b9dd)
• #1300
  • Add add_state_tick to state API, reset at end of each tick, fix #1298 (f91c300)
• #1312
  • Improve spanning of singleton substitution code, add compile-fail test, fix #1294 (404f0ac)
• #1332
  • Remove singleton referencer persist::<'static>() insertion (755e8a6)
• Uncategorized
  • Fix unnecessary & for clippy (b7275c2)

hydroflow_datalog_core v0.8.0

New Features

• add #[derive(Lattice)] derive macros, fix #1247
  This adds derive macros to allow user-created macros. Each field must be
  a lattice.

  Example usage:

  struct MyLattice<KeySet, Epoch>
  where
      KeySet: Collection,
      Epoch: Ord,
  {
      keys: SetUnion<KeySet>,
      epoch: Max<Epoch>,
  }

  Uses #[derive(Lattice)] for the lattices library Pair lattice.
  Also contains some cleanup in the lattices crate.

Bug Fixes

• add add_state_tick to state API, reset at end of each tick, fix #1298
  Option 2 of #1298

  • Main feature in the title is in src/scheduled/{context,graph}.rs
  • Codegen for some stateful operators (those which can be used as singletons, and some others) is changed to use the new API.
  • Add a test test_cartesian_product_tick_state for #1298
  • Rest of the diff is snapshot changes

  Other 'tick state will need to be cleared, but existing implementation does that when the iterator runs, which is good enough. There is only a problem if a singleton can reference the state before the iterator runs, in that case.

Bug Fixes (BREAKING)

• remove singleton referencer persist::<'static>() insertion
  Also enables singletons for persist() and ensures that only the
  'static lifetime is used

  Singletons are supposed to act like cross_join(). I.e. if we have this
  code:

  stream -> filter(|item| ... #y ...) -> ...

  It should behave equivalently to

  stream -> cj[0];
  y -> cj[1];
  cj = cross_join() -> filter(|(item, y)| ...) -> ...

  This has a very unintuitive replaying behavior, if y receives multiple
  updates:

  1. y receives an item 10
  2. stream receives an item 20
  3. (10, 20) is emitted
  4. y receives an item 11
  5. (11, 20) is emitted
     In this case the item 20 gets emitted twice.

  To emulate this unintuitive behavior, we currently ensure that a
  persist::<'static>() exists before operator that references the
  singleton (filter, in this case). (Note that this is equivalent to
  cross_join::<'static>() and not cross_join::<'tick>())

  However singletons also have had a different mechanism that affects
  this- currently singleton references create a next-stratum constraint,
  that ensures a singleton referencer must be in a later stratum than the
  singleton it is referencing.

  Note that this actually prevents the example situation above from
  happening-- the updates to y will be received all at once at the start
  of the next stratum.

  This means that actually, currently singletons are equivalent to
  something like:

  stream -> cj[0];
  y -> next_stratum() -> last() -> cj[1];
  cj = cross_join() -> filter(|(item, y)| ...) -> ...

  last() is a hypothetical operator that only keeps the most recent item
  output by y. next_stratum() -> last() is equivalent to reduce(|acc, item| *acc = item) (since that comes with a stratum barrier). So
  technically this is a slightly different behavior than just cross_join,
  but it is more intuitive.

  stream -> cj[0];
  y -> reduce(|acc, item| { *acc = item; }) -> cj[1];
  cj = cross_join() -> filter(|(item, y)| ...) -> ...

  Also fixes #1293

Refactor (BREAKING)

• require lifetime on perist*() operators

Commit Statistics

• 4 commits contributed to the release over the course of 49 calendar days.
• 59 days passed between releases.
• 4 commits were understood as conventional.
• 4 unique issues were worked on: #1250, #1295, #1300, #1332

Commit Details

view details

• #1250
  • Add #[derive(Lattice)] derive macros, fix #1247 (b3d01c2)
• #1295
  • Require lifetime on perist*() operators (67c0e51)
• #1300
  • Add add_state_tick to state API, reset at end of each tick, fix #1298 (f91c300)
• #1332
  • Remove singleton referencer persist::<'static>() insertion (755e8a6)

hydroflow_datalog v0.8.0

Unchanged from previous release.

Chore

• mark hydroflow_datalog and hydroflow_macro as unchanged for 0.8.0 release

Commit Statistics

• 1 commit contributed to the release.
• 59 days passed between releases.
• 1 commit was understood as conventional.
• 0 issues like '(#ID)' were seen in commit messages

Commit Details

view details

• Uncategorized
  • Mark hydroflow_datalog and hydroflow_macro as unchanged for 0.8.0 release (186310f)

hydroflow v0.8.0

Chore

• update pinned rust version to 2024-06-17

Documentation

• test code snipets, generate output for surface_flows, fix #814

Bug Fixes

• allow ensure_singleton_referencers_succeed_persist to ignore identity/tee/union operators, helps #1290
  Adds tests for ensure_singleton_referencers_succeed_persist persist
  insertion behavior

  Fixes some cases of #1290, when just identity operators
  (union/tee/identity) are between the singleton referencer and its
  preceding flow state.

  TODO track per-operator instead: #1311

• improve spanning of singleton substitution code, add compile-fail test, fix #1294
  Another small spanning improvement

• add add_state_tick to state API, reset at end of each tick, fix #1298
  Option 2 of #1298

  • Main feature in the title is in src/scheduled/{context,graph}.rs
  • Codegen for some stateful operators (those which can be used as singletons, and some others) is changed to use the new API.
  • Add a test test_cartesian_product_tick_state for #1298
  • Rest of the diff is snapshot changes

  Other 'tick state will need to be cleared, but existing implementation does that when the iterator runs, which is good enough. There is only a problem if a singleton can reference the state before the iterator runs, in that case.

• allow use of generics in demux_enum::<...>() op
  ensures turbofish syntax for the match clauses, which is needed when there are generic parameters in the type

  adds a test as well

• yield before collect_ready_async to ensure background async tasks can send to the stream

• make sure tasks are spawned!!!!
  fix bug introduced in #978

• Make inner for WithTop & WithBot private
  Option<T> is not a lattice, so it is unsafe to expose as public.

  I also updated documentation to lead with intention before
  implementation (minor cleanup).

Refactor

• improve diagnostics for missing generic params
  Improves diagnostic messages a bit for when no generic params are
  supplied but some are expected. Previously this would span to the entire
  macro invocation.
• improve diagnostics by re-spanning #root
  Inspired by fixing the spans in demux_enum in #1271
  • re-span #root to op_span for better diagnostics
  • use better span func.inputs in demux and demux_enum
  • clippy fixups in source_json, source_stdin
  • fix #1201 (for the most part)

Reverted

• "feat(hydroflow): Added poll_futures and poll_futures_async operators.", fix #1183
  This reverts commit 997d90a.

  We have been trying to figure out the semantics a bit, and want to give
  it more thought before we commit to maintaining it

  Can un-revert and adjust the semantics later when we use it

Test

• fix fold_keyed() and add reduce_keyed() compile-fail tests, fix #1279

Bug Fixes (BREAKING)

• remove singleton referencer persist::<'static>() insertion
  Also enables singletons for persist() and ensures that only the
  'static lifetime is used

  Singletons are supposed to act like cross_join(). I.e. if we have this
  code:

  stream -> filter(|item| ... #y ...) -> ...

  It should behave equivalently to

  stream -> cj[0];
  y -> cj[1];
  cj = cross_join() -> filter(|(item, y)| ...) -> ...

  This has a very unintuitive replaying behavior, if y receives multiple
  updates:

  1. y receives an item 10
  2. stream receives an item 20
  3. (10, 20) is emitted
  4. y receives an item 11
  5. (11, 20) is emitted
     In this case the item 20 gets emitted twice.

  To emulate this unintuitive behavior, we currently ensure that a
  persist::<'static>() exists before operator that references the
  singleton (filter, in this case). (Note that this is equivalent to
  cross_join::<'static>() and not cross_join::<'tick>())

  However singletons also have had a different mechanism that affects
  this- currently singleton references create a next-stratum constraint,
  that ensures a singleton referencer must be in a later stratum than the
  singleton it is referencing.

  Note that this actually prevents the example situation above from
  happening-- the updates to y will be received all at once at the start
  of the next stratum.

  This means that actually, currently singletons are equivalent to
  something like:

  stream -> cj[0];
  y -> next_stratum() -> last() -> cj[1];
  cj = cross_join() -> filter(|(item, y)| ...) -> ...

  last() is a hypothetical operator that only keeps the most recent item
  output by y. next_stratum() -> last() is equivalent to reduce(|acc, item| *acc = item) (since that comes with a stratum barrier). So
  technically this is a slightly different behavior than just cross_join,
  but it is more intuitive.

  stream -> cj[0];
  y -> reduce(|acc, item| { *acc = item; }) -> cj[1];
  cj = cross_join() -> filter(|(item, y)| ...) -> ...

  Also fixes #1293

Refactor (BREAKING)

• require lifetime on perist*() operators

Style (BREAKING)

• enable clippy upper-case-acronyms-aggressive
  • rename GCP -> Gcp, NodeID -> NodeId
  • update CI cargo-generate template testing to use PR's branch instead
    of whatever main happens to be

Commit Statistics

• 16 commits contributed to the release over the course of 54 calendar days.
• 59 days passed between releases.
• 16 commits were understood as conventional.
• 17 unique issues were worked on: #1143, #1216, #1244, #1260, #1271, #1273, #1274, #1280, #1283, #1295, #1296, #1297, #1300, #1309, #1312, #1332, #1345

Commit Details

view details

• #1143
  • "feat(hydroflow): Added poll_futures and poll_futures_async operators.", fix #1183 (256779a)
• #1216
  • "feat(hydroflow): Added poll_futures and poll_futures_async operators.", fix #1183 (256779a)
• #1244
  • Make inner for WithTop & WithBot private (1ad690b)
• #1260
  • Test code snipets, generate output for surface_flows, fix #814 (2a4881d)
• #1271
  • Allow use of generics in demux_enum::<...>() op (b79f1a4)
• #1273
  • Make sure tasks are spawned!!!! (39ad0d6)
• #1274
  • Yield before collect_ready_async to ensure background async tasks can send to the stream (dfb5ec9)
• #1280
  • Improve diagnostics by re-spanning #root (70a3f9e)
• #1283
  • Fix fold_keyed() and add reduce_keyed() compile-fail tests, fix #1279 (b3c233d)
• #1295
  • Require lifetime on perist*() operators (67c0e51)
• #1296
  • Improve diagnostics for missing generic params (f1442a0)
• #1297
  • Allow ensure_singleton_referencers_succeed_persist to ignore identity/tee/union operators, helps #1290 (f45b9dd)
• #1300
  • Add add_state_tick to state API, reset at end of each tick, fix #1298 (f91c300)
• #1309
  • Update pinned rust version to 2024-06-17 (3098f77)
• #1312
  • Improve spanning of singleton substitution code, add compile-fail test, fix #1294 (404f0ac)
• #1332
  • Remove singleton referencer persist::<'static>() insertion (755e8a6)
• #1345
  • Enable clippy upper-case-acronyms-aggressive (12b8ba5)

hydro_deploy v0.8.0

Refactor

• remove unneeded Arc<RwLock< wrapping of launch_binary return value (1/3)

  Curious if there was any intention behind why it was Arc<RwLock<?

  I think before some refactors we took the I/O handles instead of using broadcast channels.

• build cache cleanup

  • Replace mystery tuple with new struct BuildOutput
  • Replace Mutex and Arc-infested HashMap with memo-map crate,
    greatly simplifying build cache typing
  • Remove redundant build caching in HydroflowCrateService, expose and
    use cache parameters as BuildParams
  • Remove once_cell and async-once-cell dependencies, use std's
    OnceLock
  • Add Failed to execute command: {} context to perf error message
  • Cleanup some repeated format! expressions

Style

• rename SSH -> Ssh

Refactor (BREAKING)

• make Service::collect_resources take &self instead of &mut self
  #430 but still has RwLock wrapping

  Depends on #1347

• make Host trait use &self interior mutability to remove RwLock wrappings #430
  Depends on #1346

• Make Host::provision not async anymore
  I noticed that none of the method impls have any awaits

• make HydroflowSource, HydroflowSink traits use &self interior mutability to remove RwLock wrappings #430
  Depends on #1339

• replace async-channel with tokio::sync::mpsc::unbounded_channel
  Depends on #1339

  We could make the publicly facing stdout, stderr APIs return impl Stream<Output = String> in the future, maybe

• replace some uses of tokio::sync::RwLock with std::sync::Mutex #430 (3/3)

Style (BREAKING)

• enable clippy upper-case-acronyms-aggressive
  • rename GCP -> Gcp, NodeID -> NodeId
  • update CI cargo-generate template testing to use PR's branch instead
    of whatever main happens to be

Commit Statistics

• 10 commits contributed to the release over the course of 10 calendar days.
• 59 days passed between releases.
• 10 commits were understood as conventional.
• 10 unique issues were worked on: #1334, #1338, #1339, #1340, #1343, #1345, #1346, #1347, #1348, #1356

Commit Details

view details

• #1334
  • Build cache cleanup (0feae74)
• #1338
  • Remove unneeded Arc<RwLock< wrapping of launch_binary return value (1/3) (e3e6933)
• #1339
  • Replace some uses of tokio::sync::RwLock with std::sync::Mutex #430 (3/3) (141eae1)
• #1340
  • Rename SSH -> Ssh (947ebc1)
• #1343
  • Make Host::provision not async anymore (f536ecc)
• #1345
  • Enable clippy upper-case-acronyms-aggressive (12b8ba5)
• #1346
  • Make HydroflowSource, HydroflowSink traits use &self interior mutability to remove RwLock wrappings #430 (057a0a5)
• #1347
  • Make Host trait use &self interior mutability to remove RwLock wrappings #430 (c5a8de2)
• #1348
  • Make Service::collect_resources take &self instead of &mut self (2286558)
• #1356
  • Replace async-channel with tokio::sync::mpsc::unbounded_channel (6039078)

hydro_cli v0.8.0

Chore

• update pinned rust version to 2024-06-17

Refactor

• build cache cleanup
  • Replace mystery tuple with new struct BuildOutput
  • Replace Mutex and Arc-infested HashMap with memo-map crate,
    greatly simplifying build cache typing
  • Remove redundant build caching in HydroflowCrateService, expose and
    use cache parameters as BuildParams
  • Remove once_cell and async-once-cell dependencies, use std's
    OnceLock
  • Add Failed to execute command: {} context to perf error message
  • Cleanup some repeated format! expressions

Style

• rename SSH -> Ssh

Refactor (BREAKING)

• make Host trait use &self interior mutability to remove RwLock wrappings #430
  Depends on #1346

• make HydroflowSource, HydroflowSink traits use &self interior mutability to remove RwLock wrappings #430
  Depends on #1339

• replace async-channel with tokio::sync::mpsc::unbounded_channel
  Depends on #1339

  We could make the publicly facing stdout, stderr APIs return impl Stream<Output = String> in the future, maybe

• replace some uses of tokio::sync::RwLock with std::sync::Mutex #430 (3/3)

Style (BREAKING)

• enable clippy upper-case-acronyms-aggressive
  • rename GCP -> Gcp, NodeID -> NodeId
  • update CI cargo-generate template testing to use PR's branch instead
    of whatever main happens to be

Commit Statistics

• 8 commits contributed to the release over the course of 34 calendar days.
• 59 days passed between releases.
• 8 commits were understood as conventional.
• 8 unique issues were worked on: #1309, #1334, #1339, #1340, #1345, #1346, #1347, #1356

Commit Details

view details

• #1309
  • Update pinned rust version to 2024-06-17 (3098f77)
• #1334
  • Build cache cleanup (0feae74)
• #1339
  • Replace some uses of tokio::sync::RwLock with std::sync::Mutex #430 (3/3) (141eae1)
• #1340
  • Rename SSH -> Ssh (947ebc1)
• #1345
  • Enable clippy upper-case-acronyms-aggressive (12b8ba5)
• #1346
  • Make HydroflowSource, HydroflowSink traits use &self interior mutability to remove RwLock wrappings #430 (057a0a5)
• #1347
  • Make Host trait use &self interior mutability to remove RwLock wrappings #430 (c5a8de2)
• #1356
  • Replace async-channel with tokio::sync::mpsc::unbounded_channel (6039078)

stageleft_tool v0.2.0

Chore

• expect custom config names to prevent warnings
  See
  https://doc.rust-lang.org/nightly/cargo/reference/build-scripts.html#rustc-check-cfg

New Features

• re-compile staged sources for the macro at the top level

Commit Statistics

• 2 commits contributed to the release over the course of 43 calendar days.
• 44 days passed between releases.
• 2 commits were understood as conventional.
• 2 unique issues were worked on: #1104, #1192

Commit Details

• #1104
  • Re-compile staged sources for the macro at the top level (93fd05e)
• #1192
  • Expect custom config names to prevent warnings (b86f11a)

stageleft_macro v0.2.0

New Features

• re-compile staged sources for the macro at the top level

Bug Fixes

• handle send_bincode with local structs
  fix(hydroflow_plus): handle send_bincode with local structs

Commit Statistics

• 2 commits contributed to the release over the course of 43 calendar days.
• 48 days passed between releases.
• 2 commits were understood as conventional.
• 2 unique issues were worked on: #1104, #1151

Commit Details

• #1104
  • Re-compile staged sources for the macro at the top level (93fd05e)
• #1151
  • Handle send_bincode with local structs (0cafbdb)