Skip to content

Commit

Permalink
docs(hydroflow_plus): polish quickstart and drop stale pages (#1576)
Browse files Browse the repository at this point in the history
I've decided to leave some more challenging TODOs as comments for now,
just so we can get the live site back into decent shape.

Also changes the "Get Started" button on the landing to point to the HF+
quickstart.
  • Loading branch information
shadaj authored Nov 21, 2024
1 parent 7c99294 commit 8e1aed9
Show file tree
Hide file tree
Showing 14 changed files with 86 additions and 283 deletions.
40 changes: 0 additions & 40 deletions docs/docs/hydroflow_plus/aggregations.mdx

This file was deleted.

89 changes: 0 additions & 89 deletions docs/docs/hydroflow_plus/clusters.mdx

This file was deleted.

32 changes: 0 additions & 32 deletions docs/docs/hydroflow_plus/cycles.mdx

This file was deleted.

13 changes: 13 additions & 0 deletions docs/docs/hydroflow_plus/dataflow-programming.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
---
sidebar_position: 1
---

# Dataflow Programming
Hydroflow+ uses a dataflow programming model, which will be familiar if you have used APIs like Rust iterators. Instead of using RPCs or async/await to describe distributed computation, Hydroflow+ instead uses **asynchronous streams**, which represent data arriving over time. Streams can represent a series of asynchronous events (e.g. inbound network requests) or a sequence of data items.

Programs in Hydroflow+ describe how to **transform** entire collections of data using operators such as `map` (transforming elements one by one), `fold` (aggregating elements into a single value), or `join` (combining elements from multiple streams on matching keys).

If you are familiar with Spark, Flink or Pandas, you will find Hydroflow+ syntax familiar. However, note well that the semantics for asynchronous streams in Hydroflow+ differ significantly from bulk analytics systems like those above. In particular, Hydroflow+ uses the type system to distinguish between bounded streams (originating from finite data) and unbounded streams (originated from asynchronous input). Moreover, Hydroflow+ is designed to handle asynchronous streams of small, independent events very efficiently.

<!-- TODO(shadaj): link to collections section -->
<!-- TODO(shadaj): show example of mermaid graph -->
4 changes: 2 additions & 2 deletions docs/docs/hydroflow_plus/index.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -3,11 +3,11 @@ sidebar_position: 0
---

# Introduction
Hydroflow+ is a high-level distributed streaming framework for Rust powered by the [Hydroflow runtime](../hydroflow/index.mdx). Unlike traditional architectures such as actors or RPCs, Hydroflow+ offers _choreographic_ APIs, where expressions and functions can describe computation that takes place across many locations. It also integrates with [Hydro Deploy](../deploy/index.md) to make it easy to deploy and run Hydroflow+ programs to the cloud.
Hydroflow+ is a high-level distributed programming framework for Rust powered by the [Hydroflow runtime](../hydroflow/index.mdx). Unlike traditional architectures such as actors or RPCs, Hydroflow+ offers _choreographic_ APIs, where expressions and functions can describe computation that takes place across many locations. It also integrates with [Hydro Deploy](../deploy/index.md) to make it easy to deploy and run Hydroflow+ programs to the cloud.

Hydroflow+ uses a two-stage compilation approach. HF+ programs are standard Rust programs, which first run on the developer's laptop to generate a _deployment plan_. This plan is then compiled to individual binaries for each machine in the distributed system (enabling zero-overhead abstractions), and are then deployed to the cloud using the generated plan along with specifications of cloud resources.

Hydroflow+ has been used to write a variety of high-performance distributed system, including implementations of classic distributed protocols such as two-phase commit and Paxos. Work is ongoing to develop a distributed systems standard library that will offer these protocols and more as reusable components.
Hydroflow+ has been used to write a variety of high-performance distributed systems, including implementations of classic distributed protocols such as two-phase commit and Paxos. Work is ongoing to develop a distributed systems standard library that will offer these protocols and more as reusable components.

:::caution

Expand Down
78 changes: 0 additions & 78 deletions docs/docs/hydroflow_plus/process_streams.mdx

This file was deleted.

2 changes: 1 addition & 1 deletion docs/docs/hydroflow_plus/quickstart/_category_.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"label": "Quickstart",
"position": 1,
"position": 2,
"link": {
"type": "doc",
"id": "hydroflow_plus/quickstart/index"
Expand Down
Loading

0 comments on commit 8e1aed9

Please sign in to comment.