Skip to content

Commit

Permalink
Update Event-driven orchestration for AI systems.ipynb
Browse files Browse the repository at this point in the history
  • Loading branch information
ZmeiGorynych authored Sep 27, 2024
1 parent 72ffa3f commit 66a7e9d
Showing 1 changed file with 4 additions and 6 deletions.
10 changes: 4 additions & 6 deletions examples/Event-driven orchestration for AI systems.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -4,15 +4,13 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"# Event-driven orchestration for AI systems. Is LlamaIndex Workflows basically Kafka Streams?\n",
"# Event-driven orchestration for AI systems: new ways and old\n",
"\n",
"Data processing tools like Kafka, Spark, Flink etc. have long been used to build workflows. Today, as people create complex AI systems, they often find themselves (surprise!) needing orchestration frameworks. Is the field really so specific it needs new tools for solving the same problems? We argue that there is no difference: after all, LLMs are just APIs, and the fundamental patterns of workflow architecture remain the same.\n",
"Data processing tools like Kafka, Spark, Flink etc. have long been used to build workflows. Today, as people create complex AI systems, they unsurprisingly often find themselves needing orchestration frameworks. Is the field really so specific it needs new tools for solving the same problems? After all, LLMs are just APIs, and the fundamental patterns of workflow architecture remain the same.\n",
"\n",
"A widespread data processing paradigm is event-driven architecture. The purpose of this example is to demonstrate how you can use tried-and-tested tools like Kafka Streams (ported to Python via Faust) to build workflows, side-stepping the need to re-invent the wheel for something as fundamental as event processing. This also comes with reliability and scalability of Kafka, easing production deployment.\n",
"A widespread data processing paradigm is event-driven architecture. The purpose of this example is to demonstrate how you can use tried-and-tested tools like Kafka and [Faust](https://github.com/faust-streaming/faust) to build workflows. This also comes with reliability and scalability of Kafka, easing production deployment.\n",
"\n",
"We'll walk through the basic concepts of the newly released LlamaIndex Workflows library, and compare them to what you get with Faust.\n",
"\n",
"To get a basic understanding of event-driven workflows, we suggest you check out this guide to LlamaIndex Workflows: https://docs.llamaindex.ai/en/stable/module_guides/workflow/\n",
"We'll walk through the basic concepts of the newly released [LlamaIndex Workflows](https://docs.llamaindex.ai/en/stable/module_guides/workflow/) functionality, and compare them to what you get with Faust.\n",
"\n",
"Here we are using the well-maintained fork of Faust: https://github.com/faust-streaming/faust"
]
Expand Down

0 comments on commit 66a7e9d

Please sign in to comment.