From 7ff1046d499e592abdeb1342f5cc02a4ce2fb9af Mon Sep 17 00:00:00 2001 From: MartinRst Date: Mon, 7 Oct 2024 15:47:52 +0200 Subject: [PATCH 1/6] V1 3 years of Kestra --- content/blogs/2024-10-07-3-years-of-kestra.md | 93 +++++++++++++++++++ 1 file changed, 93 insertions(+) create mode 100644 content/blogs/2024-10-07-3-years-of-kestra.md diff --git a/content/blogs/2024-10-07-3-years-of-kestra.md b/content/blogs/2024-10-07-3-years-of-kestra.md new file mode 100644 index 0000000000..d0beda91f9 --- /dev/null +++ b/content/blogs/2024-10-07-3-years-of-kestra.md @@ -0,0 +1,93 @@ +--- +title: "3 years of Kestra, where we are from and where we go" +description: "This is the first time we, has Co-Founders, share the journey of Kestra from its beginning to the future we envision for Orchestration" +date: 2024-10-0710T12:00:00 +category: Company News +author: + name: Ludovic Dehon + image: "ldehon" + role: "CTO & Co-Founder" +image: +--- +This is the first time we, has Co-Founders, share the journey of Kestra from its beginning to the future we envision for Orchestration + +Before Kestra officially became what it is today, its roots began in my early career. I started as a self-taught developer about 20 years ago, working mainly with **PHP**, like many developers in France who were building websites at the time. Over time, I transitioned through various roles, including working in startups and doing freelance projects. My goal was always to create products that could solve complex technical problems. + +Around **2010**, while working at **Ankama**, I created the first version of what would later inspire **Kestra**. It was a simple solution to manage **distributed Cron jobs**, allowing monitoring and control of tasks across multiple servers. At the time, I called it **CronDD**, a reference to the traditional Unix **cron** jobs. The idea was to define and monitor scheduled tasks in a distributed way, but it was still a far cry from what Kestra would eventually become. + +Fast forward to **2019**, when I was at **Leroy Merlin France**. The company was migrating from **Terradata** to **Google Cloud's BigQuery**, and they needed a way to orchestrate complex workflows that interacted with a variety of data systems. **Airflow** was the obvious choice, given that it had become the leader in the orchestration space. But after spending few months analyzing it, I realized it wouldn’t fit the use cases of Leroy Merlin, the philosophy of dag defined in Python, the high constraint of security, and scalabilities needed was not meet. That’s when I started to seriously consider building my own orchestrator. + +In the evenings and on weekends, I began developing what would become **Kestra**. Starting such a huge project after working hours seems scary, I knew that the road to reach something usable would be long, but the more I worked on it, the clearer it became that there was a real need for a more flexible, scalable orchestrator that wasn’t limited by the design choices Airflow had made. By **2020**, Leroy Merlin was ready to go into production with Kestra, even though I was still the only person working on it at the time. + +At that time, Leroy Merlin was facing multiple challenges with their existing data architecture, which relied on legacy tools like Teradata, VectorWise, and Stambia for managing pipelines, with **Dollar U** and **Automic Workload Automation** for orchestration. The system was becoming increasingly outdated, and extremely complex with multiple team dependencies, especially as competitors were making swift moves to the cloud. Their data orchestration, in particular, was hindered by rigid manual processes and limited cloud connectivity. + +That’s when I decided to build something better. + +## Kestra to the Rescue + +Kestra was designed to solve the exact pain points Leroy Merlin was facing: **scalability, flexibility, and ease of use** for engineers across various teams. The goal was to create something that could handle the scale of Leroy Merlin’s data operations, while being simple enough for any engineer—whether they specialized in data, BI, or infrastructure—to use without a steep learning curve. + +After several months of development and testing, Leroy Merlin adopted Kestra, and it quickly transformed their orchestration processes. What previously required multiple teams, complex workflows, and days of waiting was now reduced to hours, with engineers empowered to manage their own workflows autonomously. + +By **August 2020**, Leroy Merlin upgraded to the **Enterprise Edition of Kestra**, unlocking even more features like role-based access control and advanced security, further streamlining their operations. + +Today, Kestra powers over **400,000 executions** and **3 million tasks** per month for Leroy Merlin, orchestrating everything from real-time data transfers to long-running data science workflows with incredible efficiency. + +## Scaling the Team and Building the Vision + +As the project grew, I reached out to **Emmanuel Darras**, the former CEO of **Ankama** and a good friend who had hired me in the past. Emmanuel had built **Ankama** into a company of over 500 people with the success of the game **Dofus**, and I knew he had the experience and vision to help Kestra scale. After discussing the project, he agreed to join me, and we officially created the company in **2021**. Together, we set out to build a platform that could solve orchestration challenges at any scale, and by **2022**, we had our first employees and began to grow the team. + +Kestra’s philosophy from the start was to build a tool that made **workflow orchestration simple**, yet powerful. We didn’t want it to be a niche product for a small group of engineers. Instead, we wanted Kestra to be **easy to adopt** while still being flexible enough to handle the most complex workflows. + +One of the core decisions we made early on was to use **YAML** for defining workflows. While some see YAML as just configuration, we saw it as a way to lower the barrier for entry. Inspired by tools like **Terraform** and **GitHub Actions**, we wanted Kestra to be a platform that developers could use without needing to learn a new programming language. That’s why Kestra is **declarative**—you declare what you want to happen, and the platform handles the execution. YAML was also a natural choice because it was already becoming the standard for configuration across many industries, especially with the rise of **Kubernetes** and **CI/CD pipelines**. + +## Early Wins: GitHub, Hacker News, and InfoQ + +Our first milestone as a company was the public release of Kestra. The response was overwhelmingly positive when we shared the project on Hacker News. That initial success brought us attention from developers and companies around the world, solidifying our belief that we were building something special. + +A second boost came from an article on **InfoQ**, which further expanded our reach and helped Kestra gain credibility in the tech community. By the time we raised our **pre-seed** funding, it was clear that Kestra was evolving beyond just a data orchestration tool — we were creating a flexible platform that could handle a wide variety of use cases, from complex data pipelines to operational workflows. + +### A Unified Approach to Orchestration + +The real power of Kestra lies in its **unified orchestration**. While many of our competitors focus on specific use cases—like orchestration for data pipelines, ML pipelines, or Business workflows—our goal is to provide a platform that can handle any type of workflow, regardless of the technology or use case. This means that whether you're orchestrating **ETL jobs**, managing **infrastructure with Terraform**, or handling **business processes**, Kestra can do it all. + +We’ve worked hard to ensure that Kestra is **language-agnostic**. You can run tasks in **Java**, **Python**, **Go**, **JavaScript**, or any other language. We achieved this by building tight integrations with **Docker** and leveraging containerization to isolate task executions. In fact, many of our clients are running highly specialized workflows, using languages like **Go** for performance-critical tasks. One of our clients even migrated all their processes from **Python to Go**, integrating it with Kestra to handle the orchestration. + +## Leveraging Plugins for Flexibility + +One of Kestra’s biggest strengths is the flexibility provided by our **plugin system**. When building Kestra, we wanted to ensure it could easily interact with a wide variety of technologies and ecosystems. That’s why we designed the platform with an architecture that makes it easy to create and extend plugins. Today, Kestra has **over 535 plugins**, and we’re aiming to double that number in the near future. + +Creating a plugin for Kestra is straightforward. We provide templates that make it simple for any **Java developer** to get started. In most cases, a plugin is 80% complete just by copying the **Quick Start guide** from the technology it’s integrating with. The heavy lifting happens behind the scenes, so the complexity is abstracted away from the developer. We also ensure high quality by running **nightly tests** on all plugins, checking for changes in technology or API updates. + +We’ve designed Kestra in such a way that users can interact with any technology through plugins, whether it's **cloud services**, **databases**, or even **custom in-house systems**. One of the major architectural choices we made was to normalize data exchange between tasks using **Amazon Ion**, a format that extends JSON but includes native support for data types like **date**, **time**, and **timestamps** with time zones. This makes sure that data is typed correctly and managed efficiently across tasks. It’s 2024, and it’s no longer acceptable for dates to be strings! + +We didn’t over-engineer the system either. While we use typed data formats like Ion, we don’t require users to define strict schemas (like Avro, Json Schema, ...) between tasks. This was a deliberate decision. We wanted to keep Kestra flexible and easy to use without adding unnecessary complexity. If users need strict data quality control, we provide plugins to handle that, but it’s not enforced by default. + +We’ve always maintained control over our core plugins to ensure quality. When community members contribute a plugin, we review it, merge it into the core if it fits, and then take over its long-term maintenance. This approach helps us avoid the risks that come with community-contributed plugins becoming outdated or inconsistent in quality. + +--- + +## Open Source at Core and Enterprise Ready + +Our open-source philosophy is in Kestra’s DNA. The open-source community is essential to our growth, and we continue to listen and evolve based on their feedback. Kestra is designed to be **open-source first**, which means that even though we offer an **Enterprise version** with features like **SSO, secret management,** and **advanced scalability options**, anyone can run the open-source version in production. + +Despite the challenges of balancing **open-source** and **commercial** strategies, we strongly believe in the benefits of open-source. It has allowed us to reach a global audience and gain adoption in industries we might never have touched otherwise. For example, we know that large companies in China have been using Kestra for years without ever contacting us directly. This is part of the nature of open-source, and while we might not convert every user to an Enterprise customer, the visibility and validation we gain through widespread usage is invaluable. + +--- + +To meet these enterprise demands, Kestra is fully **scalable** and **distributed**. We’ve split the system into microservices, with a **Scheduler**, an **Executor**, and **Workers** that can run across multiple environments. Each service can scale both **horizontally** and **vertically** to handle varying workloads. For organizations with unique infrastructure requirements, we also offer **worker groups**, which allow specific tasks to be run on specialized hardware, like **GPU** or **Windows-based** environments. + +We’ve also added **task runners** in our Enterprise version, which allow users to delegate the execution of tasks to remote engines like **Kubernetes**, **AWS Batch**, **Azure Batch**, **Google Batch**, or **Cloud Run**. This enables users to take advantage of **serverless** execution when necessary, while still keeping the core system performant for high-throughput workloads. + +--- + +## Building for the Future + +Looking ahead, our ambition is to make Kestra the go-to tool for workflow orchestration across industries. We don’t want to be known as just a **data orchestrator**. Instead, our vision is to create a **category leader**—a platform that can orchestrate everything, from **business processes** to **infrastructure automation**. + +While we’ve grown quickly, with a team nearing **20 people**, we’re far from finished. We’re continuing to build out Kestra’s capabilities, and part of that includes expanding our **Cloud offering**. Right now, we have a **private beta** of Kestra Cloud, where we’re working with select customers to ensure it meets their needs. The challenge with running Kestra in the cloud is that it needs to handle an incredibly diverse range of workloads, from high-memory tasks to network-heavy operations. We’re taking our time to get it right because we want to offer a **reliable, secure, and performant** cloud service that can scale with any workload. + +::alert{type="info"} +If you have any questions, reach out via [Slack](https://kestra.io/slack) or open [a GitHub issue](https://github.com/kestra-io/kestra). +If you like the project, give us [a GitHub star](https://github.com/kestra-io/kestra) and join [the community](https://kestra.io/slack). +:: \ No newline at end of file From f2ec75e744a3327d73ebfc7e10969277ba66c775 Mon Sep 17 00:00:00 2001 From: MartinRst Date: Mon, 7 Oct 2024 15:55:28 +0200 Subject: [PATCH 2/6] Update 2024-10-07-3-years-of-kestra.md --- content/blogs/2024-10-07-3-years-of-kestra.md | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/content/blogs/2024-10-07-3-years-of-kestra.md b/content/blogs/2024-10-07-3-years-of-kestra.md index d0beda91f9..4cb022443c 100644 --- a/content/blogs/2024-10-07-3-years-of-kestra.md +++ b/content/blogs/2024-10-07-3-years-of-kestra.md @@ -1,6 +1,6 @@ --- -title: "3 years of Kestra, where we are from and where we go" -description: "This is the first time we, has Co-Founders, share the journey of Kestra from its beginning to the future we envision for Orchestration" +title: "3 years of Kestra, From Zero to 10,000 Stars on GitHub" +description: "To mark 10k stars on GitHub, we’re sharing Kestra’s journey from its early days to what it has become today—a unified orchestration platform with a vision for the future." date: 2024-10-0710T12:00:00 category: Company News author: @@ -9,7 +9,6 @@ author: role: "CTO & Co-Founder" image: --- -This is the first time we, has Co-Founders, share the journey of Kestra from its beginning to the future we envision for Orchestration Before Kestra officially became what it is today, its roots began in my early career. I started as a self-taught developer about 20 years ago, working mainly with **PHP**, like many developers in France who were building websites at the time. Over time, I transitioned through various roles, including working in startups and doing freelance projects. My goal was always to create products that could solve complex technical problems. From 13b30f07e1ac735168c6ea2d7a27cef62832a52c Mon Sep 17 00:00:00 2001 From: Anna Geller Date: Mon, 7 Oct 2024 23:08:38 +0200 Subject: [PATCH 3/6] Update 2024-10-07-3-years-of-kestra.md --- content/blogs/2024-10-07-3-years-of-kestra.md | 76 ++++++++----------- 1 file changed, 30 insertions(+), 46 deletions(-) diff --git a/content/blogs/2024-10-07-3-years-of-kestra.md b/content/blogs/2024-10-07-3-years-of-kestra.md index 4cb022443c..ce221d5e86 100644 --- a/content/blogs/2024-10-07-3-years-of-kestra.md +++ b/content/blogs/2024-10-07-3-years-of-kestra.md @@ -1,6 +1,6 @@ --- title: "3 years of Kestra, From Zero to 10,000 Stars on GitHub" -description: "To mark 10k stars on GitHub, we’re sharing Kestra’s journey from its early days to what it has become today—a unified orchestration platform with a vision for the future." +description: "To celebrate reaching 10,000 stars on GitHub, we’re looking back at Kestra’s journey — from its early days to becoming a unified orchestration platform." date: 2024-10-0710T12:00:00 category: Company News author: @@ -10,83 +10,67 @@ author: image: --- -Before Kestra officially became what it is today, its roots began in my early career. I started as a self-taught developer about 20 years ago, working mainly with **PHP**, like many developers in France who were building websites at the time. Over time, I transitioned through various roles, including working in startups and doing freelance projects. My goal was always to create products that could solve complex technical problems. +Before Kestra became what it is today, the idea started back in the early days of my career. I began as a self-taught developer about 20 years ago, mostly working with **PHP** like many other developers in France at the time. Over the years, I took on various roles, from working in startups to freelance gigs, always aiming to create products that could solve tricky technical problems. -Around **2010**, while working at **Ankama**, I created the first version of what would later inspire **Kestra**. It was a simple solution to manage **distributed Cron jobs**, allowing monitoring and control of tasks across multiple servers. At the time, I called it **CronDD**, a reference to the traditional Unix **cron** jobs. The idea was to define and monitor scheduled tasks in a distributed way, but it was still a far cry from what Kestra would eventually become. +Around **2010**, while I was at **Ankama**, I built an early version of what would eventually inspire **Kestra**. It was a simple tool to manage **distributed Cron jobs** — a way to monitor and control tasks across multiple servers. I called it CronDD as a reference to the well-known Unix cron jobs. It was a useful solution for scheduling tasks, but it was still far from the more flexible and scalable orchestrator that Kestra would become. -Fast forward to **2019**, when I was at **Leroy Merlin France**. The company was migrating from **Terradata** to **Google Cloud's BigQuery**, and they needed a way to orchestrate complex workflows that interacted with a variety of data systems. **Airflow** was the obvious choice, given that it had become the leader in the orchestration space. But after spending few months analyzing it, I realized it wouldn’t fit the use cases of Leroy Merlin, the philosophy of dag defined in Python, the high constraint of security, and scalabilities needed was not meet. That’s when I started to seriously consider building my own orchestrator. +Fast forward to **2019**, and I was working at **Leroy Merlin France**. The company was in the middle of migrating from **Teradata** to **Google Cloud’s BigQuery**, and they needed a way to manage complex workflows that interacted with all kinds of data systems. **Airflow** was the obvious choice, but after spending months with it, I realized it wasn’t the right fit for Leroy Merlin’s needs. The way it worked, its security limitations, and the scalability issues meant it wouldn’t meet their requirements. That’s when I started to seriously think about building my own solution. -In the evenings and on weekends, I began developing what would become **Kestra**. Starting such a huge project after working hours seems scary, I knew that the road to reach something usable would be long, but the more I worked on it, the clearer it became that there was a real need for a more flexible, scalable orchestrator that wasn’t limited by the design choices Airflow had made. By **2020**, Leroy Merlin was ready to go into production with Kestra, even though I was still the only person working on it at the time. +In my spare time, I began working on what would eventually become **Kestra**. It was a huge project to take on after hours, but the more I worked on it, the more I saw the value it would bring to Leroy Merlin and potentially other companies. By **2020**, Leroy Merlin was ready to use Kestra in production. -At that time, Leroy Merlin was facing multiple challenges with their existing data architecture, which relied on legacy tools like Teradata, VectorWise, and Stambia for managing pipelines, with **Dollar U** and **Automic Workload Automation** for orchestration. The system was becoming increasingly outdated, and extremely complex with multiple team dependencies, especially as competitors were making swift moves to the cloud. Their data orchestration, in particular, was hindered by rigid manual processes and limited cloud connectivity. +Leroy Merlin was dealing with a data architecture that was getting outdated and overly complex. They were using legacy tools like Teradata, VectorWise, and Stambia to manage pipelines, with **Dollar U** and **Automic Workload Automation** handling orchestration. The system was slow and dependent on manual processes, making it hard to keep up with competitors moving to the cloud. -That’s when I decided to build something better. +## How Kestra Helped -## Kestra to the Rescue +Kestra was designed to tackle Leroy Merlin’s issues—**scalability, flexibility**, and making things easier for engineers across various teams. The idea was to build something that could handle their massive data operations but still be user-friendly for engineers from different teams, whether they worked in data, BI, or infrastructure. -Kestra was designed to solve the exact pain points Leroy Merlin was facing: **scalability, flexibility, and ease of use** for engineers across various teams. The goal was to create something that could handle the scale of Leroy Merlin’s data operations, while being simple enough for any engineer—whether they specialized in data, BI, or infrastructure—to use without a steep learning curve. +After months of development and testing, Kestra went live at Leroy Merlin, and it transformed their workflows. What used to take multiple teams days to complete was now done in hours, with engineers able to manage their own workflows independently. -After several months of development and testing, Leroy Merlin adopted Kestra, and it quickly transformed their orchestration processes. What previously required multiple teams, complex workflows, and days of waiting was now reduced to hours, with engineers empowered to manage their own workflows autonomously. +By **August 2020**, Leroy Merlin upgraded to the **Enterprise Edition of Kestra**, which brought even more features like role-based access control and advanced security, further improving their processes. Today, Kestra runs over **400,000 executions** and **3 million tasks** per month at Leroy Merlin, handling everything from real-time data transfers to long-running data science workflows. -By **August 2020**, Leroy Merlin upgraded to the **Enterprise Edition of Kestra**, unlocking even more features like role-based access control and advanced security, further streamlining their operations. +## Growing the Team and Expanding -Today, Kestra powers over **400,000 executions** and **3 million tasks** per month for Leroy Merlin, orchestrating everything from real-time data transfers to long-running data science workflows with incredible efficiency. +As Kestra gained traction, I reached out to **Emmanuel Darras**, a good friend and former CEO of **Ankama**, where I had worked before. Emmanuel had built **Ankama** into a successful company of over 500 people, and I knew his experience would be invaluable in scaling Kestra. After talking it over, he decided to join, and in **2021**, we officially founded the company. By **2022**, we had started hiring and growing the team. -## Scaling the Team and Building the Vision +Our goal from the start was to create an orchestration platform that was simple to adopt but powerful enough to handle complex workflows. One of our first decisions was to use **YAML** for defining workflows. We didn’t want developers to have to learn a new language just to use Kestra, so we went with a **declarative** approach: you tell Kestra what needs to happen, and it takes care of the rest. **YAML** was also becoming the go-to configuration format across industries, so it was the natural choice. -As the project grew, I reached out to **Emmanuel Darras**, the former CEO of **Ankama** and a good friend who had hired me in the past. Emmanuel had built **Ankama** into a company of over 500 people with the success of the game **Dofus**, and I knew he had the experience and vision to help Kestra scale. After discussing the project, he agreed to join me, and we officially created the company in **2021**. Together, we set out to build a platform that could solve orchestration challenges at any scale, and by **2022**, we had our first employees and began to grow the team. +## Early Milestones: GitHub, Hacker News, and InfoQ -Kestra’s philosophy from the start was to build a tool that made **workflow orchestration simple**, yet powerful. We didn’t want it to be a niche product for a small group of engineers. Instead, we wanted Kestra to be **easy to adopt** while still being flexible enough to handle the most complex workflows. +Our first big milestone was Kestra’s public release. The feedback was incredible when we shared it on Hacker News, and that helped us gain attention from developers and companies around the world. An article on **InfoQ** followed, which brought even more visibility and credibility to Kestra. By the time we raised our **pre-seed funding**, it was clear that Kestra was evolving beyond just a data orchestration tool — it was a flexible platform that could handle all kinds of use cases, from data pipelines to operational workflows. -One of the core decisions we made early on was to use **YAML** for defining workflows. While some see YAML as just configuration, we saw it as a way to lower the barrier for entry. Inspired by tools like **Terraform** and **GitHub Actions**, we wanted Kestra to be a platform that developers could use without needing to learn a new programming language. That’s why Kestra is **declarative**—you declare what you want to happen, and the platform handles the execution. YAML was also a natural choice because it was already becoming the standard for configuration across many industries, especially with the rise of **Kubernetes** and **CI/CD pipelines**. +### One Platform for Company-Wide Orchestration -## Early Wins: GitHub, Hacker News, and InfoQ +One of the strengths of Kestra is in how it brings everyone together. While other tools focus on specific areas such as data pipelines, ML workflows, or business processes, we wanted Kestra to be able to provide a more unified solution. Whether you’re running **ETL jobs**, managing infrastructure, or handling business processes, Kestra can orchestrate it. -Our first milestone as a company was the public release of Kestra. The response was overwhelmingly positive when we shared the project on Hacker News. That initial success brought us attention from developers and companies around the world, solidifying our belief that we were building something special. +We designed Kestra to be **language-agnostic**, meaning you can run tasks in **Java**, **Python**, **Go**, **JavaScript**, and more. By integrating with **Docker**, we made it easy to run highly specialized workflows, whether they’re in **Go** for performance reasons or in **R** for data science tasks. -A second boost came from an article on **InfoQ**, which further expanded our reach and helped Kestra gain credibility in the tech community. By the time we raised our **pre-seed** funding, it was clear that Kestra was evolving beyond just a data orchestration tool — we were creating a flexible platform that could handle a wide variety of use cases, from complex data pipelines to operational workflows. +## Plugins for Flexibility -### A Unified Approach to Orchestration +One of the things that makes Kestra stand out is its **plugin system**. We wanted the platform to work with a wide range of technologies, so we built it to be easily extendable. Today, Kestra has over **535 plugins**, with more on the roadmap. -The real power of Kestra lies in its **unified orchestration**. While many of our competitors focus on specific use cases—like orchestration for data pipelines, ML pipelines, or Business workflows—our goal is to provide a platform that can handle any type of workflow, regardless of the technology or use case. This means that whether you're orchestrating **ETL jobs**, managing **infrastructure with Terraform**, or handling **business processes**, Kestra can do it all. +Creating a plugin for Kestra is straightforward. We provide templates, so that anyone with basic **Java skills** can get started quickly. Most plugins are about 80% done just by following the **Quick Start guide**. We also run nightly tests on all plugins to make sure everything is up to date with any changes in technologies or APIs. -We’ve worked hard to ensure that Kestra is **language-agnostic**. You can run tasks in **Java**, **Python**, **Go**, **JavaScript**, or any other language. We achieved this by building tight integrations with **Docker** and leveraging containerization to isolate task executions. In fact, many of our clients are running highly specialized workflows, using languages like **Go** for performance-critical tasks. One of our clients even migrated all their processes from **Python to Go**, integrating it with Kestra to handle the orchestration. - -## Leveraging Plugins for Flexibility - -One of Kestra’s biggest strengths is the flexibility provided by our **plugin system**. When building Kestra, we wanted to ensure it could easily interact with a wide variety of technologies and ecosystems. That’s why we designed the platform with an architecture that makes it easy to create and extend plugins. Today, Kestra has **over 535 plugins**, and we’re aiming to double that number in the near future. - -Creating a plugin for Kestra is straightforward. We provide templates that make it simple for any **Java developer** to get started. In most cases, a plugin is 80% complete just by copying the **Quick Start guide** from the technology it’s integrating with. The heavy lifting happens behind the scenes, so the complexity is abstracted away from the developer. We also ensure high quality by running **nightly tests** on all plugins, checking for changes in technology or API updates. - -We’ve designed Kestra in such a way that users can interact with any technology through plugins, whether it's **cloud services**, **databases**, or even **custom in-house systems**. One of the major architectural choices we made was to normalize data exchange between tasks using **Amazon Ion**, a format that extends JSON but includes native support for data types like **date**, **time**, and **timestamps** with time zones. This makes sure that data is typed correctly and managed efficiently across tasks. It’s 2024, and it’s no longer acceptable for dates to be strings! - -We didn’t over-engineer the system either. While we use typed data formats like Ion, we don’t require users to define strict schemas (like Avro, Json Schema, ...) between tasks. This was a deliberate decision. We wanted to keep Kestra flexible and easy to use without adding unnecessary complexity. If users need strict data quality control, we provide plugins to handle that, but it’s not enforced by default. - -We’ve always maintained control over our core plugins to ensure quality. When community members contribute a plugin, we review it, merge it into the core if it fits, and then take over its long-term maintenance. This approach helps us avoid the risks that come with community-contributed plugins becoming outdated or inconsistent in quality. +We’ve kept control of core plugins to ensure quality, and we also maintain community-contributed plugins to avoid issues with outdated or low-quality code. --- -## Open Source at Core and Enterprise Ready - -Our open-source philosophy is in Kestra’s DNA. The open-source community is essential to our growth, and we continue to listen and evolve based on their feedback. Kestra is designed to be **open-source first**, which means that even though we offer an **Enterprise version** with features like **SSO, secret management,** and **advanced scalability options**, anyone can run the open-source version in production. +## Open Source and Enterprise Ready -Despite the challenges of balancing **open-source** and **commercial** strategies, we strongly believe in the benefits of open-source. It has allowed us to reach a global audience and gain adoption in industries we might never have touched otherwise. For example, we know that large companies in China have been using Kestra for years without ever contacting us directly. This is part of the nature of open-source, and while we might not convert every user to an Enterprise customer, the visibility and validation we gain through widespread usage is invaluable. - ---- +At its core, Kestra is an open-source project. We’ve always believed that open-source is key to our growth, and we continue to evolve based on feedback from the community. While we do offer an **Enterprise version** with advanced features, the open-source version of Kestra is also capable of running in single-player production environments. -To meet these enterprise demands, Kestra is fully **scalable** and **distributed**. We’ve split the system into microservices, with a **Scheduler**, an **Executor**, and **Workers** that can run across multiple environments. Each service can scale both **horizontally** and **vertically** to handle varying workloads. For organizations with unique infrastructure requirements, we also offer **worker groups**, which allow specific tasks to be run on specialized hardware, like **GPU** or **Windows-based** environments. +The open-source nature of Kestra has allowed us to reach users all over the world, including places like China, where large companies have been using Kestra for years without ever contacting us directly. We might not convert every user into an enterprise customer, but the reach and credibility we gain from open-source adoption means a lot to us. -We’ve also added **task runners** in our Enterprise version, which allow users to delegate the execution of tasks to remote engines like **Kubernetes**, **AWS Batch**, **Azure Batch**, **Google Batch**, or **Cloud Run**. This enables users to take advantage of **serverless** execution when necessary, while still keeping the core system performant for high-throughput workloads. +To meet the needs of larger organizations, Kestra is fully **scalable** and **distributed**. We’ve split the system into microservices, with each service capable of scaling to handle a variety of workloads. For teams with specific infrastructure needs, we offer **worker groups**, allowing tasks to run on specialized hardware, like **GPU** or **Windows-based** environments. --- -## Building for the Future +## What’s Next? -Looking ahead, our ambition is to make Kestra the go-to tool for workflow orchestration across industries. We don’t want to be known as just a **data orchestrator**. Instead, our vision is to create a **category leader**—a platform that can orchestrate everything, from **business processes** to **infrastructure automation**. +As we look ahead, we want to make Kestra the go-to tool for orchestration across industries. We’re not just focused on data orchestration — we’re building a platform that can manage everything from **business processes** to **infrastructure automation**. -While we’ve grown quickly, with a team nearing **20 people**, we’re far from finished. We’re continuing to build out Kestra’s capabilities, and part of that includes expanding our **Cloud offering**. Right now, we have a **private beta** of Kestra Cloud, where we’re working with select customers to ensure it meets their needs. The challenge with running Kestra in the cloud is that it needs to handle an incredibly diverse range of workloads, from high-memory tasks to network-heavy operations. We’re taking our time to get it right because we want to offer a **reliable, secure, and performant** cloud service that can scale with any workload. +While our team is growing and our cloud offering is in **private Alpha**, we’re taking our time to make sure we get it right. The goal is to offer a reliable, secure, and performant cloud service that can accommodate the needs of any organization, from startups to enterprises. ::alert{type="info"} If you have any questions, reach out via [Slack](https://kestra.io/slack) or open [a GitHub issue](https://github.com/kestra-io/kestra). If you like the project, give us [a GitHub star](https://github.com/kestra-io/kestra) and join [the community](https://kestra.io/slack). -:: \ No newline at end of file +:: From 5bc7cb38e388694b9dbadca015a32702046820da Mon Sep 17 00:00:00 2001 From: Anna Geller Date: Mon, 7 Oct 2024 23:15:17 +0200 Subject: [PATCH 4/6] cut too much --- content/blogs/2024-10-07-3-years-of-kestra.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/content/blogs/2024-10-07-3-years-of-kestra.md b/content/blogs/2024-10-07-3-years-of-kestra.md index ce221d5e86..795af5b352 100644 --- a/content/blogs/2024-10-07-3-years-of-kestra.md +++ b/content/blogs/2024-10-07-3-years-of-kestra.md @@ -62,6 +62,8 @@ The open-source nature of Kestra has allowed us to reach users all over the worl To meet the needs of larger organizations, Kestra is fully **scalable** and **distributed**. We’ve split the system into microservices, with each service capable of scaling to handle a variety of workloads. For teams with specific infrastructure needs, we offer **worker groups**, allowing tasks to run on specialized hardware, like **GPU** or **Windows-based** environments. +We’ve also added **task runners** in our Enterprise version, which allow users to delegate the execution of tasks to remote engines like **Kubernetes**, **AWS Batch**, **Azure Batch**, **Google Batch**, or **Cloud Run**. This enables users to take advantage of **serverless** execution when necessary, while still keeping the core system performant for high-throughput workloads. + --- ## What’s Next? From c653ec235a8813537ed4cb80fc89a78222f26ea0 Mon Sep 17 00:00:00 2001 From: Anna Geller Date: Mon, 7 Oct 2024 23:17:11 +0200 Subject: [PATCH 5/6] Update 2024-10-07-3-years-of-kestra.md --- content/blogs/2024-10-07-3-years-of-kestra.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/content/blogs/2024-10-07-3-years-of-kestra.md b/content/blogs/2024-10-07-3-years-of-kestra.md index 795af5b352..3a0388c927 100644 --- a/content/blogs/2024-10-07-3-years-of-kestra.md +++ b/content/blogs/2024-10-07-3-years-of-kestra.md @@ -68,9 +68,9 @@ We’ve also added **task runners** in our Enterprise version, which allow users ## What’s Next? -As we look ahead, we want to make Kestra the go-to tool for orchestration across industries. We’re not just focused on data orchestration — we’re building a platform that can manage everything from **business processes** to **infrastructure automation**. +As we look ahead, we want to make Kestra the go-to tool for orchestration across industries. We’re not just focused on data orchestration — we’re building a platform that can manage all workflows from **business processes** to **infrastructure automation**. -While our team is growing and our cloud offering is in **private Alpha**, we’re taking our time to make sure we get it right. The goal is to offer a reliable, secure, and performant cloud service that can accommodate the needs of any organization, from startups to enterprises. +While our team is growing and our cloud offering is in **private Beta**, we’re taking our time to make sure we get it right. The goal is to offer a reliable, secure, and performant cloud service that can accommodate the needs of any organization, from startups to enterprises. ::alert{type="info"} If you have any questions, reach out via [Slack](https://kestra.io/slack) or open [a GitHub issue](https://github.com/kestra-io/kestra). From 794c3d94398e1ca459b93d4d4e16eff7baec4406 Mon Sep 17 00:00:00 2001 From: Anna Geller Date: Mon, 7 Oct 2024 23:22:36 +0200 Subject: [PATCH 6/6] tone down boastful language --- content/blogs/2024-10-07-3-years-of-kestra.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/content/blogs/2024-10-07-3-years-of-kestra.md b/content/blogs/2024-10-07-3-years-of-kestra.md index 3a0388c927..3d824ae76a 100644 --- a/content/blogs/2024-10-07-3-years-of-kestra.md +++ b/content/blogs/2024-10-07-3-years-of-kestra.md @@ -24,7 +24,7 @@ Leroy Merlin was dealing with a data architecture that was getting outdated and Kestra was designed to tackle Leroy Merlin’s issues—**scalability, flexibility**, and making things easier for engineers across various teams. The idea was to build something that could handle their massive data operations but still be user-friendly for engineers from different teams, whether they worked in data, BI, or infrastructure. -After months of development and testing, Kestra went live at Leroy Merlin, and it transformed their workflows. What used to take multiple teams days to complete was now done in hours, with engineers able to manage their own workflows independently. +After months of development and testing, Kestra went live at Leroy Merlin, and it improved their workflows significantly. What used to take multiple teams days to complete was now done in hours, with engineers able to manage their own workflows independently. By **August 2020**, Leroy Merlin upgraded to the **Enterprise Edition of Kestra**, which brought even more features like role-based access control and advanced security, further improving their processes. Today, Kestra runs over **400,000 executions** and **3 million tasks** per month at Leroy Merlin, handling everything from real-time data transfers to long-running data science workflows. @@ -36,7 +36,7 @@ Our goal from the start was to create an orchestration platform that was simple ## Early Milestones: GitHub, Hacker News, and InfoQ -Our first big milestone was Kestra’s public release. The feedback was incredible when we shared it on Hacker News, and that helped us gain attention from developers and companies around the world. An article on **InfoQ** followed, which brought even more visibility and credibility to Kestra. By the time we raised our **pre-seed funding**, it was clear that Kestra was evolving beyond just a data orchestration tool — it was a flexible platform that could handle all kinds of use cases, from data pipelines to operational workflows. +Our first big milestone was Kestra’s public release. We received encouraging feedback when we shared it on Hacker News, and that helped us gain attention from developers and companies around the world. An article on **InfoQ** followed, which brought even more visibility and credibility to Kestra. By the time we raised our **pre-seed funding**, it was clear that Kestra was evolving beyond just a data orchestration tool — it was a flexible platform that could handle all kinds of use cases, from data pipelines to operational workflows. ### One Platform for Company-Wide Orchestration @@ -46,11 +46,11 @@ We designed Kestra to be **language-agnostic**, meaning you can run tasks in **J ## Plugins for Flexibility -One of the things that makes Kestra stand out is its **plugin system**. We wanted the platform to work with a wide range of technologies, so we built it to be easily extendable. Today, Kestra has over **535 plugins**, with more on the roadmap. +One of the things that makes Kestra stand out is its **plugin system**. We wanted the platform to work with a wide range of technologies, so we built it to be easily extendable. Kestra currently supports over 535 plugins, and we continue to expand that. -Creating a plugin for Kestra is straightforward. We provide templates, so that anyone with basic **Java skills** can get started quickly. Most plugins are about 80% done just by following the **Quick Start guide**. We also run nightly tests on all plugins to make sure everything is up to date with any changes in technologies or APIs. +We've made it easy to create plugins for Kestra, offering templates to help developers get started quickly. Most plugins can be built using our **Quick Start guide** and are usually about 80% complete from the get-go. To ensure everything works reliably, we regularly test our plugins to keep them up-to-date with any changes in technologies or APIs. -We’ve kept control of core plugins to ensure quality, and we also maintain community-contributed plugins to avoid issues with outdated or low-quality code. +We also maintain control over core plugins, and for community-contributed plugins, we help ensure they stay updated and reliable over time. ---