From af6854956ec66888c6994e69877cf02093bc7b7d Mon Sep 17 00:00:00 2001 From: MOHIT KUMAR KUSHWAHA Date: Mon, 10 Oct 2022 01:55:30 +0530 Subject: [PATCH] Create gcloud-dataflow.md Added information on Dataflow --- Data-streaming/gcloud-dataflow.md | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) create mode 100644 Data-streaming/gcloud-dataflow.md diff --git a/Data-streaming/gcloud-dataflow.md b/Data-streaming/gcloud-dataflow.md new file mode 100644 index 0000000..d6eb448 --- /dev/null +++ b/Data-streaming/gcloud-dataflow.md @@ -0,0 +1,16 @@ +## Google Cloud Dataflow + +Google Cloud Dataflow is a fully managed service for executing Apache Beam pipelines within the Google Cloud Platform ecosystem. + +Benefits: +- Streaming data analytics with speed: Dataflow enables fast, simplified streaming data pipeline development with lower data latency. +- Simplify operations and management: Allow teams to focus on programming instead of managing server clusters as Dataflow’s serverless approach removes operational overhead from data engineering workloads. +- Reduce total cost of ownership: Resource autoscaling paired with cost-optimized batch processing capabilities means Dataflow offers virtually limitless capacity to manage your seasonal and spiky workloads without overspending. + +Features: +- Autoscaling of resources and dynamic work rebalancing: Minimize pipeline latency, maximize resource utilization, and reduce processing cost per data record with data-aware resource autoscaling. Data inputs are partitioned automatically and constantly rebalanced to even out worker resource utilization and reduce the effect of “hot keys” on pipeline performance. +- Flexible scheduling and pricing for batch processing: For processing with flexibility in job scheduling time, such as overnight jobs, flexible resource scheduling (FlexRS) offers a lower price for batch processing. These flexible jobs are placed into a queue with a guarantee that they will be retrieved for execution within a six-hour window. +- Ready-to-use real-time AI patterns: Enabled through ready-to-use patterns, Dataflow’s real-time AI capabilities allow for real-time reactions with near-human intelligence to large torrents of events. Customers can build intelligent solutions ranging from predictive analytics and anomaly detection to real-time personalization and other advanced analytics use cases. + + +For more info: [https://cloud.google.com/dataflow](https://cloud.google.com/dataflow)