perf: tweak sample rate further

Logflare · Jun 24, 2024 · 5fea4f3 · 5fea4f3
1 parent 59bf246
commit 5fea4f3
Show file tree

Hide file tree

Showing 2 changed files with 2 additions and 1 deletion.
diff --git a/docs/docs.logflare.com/docs/concepts/ingestion/index.mdx b/docs/docs.logflare.com/docs/concepts/ingestion/index.mdx
@@ -104,7 +104,7 @@ metadata: {
 :::note
 On high ingestion volume, Logflare will sample incoming events instead of checking each event. The sample rate decreases as the ingestion rate increases. Ingestion rates are compared only on an individual local server that is performing the ingestion.
 
-From 10-100 events per second, sample rate is 0.2. From 100-1000 events per second, sample rate is 0.1. Above 1000 events per second, sample rate is 0.01.
+From 10-100 events per second, sample rate is 0.2. From 100-500 events per second, sample rate is 0.1. From 500-1000 events per second, sample rate is 0.05. Above 1000 events per second, sample rate is 0.01.
 :::
 
 ### Key Transformation

diff --git a/lib/logflare/source/bigquery/pipeline.ex b/lib/logflare/source/bigquery/pipeline.ex
@@ -195,6 +195,7 @@ defmodule Logflare.Source.BigQuery.Pipeline do
     probability =
       case PubSubRates.Cache.get_local_rates(source.token) do
         %{average_rate: avg} when avg > 1000 -> 0.01
+        %{average_rate: avg} when avg > 500 -> 0.05
         %{average_rate: avg} when avg > 100 -> 0.1
         %{average_rate: avg} when avg > 10 -> 0.2
         _ -> 1