feat: tweak schema sampling

Logflare · Jun 24, 2024 · 59bf246 · 59bf246
1 parent abb4f9e
commit 59bf246
Show file tree

Hide file tree

Showing 2 changed files with 4 additions and 3 deletions.
diff --git a/docs/docs.logflare.com/docs/concepts/ingestion/index.mdx b/docs/docs.logflare.com/docs/concepts/ingestion/index.mdx
@@ -104,7 +104,7 @@ metadata: {
 :::note
 On high ingestion volume, Logflare will sample incoming events instead of checking each event. The sample rate decreases as the ingestion rate increases. Ingestion rates are compared only on an individual local server that is performing the ingestion.
 
-From 1000-5000 events per second, sample rate is 0.1. Above 5000 events per second, sample rate is 0.01.
+From 10-100 events per second, sample rate is 0.2. From 100-1000 events per second, sample rate is 0.1. Above 1000 events per second, sample rate is 0.01.
 :::
 
 ### Key Transformation

diff --git a/lib/logflare/source/bigquery/pipeline.ex b/lib/logflare/source/bigquery/pipeline.ex
@@ -194,8 +194,9 @@ defmodule Logflare.Source.BigQuery.Pipeline do
     # random sample if local ingest rate is above a certain level
     probability =
       case PubSubRates.Cache.get_local_rates(source.token) do
-        %{average_rate: avg} when avg > 5000 -> 0.01
-        %{average_rate: avg} when avg > 1000 -> 0.1
+        %{average_rate: avg} when avg > 1000 -> 0.01
+        %{average_rate: avg} when avg > 100 -> 0.1
+        %{average_rate: avg} when avg > 10 -> 0.2
         _ -> 1
       end