Skip to content

Commit

Permalink
Support missing samples in oximeter
Browse files Browse the repository at this point in the history
- Add a `Datum::Missing` and `MissingDatum`, which records the intended
  datum type and an optional start time for a sample which could not be
  produced.
- Database upgrades which make all scalar datum columns Nullable. Array
  fields are _not_ made Nullable, since ClickHouse doesn't support
  composite types like arrays inside a Nullable wrapper type. The empty
  array is used as a sentinel, which is OK since we can't have
  zero-length array histograms in Oximeter. Add a test which will fail
  if we ever change that.
- Rework database serialization to handle Nullable types or empty
  arrays. This uses a new helper trait to convert a NULL (which has no
  type information) to the intended datum type, or an empty array to a
  histogram.
- Add a test for each measurement type that we can recover a missing
  sample of that type -- NULLs for scalar values and empty arrays for
  histograms.
  • Loading branch information
bnaecker committed Nov 28, 2023
1 parent 91b0261 commit d5095fd
Show file tree
Hide file tree
Showing 59 changed files with 834 additions and 466 deletions.
1 change: 1 addition & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

232 changes: 0 additions & 232 deletions oximeter/db/notes.txt

This file was deleted.

1 change: 1 addition & 0 deletions oximeter/db/schema/replicated/4/up01.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
ALTER TABLE oximeter.measurements_bool_local MODIFY COLUMN datum Nullable(UInt8)
1 change: 1 addition & 0 deletions oximeter/db/schema/replicated/4/up02.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
ALTER TABLE oximeter.measurements_bool MODIFY COLUMN datum Nullable(UInt8)
1 change: 1 addition & 0 deletions oximeter/db/schema/replicated/4/up03.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
ALTER TABLE oximeter.measurements_i8_local MODIFY COLUMN datum Nullable(Int8)
1 change: 1 addition & 0 deletions oximeter/db/schema/replicated/4/up04.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
ALTER TABLE oximeter.measurements_i8 MODIFY COLUMN datum Nullable(Int8)
1 change: 1 addition & 0 deletions oximeter/db/schema/replicated/4/up05.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
ALTER TABLE oximeter.measurements_u8_local MODIFY COLUMN datum Nullable(UInt8)
1 change: 1 addition & 0 deletions oximeter/db/schema/replicated/4/up06.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
ALTER TABLE oximeter.measurements_u8 MODIFY COLUMN datum Nullable(UInt8)
1 change: 1 addition & 0 deletions oximeter/db/schema/replicated/4/up07.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
ALTER TABLE oximeter.measurements_i16_local MODIFY COLUMN datum Nullable(Int16)
1 change: 1 addition & 0 deletions oximeter/db/schema/replicated/4/up08.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
ALTER TABLE oximeter.measurements_i16 MODIFY COLUMN datum Nullable(Int16)
1 change: 1 addition & 0 deletions oximeter/db/schema/replicated/4/up09.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
ALTER TABLE oximeter.measurements_u16_local MODIFY COLUMN datum Nullable(UInt16)
1 change: 1 addition & 0 deletions oximeter/db/schema/replicated/4/up10.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
ALTER TABLE oximeter.measurements_u16 MODIFY COLUMN datum Nullable(UInt16)
1 change: 1 addition & 0 deletions oximeter/db/schema/replicated/4/up11.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
ALTER TABLE oximeter.measurements_i32_local MODIFY COLUMN datum Nullable(Int32)
1 change: 1 addition & 0 deletions oximeter/db/schema/replicated/4/up12.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
ALTER TABLE oximeter.measurements_i32 MODIFY COLUMN datum Nullable(Int32)
1 change: 1 addition & 0 deletions oximeter/db/schema/replicated/4/up13.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
ALTER TABLE oximeter.measurements_u32_local MODIFY COLUMN datum Nullable(UInt32)
1 change: 1 addition & 0 deletions oximeter/db/schema/replicated/4/up14.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
ALTER TABLE oximeter.measurements_u32 MODIFY COLUMN datum Nullable(UInt32)
1 change: 1 addition & 0 deletions oximeter/db/schema/replicated/4/up15.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
ALTER TABLE oximeter.measurements_i64_local MODIFY COLUMN datum Nullable(Int64)
1 change: 1 addition & 0 deletions oximeter/db/schema/replicated/4/up16.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
ALTER TABLE oximeter.measurements_i64 MODIFY COLUMN datum Nullable(Int64)
1 change: 1 addition & 0 deletions oximeter/db/schema/replicated/4/up17.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
ALTER TABLE oximeter.measurements_u64_local MODIFY COLUMN datum Nullable(UInt64)
1 change: 1 addition & 0 deletions oximeter/db/schema/replicated/4/up18.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
ALTER TABLE oximeter.measurements_u64 MODIFY COLUMN datum Nullable(UInt64)
1 change: 1 addition & 0 deletions oximeter/db/schema/replicated/4/up19.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
ALTER TABLE oximeter.measurements_f32_local MODIFY COLUMN datum Nullable(Float32)
1 change: 1 addition & 0 deletions oximeter/db/schema/replicated/4/up20.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
ALTER TABLE oximeter.measurements_f32 MODIFY COLUMN datum Nullable(Float32)
1 change: 1 addition & 0 deletions oximeter/db/schema/replicated/4/up21.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
ALTER TABLE oximeter.measurements_f64_local MODIFY COLUMN datum Nullable(Float64)
1 change: 1 addition & 0 deletions oximeter/db/schema/replicated/4/up22.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
ALTER TABLE oximeter.measurements_f64 MODIFY COLUMN datum Nullable(Float64)
1 change: 1 addition & 0 deletions oximeter/db/schema/replicated/4/up23.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
ALTER TABLE oximeter.measurements_cumulativei64_local MODIFY COLUMN datum Nullable(Int64)
1 change: 1 addition & 0 deletions oximeter/db/schema/replicated/4/up24.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
ALTER TABLE oximeter.measurements_cumulativei64 MODIFY COLUMN datum Nullable(Int64)
1 change: 1 addition & 0 deletions oximeter/db/schema/replicated/4/up25.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
ALTER TABLE oximeter.measurements_cumulativeu64_local MODIFY COLUMN datum Nullable(UInt64)
1 change: 1 addition & 0 deletions oximeter/db/schema/replicated/4/up26.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
ALTER TABLE oximeter.measurements_cumulativeu64 MODIFY COLUMN datum Nullable(UInt64)
1 change: 1 addition & 0 deletions oximeter/db/schema/replicated/4/up27.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
ALTER TABLE oximeter.measurements_cumulativef32_local MODIFY COLUMN datum Nullable(Float32)
1 change: 1 addition & 0 deletions oximeter/db/schema/replicated/4/up28.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
ALTER TABLE oximeter.measurements_cumulativef32 MODIFY COLUMN datum Nullable(Float32)
1 change: 1 addition & 0 deletions oximeter/db/schema/replicated/4/up29.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
ALTER TABLE oximeter.measurements_cumulativef64_local MODIFY COLUMN datum Nullable(Float64)
1 change: 1 addition & 0 deletions oximeter/db/schema/replicated/4/up30.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
ALTER TABLE oximeter.measurements_cumulativef64 MODIFY COLUMN datum Nullable(Float64)
1 change: 1 addition & 0 deletions oximeter/db/schema/replicated/4/up31.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
ALTER TABLE oximeter.measurements_string_local MODIFY COLUMN datum Nullable(String);
1 change: 1 addition & 0 deletions oximeter/db/schema/replicated/4/up32.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
ALTER TABLE oximeter.measurements_string MODIFY COLUMN datum Nullable(String);
Loading

0 comments on commit d5095fd

Please sign in to comment.