Code

Live waveform data in Emap#66

Code

Open

jeremyestein wants to merge 108 commits intodevelopSAFEHR-data/emap:developfrom sk/waveform-devSAFEHR-data/emap:sk/waveform-devCopy head branch name to clipboard

+4,284-163

Commits

Commits on Jun 13, 2024

• First pass at adding waveform data to Emap

  jeremyestein

  committed
  be8944d

• Write test which checks some of the data and passes

  jeremyestein

  committed
  7911d23

• checkstyle fixes and make test quicker

  jeremyestein

  committed
  50d34f8

Commits on Jun 14, 2024

• Merge branch 'develop' into jeremy/hf-data

  jeremyestein

  authored
  36f7a84

Commits on Jun 19, 2024

• Clarify validation run docs

  jeremyestein

  committed
  4aff3a9

• In new version of checkstyle, the encoding tag is no longer valid, and

  Show description for 5407881
  jeremyestein

  committed
  5407881

Commits on Jun 20, 2024

• Remove duplicated file set_mvn_proxy.sh

  jeremyestein

  committed
  a6e11d9

• Bare bones waveform-reader service. Docker image builds, but is

  Show description for 39a1fa7
  jeremyestein

  committed
  39a1fa7

Commits on Jun 25, 2024

• Initialise the fake UDS properly; add fake UDS and waveform reader to

  Show description for cf27fe5
  jeremyestein

  committed
  cf27fe5

• Temporarily break the core proc RabbitMQ listener by only listening to

  Show description for ff7db6f
  jeremyestein

  committed
  ff7db6f

Commits on Jun 26, 2024

• Make data more realistic in terms of volume. Had to add some layers to

  Show description for d1c1c9c
  jeremyestein

  committed
  d1c1c9c

Commits on Jun 27, 2024

• Make message IDs more descriptive to enable better performance tracking

  jeremyestein

  committed
  b7f10f2

• Ten seconds of data per message

  jeremyestein

  committed
  a3db3fc

• call saveAll rather than save many times

  jeremyestein

  committed
  a16adbf

• Batch single-row INSERTs

  jeremyestein

  committed
  a209f90

• Couple of other tweaks, not sure what it'll do, but need a baseline

  Show description for 9615106
  jeremyestein

  committed
  9615106

• Add reWriteBatchedInserts option to db connection string so that batched

  Show description for e9bff1f
  jeremyestein

  committed
  e9bff1f

• Does this help?

  jeremyestein

  committed
  3193369

• Expicitly specify sequence Id generation so we don't have to query the

  Show description for 46ff9a4
  jeremyestein

  committed
  46ff9a4

Commits on Jul 1, 2024

• Can't use existing sequence as it has a different increment value

  jeremyestein

  committed
  5361af2

Commits on Jul 3, 2024

• Use SQL arrays (of double precision DB type)

  jeremyestein

  committed
  e2635c8

• Fix copy constructor

  jeremyestein

  committed
  7b9b01a

• Copy constructor checker didn't know about Double[]

  jeremyestein

  committed
  182d2c2

• Put data from multiple patients to test it doesn't get mixed up. Shuffle

  Show description for 5558e48
  jeremyestein

  committed
  5558e48

Commits on Jul 16, 2024

• Make observation time in synthetic data evenly spread out, as real data

  Show description for 49d93d9
  jeremyestein

  committed
  49d93d9

Commits on Jul 17, 2024

• Make number of patients, start date, and warp factor configurable to

  Show description for cd5567f
  jeremyestein

  committed
  cd5567f

Commits on Jul 18, 2024

• Add indexes to Waveform table

  jeremyestein

  committed
  55e0c60

• Cassandra is still running out of memory

  jeremyestein

  committed
  73619d4

Commits on Jul 19, 2024

• Make configurable the rabbitmq queues that core will listen on,

  Show description for 1c3cc09
  jeremyestein

  committed
  1c3cc09

• Make the synthetic data values use a variable amount of digits to test

  Show description for 52dbc3a
  jeremyestein

  committed
  52dbc3a

Commits on Jul 22, 2024

• Set an end date time for synthetic data and exit when this is reached,

  Show description for cd56584
  jeremyestein

  committed
  cd56584

• Add waveform-reader to validation script

  jeremyestein

  committed
  3fa518a

Commits on Jul 24, 2024

• Fix some copy paste errors

  jeremyestein

  committed
  f5f914b

Commits on Jul 25, 2024

• Schedule a task in the core proc to periodically delete old waveform

  Show description for 0b5b576
  jeremyestein

  committed
  0b5b576

Commits on Jul 26, 2024

• Receive messages on a TCP server port, with source IP filtering

  jeremyestein

  committed
  4113ba8

• Split into two services, one a fake for generating HL7 messages, and

  Show description for 8a9a2b0
  jeremyestein

  committed
  8a9a2b0

Commits on Jul 29, 2024

• Copied from project board

  Show description for 1a3442d
  jeremyestein

  committed
  1a3442d

Commits on Jul 30, 2024

• Merge pull request #42 from UCLH-DHCT/jeremy/hf-data

  Show description for ba238c2
  skeating

  authored
  ba238c2

Commits on Aug 8, 2024

• In a seperate process, generate synthetic messages to simulate an HL7

  Show description for 1d73817
  jeremyestein

  committed
  1d73817

Commits on Aug 9, 2024

• Generator does not need interchange

  jeremyestein

  committed
  51af8d9

• Remove more unnecessary dependencies

  jeremyestein

  committed
  cf878e6

Commits on Aug 12, 2024

• Merge pull request #22 from UCLH-DHCT/jeremy/hf-data

  Show description for 5eecc32
  jeremyestein

  authored
  5eecc32

Commits on Aug 14, 2024

• Infer location visit from time and location string

  jeremyestein

  committed
  25ba727

Commits on Aug 15, 2024

• Add visit observation type field (+ tests) so we know what the

  Show description for 27a154a
  jeremyestein

  committed
  27a154a

• Differentiate between source and mapped location in test

  jeremyestein

  committed
  8bb5ff0

• Test that the mapped stream name gets through too

  jeremyestein

  committed
  1dd7fbc

• Update emap-tests.yaml

  skeating

  authored
  f8e2840

• Fix copy constructor

  jeremyestein

  committed
  945b946

Commits on Aug 19, 2024

• Merge pull request #49 from UCLH-DHCT/jeremy/hf-data

  Show description for ddbfc4e
  jeremyestein

  authored
  ddbfc4e

• Generate real HL7 in the synth generator. Parse real HL7 with HAPI in

  Show description for f62dbad
  jeremyestein

  committed
  f62dbad

Commits on Aug 20, 2024

• Use correct line endings for generated HL7 messages

  jeremyestein

  committed
  52f0742

• Thin out the logging a bit

  jeremyestein

  committed
  ee34e0d

• Give the terser explicit indexes rather than building a query string;

  Show description for e08dcdb
  jeremyestein

  committed
  e08dcdb

Commits on Aug 21, 2024

• Use a simple, ad hoc parser for HL7. It's much quicker.

  jeremyestein

  committed
  f152dda

• Use the correct year in the date format!

  jeremyestein

  committed
  ceafdae

• Generate message timestamp as slightly after the observation timestamp

  jeremyestein

  committed
  fd45fed

• Make HL7 messages much, much smaller, more in line with the real ones.

  Show description for 33abe68
  jeremyestein

  committed
  33abe68

Commits on Aug 27, 2024

• Merge pull request #51 from UCLH-DHCT/jeremy/hf-data

  Show description for 3b901f3
  jeremyestein

  authored
  3b901f3

Commits on Aug 30, 2024

• First go at collating tiny Waveform messages into bigger messages that

  Show description for 3484c15
  jeremyestein

  committed
  3484c15

• Use sample count rather than time as the message size target/limit.

  Show description for 1b236b3
  jeremyestein

  committed
  1b236b3

• More fine-grained locking, should allow better multithreaded message

  Show description for ca110b5
  jeremyestein

  committed
  ca110b5

Commits on Sep 2, 2024

• Check number of samples in generated messages. Test what happens when

  Show description for 8ed5ae7
  jeremyestein

  committed
  8ed5ae7

Commits on Sep 9, 2024

• Fix magic number in gap checking, tidy up tests and other things

  jeremyestein

  committed
  a96a8b2

Commits on Sep 10, 2024

• Bring in metadata from ICU_HL7_Data_Extraction repo commit 05676f0aeb799919d40906dba7c4bc8ad7b6d2dc

  jeremyestein

  committed
  aa782f9

Commits on Sep 12, 2024

• Read stream metadata from CSV file and use it to describe the streams in

  Show description for af9dcf7
  jeremyestein

  committed
  af9dcf7

Commits on Sep 17, 2024

• Need the stream id to make a unique message ID! Now all the messages get

  Show description for e434619
  jeremyestein

  committed
  e434619

Commits on Sep 19, 2024

• Remove/downgrade excessive logging

  jeremyestein

  committed
  2f156df

• Add thread pools on the TCP listener and the HL7 message handler to see

  Show description for ee74b95
  jeremyestein

  committed
  ee74b95

• We need to allow lists of messages to queue to be from different

  Show description for 6ed92f2
  jeremyestein

  committed
  6ed92f2

• Merge pull request #52 from UCLH-DHCT/jeremy/hf-data-collate

  Show description for ea5098e
  jeremyestein

  authored
  ea5098e

Commits on Sep 23, 2024

• Define expected location mappings with realistic location strings

  jeremyestein

  committed
  d6fb58c

• Implement location mapping

  jeremyestein

  committed
  e47325c

Commits on Sep 25, 2024

• Fix Capsule bed names

  jeremyestein

  committed
  50a17a9

• Before looking at the gap validation problem, make the synth data more up to the current

  Show description for 2e0c48c
  jeremyestein

  committed
  2e0c48c

• Test the created visit_observation_type rows

  jeremyestein

  committed
  5d7f255

• Use only the unrounded timestamp when detecting gaps. Differentiate

  Show description for 390e03c
  jeremyestein

  committed
  390e03c

Commits on Sep 26, 2024

• Check that data out of the database resembles the original sine wave,

  Show description for 4673b45
  jeremyestein

  committed
  4673b45

• Add unit to waveform data

  jeremyestein

  committed
  6cdd7f5

Commits on Oct 1, 2024

• Message separator is not actually CRLF

  jeremyestein

  committed
  8e7ed08

Commits on Oct 2, 2024

• Read from a file containing an HL7 message dump from Smartlinx

  jeremyestein

  committed
  70d8e07

Commits on Oct 3, 2024

• Merge pull request #55 from UCLH-DHCT/jeremy/hf-data-metadata

  Show description for 2eba4de
  jeremyestein

  authored
  2eba4de

• Merge pull request #58 from UCLH-DHCT/jeremy/hf-data-location

  Show description for bc6132e
  jeremyestein

  authored
  bc6132e

• Merge pull request #59 from UCLH-DHCT/jeremy/hf-data-gaps

  Show description for a7f769f
  jeremyestein

  authored
  a7f769f

• Need to map the directory containing test data

  jeremyestein

  committed
  dfed36c

• Python setup script - allow independent switching of hoover, hl7 adt and

  Show description for 360fe56
  jeremyestein

  committed
  360fe56

• BooleanOptionalAction doesn't exist on python 3.8, do it ourselves

  jeremyestein

  committed
  ecba32f

• synth vs reader confusion

  jeremyestein

  committed
  9902814

• Don't parse values as numeric unless they are in fact numeric

  jeremyestein

  committed
  e178b6d

Commits on Oct 4, 2024

• Log, skip and continue if HL7 messages cause parsing errors

  jeremyestein

  committed
  387f9d4

• Logic for checking empty queues seems to be inverted

  jeremyestein

  committed
  c9f49fd

• microsecond/second mixup meant that timeout was never reached

  jeremyestein

  committed
  68b8ad1

• Handle missing segments without crashing

  jeremyestein

  committed
  21c4259

• String.substring needs to be in bounds

  jeremyestein

  committed
  c5a2465

• Translate more assorted parsing errors to Hl7ParseException for better

  Show description for 10474e0
  jeremyestein

  committed
  10474e0

Commits on Oct 7, 2024

• Fix ends of files

  jeremyestein

  committed
  1501016

• Merge pull request #61 from UCLH-DHCT/jeremy/hf-data-misc

  Show description for 4ed7c1a
  jeremyestein

  authored
  4ed7c1a

• Don't use waveform generator in validation

  jeremyestein

  committed
  4b96678

• Reproduce issue #25

  jeremyestein

  committed
  2d5872d

• Fix issue #25 - don't delete existing repo without warning when running

  Show description for e0d9c99
  jeremyestein

  committed
  e0d9c99

• Bring design doc more up to date. Still needs more work though.

  jeremyestein

  committed
  fd9d987

Commits on Oct 8, 2024

• Address review feedback

  jeremyestein

  committed
  b7819da

• Add tests for recently changed code

  jeremyestein

  committed
  498a495

Commits on Oct 9, 2024

• Update docs and rename property for clarity

  jeremyestein

  committed
  d2b0c8f

• Merge pull request #64 from UCLH-DHCT/jeremy/hf-data-setup

  Show description for 4d87fac
  jeremyestein

  authored
  4d87fac

Commits on Oct 23, 2024

• Address some review comments

  jeremyestein

  committed
  c635859

Commits on Oct 24, 2024

• Remove JPA options relating to insert/update batching, since waveform

  Show description for fa0936b
  jeremyestein

  committed
  fa0936b

• Make cassandra settings configurable with unchanged defaults suitable for GAE

  jeremyestein

  committed
  e4e6460

• Make timeout configurable as per review feedback

  jeremyestein

  committed
  7c4d9e3