RSVP Copy Phrase Simulator #315

sreekaroo · 2024-02-16T00:13:29Z

Overview

BciPy Simulator infrastructure.

Issues

https://www.pivotaltracker.com/story/show/185958637

Discussion

See the README for usage information, key components, and current limitations.

…ping workssss

…ype being a DataFrame rather than a List[Series]

…mentation.

…ided DataEngine with a query interface to allow the engine to abstract out the details about how the data is stored. Added a mechanism to the Copy Phrase simulation Task to match SignalModels to Samplers and used this in the compute_device_evidence method to support gathering evidence from multiple sources.

Multimodal simulator

Simulator Cleanup

celikbasak · 2024-09-23T23:49:58Z

bcipy/simulator/data/data_engine.py

+        """Query the data."""
+
+
+def convert_trials(data_source: ExtractedExperimentData) -> List[Trial]:


Can this method also provide a list of Inquiries? Or, can it leave the structure as-is if the data is extracted by inquiries already? (Inquiries would later be useful for gaze simulation)

celikbasak · 2024-09-23T23:53:41Z

bcipy/simulator/data/sampler.py

+        trials = self.sample(state)
+        return self.reshaped(trials)
+
+    def sample_with_context(self,


Does the context here include the information about where the sample is taken from (e.g. experiment #, trigger #) ?

Yeah, that's the idea. I expect some of this will require refinement when we implement the specifics for Gaze.

celikbasak · 2024-09-23T23:56:05Z

bcipy/simulator/task/copy_phrase.py

+    return model.metadata.evidence_type or DEFAULT_EVIDENCE_TYPE
+
+
+class SimulatorCopyPhraseTask(RSVPCopyPhraseTask):


I really like the clear method names here.

celikbasak · 2024-09-23T23:57:44Z

bcipy/simulator/task/copy_phrase.py

+
+        for model in self.signal_models:
+            sampler = self.samplers[model]
+            # This assumes that sampling is independent. Changes to the sampler API are needed if


Does that mean we assume a uniform LM throughout the simulation?

The language model evidence gets added in a different method (in the copy phrase wrapper initialize_series). This method specifically gets evidence from devices. So any language model can be configured for simulation.

celikbasak · 2024-09-24T00:00:03Z

bcipy/simulator/task/copy_phrase.py

+    def exit_display(self) -> None:
+        """Close the UI and cleanup."""
+
+    def elapsed_seconds(self) -> float:


Is it possible to estimate the just the computation time per symbol selection / next symbol querying? I think it would be more realistic if the formula here becomes something like: elapsed_time = inquiry_length + computation_time

The computation of time is surprisingly complicated. There is an estimate in the acquisition helper for how much time an inquiry will take (for acquisition buffering purposes), but there are some confounding factors. I will create a followup ticket because I want to give this more thought and perhaps provide a way for this to be configured.

celikbasak

Looks great! The class names and methods are much more clear after cleanup/refactoring.

tab-cmd

This is a considerable feature addition! Thank you for carrying this over for us. It seems beneficial enough to add some documentation to the main README and potentially to the install script itself for top-level client use.

    entry_points={
        'console_scripts': ['bcipy = bcipy.main:bcipy_main', 'bcipy-sim = bcipy.simulator'],
    }

# OR if removing __main__

    entry_points={
        'console_scripts': ['bcipy = bcipy.main:bcipy_main', 'bcipy-sim = bcipy.simulator.task.task_runner:main'],
    },

Then one could use,

bcipy-sim <args>

tab-cmd · 2024-09-24T17:39:36Z

bcipy/helpers/stimuli.py

@@ -401,12 +401,11 @@ def __call__(self,
        return np.stack(reshaped_trials, 1), targetness_labels


-def update_inquiry_timing(timing: List[List[float]], downsample: int) -> List[List[float]]:


We would want to update the expected return from our reshapers to make this compatible. I think this is still correct, given we expect a sample instead of a time at this stage?

tab-cmd · 2024-09-24T17:41:01Z

bcipy/simulator/README.md

+
+* Only provides EEG support
+* Only one sampler maybe provided for all devices. Ideally we should support a different sampling strategy for each device.
+* Only the RSVP Copy Phrase may be simulated


You list Matrix CopyPhrase in the documentation, and if this is true, we should update the example above to avoid confusion. It seems like we could do MatrixCopyPhrase?

tab-cmd · 2024-09-24T17:42:53Z

bcipy/simulator/__main__.py

@@ -0,0 +1,6 @@
+""" Entry point to run Simulator """
+
+from bcipy.simulator.task import task_runner


Is this behavior needed? I try to avoid putting functionality in init or main as much as possible since it can be harder to trace and understand.

This allows you to run the module. It's not necessary, but if not here I would change the usage instructions. I don't have a preference here so I'm fine changing it.

tab-cmd · 2024-09-24T17:43:12Z

bcipy/simulator/data/data_engine.py

+        eeg - EEG data associated with this trial
+    """
+    source: str
+    # series_n: int


tab-cmd · 2024-09-24T17:44:26Z

bcipy/simulator/data/data_engine.py

+    for querying the processed data."""
+
+    def load(self):
+        """Load data from sources."""


It's not necessary, but often we use the ... when no return is set.

tab-cmd · 2024-09-24T18:09:28Z

bcipy/task/paradigm/rsvp/copy_phrase.py

-             'feedback_stim_height': self.parameters['info_height'],
-             'feedback_duration': self.parameters['feedback_duration']},
-            self.experiment_clock)
+        self.feedback = self.init_feedback()


I like this too!

tab-cmd · 2024-09-24T18:09:59Z

bcipy/task/paradigm/rsvp/copy_phrase.py

        if save:
            self.write_session_data()
        if decision_made:
            self.session.add_series()

+    def elapsed_seconds(self) -> float:


tab-cmd · 2024-09-24T18:16:00Z

scripts/python/replay_session.py

-    DEFAULT_DEVICE_SPEC_FILENAME,
-)
-from bcipy.helpers.acquisition import analysis_channels
+


I like that we can still use this script, but I do worry about its long-term survival. Could we add a replay feature to the simulator? It would permit slightly different questions and add a sampling strategy. The visualization was also beneficial (and the most used to communicate outwardly) and, I think, could be used more generally in the simulator output.

I'd be okay with deprecating this script (and most others) before the 2.0 release, even if it can't be rolled in.

I didn't actually make any changes to this script, but saving it organized the imports in my editor. Replicating this behavior should be fairly easy to do in the simulator. I think it's just a matter of having a stateful sampler.

tab-cmd · 2024-09-24T18:23:29Z

.gitignore

+
+bcipy/simulator/tests/resource/
+!bcipy/simulator/data


We have /data earlier in the file, but this was too general and ignoring the data module in the simulator directory. The ! indicates not, so don't ignore this one.

tab-cmd · 2024-09-24T18:27:21Z

bcipy/simulator/README.md

+`main.py` is the entry point for program. After following `BciPy` readme steps for setup, run the module from terminal:
+
+```
+(venv) $ python bcipy/simulator -h


We would want to document other clients here or in a linked README (ex. helpers/state). Otherwise, we could move those clients to a demo if they will not be used often.

sreekaroo and others added 30 commits September 7, 2023 13:06

replay session refactored.1 + LM evidence

1e3c3e6

replay session refactored.1 minus LM evidence support

f6f5bb0

Refactored replay_session into Simulator object basic functionality

eaf4bf7

simulator with sampling design

d666df5

documentation and seperated files

e377505

resolve PR comments

bc202ad

run loop logic and decision making. draft

251cf43

integrated evidence fuser to calculate likelihoods

bcd3a2c

fuser name fix

0d36823

using best selection for inqiuiry stimuli

e391010

Logging output to file. Sampler reshaper fixes. Single data source ty…

8731cf5

…ping workssss

prettier logging

f4faea1

typing incorrect decisions and backspace support

5713b7c

cli runnable simulator

224eb32

time step sleep in sim_parameters.json

bfb4ecc

recording metrics and storing series_results bug fix

5311d97

Fixes an issue with logging samples that was caused by the provided t…

be6b292

…ype being a DataFrame rather than a List[Series]

sim README.md

4040285

deleting first sim draft

dd5932c

linting

810b6c9

pylint clean ups

f5932a6

model handler pickle load in constructor

a03ae36

language model support + evidence fusing bug fix

9b946ae

singular data folder wrapper

867c7d9

supporting trial_length param for data processing

9d7f3a6

adding resource file to data_engine test

6cc0c51

backspace prob included in lm evidence

e91f1ff

total_time_spent estimate

ee7bb01

refactor logging configuration

95c16fd

minor log clean

0671627

lawhead added 15 commits September 3, 2024 17:04

Refactored TaskFactory into its own module. Cleaned up types and docu…

0887d8c

…mentation.

Sim task cleanup

7066c35

Fixed unit tests and linting errors

bd45e48

Merge pull request #345 from CAMBI-tech/data-loading-refinements

8d80504

Multimodal simulator

Delete unused or redundant code

96a8382

Improved documentation

c0fb47e

Set simulator output to data folder. Re-organized modules

36ebc40

Fix issue with gitignore

980b93a

Added functionality for debugging simulator output

bd29127

Fixed linting issue

d38dd66

Merge pull request #350 from CAMBI-tech/clean-unused

4c898c5

Simulator Cleanup

Merged upstream changes

7fc3a01

Add missing parameters to the simulation params file

226c4cc

Revert changes on non-simulator modules

4af8825

lawhead marked this pull request as ready for review September 23, 2024 23:15

lawhead requested review from tab-cmd and celikbasak September 23, 2024 23:35

celikbasak reviewed Sep 23, 2024

View reviewed changes

celikbasak reviewed Sep 24, 2024

View reviewed changes

celikbasak approved these changes Sep 24, 2024

View reviewed changes

tab-cmd approved these changes Sep 24, 2024

View reviewed changes

lawhead added 3 commits September 25, 2024 15:53

Cleanup based on PR feedback

c211198

Added logging for missing parameters

2aeb419

Updated documentation

45524bb

lawhead merged commit edf838b into 2.0.0rc4 Sep 26, 2024
6 checks passed

lawhead deleted the s_simple_sampler_simulator branch September 26, 2024 00:19

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

RSVP Copy Phrase Simulator #315

RSVP Copy Phrase Simulator #315

sreekaroo commented Feb 16, 2024 •

edited by lawhead

Loading

celikbasak Sep 23, 2024

celikbasak Sep 23, 2024

lawhead Sep 24, 2024

celikbasak Sep 23, 2024

celikbasak Sep 23, 2024

lawhead Sep 24, 2024

celikbasak Sep 24, 2024 •

edited

Loading

lawhead Sep 24, 2024

celikbasak left a comment •

edited

Loading

tab-cmd left a comment

tab-cmd Sep 24, 2024

tab-cmd Sep 24, 2024

tab-cmd Sep 24, 2024

lawhead Sep 24, 2024

tab-cmd Sep 24, 2024

tab-cmd Sep 24, 2024

tab-cmd Sep 24, 2024

tab-cmd Sep 24, 2024

tab-cmd Sep 24, 2024

lawhead Sep 24, 2024

tab-cmd Sep 24, 2024

lawhead Sep 24, 2024

tab-cmd Sep 24, 2024

		"""Query the data."""


		def convert_trials(data_source: ExtractedExperimentData) -> List[Trial]:

		return model.metadata.evidence_type or DEFAULT_EVIDENCE_TYPE


		class SimulatorCopyPhraseTask(RSVPCopyPhraseTask):

		@@ -401,12 +401,11 @@ def __call__(self,
		return np.stack(reshaped_trials, 1), targetness_labels


		def update_inquiry_timing(timing: List[List[float]], downsample: int) -> List[List[float]]:

		@@ -0,0 +1,6 @@
		""" Entry point to run Simulator """

		from bcipy.simulator.task import task_runner


		bcipy/simulator/tests/resource/
		!bcipy/simulator/data

RSVP Copy Phrase Simulator #315

RSVP Copy Phrase Simulator #315

Conversation

sreekaroo commented Feb 16, 2024 • edited by lawhead Loading

Overview

Issues

Discussion

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

celikbasak Sep 24, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

celikbasak left a comment • edited Loading

Choose a reason for hiding this comment

tab-cmd left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sreekaroo commented Feb 16, 2024 •

edited by lawhead

Loading

celikbasak Sep 24, 2024 •

edited

Loading

celikbasak left a comment •

edited

Loading