Enabling reading "calibrated" data

[fschlueter]

Hi all,

I thought it makes sense to collimate the discussion around "providing a calibrated dataset" with mattak in an issue. For the moment I want to limit the discussion to how we apply the necessary "calibrations" on the fly rather than storing calibrated data to disc (this is a further discussion to have once we decide we want to do that).

Maybe first discuss the context of what "calibrations" we can or want to apply in mattak. Obvious is the voltage calibration, this is already implemented and thus I think we do not need to further discuss this. The second thing would the the cable delays. This is something which has been requested many times and I do see that this could be useful for some analysis (e.g., calculating if an impulse event arrived as a plane wave). However, I think we have to acknowledge that the cable delays are only truly meaningful when we also know the positions of the antennas (and to a lesser extend the ice model!)! Providing all this information exceeds what mattak should do in my opinion - for this we have NuRadio. Nonetheless, as I said introducing some cable delays might be beneficial for some analysis which do not need a high level of accuracy as no such high accuracy could be archived. As a consequence those cable delays do not need to be very accurate - this is also important for a suggestions below. I would be interested in opinions about this!

If we decide we want to add cable delays to mattak that would probably happen in form of an interface where one can ask the dataset to get the trace_start_times. A more important question is, where do we get this data from? One could of course ask the NuRadio detector class but this would add NuRadio as a significant dependency. If you use the hardware DB that also means that you need internet access if you do not use exported files. Another way would be to provide a simple lightway text or json files which stores these information and which could be added to this repository. The issue with that solution is versioning. And here comes a suggestion: If we store the cable delays as files in the repository those files are not versioned. Which in the end means we should not relay on them in an analysis we want to reproduce (or this analysis has to version those files/values explicitly themself which is a slippery road to go). Philipp suggested to never change them (and thus they are "versioned"). This is in principle possible since the accuracy requirements are not that high since we can not archive highest accuracy without channel positions anyway. However, this fails if we decommission a channel and recommission it.

Maybe the best way would be to implement both. A default which relies on un-versioned files and an optional mode which asks the detector class....

@cozzyd @philippwindischhofer

[cozzyd]

My feeling is that the best thing to do is to write a mattak::Detector class that contains cable delays and antenna geometry information (though, I suppose it could be extended to include impulse responses too in the future, if we really wanted to). This would include in it some sort of version tag.

This class could be instantiated from the database (which would require mongodb libraries/header files to be present on the system), from a saved instance in a ROOT file, or perhaps from a .json-like detector description (that... would include a version tag as well!).

Since the Detector class would know its version and can be saved with the data, I don't see a strong versioning problem with doing something like this.

I actually started implementing something like this in a private branch, but stopped when I realized the database does not currently have the cable delay / antenna geometry information needed yet.

[fschlueter]

maybe first things first. The database does include the antenna geometry and the cable delays.

[cozzyd]

Not the calibrated cable delays or geometry though (or maybe it does now?), which is what you would want?

[fschlueter]

How would you export that version tag when most people work with the python backends? My feeling is that this would create a lot of "redundant" work since we have something like this already in NuRadio. Maybe the question than is "what is mattak". I thought it is mostly a data format which includes some (limited) low-level data processing capabilities). What you are describing, to me, would move mattak more into a direction of a reconstruction framework.

[fschlueter]

Not the calibrated cable delays or geometry though (or maybe it does now?), which is what you would want?

well they are not yet in there because the calibration is not finished. But once they are in they will be "replacing" what is in the DB right now. Or I should but it differently, by default the NuRadio detector class will return with the same interface the calibrated values once they are added to the DB

[philippwindischhofer]

I think I agree with Felix that the foremost question we should answer is one of scope. Do we want mattak to calibrate out all sorts of detector effects, or limit it to its present role of providing an efficient low-level data access mechanism?

Both ways have their advantages and disadvantages, and out preference can evolve with time, of course.

In the short term, we have e.g. Bryan's and Jakob's CR analyses that are starting to fold in detector information from the DB. In that context, at least, I can see some value in making it as easy as possible to remove a rough measure of our cable delays. (Which doesn't factor in a lot of dependencies but would already be helpful.)

What do you think about discussing this with the current analyzers (and related people) in one of the upcoming calibration / analysis calls?