Implement support for Tr matrix with relative times in OPUS #67
Comments
|
Thanks for opening, and welcome here! Cheers, |
|
I've included a recording of ethanol in the attached zip. I think it is recorded in OPUS 8. I took an effort to scrap the name of the operator (and verified in OPUS 7 and using BrukerOpusReader (my fork) that the file is not corrupted) This plot shows the spectra of the two channels, with the time on the z-axis. Not a lot happens over time, except for some evaporation of the ethanol, but it is just as a simple example:
There is a total of 405 spectra per channel, so there must be 405 timesteps somewere as well. We saw in OPUS the TRC block (?):
If we open it we see a matrix with the times. The SRT column contains the relative times for the ATR channel, the ERT for the Transmission channel: I hope this example is clear. Thank you for your time! Cheers, Luuk |
|
Hi Luuk, thanks for the excellent example. I finally have some free time to work a bit on it and get some stuff testing. I'll get back to you once I make some progress/find something useful to share here and/or implement. |
|
Cool graph btw 💯 |
|
Just briefly reporting current state without time support. devtools::load_all()
#> ℹ Loading opusreader2
file <- system.file("extdata/new_data", "PyOReader-Testsample-EtOH.0",
package = "opusreader2"
)
data <- read_opus(dsn = file)
#> Error: Unknown header entry.
#> The following 'composite key' is not yet mapped in the {opusreader2} key-value map of the header:
#> * "b16-c28-t80-a0"
#> We encourage your contribution to feature this new OPUS block by opening a new issue on
#> https://github.com/spectral-cockpit/opusreader2/issues
#> Please
#> 1. report reproducibly, using short code with {opusreader2}
#> (recommended: https://reprex.tidyverse.org)
#> 2. describe briefly
#> a) Bruker instrument used
#> b) equipment
#> c) measurement mode and spectral blocks saved (OPUS settings)
#> d) OPUS software version
#> e) your general workflow for spectroscopic diagnostics
#> 3. provide an example OPUS binary file uploaded for public access
#> on GitHub (best in issue)
#> 4. to facilitate widespread support of Bruker devices in open source
#> based infrastructure, show the data blocks as print screens in the
#> Bruker OPUS software (right-click in Viewer). Please upload the
#> contents of all OPUS blocks in individual screenshots.Created on 2023-02-11 by the reprex package (v2.0.1) => starting to have a look at new block type 16. |
philipp-baumann
changed the title
|
for the moment, I need to name the devtools::load_all()
#> ℹ Loading opusreader2
file <- system.file("extdata/new_data", "PyOReader-Testsample-EtOH.0",
package = "opusreader2"
)
data <- read_opus(dsn = file)
#> Warning in readBin(con, what = "character", n = n, size = 1, endian =
#> "little"): incomplete string at end of file has been discarded
#> Warning in readBin(con, what = "character", n = n, size = 1, endian =
#> "little"): incomplete string at end of file has been discarded
names(data)
#> [1] "sc_ref_data_param" "sc_ref"
#> [3] "ig_ref_data_param" "ig_ref"
#> [5] "optics" "optics_ref"
#> [7] "acquisition_ref" "fourier_transformation_ref"
#> [9] "fourier_transformation" "sample"
#> [11] "acquisition" "instrument_ref"
#> [13] "instrument" "history"
str(data$sc_ref)
#> List of 10
#> $ block_type : int 11
#> $ channel_type : int 4
#> $ text_type : int 0
#> $ additional_type: int 0
#> $ offset : int 2803976
#> $ next_offset : int 2810640
#> $ chunk_size : int 1666
#> $ block_type_name: chr "sc_ref"
#> $ data : num [1, 1:1665] 0.225 0.212 0.207 0.221 0.235 ...
#> ..- attr(*, "dimnames")=List of 2
#> .. ..$ : NULL
#> .. ..$ : chr [1:1665] "3805.05761897564" "3803.12905454636" "3801.20049011707" "3799.27192568779" ...
#> $ wavenumbers : num [1:1665] 3805 3803 3801 3799 3797 ...
#> - attr(*, "class")= chr "data"
str(data$sc_ref_data_param)
#> List of 9
#> $ block_type : int 27
#> $ channel_type : int 4
#> $ text_type : int 0
#> $ additional_type: int 0
#> $ offset : int 2803800
#> $ next_offset : int 2803976
#> $ chunk_size : int 176
#> $ block_type_name: chr "sc_ref_data_param"
#> $ parameters :List of 10
#> ..$ DPF:List of 4
#> .. ..$ parameter_name : chr "DPF"
#> .. ..$ parameter_name_long: chr "Data Point Format"
#> .. ..$ parameter_value : int 1
#> .. ..$ parameter_type : chr "int"
#> ..$ NPT:List of 4
#> .. ..$ parameter_name : chr "NPT"
#> .. ..$ parameter_name_long: chr "Number of Data Points"
#> .. ..$ parameter_value : int 1665
#> .. ..$ parameter_type : chr "int"
#> ..$ FXV:List of 4
#> .. ..$ parameter_name : chr "FXV"
#> .. ..$ parameter_name_long: chr "Frequency of First Point"
#> .. ..$ parameter_value : num 3805
#> .. ..$ parameter_type : chr "float"
#> ..$ LXV:List of 4
#> .. ..$ parameter_name : chr "LXV"
#> .. ..$ parameter_name_long: chr "Frequency of Last Point"
#> .. ..$ parameter_value : num 596
#> .. ..$ parameter_type : chr "float"
#> ..$ CSF:List of 4
#> .. ..$ parameter_name : chr "CSF"
#> .. ..$ parameter_name_long: chr "Y - Scaling Factor"
#> .. ..$ parameter_value : num 1
#> .. ..$ parameter_type : chr "float"
#> ..$ MXY:List of 4
#> .. ..$ parameter_name : chr "MXY"
#> .. ..$ parameter_name_long: chr "Y - Maximum"
#> .. ..$ parameter_value : num 0.903
#> .. ..$ parameter_type : chr "float"
#> ..$ MNY:List of 4
#> .. ..$ parameter_name : chr "MNY"
#> .. ..$ parameter_name_long: chr "Y - Minimum"
#> .. ..$ parameter_value : num 0.00129
#> .. ..$ parameter_type : chr "float"
#> ..$ DAT:List of 4
#> .. ..$ parameter_name : chr "DAT"
#> .. ..$ parameter_name_long: chr "Date of Measurement"
#> .. ..$ parameter_value : chr "02/02/2023"
#> .. ..$ parameter_type : chr "str"
#> ..$ TIM:List of 4
#> .. ..$ parameter_name : chr "TIM"
#> .. ..$ parameter_name_long: chr "Time of Measurement"
#> .. ..$ parameter_value : chr "09:40:00.289 (GMT+1)"
#> .. ..$ parameter_type : chr "str"
#> ..$ DXU:List of 4
#> .. ..$ parameter_name : chr "DXU"
#> .. ..$ parameter_name_long: chr "X Units"
#> .. ..$ parameter_value : chr "WN"
#> .. ..$ parameter_type : chr "str"
#> - attr(*, "class")= chr "parameter"
sessioninfo::session_info()
#> ─ Session info ───────────────────────────────────────────────────────────────
#> setting value
#> version R version 4.1.3 (2022-03-10)
#> os Ubuntu 22.04.1 LTS
#> system x86_64, linux-gnu
#> ui X11
#> language (EN)
#> collate en_US.UTF-8
#> ctype en_US.UTF-8
#> tz Europe/Zurich
#> date 2023-02-13
#> pandoc 2.9.2.1 @ /usr/bin/ (via rmarkdown)
#>
#> ─ Packages ───────────────────────────────────────────────────────────────────
#> ! package * version date (UTC) lib source
#> backports 1.4.1 2021-12-13 [1] CRAN (R 4.1.2)
#> brio 1.1.3 2021-11-30 [1] CRAN (R 4.1.2)
#> cachem 1.0.6 2021-08-19 [1] CRAN (R 4.1.1)
#> callr 3.7.3 2022-11-02 [1] CRAN (R 4.1.3)
#> cli 3.6.0 2023-01-09 [1] CRAN (R 4.1.3)
#> crayon 1.5.2 2022-09-29 [1] CRAN (R 4.1.3)
#> desc 1.4.2 2022-09-08 [1] CRAN (R 4.1.3)
#> devtools 2.4.5 2022-10-11 [1] CRAN (R 4.1.3)
#> digest 0.6.31 2022-12-11 [1] CRAN (R 4.1.3)
#> ellipsis 0.3.2 2021-04-29 [1] CRAN (R 4.1.2)
#> evaluate 0.20 2023-01-17 [1] CRAN (R 4.1.3)
#> fansi 1.0.4 2023-01-22 [1] CRAN (R 4.1.3)
#> fastmap 1.1.0 2021-01-25 [1] CRAN (R 4.1.1)
#> fs 1.6.1 2023-02-06 [1] CRAN (R 4.1.3)
#> glue 1.6.2 2022-02-24 [1] CRAN (R 4.1.2)
#> htmltools 0.5.4 2022-12-07 [1] RSPM (R 4.1.0)
#> htmlwidgets 1.6.1 2023-01-07 [1] CRAN (R 4.1.3)
#> httpuv 1.6.8 2023-01-12 [1] CRAN (R 4.1.3)
#> knitr 1.42 2023-01-25 [1] CRAN (R 4.1.3)
#> later 1.3.0 2021-08-18 [1] CRAN (R 4.1.1)
#> lifecycle 1.0.3 2022-10-07 [1] CRAN (R 4.1.3)
#> magrittr 2.0.3 2022-03-30 [1] CRAN (R 4.1.3)
#> memoise 2.0.1 2021-11-26 [1] CRAN (R 4.1.2)
#> mime 0.12 2021-09-28 [1] CRAN (R 4.1.2)
#> miniUI 0.1.1.1 2018-05-18 [1] CRAN (R 4.1.2)
#> VP opusreader2 * 0.1.0 2022-12-23 [?] Github (spectral-cockpit/opusreader2@102bde3) (on disk 0.0.0.9000)
#> pillar 1.8.1 2022-08-19 [1] CRAN (R 4.1.3)
#> pkgbuild 1.4.0 2022-11-27 [1] CRAN (R 4.1.3)
#> pkgconfig 2.0.3 2019-09-22 [1] CRAN (R 4.1.2)
#> pkgload 1.3.2 2022-11-16 [1] CRAN (R 4.1.3)
#> prettyunits 1.1.1 2020-01-24 [1] CRAN (R 4.1.2)
#> processx 3.8.0 2022-10-26 [1] CRAN (R 4.1.3)
#> profvis 0.3.7 2020-11-02 [1] CRAN (R 4.1.3)
#> promises 1.2.0.1 2021-02-11 [1] CRAN (R 4.1.2)
#> ps 1.7.2 2022-10-26 [1] CRAN (R 4.1.3)
#> purrr 1.0.1 2023-01-10 [1] CRAN (R 4.1.3)
#> R6 2.5.1 2021-08-19 [1] CRAN (R 4.1.2)
#> Rcpp 1.0.10 2023-01-22 [1] CRAN (R 4.1.3)
#> remotes 2.4.2 2021-11-30 [1] CRAN (R 4.1.3)
#> reprex 2.0.1 2021-08-05 [1] CRAN (R 4.1.2)
#> rlang 1.0.6 2022-09-24 [1] CRAN (R 4.1.3)
#> rmarkdown 2.20 2023-01-19 [1] CRAN (R 4.1.3)
#> rprojroot 2.0.3 2022-04-02 [1] CRAN (R 4.1.3)
#> rstudioapi 0.14 2022-08-22 [1] CRAN (R 4.1.3)
#> sessioninfo 1.2.2 2021-12-06 [1] CRAN (R 4.1.2)
#> shiny 1.7.4 2022-12-15 [1] RSPM (R 4.1.0)
#> stringi 1.7.12 2023-01-11 [1] CRAN (R 4.1.3)
#> stringr 1.5.0 2022-12-02 [1] CRAN (R 4.1.3)
#> styler 1.5.1 2021-07-13 [1] CRAN (R 4.1.1)
#> testthat * 3.1.6 2022-12-09 [1] CRAN (R 4.1.3)
#> tibble 3.1.8 2022-07-22 [1] CRAN (R 4.1.3)
#> urlchecker 1.0.1 2021-11-30 [1] CRAN (R 4.1.3)
#> usethis 2.1.6 2022-05-25 [1] CRAN (R 4.1.3)
#> utf8 1.2.3 2023-01-31 [1] CRAN (R 4.1.3)
#> vctrs 0.5.2 2023-01-23 [1] CRAN (R 4.1.3)
#> withr 2.5.0 2022-03-03 [1] CRAN (R 4.1.3)
#> xfun 0.37 2023-01-31 [1] CRAN (R 4.1.3)
#> xtable 1.8-4 2019-04-21 [1] CRAN (R 4.1.2)
#> yaml 2.3.7 2023-01-23 [1] CRAN (R 4.1.3)
#>
#> [1] /home/philipp/R/x86_64-pc-linux-gnu-library/4.1
#> [2] /opt/R/4.1.3/lib/R/library
#>
#> V ── Loaded and on-disk version mismatch.
#> P ── Loaded and on-disk path mismatch.
#>
#> ──────────────────────────────────────────────────────────────────────────────Created on 2023-02-13 by the reprex package (v2.0.1) |
|
This is an example of a new key: opusreader2/R/create_dataset.R Line 36 in fb738b3 |
|
Hi @luukvandervelden found some stuff! are you familiar with R? |
|
Hi @philipp-baumann I am not familiar with R (I use Python and MATLAB), I am curious about what you've found and how you have found it! |
|
shall we have a short meeting on Friday or you prefer here? |
|
A meeting on Friday would be nice! My schedule this Friday is still completely empty, what time suits you? |
or let me prepare sth for Friday to put here # (have some imp stuff to do for opengis.ch ) |
|
Sure, no rush, just give me a ping when you have some time @philipp-baumann |
philipp-baumann
added
the
opus-data-support
|
The data you are looking for is located in a mini-header present in all 3D data blocks. The data is saved as a fixed-data structure that includes SRT (start time) and ERT (end time) for each 3D data block (among other parameters in array format). You can see the full decoded data structure in my code: https://github.com/joshduran/brukeropus The relevant information is in @luukvandervelden, since you are using python you can try my code out. You access this data in the following way: from brukeropus import read_opus
data = read_opus(filepath)
# Absorbance data
x = data.a.x # Wavenumbers
z = data.a.srt # Start time
y = data.a.y # Absorbance values (2D matrix)Since the 3D header block does not include the three char parameter keys (e.g. SRT, ERT), I had to guess what they were and made a mistake for ERT. To access ERT in the current version, use
|
Hi @joshduran thank you for the ping and for the excellent news. I've tried your code and it looks like you have solved the puzzle! |
I'm working with the python codebase of opusreader and am trying to figure out the time data of my spectra.
I have a two channels of timeseries of absorbance spectra and am able to extract them from the .0 files correctly.
When I export the data from OPUS to a MAT file, both the absorbance data AB matrix and the time data Tr matrix are exported.
Now I wonder how to get this matrix with relative time data from the binary files. Considering this R repository is rather active, I hope someone can help me. Does anyone have experience with this?
Some info about the Tr matrix. It contains 5 columns. I figured out that column 2 and 3 belong to the absorbance spectra (dual channel, a simultaneous ATR and transmissions measurement). The first 10 rows look like this:
the first 5 datapoints are just after depositing the sample. Then there is a wait time of approximately 60 seconds, after which the measurement starts with a reasonably high acquisition rate.
The text was updated successfully, but these errors were encountered: