-
Notifications
You must be signed in to change notification settings - Fork 3
/
Copy pathsigDataExtr.Rmd
144 lines (102 loc) · 9.81 KB
/
sigDataExtr.Rmd
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
# Signal data extraction [^1-chap:sigDataExtr]{#chap:sigDataExtr}
[^1-chap:sigDataExtr]: Parts of this chapter have been published in @winkelmann:2017aa.
```{r fig.align="center", echo=FALSE, out.width = "65%"}
knitr::include_graphics("pics/EMU-webAppEmu_sigDataExtr.png")
```
As mentioned in the default workflow of Chapter \@ref(chap:overview), after querying the symbolic annotation structure and dereferencing its time information, the result is a set of items with associated time stamps. It was necessary that the `emuR` package contain a mechanism for extracting signal data corresponding to this set of items. As illustrated in Chapter \@ref(chap:wrassp), `wrassp` provides the R ecosystem with signal data file handling capabilities as well as numerous signal processing routines. `emuR` can use this functionality to either obtain pre-stored signal data or calculate derived signal data that correspond to the result of a query. Figure \@ref(fig:sigDataExtr)A shows a snippet of speech with overlaid annotations where the resulting `SEGMENT` of an example query (e.g., `"Phonetic == ai"`) is highlighted in yellow. Figure \@ref(fig:sigDataExtr)B displays a time parallel derived signal data contour as would be returned by one of `wrassp`'s file handling or signal processing routines. The yellow segment in Figure \@ref(fig:sigDataExtr)B marks the corresponding samples that belong to the `ai` segment of Figure \@ref(fig:sigDataExtr)A.
```{r sigDataExtr, fig.cap="Segment of speech with overlaid annotations and time parallel derived signal data contour.", fig.align="center", echo=FALSE, out.width = "75%"}
knitr::include_graphics("pics/sigDataExtr.png")
```
The R code snippet below shows how to create the demo data that will be used throughout this chapter.
```{r results='hide', message=FALSE}
# load the package
library(emuR)
# create demo data in directory provided by the tempdir() function
create_emuRdemoData(dir = tempdir())
# get the path to a emuDB called "ae" that is part of the demo data
path2directory = file.path(tempdir(),
"emuR_demoData",
"ae_emuDB")
# load emuDB into current R session
ae = load_emuDB(path2directory)
```
## Extracting pre-defined tracks
To access data that are stored in files, the user has to define tracks for a database that point to sequences of samples in files that match a user-specified file extension. The user-defined name of such a track can then be used to reference the track in the signal data extraction process. Internally, `emuR` uses `wrassp` to read the appropriate files from disk, extract the sample sequences that match the result of a query and return values to the user for further inspection and evaluation. The R code snippet below shows how a signal track that is already defined in the `ae` demo database can be extracted for all annotation items on the `Phonetic` level containing the label *ai*.
```{r, message=F, warning=F, error=F}
# list currently available tracks
list_ssffTrackDefinitions(ae)
# query all "ai" phonetic segments
ai_segs = query(ae, "Phonetic == ai")
# get "fm" track data for these segments
# (verbose = F is only set to avoid additional output in manual)
ai_td_fm = get_trackdata(ae,
seglist = ai_segs,
ssffTrackName = "fm",
verbose = FALSE)
# show ai_td_fm (F1-F4 in columns T1-T4)
ai_td_fm
```
Being able to access data that is stored in files is important for two main reasons. Firstly, it is possible to generate files using external programs such as VoiceSauce [@shue:2011a], which can export its calculated output to the general purpose SSFF file format. This file mechanism is also used to access data produced by EMA, EPG or many other forms of signal data recordings. Secondly, it is possible to track, save and access manipulated data such as formant values that have been manually corrected. It is also worth noting that the `get_trackdata()` function has a predefined track which is always available without it having to be defined. The name of this track is `MEDIAFILE_SAMPLES` which references the actual samples of the audio files of the database. The R code snippet below shows how this predefined track can be used to access the audio samples belonging to the segments in `ai_segs`.
```{r}
# get media file samples
# (verbose = F is only set to avoid additional output in manual)
ai_td_mfs = get_trackdata(ae,
seglist = ai_segs,
ssffTrackName = "MEDIAFILE_SAMPLES",
verbose = FALSE)
# ai_td_mfs (sample values in column T1)
ai_td_mfs
```
## Adding new tracks
As described in detail in Section \@ref(sec:wrassp-emu-sdms), the signal processing routines provided by the `wrassp` package can be used to produce SSFF files containing various derived signal data (e.g., formants, fundamental frequency, etc.). The R code snippet below shows how the `add_ssffTrackDefinition()` can be used to add a new track to the `ae` `emuDB`. Using the `onTheFlyFunctionName` parameter, the `add_ssffTrackDefinition()` function automatically executes the `wrassp` signal processing function `ksvF0` (`onTheFlyFunctionName = "ksvF0"`) and stores the results in SSFF files in the bundle directories.
```{r}
# add new track and calculate
# .f0 files on-the-fly using wrassp::ksvF0()
# (verbose = F is only set to avoid additional output in manual)
add_ssffTrackDefinition(ae,
name = "F0",
onTheFlyFunctionName = "ksvF0",
verbose = FALSE)
# show newly added track
list_ssffTrackDefinitions(ae)
# show newly added files
list_files(ae, fileExtension = "f0")
# extract newly added trackdata
# (verbose = F is only set to avoid additional output in manual)
ai_td = get_trackdata(ae,
seglist = ai_segs,
ssffTrackName = "F0",
verbose = FALSE)
# show ai_td (F0 values in column T1)
ai_td
```
## Calculating tracks on-the-fly
With the `wrassp` package, we were able to implement a new form of signal data extraction which was not available in the legacy system. The user is now able to select one of the signal processing routines provided by `wrassp` and pass it on to the signal data extraction function. The signal data extraction function can then apply this `wrassp` function to each audio file as part of the signal data extraction process. This means that the user can quickly manipulate function parameters and evaluate the result without having to store to disk the files that would usually be generated by the various parameter experiments. In many cases this new functionality eliminates the need for defining a track definition for the entire database for temporary data analysis purposes. The R code snippet below shows how the `onTheFlyFunctionName` parameter of the `get_trackdata()` function is used.
```{r}
# (verbose = F is only set to avoid additional output in manual)
ai_td_pit = get_trackdata(ae,
seglist = ai_segs,
onTheFlyFunctionName = "mhsF0",
verbose = FALSE)
# show ai_td_pit (F0 values in column T1)
ai_td_pit
```
## The resulting object: `tibble`
As of version 2.0.0 of `emuR` the default resulting object of a call to `get_trackdata()` is of class `tibble` (see R code snippet below).
```{r}
# show class vector of ai_td_pit
class(ai_td_pit)
# show ai_td_pit (F0 values in column T1)
ai_td_pit
```
As can be seen by the first row output of the R code snippet above, the `tibble` object is an amalgamation of both a segment list and the actual signal data. The first `sl_rowIdx` column of the `ai_emuRtd_pit` object indicates the row index of the segment list the current row belongs to, the `times_rel` and `times_orig` columns represent the relative time and the original time of the samples contained in the current row (see above R code snippet) and T1 (to Tn in n-dimensional tracks) contains the actual signal sample values. As is often the case with tabular data, the `tibble` object carries certain redundant information (e.g. segment start and end times). However, the benefit of having a `data.frame` object that contains all the information needed to process the data is the ability to replace package specific functions (e.g. the legacy `eplot()` etc.) with standardized `data.frame` processing and visualization procedures that can be applied to any `data.frame` object independent of the package that generated it. Therefore, the knowledge that is necessary to process a `tibble` object can be transferred to/from other packages which was not the case for the legacy `trackdata` object. For examples on how functions provided by packages belonging to the [tidyverse](https://www.tidyverse.org/) can be used to replace the legacy `eplot()` and `dplot()` functions see \@ref(recipe:plottingSnippets). The legacy `dcut()` can simply be replaced using `normalize_length()` in combination with `dplyr::filter()`. Finally, it is worth noting that for backward compatibility the legacy `trackdata` object is still available by explicitly setting the `resultType` parameter of `get_trackdata()`.
## Conclusion
This chapter introduced the signal data extraction mechanics of the `emuR` package. The combination of the `get_trackdata()` function and the file handling and signal processing abilities of the `wrassp` package (see Chapter \@ref(chap:wrassp) for further details) provide the user with a flexible system for extracting derived or complementary signal data belonging to their queried annotation items.
```{r echo=FALSE, results='hide', message=FALSE}
# disconnect to avoid file locking to sqliteDB that causes unlink
# to fail under windows
# DBI::dbDisconnect(dbHandle$connection)
# clean up emuR_demoData
unlink(file.path(tempdir(), "emuR_demoData"), recursive = TRUE)
unlink(file.path(tempdir(),'myTGcolDB_emuDB'), recursive = TRUE)
```