diff --git a/ObscoreTimeExtension.pdf b/ObscoreTimeExtension.pdf index 291c1a5..b1d30b9 100644 Binary files a/ObscoreTimeExtension.pdf and b/ObscoreTimeExtension.pdf differ diff --git a/ObscoreTimeExtension.tex b/ObscoreTimeExtension.tex index 755473f..baf0b0d 100644 --- a/ObscoreTimeExtension.tex +++ b/ObscoreTimeExtension.tex @@ -91,13 +91,13 @@ \begin{abstract} -This IVOA specification details a list of metadata dealing with time-related features needed for discovery -of time series data sets, in the context of ObsTAP services. +This IVOA specification details a list of metadata dealing with time-related features needed for discovery +of time series data sets, in the context of ObsTAP services. It is based on science cases explained in a previous IVOA note prepared in 2018 and recently revised \citep{note:TSSerialisationNote}. -Here we discuss various use-cases. -We highlight first which existing time related metadata in the ObsCore standard version 1.1 can be used, -and second propose new features needed for an ObsCore time extension in order to allow more search criteria -for time-resolved data sets. +Here we discuss various use-cases. +We highlight first which existing time related metadata in the ObsCore standard version 1.1 can be used, +and second propose new features needed for an ObsCore time extension in order to allow more search criteria +for time-resolved data sets. \end{abstract} @@ -127,17 +127,17 @@ \section*{Conformance-related definitions} \section{Introduction} Time domain astronomy studies astrophysical phenomenae that vary in different time stamps and hence, in order to study the different physical underlying mechanisms a user might need to collect and analyse data from different missions and of different nature. Therefore she/he needs to search across various archives based on time related criteria. -ObsCore and ObsTAP \cite{2017ivoa.spec.0509L} have proven their efficiency for the discovery of astronomical data sets in the IVOA. +ObsCore and ObsTAP \cite{2017ivoa.spec.0509L} have proven their efficiency for the discovery of astronomical data sets in the IVOA. In this specification we consider how the ObsCore metadata profile can be extended to include time-related properties of the data, specific to time series and not yet covered. -In this specification we examine how to enhance data discovery and data selection of time sampled data sets in the context of the ObsCore data model and its TAP implementations. +In this specification we examine how to enhance data discovery and data selection of time sampled data sets in the context of the ObsCore data model and its TAP implementations. The ObsCore Specification \cite{2017ivoa.spec.0509L} proposes a set of features to describe the data present in a data set as well as metadata about its acquisition, creation and publication (curation). -The physical in terms of spatial, spectral, temporal, polarimetry, and observable measure are also described by a group of features dedicated to each axis, and considered independant from each other. The idea is to provide a physical feature profile for each axis with coverage, sampling, resolution, etc. -Search criteria in ObsTAP are based on these features. +The physical in terms of spatial, spectral, temporal, polarimetry, and observable measure are also described by a group of features dedicated to each axis, and considered independant from each other. The idea is to provide a physical feature profile for each axis with coverage, sampling, resolution, etc. +Search criteria in ObsTAP are based on these features. We examine in section \ref{sec:alreadythere} how the set of time parameters already present in ObsCore v1.1 can be used for time series discovery. In section \ref{sec:timeext} we consider specific time related uses cases and propose new parameters to be included for the tables extension in ObsCore. -The extension mechanism in TAP is discussed in section \ref{sec:comext} with user queries examples +The extension mechanism in TAP is discussed in section \ref{sec:comext} with user queries examples \subsection{Role within the VO Architecture} @@ -150,26 +150,26 @@ \subsection{Role within the VO Architecture} % Notes don't generally need this. If you don't copy role_diagram.xml, % you must remove role_diagram.pdf from SOURCES in the Makefile. -\includegraphics[width=0.9\textwidth]{myrole_diagram.pdf} +%\includegraphics[width=0.9\textwidth]{myrole_diagram.pdf} \caption{Architecture diagram for this document} \label{fig:archdiag} \end{figure} Fig.~\ref{fig:archdiag} shows the role this document plays within the -IVOA architecture \citep{2021ivoa.spec.1101D}. This specification builds up a metadata profile that must be used in a TAP service based on the ObsTAP TAP schema. -It relies on fundamental standards like TAP, VOTable, UCD, VOUnits and Vocabularies defined in the IVOA for data product type and TimeReference systems +IVOA architecture \citep{2021ivoa.spec.1101D}. This specification builds up a metadata profile that must be used in a TAP service based on the ObsTAP TAP schema. +It relies on fundamental standards like TAP, VOTable, UCD, VOUnits and Vocabularies defined in the IVOA for data product type and TimeReference systems for instance. \section{Time Series} \label{sect:metadata} -In this section we describe what Time Series data is in a wide context, describing the most relevant parameters that define it. We describe the common requirements of the different science use cases collected by the Science Priority Committee \cite{SPC_UC}. A common frame for time is defined with the minimum set of parameters taken from and compatible with the definition of SpaceTime coordinates and Coords DM. We then compare the defined fields describing time with the fields content of ObsCore and EPNcore. +In this section we describe what Time Series data is in a wide context, describing the most relevant parameters that define it. We describe the common requirements of the different science use cases collected by the Science Priority Committee \cite{SPC_UC}. A common frame for time is defined with the minimum set of parameters taken from and compatible with the definition of SpaceTime coordinates and Coords DM. We then compare the defined fields describing time with the fields content of ObsCore and EPNcore. \subsection{Definition} -Time Series can be defined in a very large sense as a collection of any kind of data over time for a particular source (e.~g. star, binary, QSO) or part of a source (e.~g. sun spots), independent on the type of data (images, light-curves, radial velocity, polarisation states or degrees, positions, number of sunspots, densities,...), the duration of the signal integration or the cadence. -To clarify the vocabulary here we consider a time series as a sequence of signal integrations, or snap-shots observing an object or phenomenon over time, so diffrent observations over time. -Considering how observations in general can be spanned along the time axis, we can sketch Time Series data as shown in Fig.~\ref{fig:time-series}. Time Series data is composed of a set of observations (n\_observations = 3 in this example), each with a different exposure or integration time (t\_exp). Although in some cases the cadence or time span between each signal intergration (delta\_t) is fixed, in the general case it can be different and we can therefore define a minimum and a maximum value (delta\_t\_min, delta\_t\_max). Each observation has it's own time stamp (\emph{t\_i)} with a given precision or resolution (t\_resolution). As can be seen from this figure the duration of the observation can be defined in different ways: a) as the total integration or exposure time, i.~e. the sum of all the exposure times: \emph{t\_exp\_total }= $\sum$ \emph{t\_exp} ; or b) as the time span between the beginning and the end of the observations: \emph{t\_exp\_total} = \emph{t\_max} - \emph{time\_min}). Note that in the case that the exposure time is constant for all the observations then \emph{t\_exp\_total }= n\_observations $\times$ \emph{t\_exp}. The situation can be more complicated, for instance during the observation there could be clouds and we therefore pause the exposure for a while and resume once the cloud has passed or we might want to remove parts of the observation due to artefacts in the data. In any case these values can be taken as approximative of the minimum and the maximum value this specific field can have. +Time Series can be defined in a very large sense as a collection of any kind of data over time for a particular source (e.~g. star, binary, QSO) or part of a source (e.~g. sun spots), independent on the type of data (images, light-curves, radial velocity, polarisation states or degrees, positions, number of sunspots, densities,...), the duration of the signal integration or the cadence. +To clarify the vocabulary here we consider a time series as a sequence of signal integrations, or snap-shots observing an object or phenomenon over time, so diffrent observations over time. +Considering how observations in general can be spanned along the time axis, we can sketch Time Series data as shown in Fig.~\ref{fig:time-series}. Time Series data is composed of a set of observations (n\_observations = 3 in this example), each with a different exposure or integration time (t\_exp). Although in some cases the cadence or time span between each signal intergration (delta\_t) is fixed, in the general case it can be different and we can therefore define a minimum and a maximum value (delta\_t\_min, delta\_t\_max). Each observation has it's own time stamp (\emph{t\_i)} with a given precision or resolution (t\_resolution). As can be seen from this figure the duration of the observation can be defined in different ways: a) as the total integration or exposure time, i.~e. the sum of all the exposure times: \emph{t\_exp\_total }= $\sum$ \emph{t\_exp} ; or b) as the time span between the beginning and the end of the observations: \emph{t\_exp\_total} = \emph{t\_max} - \emph{time\_min}). Note that in the case that the exposure time is constant for all the observations then \emph{t\_exp\_total }= n\_observations $\times$ \emph{t\_exp}. The situation can be more complicated, for instance during the observation there could be clouds and we therefore pause the exposure for a while and resume once the cloud has passed or we might want to remove parts of the observation due to artefacts in the data. In any case these values can be taken as approximative of the minimum and the maximum value this specific field can have. The most relevant fields of Time Series metadata are summarized in Table~\ref{tab:fields}. @@ -177,12 +177,12 @@ \subsection{Definition} \begin{center} \includegraphics[width=0.8\textwidth]{figs/fig1.png} -\caption{Simple representation of Time Series data.} +\caption{Simple representation of Time Series data.} \label{fig:time-series} \end{center} \end{figure} -\begin{table}[hb] +\begin{table}[hb] \begin{center} \caption{Time Series metadata fields.} \label{tab:fields} @@ -190,7 +190,7 @@ \subsection{Definition} \sptablerule \textbf{Field} & \textbf{Explanation} \\\sptablerule (RA,Dec) & Coordinates$^1$ \\ - target\_name & Target name$^1$ \\ + target\_name & Target name$^1$ \\ t\_min & Date of the begining of the time series \\ t\_max & Date of the end of the time series \\ t\_exp\_min & Minimum exposure time \\ @@ -204,24 +204,24 @@ \subsection{Definition} \sptablerule \end{tabular} \end{center} - \textbf{Note:} $^1$For SSO or moving objects coordinates might not be enough or relevant. + \textbf{Note:} $^1$For SSO or moving objects coordinates might not be enough or relevant. \end{table} In many cases time series data is composed of only three columns: \emph{Time, Magnitude, Magnitude Error}. -This is the simplest kind of data set, which is identified in the data product type vocabulary as 'light-curve'. +This is the simplest kind of data set, which is identified in the data product type vocabulary as 'light-curve'. See the IVOA product-type vocabulary at \url{https://www.ivoa.net/rdf/product-type/2024-03-22/product-type.html}. For this data to be fully exploitable and reusable (interoperable) it has to be properly documented. In this specific case the minimum information that needs to be provided is: the object coordinates (or name), the filter in which the observations have been carried out, and the time frame and offset (if applicable). -However, the dimensionality of what is observed at the time stamps' sequence may correspond to 1D or 2D observations, like spectra or images as well. +However, the dimensionality of what is observed at the time stamps' sequence may correspond to 1D or 2D observations, like spectra or images as well. That's why the dataproduct type defined in ObsCore 1.1 should be more precise and eventually rely on the IVOA product-type vocabulary. -In addition, a mechanism should be defined to clarify what part of the data is varying with time, as described further in section \ref{sec:timevariant}. +In addition, a mechanism should be defined to clarify what part of the data is varying with time, as described further in section \ref{sec:timevariant}. \subsection{Science use cases} \label{sect:usecases} -Different science use cases for Time Series have been collected and described in by E. Solano at \url{http://wiki.ivoa.net/twiki/bin/view/IVOA/CSPTimeSeries}. +Different science use cases for Time Series have been collected and described in by E. Solano at \url{http://wiki.ivoa.net/twiki/bin/view/IVOA/CSPTimeSeries}. They highlight the case of optical light curves but can be generalized to all spectral regimes ( xray, gamma ray, radio, multi-messengers) where time dependent measures have been taken. -Science cases are grouped according to their common requirements summarized as: +Science cases are grouped according to their common requirements summarized as: \begin{itemize} \item \textbf{Group A} Combine photometry and light curves of a given object/list of objects in the \emph{same} photometric band \item \textbf{Group B} Combine photometry and light curves of a given object/list of objects in \emph{different} photometric bands @@ -249,21 +249,21 @@ \subsection{Science use cases} \item \emph{Is it possible to discover long/short term variability within the data?} \end{enumerate} -To answer the first question a user needs to be sure that dates are comparable, that is time has to be brought into a common time frame. -To answer the second question we need to keep track of the minimum and maximum time span. +To answer the first question a user needs to be sure that dates are comparable, that is time has to be brought into a common time frame. +To answer the second question we need to keep track of the minimum and maximum time span. \subsection{Using a common time frame} \label{sec:comtimeframe} -To compare datasets from different missions or archives a common representation of time is needed. In order to do so we propose to map time into a pivot format. Following \cite{2015A+A...574A..36R} and \cite{2007ivoa.spec.1030R} we propose a set of minimum metadata to be added for serializations of Time Series (see Table~\ref{tab:metadata}). +To compare datasets from different missions or archives a common representation of time is needed. In order to do so we propose to map time into a pivot format. Following \cite{2015A+A...574A..36R} and \cite{2007ivoa.spec.1030R} we propose a set of minimum metadata to be added for serializations of Time Series (see Table~\ref{tab:metadata}). -\begin{table}[!htb] +\begin{table}[!htb] \begin{center} \caption{Metadata for time in Time Series data serialisation.} \label{tab:metadata} \begin{tabular}{p{0.35\textwidth}p{0.64\textwidth}} \sptablerule \textbf{Parameter proposal} & \textbf{Explanation} \\\sptablerule - t\_scale & Time frame scale is the scale used to measure time. IAU definition: "A time scale is simply a well defined way of measuring time based on a specific periodic natural phenomenon.'' See \url{http://aa.usno.navy.mil/publications/docs/Circular_179.pdf}. + t\_scale & Time frame scale is the scale used to measure time. IAU definition: "A time scale is simply a well defined way of measuring time based on a specific periodic natural phenomenon.'' See \url{http://aa.usno.navy.mil/publications/docs/Circular_179.pdf}. Recognized time scale values and their meaning are listed in Table~\ref{tab:scales}. If we don't know use UNKOWN. \\ t\_ref\_position & Time Frame Position is the place where the time is measured. Standard values are listed in Table~\ref{tab:positions}. If we don't know use UNKOWN. \\ t\_uncertainty & Resolution or uncertainty of the time stamps. \\ @@ -280,18 +280,18 @@ \subsection{Using a common time frame} \begin{center} JD(TT;BARYCENTER) \end{center} -We also give some values that can be used as default in the case that some information is not known and impossible to recover. We minimize the impact of doing this by adding a systematic error to time when those values are unknown. +We also give some values that can be used as default in the case that some information is not known and impossible to recover. We minimize the impact of doing this by adding a systematic error to time when those values are unknown. \section{Extension of ObsCore} -ObsCore has a normalized description of the data content along the various physical axes where the data are projected. +ObsCore has a normalized description of the data content along the various physical axes where the data are projected. The spatial properties are described in the \emph{s\_*} group, the spectral ones in \emph{em\_*} group, the temporal ones in \emph{t\_*}, etc. For each data set there is a minimal set of metadata to describe its sky position, spectral band, time interval, etc. which are independent from each other. -This allows to enhance time sampling description by adding new parameters to the time group without putting the ObsCore existing model at risk. +This allows to enhance time sampling description by adding new parameters to the time group without putting the ObsCore existing model at risk. \subsection{Extension of ObsCore based on EPNCore} -Astronomy and space science both consider time series data and have proposed metadata data description for it. Some metadata have already been defined and used in the context of data discovery using ObsCore \cite{2017ivoa.spec.0509L}, and the remaining ones have been defined in the context of planetary data in the EPNcore specification \cite{2022ivoa.spec.0822E}. In Table~\ref{tab:obs_epn} we show the equivalence between the fields we require here and those existing in ObsCore and EPNcore specifications. +Astronomy and space science both consider time series data and have proposed metadata data description for it. Some metadata have already been defined and used in the context of data discovery using ObsCore \cite{2017ivoa.spec.0509L}, and the remaining ones have been defined in the context of planetary data in the EPNcore specification \cite{2022ivoa.spec.0822E}. In Table~\ref{tab:obs_epn} we show the equivalence between the fields we require here and those existing in ObsCore and EPNcore specifications. -\begin{table}[!htb] +\begin{table}[!htb] \begin{center} \begin{small} \caption{Equivalence and complementarity between Time Series data fields and ObsCore and EPNCore fields} @@ -330,27 +330,27 @@ \subsection{Extension of ObsCore based on EPNCore} \end{small} \end{center} - \end{table} - -\textbf{ Note:} \emph{t\_resolution} in ObsCore needs some clarification and the dataproduct\_type labels defined in ObsCore and EPNCore are different currently. + \end{table} + +\textbf{ Note:} \emph{t\_resolution} in ObsCore needs some clarification and the dataproduct\_type labels defined in ObsCore and EPNCore are different currently. That is why \emph{dataproduct\_type} should be enriched in ObsCore, and harmonized with the product type IVOA vocabulary maintained at \url{ivoa.net/rdf/}. \subsection{Mentioning what part of the dataset varies with time } \label{sec:timevariant} Obscore 1.1 uses the attribute \emph{o\_ucd} to describe what is the quantity observed depending on the various physical axes of the data product. The UCD string corresponding to the observable in a one dimensional dataset is easy to choose in the UCD list. We propose to extend this definition to generalize for time series of multiple dimensional data sets and add a \emph{time\_variant} attribute in ObsCore. -In a time series, the principal axis considered is the Time axis. The time variant component can be either one dimensional, like for a light curve or velocity curve, or multi-dimensional. The time series is viewed as time dependent sequence of components, which can be characterized by a data product type, such as an image, a spectrum, a spectral cube, etc., also defined in the product-type vocabulary. Table \ref{tab:timevar} summarizes the use of \emph{ time\_variant} in various cases. -This parameter is worth to include in the Time ObsCore extension table. +In a time series, the principal axis considered is the Time axis. The time variant component can be either one dimensional, like for a light curve or velocity curve, or multi-dimensional. The time series is viewed as time dependent sequence of components, which can be characterized by a data product type, such as an image, a spectrum, a spectral cube, etc., also defined in the product-type vocabulary. Table \ref{tab:timevar} summarizes the use of \emph{ time\_variant} in various cases. +This parameter is worth to include in the Time ObsCore extension table. \begin{table}[!htb] \begin{center} \caption{Time variant property of a temporally sampled dataset. \label{tab:timevar} } - + \begin{small} \begin{tabular}{|l|l|l|} \sptablerule \textbf{Main dataproduct\_type} & \textbf{o\_ucd} &\textbf{time\_variant component } \\ \hline light curve & phot.flux & scalar value \\ \hline -velocity curve & doppler.veloc & scalar value \\ \hline +velocity curve & doppler.veloc & scalar value \\ \hline trajectory & pos.eq & sky position (vector) \\ \hline dynamic spectrum & phot.flux & spectrum \\ \hline movie & phot.flux & image \\ \hline @@ -358,21 +358,21 @@ \subsection{Mentioning what part of the dataset varies with time } \end{tabular} \end{small} \end{center} - \end{table} - + \end{table} + \subsection{Time parameters defined in ObsCore v1.1} \label{sec:alreadythere} - We have seen the data product type helps to search for time sampled data sets. + We have seen the data product type helps to search for time sampled data sets. In order to describe properties of the data set along the time axis, we can reuse the axis properties defined in the Characterization data model \cite{2008ivoa.spec.0325L}. The idea is to describe how the time stamps are spanned along the time axis, with time duration and cadence. \subsubsection{t\_min, t\_max} These parameters provide the bounds of the time coverage for this data set. For a light-curve it is the beginning of the first time sample and the end of the last sample. \subsubsection{t\_exptime} - This parameter represents the duration or live time of the observation. + This parameter represents the duration or live time of the observation. For a light-curve it is the sum of all valid time samples. For instance for a time-cube it is the total exposure time summing up all the poses. \subsubsection{t\_resolution} t\_resolution can be defined as the time limit under which two observable quantities cannot be distinguished from each other. - This works for event-list, light-curve, time-cube data sets, etc. + This works for event-list, light-curve, time-cube data sets, etc. \subsubsection{t\_xel, number of time stamps} This parameter entails the number of observations in the time series. It is important to query for guessing how rich is the dataset, especially for observing variability. @@ -386,19 +386,19 @@ \subsection{Mentioning what part of the dataset varies with time } \begin{flushleft} \caption{Definitions of ObsCore time related attributes in the current specification. \label{tab:timeinobscore} } \begin{scriptsize} - \begin{tabular}{|l|l|l|l|l|} + \begin{tabular}{|l|l|l|l|l|} \sptablerule \textbf{Name} & \textbf{Definition \&Utype} & \textbf{UCD} & \textbf{Units}& \textbf{Status} \\ \hline -\emph{t\_min} & Time start of the sequence (MJD) & time.start;obs.sequence & d &man\\ +\emph{t\_min} & Time start of the sequence (MJD) & time.start;obs.sequence & d &man\\ & {\color{blue} Char.TimeAxis.Coverage.Bounds.Limits.LoLim} & & & \\ \hline -\emph{t\_max} & Time end of the sequence & time.end;obs.sequence & d & man \\ +\emph{t\_max} & Time end of the sequence & time.end;obs.sequence & d & man \\ & {\color{blue}Char.TimeAxis.Coverage.Bounds.Limits.HiLim} & & & \\ \hline \emph{t\_exptime} & Exposure time (sum of multiple exposures)& time.duration;obs.exposure &s &man \\ & {\color{blue}Char.TimeAxis.Support.Extent} & & & \\ \hline -\emph{t\_resolution} & Minimal interpretable time difference & time.resolution & s & man \\ - & {\color{blue}Char.TimeAxis.Resolution.Refval I}& & & \\ \hline -\emph{t\_xel} & Number of time stamps in the series & meta.number & null & man\\ - & {\color{blue}Char.TimeAxis.numBins} & & & \\ \hline +\emph{t\_resolution} & Minimal interpretable time difference & time.resolution & s & man \\ + & {\color{blue}Char.TimeAxis.Resolution.Refval I}& & & \\ \hline +\emph{t\_xel} & Number of time stamps in the series & meta.number & null & man\\ + & {\color{blue}Char.TimeAxis.numBins} & & & \\ \hline \end{tabular} \end{scriptsize} \end{flushleft} @@ -406,24 +406,24 @@ \subsection{Mentioning what part of the dataset varies with time } \subsection{Time series uses cases already covered by ObsCore1.1} Several uses-cases for time series discoveries were considered in the ObsCore 1.1 specification, built on its short list of time related features. -They are available in appendix A in section A.4. Discovering time series. -Here the \emph{dataproduct\_type} value is "timeseries", very general, but the same uses cases can be applied for more specific time sampled datasets like "time-cube" or or "light-curve" available now in the \textbf{product-type} vocabulary . +They are available in appendix A in section A.4. Discovering time series. +Here the \emph{dataproduct\_type} value is "timeseries", very general, but the same uses cases can be applied for more specific time sampled datasets like "time-cube" or or "light-curve" available now in the \textbf{product-type} vocabulary . ObsCore uses cases are also provided in a web page available at : \url{http://saada.unistra.fr/voexamples/show/ObsCore/}. \section{Time parameters proposed for Obscore Extension } \label{sec:timeext} - + \subsection{Time Frame description} As mentioned in section \ref{sec:comtimeframe} the Time Frame description is essential for comparing various time series data sets. This metadata was described first in the STC data model \citep{2007ivoa.spec.1030R}, then in the Coords DM \citep{2022ivoa.specQ1004R}, and serialized in the VOTABLE format in the TimeSYS element. -Up to now, this metadata was not defined in ObsCore1.1. It is coded into the VOTable metadata of the dataset. +Up to now, this metadata was not defined in ObsCore1.1. It is coded into the VOTable metadata of the dataset. Having it as part of the query response coming back for a search for time series would help the user application to interpret time stamps precisely. -MJD is the time format used for an ObsTAP query related to time. +MJD is the time format used for an ObsTAP query related to time. We propose to add the time frame parameters in the Time Obscore extension. -These various definitions are harmonized in the proposal given in table \ref{tab:timereff}. We list the corresponding terms used in the Coords Data model and in the UCD vocabulary, as well as the attribute of the TIMESYS param defined for VOTable serialization. -All terms are proposed as mandatory, but can be set to UNKNOWN if not available. -With the expansion of massive time series datasets, where efficient data discovery will serve the selection of big training sets for analysis workflows, -such parameters are highly recommended especially for new data collections. +These various definitions are harmonized in the proposal given in table \ref{tab:timereff}. We list the corresponding terms used in the Coords Data model and in the UCD vocabulary, as well as the attribute of the TIMESYS param defined for VOTable serialization. +All terms are proposed as mandatory, but can be set to UNKNOWN if not available. +With the expansion of massive time series datasets, where efficient data discovery will serve the selection of big training sets for analysis workflows, +such parameters are highly recommended especially for new data collections. \begin{sidewaystable}[!htb] \caption{ Time extension Table Summary. } %%%%%%%%%%%%%%%%% @@ -436,17 +436,17 @@ \subsection{Mentioning what part of the dataset varies with time } \hline t-obs attribute& Definition & VODML-ID & TIMESYS &UCD & Units&Status\\ & & in Coords DM & attribute& & &\\ \hline -% Time Coordinate system +% Time Coordinate system {\color{blue}t\_origin} & Time frame origin & TimeOffset.time0 & timeorigin & time.epoch & & man \\ \hline -{\color{blue}t\_scale} & Time frame scale & TimeFrame.timeScale ×cale& time.scale & ?&man \\ \hline -{\color{blue}t\_refPosition} & Time reference position & TimeFrame.refPosition& refposition & & &man \\ \hline -{\color{blue}t\_refDirection} &Time reference direction&TimeFrame.refDirection&refdirection & & & man \\ \hline -%Time representation ISOtime, MJD, JD, ? -{\color{blue}t\_format}& Time representation & & & time;meta.code.class& null &man \\ \hline +{\color{blue}t\_scale} & Time frame scale & TimeFrame.timeScale ×cale& time.scale & ?&man \\ \hline +{\color{blue}t\_refPosition} & Time reference position & TimeFrame.refPosition& refposition & & &man \\ \hline +{\color{blue}t\_refDirection} &Time reference direction&TimeFrame.refDirection&refdirection & & & man \\ \hline +%Time representation ISOtime, MJD, JD, ? +{\color{blue}t\_format}& Time representation & & & time;meta.code.class& null &man \\ \hline \end{tabular} - + \bigskip \subcaption{Sampling properties along Time axis. \label{tab:time_ext} } \begin{tabular}{|l|l|l|l|l|l|} @@ -455,19 +455,19 @@ \subsection{Mentioning what part of the dataset varies with time } % % %{\color{blue}t\_origin} & Time frame origin & TimeOffset.time0 & time.epoch & & opt \\ \hline -%{\color{blue}t\_scale} & Time frame scale & TimeFrame.timeScale & time.scale & ?&opt \\ \hline -%{\color{blue}t\_refPosition} & Time reference position & TimeFrame.refPosition & & &opt \\ \hline -%{\color{blue}t\_refDirection} &Time reference direction&TimeFrame.refDirection & & & opt \\ \hline +%{\color{blue}t\_scale} & Time frame scale & TimeFrame.timeScale & time.scale & ?&opt \\ \hline +%{\color{blue}t\_refPosition} & Time reference position & TimeFrame.refPosition & & &opt \\ \hline +%{\color{blue}t\_refDirection} &Time reference direction&TimeFrame.refDirection & & & opt \\ \hline \hline {\color{blue} t\_variant } & sub product attached to a time stamp & & meta.code.class & & opt\\ \hline -{\color{blue} t\_exp\_min} & minimal length of time sample & Char.TimeAxis.Sampling.Extent.loLim & time.duration; & s & man\\ +{\color{blue} t\_exp\_min} & minimal length of time sample & Char.TimeAxis.Sampling.Extent.loLim & time.duration; & s & man\\ & (min integration time)& & obs.sequence;stat.min& & \\ \hline -{\color{blue}t\_exp\_max} & maximal length of time sample & Char.TimeAxis.Sampling.Extent.hiLim & time.duration; & s & man\\ +{\color{blue}t\_exp\_max} & maximal length of time sample & Char.TimeAxis.Sampling.Extent.hiLim & time.duration; & s & man\\ & (max integration time) & &bs.sequence;stat.max & & \\ \hline -%time space between 2 time samples / cadence -{\color{blue}t\_delta\_min} & minimal length of time interval & Char.TimeAxis.Sampling.Period.loLim & time.interval; & s & man \\ +%time space between 2 time samples / cadence +{\color{blue}t\_delta\_min} & minimal length of time interval & Char.TimeAxis.Sampling.Period.loLim & time.interval; & s & man \\ & cadence (min)& &obs.sequence;stat.min & & \\ \hline -{\color{blue}t\_delta\_max} & maximal length of time interval & Char.TimeAxis.Sampling.Period.hiLim & time.interval;& s & man\\ +{\color{blue}t\_delta\_max} & maximal length of time interval & Char.TimeAxis.Sampling.Period.hiLim & time.interval;& s & man\\ & cadence (max)& & obs.sequence;stat.max& & \\ \hline {\color{blue} t\_fold\_period}& folding period length & & time.period&d & man \\ \hline {\color{blue} t\_fold\_phaseReference}& time stamp of folding start in time series & & meta.ref;&d & opt \\ @@ -476,88 +476,88 @@ \subsection{Mentioning what part of the dataset varies with time } \end{tabular} \end{small} - \end{sidewaystable} - - -Values to fill these terms should rely on the terms defined in IVOA vocabularies, namely for time scales and time reference position. + \end{sidewaystable} + + +Values to fill these terms should rely on the terms defined in IVOA vocabularies, namely for time scales and time reference position. As an example Appendix A summarize the definitions listed in previous models like STC. - + \subsection{Time axis sampling description} \emph{t\_delta\_min }, \emph{t\_delta\_max} represent the minimal (resp. maximal) time interval between two time samples. -This concept is covered in the Characterization data model \citep{2008ivoa.spec.0325L} and designated as sampling period along the Time axis. -The cadence of the observations in the time series can be assumed from theses parameters. - - The Time Axis Sampling Extent defined in Characterization DM is the duration of each sample and may vary along the time sequence. - During the observation process, it corresponds to an exposure time. - If the sampling is not regular the minimal and maximal value described in \emph{ t\_exp\_min, t\_exp\_max} give the bounds values of the sampling extent. -When the sampling is even, all samples have the same duration and t\_exp\_min, t\_exp\_max have the same value. -When the sampling period, or cadence is even, \emph{t\_delta\_min }, \emph{t\_delta\_max} have the same value. +This concept is covered in the Characterization data model \citep{2008ivoa.spec.0325L} and designated as sampling period along the Time axis. +The cadence of the observations in the time series can be assumed from theses parameters. + + The Time Axis Sampling Extent defined in Characterization DM is the duration of each sample and may vary along the time sequence. + During the observation process, it corresponds to an exposure time. + If the sampling is not regular the minimal and maximal value described in \emph{ t\_exp\_min, t\_exp\_max} give the bounds values of the sampling extent. +When the sampling is even, all samples have the same duration and t\_exp\_min, t\_exp\_max have the same value. +When the sampling period, or cadence is even, \emph{t\_delta\_min }, \emph{t\_delta\_max} have the same value. -In general the \emph{t\_resolution}, the minimal distinguishable time interval between two time stamps is much finer than the chosen cadence in the instrument. % ZTF ? LSST? typical values? +In general the \emph{t\_resolution}, the minimal distinguishable time interval between two time stamps is much finer than the chosen cadence in the instrument. % ZTF ? LSST? typical values? \subsection{Time axis mode, folding period and phase reference} -Time series may be distributed in two modes, "search mode" or "folded". +Time series may be distributed in two modes, "search mode" or "folded". The folding allows to improve the SNR and to analyse further the periodicity of the observed phenomenon. For data discovery purpose one parameter may be introduced : \emph{t\_fold\_period}, the time duration of the folding. A \emph{t\_fold\_period} parameter set to zero means that the time axis is not folded and then indicates it belongs to search mode . \subsubsection{ t\_fold\_period, t\_fold\_phaseReference} -This metadata gives the length of the folding interval. It is given in the same time units as the time stamps along the sequence. -The time origin at which the folding starts is another important metadata and stored in +This metadata gives the length of the folding interval. It is given in the same time units as the time stamps along the sequence. +The time origin at which the folding starts is another important metadata and stored in \emph{t\_fold\_phaseReference}. It is given as a time stamp within the \emph{t\_min, t\_max} interval of the time series before folding. -This value is usually chosen according to the transient phenomenon under study, on peak or gap, etc. and cannot be standardized, that is why the time reference in the original curve is more convenient. +This value is usually chosen according to the transient phenomenon under study, on peak or gap, etc. and cannot be standardized, that is why the time reference in the original curve is more convenient. Both attributes enable to study the periodicity of the signal and compare between various light curves. -% -% \begin{sidewaystable}[!htb] +% +% \begin{sidewaystable}[!htb] % % \begin{center} % \caption{Table of Obscore Time extension attributes, \\ Here are the extended features for comparing time sampled data sets.\label{tab:time_ext}} % \begin{small} % \begin{tabular}{|l|l|l|l|l|l|} % \hline %\bf{Name} & \bf{Definition} & Utype& UCD &Units& Status \\ \hline -%t\_variant & datatype of the leaf data set of time stamps & & meta.code.class & & opt\\ +%t\_variant & datatype of the leaf data set of time stamps & & meta.code.class & & opt\\ %& (min integration time)& & & & \\ \hline -% t\_exp\_min & minimal length of time sample & Char.TimeAxis.Sampling.Extent.loLim & time.duration; & s & opt\\ +% t\_exp\_min & minimal length of time sample & Char.TimeAxis.Sampling.Extent.loLim & time.duration; & s & opt\\ %& (min integration time)& & obs.sequence;stat.min& & \\ \hline -%t\_exp\_max & maximal length of time sample & Char.TimeAxis.Sampling.Extent.hiLim & time.duration; & s & opt\\ +%t\_exp\_max & maximal length of time sample & Char.TimeAxis.Sampling.Extent.hiLim & time.duration; & s & opt\\ %& (max integration time) & &bs.sequence;stat.max & & \\ \hline -%%time space between 2 time samples / cadence -%t\_delta\_min & minimal length of time interval & Char.TimeAxis.Sampling.Period.loLim & time.interval; & s & opt \\ +%%time space between 2 time samples / cadence +%t\_delta\_min & minimal length of time interval & Char.TimeAxis.Sampling.Period.loLim & time.interval; & s & opt \\ %& cadence (min)& &obs.sequence;stat.min & & \\ \hline -%t\_delta\_max & maximal length of time interval & Char.TimeAxis.Sampling.Period.hiLim & time.interval;& s & opt \\ +%t\_delta\_max & maximal length of time interval & Char.TimeAxis.Sampling.Period.hiLim & time.interval;& s & opt \\ %& cadence (max)& & obs.sequence;stat.max& & \\ \hline %t\_fold\_period& Folding period length & & time.period&d & opt \\ \hline % \end{tabular} % \end{small} % % \end{center} -% \end{sidewaystable} +% \end{sidewaystable} \section{Extension mechanism in ObsTAP } \label{sec:comext} ObsCore is mostly implemented in the TAP protocol \citep{2019ivoa.spec.0927D}. % note:TSSerialisationNote An ObsTAP service is considered compliant to the standard if it serves all the attributes tagged as mandatory in the specification. - These are gathered in the TAP\_SCHEMA in the table usually named \emph{ivoa.obscore}. + These are gathered in the TAP\_SCHEMA in the table usually named \emph{ivoa.obscore}. Following the practice introduced for EPNTap, the utype column in \emph{ivoa.obscore} should be the standard identifier of the specification supported by the table content, so here \texttt{ivo://ivoa.net/std/obscore\#table-1.1} - + This table can also hold more columns corresponding to optional attributes, as summarized in the Table 7 - Optional Parameters of the ObsCore specification. - There is no guarantee that an optional parameter will be filled in an ObsTAP service; this must be checked first by the user. - - Therefore the Time extension for Obscore should rely on mandatory parameters. + There is no guarantee that an optional parameter will be filled in an ObsTAP service; this must be checked first by the user. + + Therefore the Time extension for Obscore should rely on mandatory parameters. They may be set UNKNOWN if unknown. In order to warn users that extra time parameters have been included in ObsTAP, we propose to gather them in another table named \emph{ivoa.t-obs} - for services that distribute time sampled data sets. + for services that distribute time sampled data sets. The utype column in \emph{ivoa.t\_obs} should be the standard identifier of this specification, so here \texttt{ivo://ivoa.net/std/obscore\#t-obs-1.0}. If this table contains an identifier for the corresponding dataset described in main \emph{ivoa.obscore} table, then it is easy to join general Obscore properties to the time specific ones in an ADQL query. - Here is a query example : ( to be checked) + Here is a query example : ( to be checked) \begin{lstlisting} [language=SQL, caption= Query example with a JOIN between the main ObsCore table and the Time extension table] - SELECT obs_id, t_min, t_max, obs_publisher_did, obs_collection, access.reference FROM ivoa.obscore + SELECT obs_id, t_min, t_max, obs_publisher_did, obs_collection, access.reference FROM ivoa.obscore WHERE dataproduct_type=='light-curve' - AND t_min > 55197 + AND t_min > 55197 AND t_max < 55204 JOIN ivoa.t-obs as tt ON obs_publisher_did==tt.obs_publisher_did @@ -566,13 +566,13 @@ \subsubsection{ t\_fold\_period, t\_fold\_phaseReference} Other examples of queries using these extra parameters are proposed in Appendix \ref{sec:query_examples}. -More generally, other extensions can be considered in ObsTAP, like the radio extension or high energy extension specific to these spectral domains and instrumentations. -In an extended ObsTAP service the main ObsCore table and the other extension tables must be gathered in a TAP\_SCHEMA with utype \\ \texttt{ivo://ivoa.net/std/obscore1.1}, for version 1.1 and containing the different tables : ivoa.obscore, ivoa.t-obs, ivoa.radio, ivoa.heig etc.... when needed. +More generally, other extensions can be considered in ObsTAP, like the radio extension or high energy extension specific to these spectral domains and instrumentations. +In an extended ObsTAP service the main ObsCore table and the other extension tables must be gathered in a TAP\_SCHEMA with utype \\ \texttt{ivo://ivoa.net/std/obscore1.1}, for version 1.1 and containing the different tables : ivoa.obscore, ivoa.t-obs, ivoa.radio, ivoa.heig etc.... when needed. This would guarantee that all the tables for data discovery are grouped together and discovered, and that the service can allow the table joins. % exemples of joins -%SELECT TOP 10 lat, long, flux -%FROM lightmeter.measurements +%SELECT TOP 10 lat, long, flux +%FROM lightmeter.measurements %JOIN lightmeter.stations %USING (stationid) @@ -582,24 +582,24 @@ \subsubsection{ t\_fold\_period, t\_fold\_phaseReference} \bibliography{ivoatex/ivoabib, ivoatex/docrepo, ivoatex/myref} % note:TSSerialisationNote - + \appendix \include{appendix_table_time_reference_Ada} - + \section{Query examples for join tables}\label{sec:query_examples} \todo{Other examples of join and uses cases} -\section{Previous work on the Time series characterization and description}. +\section{Previous work on the Time series characterization and description}. -\begin{itemize} +\begin{itemize} \item Very initial draft initiated by D. Tody. \url{https://wiki.ivoa.net/internal/IVOA/LightCurves/STSP.pdf} \item Table reference for time position : Table 1, p15 in Space-Time Coordinate Metadata for the Virtual Observatory -Version 1.33, \url{https://www.ivoa.net/documents/REC/DM/STC-20071030.pdf} +Version 1.33, \url{https://www.ivoa.net/documents/REC/DM/STC-20071030.pdf} \item Table reference for time scales : Table 2, p17 in Space-Time Coordinate Metadata for the Virtual Observatory Version 1.33 -\item -\end{itemize} +\item +\end{itemize} \section{Vocabulary enhancement} \url{https://www.ivoa.net/rdf/product-type/2024-03-22/product-type.html}