Registry matters #11
Registry matters #11
Conversation
(I'm hoping the flesh of this will be filled in by Pierre).
In particular, language on interface/@ROLE, which now allows non-standard interfaces on other types. Also, we're now requiring our interface/@stdID to be in the RegTAP res_details.
|
There is a discussion about this PR and the issues it raises on the time-domain mailing list. |
We do have a choice. As described on the mailing list[1], we can change the wording in this PR to remove the MUST and change it to say "IF you are registering a stream, this is how". [1] http://mail.ivoa.net/pipermail/voevent/2020-October/003282.html |
|
On Mon, Nov 02, 2020 at 08:11:24AM -0800, Zarquan wrote:
> There's also the "Public VOEvent Streams MUST be registered" matter. Again, I'd suggest to merge into the document as-is, because there's really little choice in this matter as long as event ids are ivoids.
We do have a choice. As described on the mailing list, we can change the wording in this PR to remove the MUST and change it to say "IF you are registering a stream, this is how".
The trouble with this is that then the identifiers of the VOEvents
are invalid (or at least in contradiction to IVOA Identifiers);
there, it's clear that if you have a URI in the ivo scheme, the URI
(as usual, excluding query and fragment) must resolve in a registry.
One could conceivably to say "Register something else and build your
event ids from this"; that's still a mild violation of the URI
spirit, though (the VOEvents pretend to be "generatable" from the
registered URI.
So -- let's do the right thing and drop the ivoid requirement for
VOEvents. But that's beyond this PR and needs to be discussed after
we merge this, together with an explicit statement that
non-registered VOEvent streams are use case. The problem will also
become a bit easier to state then, because we can say "the id of a
VOEvent from a registered VOEvent stream SHOULD consist of the
stream's ivoid and an arbitrary fragment identifiers. While other
identifier schemes are allowed, VOEvent identifiers MUST NOT be built
using ivoids other that the VOEvent stream's.
Once something like that is done, we can tone down (or rather, make
more concrete, as we don't really explain what the "public" is in the
contentious sentence) the language in the registry chapter, too.
|
|
For a “VOEvent outsider” (=me, a few years ago, and other colleagues now), it is impossible to find event streams if you don’t know the people that are producing them. If I understand well, the reasons for not registering VOEvents are:
I think it is ok to have "unregistered" streams for prototyping purposes. However, if we want to raise the VOEvent standard in the era of FAIR science, we need to be able to find metadata from a VOEvent using its ID. This implies that the VOEvent stream metadata is available somewhere. To me the IVOA Registry is the natural place for the IVOA ecosystem. It would be nice to have a Note on how to do this another way (other URIs, other registry...) |
|
Another good thing with having the VOEvent stream registered in the Registry is the capability to advertise for a service that serves past events produced by that stream. For instance: |
|
I think we are just disagreeing about the sequence we do the steps.
Your proposal is to accept this PR now, as-is, with the MUST
requirement, and then look at changing the rest of the standard to
resolve the larger question about service discovery and query later.
I'm wary of adding more text reinforcing the MUST requirement because
past experience says it will be hard to remove later. I'd prefer to make
the text more permissive, using IF and SHOULD rather than MUST, and
allow people to choose.
I'd like to find a way of avoiding adding another MUST, and make a start
on moving the main text towards a more flexible and inclusive form that
enables people to have public (registered) and private (internal)
services.
If we make it easy and useful to register, then most people will
register them, without us having to resort to MUST in the specification.
If we add more MUSTs to the specification and people still don't
register their services, then we end up with more non-standard services.
As far as I know this is the only IVOA specification that actually
requires the service to be registered. It is perfectly legitimate to
create a local TAP service that is 100% standards compliant that can be
used by PyVO, Aladin and TopCat based on just the endpoint URL.
The DALI/TAP specification allows (MAY) a TAP service to include an INFO
element in the VOTable results RESOURCE element with details of the
original query, but DALI/TAP doesn't require (MUST) the results to
include a resolvable IVORN of the service.
Perhaps the DALI/TAP specification should include a include a resolvable
IVORN of the service in the results, but if it did, it would likely be a
MAY not a MUST.
…-- Dave
On 2020-11-03 08:40, msdemlei wrote:
On Mon, Nov 02, 2020 at 08:11:24AM -0800, Zarquan wrote:
> > There's also the "Public VOEvent Streams MUST be registered" matter. Again, I'd suggest to merge into the document as-is, because there's really little choice in this matter as long as event ids are ivoids.
>
> We do have a choice. As described on the mailing list, we can change
> the wording in this PR to remove the MUST and change it to say "IF you
> are registering a stream, this is how".
The trouble with this is that then the identifiers of the VOEvents
are invalid (or at least in contradiction to IVOA Identifiers);
there, it's clear that if you have a URI in the ivo scheme, the URI
(as usual, excluding query and fragment) must resolve in a registry.
One could conceivably to say "Register something else and build your
event ids from this"; that's still a mild violation of the URI
spirit, though (the VOEvents pretend to be "generatable" from the
registered URI.
So -- let's do the right thing and drop the ivoid requirement for
VOEvents. But that's beyond this PR and needs to be discussed after
we merge this, together with an explicit statement that
non-registered VOEvent streams are use case. The problem will also
become a bit easier to state then, because we can say "the id of a
VOEvent from a registered VOEvent stream SHOULD consist of the
stream's ivoid and an arbitrary fragment identifiers. While other
identifier schemes are allowed, VOEvent identifiers MUST NOT be built
using ivoids other that the VOEvent stream's.
Once something like that is done, we can tone down (or rather, make
more concrete, as we don't really explain what the "public" is in the
contentious sentence) the language in the registry chapter, too.
|
|
I agree, this is indeed a good reason for registering services in the
registry.
…-- Dave
On 2020-11-03 09:00, Baptiste Cecconi wrote:
Another good thing with having the VOEvent stream registered in the
Registry is the capability to advertise for a service that serves past
events emitted from that steam. For instance:
```
<relationship>
<!-- TAP service containing/archiving all previous
events produced by this stream -->
<relationshipType>IsSourceOf</relationshipType>
<relatedResource
ivo-id="ivo://vopdc.obspm/imcce/meteorshower/epn"
>Meteor Shower prediction EPN-TAP
service</relatedResource>
</relationship>
```
|
|
On 2020-11-03 08:58, Baptiste Cecconi wrote:
I think it is ok to have "unregistered" streams for prototyping
purposes. However, if we want to raise the VOEvent standard in the era
of FAIR science, we need to be able to find metadata from a VOEvent
using its ID. This implies that the VOEvent stream metadata is
available somewhere. To me the IVOA Registry is the natural place for
the IVOA ecosystem. It would be nice to have a Note on how to do this
another way (other URIs, other registry...)
I agree, but, I think it should be MAY not MUST.
There are many use cases for having private (unregistered) services.
Someone may want to set up a service for propagating events along an
internal data processing pipeline that is not available to the public.
Someone may want to set up a service for propagating events to a data
analysis platform that are only intended to be available to users
running notebooks on a JupyterHub inside that cloud compute platform.
Someone may want to set up a service for propagating events that are
nothing to do with astronomy.
All of which are possible with a TAP service and still remain standard
compliant, but they are are not possible with a standards compliant
VOEvent service because we entangle the event identifier with the
service registration.
Dis-entangling the event identifier from the service registration is
likely to be a gradual process, with many changes to the specification.
It would be good if we could make this PR into a step along that path,
by changing the emphasis of the wording to be IF and MAY rather than
MUST.
…-- Dave
--------
Dave Morris
Research Software Engineer
Wide Field Astronomy Unit
Institute for Astronomy
University of Edinburgh
--------
|
|
On Tue, Nov 03, 2020 at 06:46:05AM -0800, Zarquan wrote:
I think we are just disagreeing about the sequence we do the steps.
Your proposal is to accept this PR now, as-is, with the MUST
requirement, and then look at changing the rest of the standard to
resolve the larger question about service discovery and query later.
I'm wary of adding more text reinforcing the MUST requirement because
past experience says it will be hard to remove later. I'd prefer to make
Don't worry. If it doesn't go into REC, it'll be easy to remove it.
Promised. I'll write the PR myself if nobody else jumps in right
after we've worked out how we'd like to tear the IVOIDs out of the
VOEvent core.
I'd like to find a way of avoiding adding another MUST, and make a start
on moving the main text towards a more flexible and inclusive form that
enables people to have public (registered) and private (internal)
services.
As long as we don't get rid of the IVOIDs as VOEvent ids, I don't
think there is a non-insane way. The requirement to register is
already in the current spec, it's just not spelled out. True, it's a
bit odd that there's not a single registered VOEvent stream
nevertheless, but that's why we're here.
As far as I know this is the only IVOA specification that actually
requires the service to be registered. It is perfectly legitimate to
If a standard requires the use of an IVOID, it requires people to
register; I think you're right that no other DAL standard exactly
does this. As I said: getting VOEvent to where the DAL standards are
is the goal of the next step.
The DALI/TAP specification allows (MAY) a TAP service to include an INFO
element in the VOTable results RESOURCE element with details of the
original query, but DALI/TAP doesn't require (MUST) the results to
include a resolvable IVORN of the service.
[Incidentially, we've deprecated the word IVORN in Identifiers 2;
let's just converge on IVOID]. Right, they can afford to be
cavalier with the registration because they don't do this. But if we
try to go from fixing the current breakage (implicit registry
requirement) to having optional registry involvement (and hence an
alternative, non-IVOID identifier scheme) in one step, it'll take
forever, and the discussions will be a lot more complex.
I'm sure that isolating the discussion of how to register from how to
have enable standards-compliant unregistered VOEvent services (which
we currently don't have) really helps.
|
|
I agree with @msdemlei that it is only a PR, not a REC. The current REC specification V2.0 states:
This paragraph says the VOEvent In case (a), the packets are registered, and so should be the stream. The proposed general form implies that the stream is registered and a mechanism is provided to access to packets (for instance, using an related TAP service, as in the meteor showers prediction example). In case (b), i.e., using an external VOEvent repository, it would be nice to have a place in the registry to advertise this if the stream is public and persistent. Further down is the document, we read:
So, the goal is to register VOEvent streams in the VO Registry, so that packet metadata and content can be found retrospectively. I thus tend to agree with @msdemlei about the general requirement of having a registry record for a stream, when a stream is intended to be discoverable and interoperable. I also agree with @Zarquan that we should have a way to have non-public and unregistered streams (such as demo or prototype services), implying that the stream will not be findable (hence the packets) unless people have the details at hand. In the latter case, the For the identifier topic: I have raised an issue about allow other |
|
Dave,
On Tue, Nov 03, 2020 at 03:11:03PM +0000, Dave Morris wrote:
On 2020-11-03 08:58, Baptiste Cecconi wrote:
Dis-entangling the event identifier from the service registration is likely
to be a gradual process, with many changes to the specification.
Let's see -- I don't think it's that hard, because, really, no actual
way to "resolve" VOEvents in a meaningful way is forseen in the
current draft either, as there is no archiving facility. Hence, as
long as we suggest a way alternative identifier schemes still have
(strong-ish) guarantees of identifier uniqueness, I'd say they ought
to just work.
It would be good if we could make this PR into a step along that path, by
changing the emphasis of the wording to be IF and MAY rather than MUST.
I'll not quarrel on forever, and it's a minor point as the wording
won't stand forever either way, but one last attempt: Won't you agree
that it's preferable if each step in document evolution is consistent
within itself? If we say MAY here while there's a MUST on ivoids,
the draft has an internal inconsistency. I'd like to avoid that, in
particular with a view to discussing why we'd like to open up to
other identifier schemes.
But, really, few people will see that intermediate draft, so from my
end if you can't live with the MUST, change it in some appropriate
way before the merge -- but let's merge, and then try things out
while continuing on the identifiers issue where that belongs.
…-- Markus
|
This PR adds a section on how to register VOEvent streams. It introduces a simple registry extension so interfaces using different protocols (VTP, kafka, XMPP) can be told apart.
An open question for (I'd suggest) later resolution: Do we need to define the format of the events in the stream ("VOEvent 2 in XML serialisation") already at this point?
There's also the "Public VOEvent Streams MUST be registered" matter. Again, I'd suggest to merge into the document as-is, because there's really little choice in this matter as long as event ids are ivoids. So, if we want to relax this, we'll have to discuss the question of VOEvent ids as well.