Feature: v2 Course & Resources APIs

[panaaj]

This PR supercedes PR #1381 Course and Resources API as it is a merge of #1381 with the current main branch (v1.45.0).

Implements:

• Resources API
• Course API

Course API emits v1 courseGreatCircle and courseRhumbline deltas for backwards compatibility.

Reference the following for tasks to address deprecating v1 paths at the desired time.

[panaaj]

@tkurki @sbender9 Interested in your thoughts on the following....

As I see it there are currently two main sources of information to populate the SK data model:

1. NMEA data stream
2. SK API

This PR currently addresses the population and maintenance of SK paths related to resources and setting a course via 2. SK API.

So the question is... what should happen, if anything, to course / destination information coming in via NMEA that is destined for SK paths set via the course API? as the incoming NMEA values may:

• conflict with values set via API (e.g. the source NMEA device may have a different setting)
• not provide values for all required SK paths.

I would think that the values set via the API should prevail.

[tkurki]

A good point.

I don't quite follow your reasoning - why would setting things via API be different from setting navigation destination via some other means? In the ideal world I should be able to set navigation mode and params, say Go To Point, on any device and things should JustWork ™️ .

We should think about and experiment with the different scenarios and how to tackle them. A setting for navigation priorities? Notifications about quarrelling inputs and a way to resolve them?

One way forward would be to list some of these scenarios, on what we know how things work with 0183 and N2K, and how they could be resolved with minimal confusion & fuss.

One way to resolve this issue would be altering the data model so that we can capture multiple navigation setups: the internal. The same issue rises with multiple MFDs on the bus. I wonder how that scenario works today?

We got multiple values not quite right in v1. We should do better this time.

[tkurki]

As it stands this PR adds quite a bunch of deltas to the initial set that is sent upon ws connection. This is added even if the user has never used course api. I think we should instead be lazy: you should start seeing course API data only after you start using it. That should be perfectly possible, given that the course api state is persisted separately.

[tkurki]

...forgot to say that I can have a look at suppressing the course api data until activation.

[tkurki]

Another point that popped up when coming back to this branch after a while: I found it odd that for example POSTing and GETting waypoints use different data formats. I am starting to think they should both use the same Geojson format. Writing one and getting another is definitely not the principle of least surprise....

[tkurki]

@wellenvogel AvNav is a pretty complete UI for the features in this branch: creating and modifying waypoints and routes & activating & stopping navigation as well as associated operations such as setting/advancing the next point. I would love to have your insight on these apis!

The easiest to way to get an overview is probably running the docker branch from this image: docker run -it --rm -p 3000:3000 --entrypoint /home/node/signalk/bin/signalk-server signalk/signalk-server:resources_course_api --sample-nmea0183-data. Then the Swagger UIs are at http://localhost:3000/admin/openapi/?urls.primaryName=course and http://localhost:3000/admin/openapi/?urls.primaryName=resources

[panaaj]

Another point that popped up when coming back to this branch after a while: I found it odd that for example POSTing and GETting waypoints use different data formats. I am starting to think they should both use the same Geojson format. Writing one and getting another is definitely not the principle of least surprise....

The thinking here was to allow simple resource creation by providing minimal data (abstracted from the schema) e.g. create a waypoint by providing only lat/lon and the API takes care of the rest.
This makes most sense when creating resources hence POST.

PUT allows the use of GeoJSON to create or replace a record... so in this way both scenarios are covered providing some flexibility.

[tkurki]

Yeah, my point is that I think consistency in the data format for create and read operations outweigh the handiness of the more compact format.

i can do the change on the server (once I fix the mysterious test failures) if you could do Freeboard afterwards?

[tkurki]

Make that docker run -it --rm -p 3000:3000 --init --entrypoint /usr/lib/node_modules/signalk-server/bin/signalk-server signalk/signalk-server:v2_api_base --sample-nmea0183-data for the latest version.

[wellenvogel]

Currently I'm busy with some project on my side.
Anyway will try to have a look...
What always seemed to be important to me:
If you have an app (like AvNav) that has some own storage for things like routes, way points and similar it is crucial to be able to sync both data storages in a simple way.
Currently this is rather complicated (had a hard time doing this for data that I currently integrate - especially for notifications).
I would prefer an option to get notified about any changes on the server side (similar to the deltas) but with additionally getting all the data at regular intervals (to recover from any lost data situations).
And a clear concept of source identifiers is necessary so that you can easily detect whether a change has been issued by yourself or not.
Just imagine a simple situation like editing a route at 2 clients at the same time - whenever a client receives a change from the server it has to know if it was triggered by himself.

Functional question:
Do you intend to implement some routing function in the server?
Could be some plugin of course...
And if yes it needs a clear concept of "mastership" - will the routing function honor changes of the current routing related data or will it only write this data.

[tkurki]

intend to implement some routing function in the server?

To me routing or autorouting means that something can create a (safe) route from point a to point b. That is not in the scope of this PR nor in the plans. This PR should expose the APIs for a plugin to manipulate routes, so a plugin could do that.

What is in scope is broadcasting changes in the navigation state: when you activate/deactivate navigation (to a point/waypoint or along a route) the server emits data like so https://github.com/SignalK/signalk-server/pull/1477/files#diff-1751c3e712c0afe5b944ef930a6ec6eafce600e8132886fe7f336c52a11468d4R735-R793

[wellenvogel]

Ok, autorouting was not the point...
But typically you need some routing engine that switches to the next way point in a route once you passed the previous one.

[panaaj]

Currently this is delegated to the Course Provider plugin which does the course calculations and raises a "notification" when perpendicular is passed. This can then be acted on accordingly.
The auto-advance could be built into the server behind a "setting" to enable/disable as required but since the calculations are occuring in a plugin, it makes more sense for the plugin to provide the auto-advance functionality.

[panaaj]

Currently this is rather complicated (had a hard time doing this for data that I currently integrate - especially for notifications).
I would prefer an option to get notified about any changes on the server side (similar to the deltas) but with additionally getting all the data at regular intervals (to recover from any lost data situations).

All resource adds, updates, deletes are emitted as deltas so changes to resources are available to clients by subscribing to /resources/*

[panaaj]

@tkurki Since the refactor it appears that requests to add, update, delete resources are being met with a 403 Unauthorised response due to updateAllowed() returning false.

signalk-server/src/api/resources/index.ts

Line 144 in 3b3c1af

const updateAllowed = (req: Request): boolean => {

This occurs even if security is not enabled.
Could you please review.

[tkurki]

Without security POST, GET and PUT work for me from the Swagger UI with the server launched via the above mentioned docker method and locally built native node.

[tkurki]

The resource ids all have the urn:mrn:signalk:uuid: prefix that was added to the v1 resource schema back in 2016. I am not aware of it actually being used anywhere or being useful somewhere. I think this was inspired by somebody who was working with what eventually became https://datatracker.ietf.org/doc/html/draft-knielsen-mrn-urn-01.

In practise it adds some, mostly meaningless extra characters to the already pretty unwieldy uuid values.

I'd be willing to go with the naked uuids as resource ids. I think this could be the time to do this, as we are setting fundamentals for v2 resource api.

What do you think @panaaj ?

[panaaj]

docker run -it --rm -p 3000:3000 --init --entrypoint /usr/lib/node_modules/signalk-server/bin/signalk-server signalk/signalk-server:v2_api_base

It works for me OK when using docker but using the local code is where I have the issue (where it wasn't an issue before).
Clearly something local.

[panaaj]

I'd be willing to go with the naked uuids as resource ids. I think this could be the time to do this, as we are setting fundamentals for v2 resource api.

I have no issues with moving to just a uuid.

The impacts I foresee for existing implementations / applications are:

• stored resources with references to other resources (e.g. notes referencing a region)
• stored settings where resource ids are persisted in preferences / settings (e.g. selected resources for display on map)

These would require apps to detect v2 api and make the necessary adjustments.

[tkurki]

Since we don’t do any validation (?) this mostly affects assigning new ids on POSTs and examples in the api specs I think.

[panaaj]

I have started work on removing the prefix in the API and also the resource-provider plugin.

[panaaj]

@tkurki I have pushed updates to

1. update resource POST payloads to align with those of PUT requests
2. Removed the uuid prefix from resources.

[tkurki]

This does not look correct. Missing a period? Same below.

[panaaj]

fixed

[tkurki]

Any ideas where the value string is coming from? I created a route and activated it.

[panaaj]

Does the route have a name (in it's definition)?
The value should be the name of the route

Line 466: src/api/course/index.ts

    if (route.href) {
      rte = await this.getRoute(route.href)
      if (!rte) {
        throw new Error(
          `** Could not retrieve route information for ${route.href}`
        )
      }
      if (!Array.isArray(rte.feature?.geometry?.coordinates)) {
        throw new Error(`Invalid route coordinate data! (${route.href})`)
      }
    } else {
      throw new Error('Route information not supplied!')
    }

    const newCourse: CourseInfo = { ...this.courseInfo }

    // set activeroute
    newCourse.activeRoute.href = route.href

    newCourse.startTime = new Date().toISOString()

    newCourse.activeRoute.name = rte.name
    newCourse.activeRoute.waypoints = this.getRoutePoints(rte)

It seems to be working for me using Freeboard beta 4.
(I have just published Freeboard beta 4 which contains updates for the most recent changes in this PR.)

How did you activate it... using Freeboard or curl?

Here is the result I get

{
	"activeRoute": {
		"href": "/resources/routes/6b23c37c-a13c-4cad-a774-562b4ce134ec",
		"pointIndex": 0,
		"pointTotal": 4,
		"reverse": false,
		"name": "rt3",
		"waypoints": [{
			"position": {
				"latitude": -35.04392514341443,
				"longitude": 138.31563470807106
			}
		}, {
			"position": {
				"latitude": -35.087453067356904,
				"longitude": 138.3175511229639
			}
		}, {
			"position": {
				"latitude": -35.10155705655288,
				"longitude": 138.3549528776733
			}
		}, {
			"position": {
				"latitude": -35.066672359569836,
				"longitude": 138.37743908836438
			}
		}]
	},
	"nextPoint": {
		"href": null,
		"type": "RoutePoint",
		"position": {
			"latitude": -35.04392514341443,
			"longitude": 138.31563470807106
		},
		"arrivalCircle": 4000
	},
	"previousPoint": {
		"href": null,
		"type": "VesselPosition",
		"position": {
			"longitude": 138.40021242628814,
			"latitude": -35.04938213007765
		}
	},
	"startTime": "2023-01-01T05:58:25.620Z",
	"targetArrivalTime": "2022-08-07T15:32:03Z"
}

[tkurki]

Ooops, I did that in Swagger and name and description had no example defined in ´RoutePostModel` (that is now gone per your restructuring). Re-added sample data in 058daec.