Access more spinedb_api functionality in SpineInterface

[DillonJ]

This was discussed at our roadmap session earlier today.

Currently SpineOpt allows us to read the contents of a data store and also gives us the functionality of Spinedb_api.import_data. However there are many use cases where we would like to access more api functionlity like renaming object classes for example. This would allow us to create database migration scripts within SpineInterface.

A problem in all of this is that spinedb_api is written in python and we are trying to minimise use of Julia's PyCall. For this, communications between spinedb_api and SpineInterface happen via sockets where possible.

The options to expose more spinedb_api functionilty to SpineInterface were as follows:

• re-implement spinedb_api in a language with bindings to both Python and Julia (e.g. C++ doesn't have SQAlchemy)
• develop the sockets functionality so it allows us to do more in SpineInterface. This might require enablement of a standalone socket server

@manuelma have I covered the options here? I may have missed something.

[manuelma]

I think so @DillonJ

The C bindings are certainly interesting but I have no idea how it would work. @soininen ?
The socket API extension is doable now. I'm thinking that a simple approach would be to add an entry point to call a DatabaseMapping method by name.

[manuelma]

I think we need to wait for the actual use-case to become apparent, before implementing any of this. I know we agree that it sounds like a good idea, but it's a little bit of work to implement just in case someone wants to use it. We haven't decided if we want to write migration scripts in Julia yet.

[DillonJ]

Well, one of the use cases was the data migration script - so we could do it in Julia rather than in toolbox... but perhaps that's not enough right now?

[soininen]

The C bindings are certainly interesting but I have no idea how it would work.

It is a pretty standard techinque to provide access to certain library from other programming languages. What you need is a wrapper that translates calls from you programming language to the target library. A familiar example would be PySide2 which provides Python bindings for the Qt API. You can write the bindings manually (see e.g. Python's ctypes module) but usually a generator is used, for instance PySide2 is generated by Shiboken2.

[manuelma]

Thank you @soininen for the explanation.

Ok, so this would mean 'packing' spinedb_api as a C library that can be distributed together with/made accessible to SpineInterface.jl, plus implementing/generating the julia bindings. Right?

Feels powerful but nothing I would want to implement right now.

[manuelma]

The socket server now has a couple of additional entry points that allow one to do everything that's available in the DatabaseMapping class, so same set of db manipulations. The only caveat is we can't pass keyword args at the moment.

[manuelma]

What does it mean to enable a standalone socket server @DillonJ ?

[soininen]

Right?

Right. I agree it would be such a big focus shift that we should concentrate on spinedb_api-as-server for now.

What does it mean to enable a standalone socket server?

I guess this means SpineDBServer. It should be possible to run that outside of Toolbox and that is already the case if I got this right.

[manuelma]

I guess this means SpineDBServer. It should be possible to run that outside of Toolbox and that is already the case if I got this right.

That's right, you can import spinedb_api and then run spinedb_api.start_spine_db_server(db_url) or something like that, so it's good.

I think we can close this one and then open another if my API extension was insufficient, but as I said, you can now do anything that you can do with the DatabaseMapping class (except passing keyword arguments).

[soininen]

I wonder if we want to provide a server 'executable' in the long run: a python script that would start the server as a daemon or service that listens to a port on localhost. The server could then be started from any other programming language.

[DillonJ]

What does it mean to enable a standalone socket server?

I think this is largely irrelevant now, but this was just a note I took from our discussion and I have absolutely no deeper knowledge than that :-)