Explore adopting autogenerated python SDK

[nitrosx]

SciCat backend has a matching python SDK autogenerated for each new release:

• https://github.com/SciCatProject/scicat-backend-next/deployments/python-pydantic-v1-sdk-package
• https://github.com/SciCatProject/scicat-backend-next/deployments/python-sdk-package
  which are also published on Pypi
• https://pypi.org/project/scicat-sdk-pydantic/
• https://pypi.org/project/scicat-sdk-py/

I wonder if we could adopt one of this new SDK in Scitacean

[jl-wynen]

The SDK seems broken. Hyphens are not allowed in Python package names, so it can't be imported.

Is there a documentation somewhere about how to use it? The interface looks quite low level.

[nitrosx]

We should address this issue and brainstorm with @Junjiequan. He is the scicat SDK guru

[Junjiequan]

# scicat-sdk-py
This is the API for the SciCat Backend

This Python package is automatically generated by the [OpenAPI Generator](https://openapi-generator.tech) project:

- API version: api/v3
- Package version: latest
- Generator version: 7.10.0-SNAPSHOT
- Build package: org.openapitools.codegen.languages.PythonClientCodegen

## Requirements.

Python 3.7+

## Installation & Usage
### pip install

If the python package is hosted on a repository, you can install directly using:

```sh
pip install git+https://github.com/GIT_USER_ID/GIT_REPO_ID.git

(you may need to run pip with root permission: sudo pip install git+https://github.com/GIT_USER_ID/GIT_REPO_ID.git)

Then import the package:

import scicat-sdk-py

Setuptools

Install via Setuptools.

python setup.py install --user

(or sudo python setup.py install to install the package for all users)

Then import the package:

import scicat-sdk-py

Tests

Execute pytest to run the tests.

Getting Started

Please follow the installation procedure and then run the following:

import scicat-sdk-py
from scicat-sdk-py.rest import ApiException
from pprint import pprint

# Defining the host is optional and defaults to http://localhost
# See configuration.py for a list of all supported configuration parameters.
configuration = scicat-sdk-py.Configuration(
    host = "http://localhost"
)



# Enter a context with an instance of the API client
with scicat-sdk-py.ApiClient(configuration) as api_client:
    # Create an instance of the API class
    api_instance = scicat-sdk-py.AdminApi(api_client)

    try:
        api_response = api_instance.admin_controller_get_config()
        print("The response of AdminApi->admin_controller_get_config:\n")
        pprint(api_response)
    except ApiException as e:
        print("Exception when calling AdminApi->admin_controller_get_config: %s\n" % e)

[Junjiequan]

Documentation for API Endpoints

All URIs are relative to http://localhost

Class	Method	HTTP request	Description
AdminApi	admin_controller_get_config	GET /api/v3/admin/config
AdminApi	admin_controller_get_theme	GET /api/v3/admin/theme
AuthApi	auth_controller_ldap_login	POST /api/v3/auth/ldap	Endpoint to authenticate users through an ldap service.
AuthApi	auth_controller_login	POST /api/v3/auth/login
AuthApi	auth_controller_login_callback	GET /api/v3/auth/oidc/callback
AuthApi	auth_controller_logout	POST /api/v3/auth/logout	It logs the current user out.
AuthApi	auth_controller_msad_login	POST /api/v3/auth/msad	Legacy endpoint to authenticate users through an ldap service.
AuthApi	auth_controller_oidc_login	GET /api/v3/auth/oidc
AuthApi	auth_controller_whoami	GET /api/v3/auth/whoami
DatasetsApi	datasets_controller_append_to_array_field	POST /api/v3/datasets/{pid}/appendToArrayField	It appends a new value to the specific field.
DatasetsApi	datasets_controller_count	GET /api/v3/datasets/count	It returns the number of datasets.
DatasetsApi	datasets_controller_create	POST /api/v3/datasets	It creates a new dataset which can be a raw or derived one.
DatasetsApi	datasets_controller_create_attachment	POST /api/v3/datasets/{pid}/attachments	It creates a new attachement for the dataset specified.
DatasetsApi	datasets_controller_create_datablock	POST /api/v3/datasets/{pid}/datablocks	It creates a new datablock for the dataset specified.
DatasetsApi	datasets_controller_create_orig_datablock	POST /api/v3/datasets/{pid}/origdatablocks	It creates a new origDatablock for the dataset specified.
DatasetsApi	datasets_controller_find_all	GET /api/v3/datasets	It returns a list of datasets.
DatasetsApi	datasets_controller_find_all_attachments	GET /api/v3/datasets/{pid}/attachments	It returns all the attachments for the dataset specified.
DatasetsApi	datasets_controller_find_all_datablocks	GET /api/v3/datasets/{pid}/datablocks	It returns all the datablock for the dataset specified.
DatasetsApi	datasets_controller_find_all_orig_datablocks	GET /api/v3/datasets/{pid}/origdatablocks	It returns all the origDatablock for the dataset specified.
DatasetsApi	datasets_controller_find_by_id	GET /api/v3/datasets/{pid}	It returns the dataset requested.
DatasetsApi	datasets_controller_find_by_id_and_delete	DELETE /api/v3/datasets/{pid}	It deletes the dataset.
DatasetsApi	datasets_controller_find_by_id_and_replace	PUT /api/v3/datasets/{pid}	It updates the dataset.
DatasetsApi	datasets_controller_find_by_id_and_update	PATCH /api/v3/datasets/{pid}	It partially updates the dataset.
DatasetsApi	datasets_controller_find_logbook_by_pid	GET /api/v3/datasets/{pid}/logbook	Retrive logbook associated with dataset.
DatasetsApi	datasets_controller_find_one	GET /api/v3/datasets/findOne	It returns the first dataset found.
DatasetsApi	datasets_controller_find_one_attachment_and_remove	DELETE /api/v3/datasets/{pid}/attachments/{aid}	It deletes the attachment from the dataset.
DatasetsApi	datasets_controller_find_one_attachment_and_update	PUT /api/v3/datasets/{pid}/attachments/{aid}	It updates the attachment specified for the dataset indicated.
DatasetsApi	datasets_controller_find_one_datablock_and_remove	DELETE /api/v3/datasets/{pid}/datablocks/{did}	It deletes the datablock from the dataset.
DatasetsApi	datasets_controller_find_one_datablock_and_update	PATCH /api/v3/datasets/{pid}/datablocks/{did}	It updates the datablocks specified for the dataset indicated.
DatasetsApi	datasets_controller_find_one_orig_datablock_and_remove	DELETE /api/v3/datasets/{pid}/origdatablocks/{oid}	It deletes the origdatablock from the dataset.
DatasetsApi	datasets_controller_find_one_orig_datablock_and_update	PATCH /api/v3/datasets/{pid}/origdatablocks/{oid}	It updates the origDatablocks specified for the dataset indicated.
DatasetsApi	datasets_controller_fullfacet	GET /api/v3/datasets/fullfacet	It returns a list of dataset facets matching the filter provided.
DatasetsApi	datasets_controller_fullquery	GET /api/v3/datasets/fullquery	It returns a list of datasets matching the query provided.
DatasetsApi	datasets_controller_is_valid	POST /api/v3/datasets/isValid	It validates the dataset provided as input.
DatasetsApi	datasets_controller_metadata_keys	GET /api/v3/datasets/metadataKeys	It returns a list of metadata keys contained in the datasets matching the filter provided.
DatasetsApi	datasets_controller_orig_datablock_is_valid	POST /api/v3/datasets/{pid}/origdatablocks/isValid	It validates the origDatablock values passed.
DatasetsApi	datasets_controller_thumbnail	GET /api/v3/datasets/{pid}/thumbnail	It returns the thumbnail associated with the dataset.
DefaultApi	health_controller_check	GET /api/v3/health
ElasticSearchApi	elastic_search_service_controller_create_index	POST /api/v3/elastic-search/create-index
ElasticSearchApi	elastic_search_service_controller_delete_index	POST /api/v3/elastic-search/delete-index
ElasticSearchApi	elastic_search_service_controller_fetch_es_results	POST /api/v3/elastic-search/search
ElasticSearchApi	elastic_search_service_controller_get_index	GET /api/v3/elastic-search/get-index
ElasticSearchApi	elastic_search_service_controller_sync_database	POST /api/v3/elastic-search/sync-database
ElasticSearchApi	elastic_search_service_controller_update_index	POST /api/v3/elastic-search/update-index
InstrumentsApi	instruments_controller_create	POST /api/v3/instruments
InstrumentsApi	instruments_controller_find_all	GET /api/v3/instruments
InstrumentsApi	instruments_controller_find_by_id	GET /api/v3/instruments/{id}
InstrumentsApi	instruments_controller_find_one	GET /api/v3/instruments/findOne	It returns the first instrument found.
InstrumentsApi	instruments_controller_remove	DELETE /api/v3/instruments/{id}
InstrumentsApi	instruments_controller_update	PATCH /api/v3/instruments/{id}
JobsApi	jobs_controller_create	POST /api/v3/jobs
JobsApi	jobs_controller_find_all	GET /api/v3/jobs
JobsApi	jobs_controller_find_one	GET /api/v3/jobs/{id}
JobsApi	jobs_controller_fullfacet	GET /api/v3/jobs/fullfacet
JobsApi	jobs_controller_fullquery	GET /api/v3/jobs/fullquery
JobsApi	jobs_controller_remove	DELETE /api/v3/jobs/{id}
JobsApi	jobs_controller_update	PATCH /api/v3/jobs/{id}
LogbooksApi	logbooks_controller_find_all	GET /api/v3/logbooks
LogbooksApi	logbooks_controller_find_by_name	GET /api/v3/logbooks/{name}
OrigdatablocksApi	orig_datablocks_controller_create	POST /api/v3/origdatablocks	It creates a new orig datablock for the specified dataset.
OrigdatablocksApi	orig_datablocks_controller_find_all	GET /api/v3/origdatablocks	It returns a list of orig datablocks.
OrigdatablocksApi	orig_datablocks_controller_find_by_id	GET /api/v3/origdatablocks/{id}	It retrieve the origdatablock.
OrigdatablocksApi	orig_datablocks_controller_fullfacet	GET /api/v3/origdatablocks/fullfacet
OrigdatablocksApi	orig_datablocks_controller_fullfacet_files	GET /api/v3/origdatablocks/fullfacet/files
OrigdatablocksApi	orig_datablocks_controller_fullquery	GET /api/v3/origdatablocks/fullquery
OrigdatablocksApi	orig_datablocks_controller_fullquery_files	GET /api/v3/origdatablocks/fullquery/files
OrigdatablocksApi	orig_datablocks_controller_is_valid	POST /api/v3/origdatablocks/isValid	It validates the orig datablock provided as input.
OrigdatablocksApi	orig_datablocks_controller_remove	DELETE /api/v3/origdatablocks/{id}	It deletes the origdatablock.
OrigdatablocksApi	orig_datablocks_controller_update	PATCH /api/v3/origdatablocks/{id}	It updates the origdatablock.
PoliciesApi	policies_controller_count	GET /api/v3/policies/count
PoliciesApi	policies_controller_create	POST /api/v3/policies
PoliciesApi	policies_controller_find_all	GET /api/v3/policies
PoliciesApi	policies_controller_find_one	GET /api/v3/policies/{id}
PoliciesApi	policies_controller_remove	DELETE /api/v3/policies/{id}
PoliciesApi	policies_controller_update	PATCH /api/v3/policies/{id}
PoliciesApi	policies_controller_update_where	POST /api/v3/policies/updateWhere
ProposalsApi	proposals_controller_create	POST /api/v3/proposals	It creates a new proposal.
ProposalsApi	proposals_controller_create_attachment	POST /api/v3/proposals/{pid}/attachments	It creates a new attachement for the proposal specified.
ProposalsApi	proposals_controller_find_all	GET /api/v3/proposals	It returns a list of proposals.
ProposalsApi	proposals_controller_find_all_attachments	GET /api/v3/proposals/{pid}/attachments	It returns all the attachments for the proposal specified.
ProposalsApi	proposals_controller_find_all_datasets	GET /api/v3/proposals/{pid}/datasets	It returns all the datasets associated with the proposal indicated.
ProposalsApi	proposals_controller_find_by_id	GET /api/v3/proposals/{pid}	It returns the proposal requested.
ProposalsApi	proposals_controller_find_by_id_access	GET /api/v3/proposals/{pid}/authorization	Check user access to a specific proposal.
ProposalsApi	proposals_controller_find_one_attachment_and_remove	DELETE /api/v3/proposals/{pid}/attachments/{aid}	It deletes the attachment from the proposal.
ProposalsApi	proposals_controller_find_one_attachment_and_update	PATCH /api/v3/proposals/{pid}/attachments/{aid}	It updates the attachment specified for the proposal indicated.
ProposalsApi	proposals_controller_fullfacet	GET /api/v3/proposals/fullfacet	It returns a list of proposal facets matching the filter provided.
ProposalsApi	proposals_controller_fullquery	GET /api/v3/proposals/fullquery	It returns a list of proposals matching the query provided.
ProposalsApi	proposals_controller_is_valid	POST /api/v3/proposals/isValid	It validates the proposal provided as input.
ProposalsApi	proposals_controller_remove	DELETE /api/v3/proposals/{pid}	It deletes the proposal.
ProposalsApi	proposals_controller_update	PATCH /api/v3/proposals/{pid}	It updates the proposal.
PublishedDataApi	published_data_controller_count	GET /api/v3/publisheddata/count
PublishedDataApi	published_data_controller_create	POST /api/v3/publisheddata
PublishedDataApi	published_data_controller_find_all	GET /api/v3/publisheddata
PublishedDataApi	published_data_controller_find_one	GET /api/v3/publisheddata/{id}	It returns the published data requested.
PublishedDataApi	published_data_controller_form_populate	GET /api/v3/publisheddata/formpopulate
PublishedDataApi	published_data_controller_register	POST /api/v3/publisheddata/{id}/register
PublishedDataApi	published_data_controller_remove	DELETE /api/v3/publisheddata/{id}
PublishedDataApi	published_data_controller_resync	POST /api/v3/publisheddata/{id}/resync	Edits published data.
PublishedDataApi	published_data_controller_update	PATCH /api/v3/publisheddata/{id}
SamplesApi	samples_controller_create	POST /api/v3/samples	It creates a new sample.
SamplesApi	samples_controller_create_attachments	POST /api/v3/samples/{id}/attachments	It creates a new attachment related with this sample.
SamplesApi	samples_controller_find_all	GET /api/v3/samples	It returns a list of samples
SamplesApi	samples_controller_find_all_attachments	GET /api/v3/samples/{id}/attachments	It returns the attachments related to a specific sample.
SamplesApi	samples_controller_find_all_datasets	GET /api/v3/samples/{id}/datasets	It returns the datasets related to a specific sample.
SamplesApi	samples_controller_find_by_id	GET /api/v3/samples/{id}	It returns the sample requested.
SamplesApi	samples_controller_find_by_id_access	GET /api/v3/samples/{id}/authorization	Check user access to a specific sample.
SamplesApi	samples_controller_find_one	GET /api/v3/samples/findOne	It returns a sample matching the query provided.
SamplesApi	samples_controller_find_one_attachment	GET /api/v3/samples/{id}/attachments/{fk}	It returns the attachment related to a specific sample.
SamplesApi	samples_controller_find_one_attachment_and_remove	DELETE /api/v3/samples/{id}/attachments/{fk}	It deletes the attachment related to a specific sample.
SamplesApi	samples_controller_fullquery	GET /api/v3/samples/fullquery	It returns a list of samples matching the query provided.
SamplesApi	samples_controller_metadata_keys	GET /api/v3/samples/metadataKeys	It returns a list of sample metadata keys matching the query provided.
SamplesApi	samples_controller_remove	DELETE /api/v3/samples/{id}	It deletes the sample.
SamplesApi	samples_controller_update	PATCH /api/v3/samples/{id}	It updates the sample.
UserIdentitiesApi	user_identities_controller_find_one	GET /api/v3/useridentities/findOne	It returns the user identity profile of the first user matching the query
UserIdentitiesApi	user_identities_controller_is_valid_email	GET /api/v3/useridentities/isValidEmail	It returns true if the emailed passed in is linked to any registered users
UsersApi	users_controller_can_user_create_dataset	GET /api/v3/users/{id}/authorization/dataset/create
UsersApi	users_controller_create_custom_jwt	POST /api/v3/users/{id}/jwt	It creates a new jwt token for the user specified.
UsersApi	users_controller_create_settings	POST /api/v3/users/{id}/settings
UsersApi	users_controller_find_by_id	GET /api/v3/users/{id}
UsersApi	users_controller_get_my_settings	GET /api/v3/users/my/settings
UsersApi	users_controller_get_my_user	GET /api/v3/users/my/self	Returns the information of the user currently logged in.
UsersApi	users_controller_get_my_user_identity	GET /api/v3/users/my/identity
UsersApi	users_controller_get_settings	GET /api/v3/users/{id}/settings
UsersApi	users_controller_get_user_identity	GET /api/v3/users/{id}/userIdentity
UsersApi	users_controller_get_user_jwt	POST /api/v3/users/jwt	It creates a new jwt token.
UsersApi	users_controller_login	POST /api/v3/users/login	This endpoint is deprecated and will be removed soon. Use /auth/login instead
UsersApi	users_controller_logout	POST /api/v3/users/logout	It logs the current user out.
UsersApi	users_controller_patch_external_settings	PATCH /api/v3/users/{id}/settings/external
UsersApi	users_controller_patch_settings	PATCH /api/v3/users/{id}/settings
UsersApi	users_controller_remove_settings	DELETE /api/v3/users/{id}/settings
UsersApi	users_controller_update_settings	PUT /api/v3/users/{id}/settings

[Junjiequan]

@nitrosx @jl-wynen
I did a quick search, it seems there's no easy way to import python package with hyphon directly.
Maybe the package name change is needed, e.g, scicat_sdk_py. Other than that, we also need to include this documentation somewhere. Currently the documentation can only be observed from the generated SDK folder.

[nitrosx]

@Junjiequan either scicat_sdk_py or scicatSdkPy works for me.
Unless @jl-wynen, being the python guru, has any suggestions, please select the one that you like the best.

[nitrosx]

One more question: do we need the long names, like users_controller_find_all?
Could we configure the generation to create function with names like user_find_all? aka remove controller from the names.

[Junjiequan]

@nitrosx
From the configuration provided by open API officially
I do not see a straightforward way to shorten the naming

[nitrosx]

@Junjiequan OK. Let's not worry about that.
Let's get all the relevant feedback from @jl-wynen regarding names and pypi package and make sure that we include those, so he can evaluate using the sdk in scitacean.

Thank you both

[jl-wynen]

The package has to be renamed. It is possible to keep the name on pypi.org with kebab case and only rename the actual folder in the project. The name that is used on PyPI and when pip installing comes from project.name in pyproject.toml. The file names are simply the names in the source tree. Although it is possible possible to have different names, I personally would advocate for changing both and keeping them the same to avoid confusion.

As to the convention, Python packages names normally use snake_case. CamelCase is only used for class names in Python.

[nitrosx]

Thanks for the feedback @jl-wynen.
We might ask for your help if we run into issues.

[Junjiequan]

@jl-wynen New SDK package is uploaded with underscore, and now you should be able to import the package.
The project name however cannot contain any underscore https://pybuilder.io/documentation/publishing-plugins#:~:text=Pip%20replaces%20underscores%20with%20dashes,a%20syntax%20error%20at%20import.

[jl-wynen]

It doesn't work:

import scicat_sdk_pydantic

cfg = scicat_sdk_pydantic.Configuration(
    host="https://staging.scicat.ess.eu"
)

with scicat_sdk_pydantic.ApiClient(cfg) as client:
    inst = scicat_sdk_pydantic.DatasetsApi(client)
    response = inst.datasets_controller_find_by_id("20.500.12269/a76901e4-7e3a-4bd1-be09-6845f1f29b83")
    print(response)

raises

pydantic.error_wrappers.ValidationError: 1 validation error for OutputDatasetObsoleteDto
investigator
  none is not an allowed value (type=type_error.none.not_allowed)

This seems to come down to the schema being broken (SciCatProject/scicat-backend-next#1474). So maybe once the schema is fixed, this SDK will work. Do you test it in any way?

The API location is hard coded in the package to api/v3 which is appended to the user-provided host. IIRC this is not how it works everywhere. So is this an ESS SDK or do you envision it being used elsewhere as well?

[nitrosx]

@jl-wynen
Regarding the validation error... You are correct that the schema needs to be fixed.
That said, I see this one as a warning that the object returned by scicat has a field that scitacean did not expect.
You can still create the scitacean object.

[nitrosx]

@jl-wynen

Can you clarify and elaborate on this:

The API location is hard coded in the package to api/v3 which is appended to the user-provided host. IIRC this is not how it works everywhere. So is this an ESS SDK or do you envision it being used elsewhere as well?

Thanks

[jl-wynen]

That said, I see this one as a warning that the object returned by scicat has a field that scitacean did not expect.
You can still create the scitacean object.

In scitacean, yes. It falls back to loading only known fields. Although some fields may have an unexpected type because they were not parsed correctly. In the generated SDK, this is a hard failure and you cannot (to my knowledge) get the data from the API.

Can you clarify and elaborate on this:

The API location is hard coded in the package to api/v3 which is appended to the user-provided host. IIRC this is not how it works everywhere. So is this an ESS SDK or do you envision it being used elsewhere as well?

Thanks

In the SDK, all APi calls are made to {host}/api/v3/{endpoint} where host is provided by the user. I seem to remember that some facilities, e.g., PSI, don't have their API at a URL like this. Is this correct?

[Junjiequan]

In the SDK, all APi calls are made to {host}/api/v3/{endpoint} where host is provided by the user. I seem to remember that some facilities, e.g., PSI, don't have their API at a URL like this. Is this correct?

If you are asking is it a new thing with SDK then the answer is no. It's a global prefix for all the routers within the application.