[CLI] Generate full REST API (model+repository+controller) from model schema

[orshlom]

Description / Steps to reproduce / Feature proposal

The intention of this proposal is to create a fully functional REST API with ease imported from a schema and using an existing CLI tool lb4 openapi.

Current Behavior

Currently, the OpenAPI generator lb4 openapi is missing several functionalities:

• The command creates only partial properties information for models listed in the yaml schema.
  For example:

// IN CUSTOMER.JSON MODEL FILE (LB3):
...
"properties": {
    "id": {
        "type": "string",
        "id": true,
        "description": "Customer ID",
        "defaultFn": "uuid",
        "postgresql": {
            "owner": "public",
            "tableName": "customer",
            ...
        }
    ...
}
...
 
// TRANSLATES INTO THIS IN CUSTOMER.MODEL.TS MODEL FILE (LB4):
...
@property({name: 'id'})
id: string;
...

• No repositories or datasources are created with the command.
• All controller CRUD methods are implemented with throwing a "Not implemented" error.

Expected Behavior

In order to provide basic but functional application, this is my proposal after reading this comment:

• Create a default repository file to provide default CRUD operations (same as using the CLI command lb4 repository):
  For each model provided in the yaml schema, create a <model_name>.repository.ts file that contain basic CRUD operations.
• Use the newly created repository to provide basic functionality in the Controller file (instead of throwing "not implemented" errors).
• Add support for custom attributes in the yaml schema for relations and datasource:
  current generator does not create a datasource connector, and there is no relations creation (although there is an issue #1359 regarding this).
  The datasource file should be generated the same as creating one using the CLI command lb4 datasource.
  It is possible using the OpenAPI extentions support to define properties for relations and datasource. For example, in the OpenAPI yaml file:

...
Customer:
    properties:
        id:
            description: Customer ID
            type: string
        name:
            type: string
        required:
            - id
    additionalProperties: false
    x-lb-relations:                 <==== NEW ATTRIBUTE HERE
        orders:
            type: hasMany
            model: Order
    x-lb-datasource: postgreSQL     <==== NEW ATTRIBUTE HERE
Order:
    properties:
        id:
            description: Customer ID
            type: string
        name:
            type: string
        required:
            - id
    additionalProperties: false
    x-lb-relations:                 <==== NEW ATTRIBUTE HERE
        customer:
            type: belongsTo
            model: Customer
    x-lb-datasource: postgreSQL     <==== NEW ATTRIBUTE HERE
...

[bajtos]

Thank you @orshlom for the proposal.

Could you please describe the scenario you are trying to address in more details, from the point of view of LB4 app developer?

Are you trying to import Swagger spec generated from a LB3 application into a LB4 application, as a tool for migrating from LB3 to LB4? Or is the intention to import arbitrary OpenAPI Schema, allowing LB4 app developers to maintain additional metadata inside the OpenAPI spec (as opposed to having the metadata in TypeScript code)?

I'd also like to improve the GH issue title to better describe the specifics. Would the following text describe your scenario correctly?

[CLI] Generate full REST API (model+repository+controller) from model schema

Also, how do you envision keeping the model schema and the generated TypeScript artifacts in sync? For example, when a new property is added to the model schema, how would you like to update the generated model TS file? When a new relation is added, how would you like to update the generated repository TS file?

If the intention is to use some sort of a schema as the single source of truth for the data access (model + repository), then it may be better to define models & repositories at runtime (similarly to how LB3 works) and skip the code generation. See #2036 and #565

[elv1s]

Another option might be for x-lb-relations to instead use openapi links
https://swagger.io/docs/specification/links/

The openapi spec is not explicit in the kind of link (one, many, belongs, etc.) so openapi extensions could be used to explain what kind of relationship (link) exists.

[raymondfeng]

@orshlom Thank you for kicking off the discussion. +1 to introduce loopback extensions to OpenAPI spec for richer metadata.

Please also check out strongloop/loopback-swagger#81. It added some extensions to lb swagger to generate implementation code from so-called templates.

[orshlom]

Thanks for commenting @bajtos and @raymondfeng.

I agree with the new title name, as it better suits this proposal.
@bajtos the purpose of this proposal is to have the ability of creating an importer that can generate functional REST API for a model from a single yaml file. By functional I mean creating a model-repository-controller files in LB4 that have default CRUD functionality as created with lb4 model, lb4 repository and lb4 controller CLI commands.

The #2036 you have mentioned discusses about creating functionality in runtime. My intention, however, is to have the relevant files for keeping the ability to extend the basic functionality of the models.

Some elaboration for the proposal, based on your comments:

• Datasource custom attribute - should list an existing datasource in the yaml file, in order to create a repository for the model. If there is no such datasource already created, return an error in the CLI.
• Maybe there should be an included special attribute under each operation specified for a path, such as x-default-crud, that specifies whether to create a default CRUD implementation in the controller.

[bajtos]

The #2036 you have mentioned discusses about creating functionality in runtime. My intention, however, is to have the relevant files for keeping the ability to extend the basic functionality of the models.

Thank you for describing the difference, I better understand your use case now.

There is still one aspect that's not clear to me:

Also, how do you envision keeping the model schema and the generated TypeScript artifacts in sync? For example, when a new property is added to the model schema, how would you like to update the generated model TS file? When a new relation is added, how would you like to update the generated repository TS file?

Are we talking about the initial scaffolding only, where the developer takes the YAML file and produces a set of model/repository/controller files, and any further updates must be done in the scaffolded files?

If that's the goal, then I am proposing to investigate non-interactive JSON model offered by our CLI (CLI args --yes and --config.) The idea is to write a small script that will read your YAML definition and produce three JSON files to drive CLI execution. The script parsing YAML can use child_process API to invoke CLI and then use stdin stream to provide the JSON config.

Conceptually:

$ cat model-config.json | lb4 model --yes --config
$ cat repository-config.json | lb4 repository --yes --config
$ cat controller-config.json | lb4 controller --yes --config

I am afraid our current documentation in Command line interface does not correctly describe configuration formats used by different commands. I guess you will have to discover this from source code. It would be great if you could contribute this knowledge into our documentation afterwards.

[elv1s]

Thanks @bajtos ! The CLI --yes and --config are good to know about.

Are we talking about the initial scaffolding only, where the developer takes the YAML file and produces a set of model/repository/controller files, and any further updates must be done in the scaffolded files?

Not really. For our use case, the generated code is something that should not be edited since the build process we will put in place will likely overwrite any changes made to generated code. So this would be more than initial scaffolding. The files would already include the necessary updates for default CRUD.

[bajtos]

Not really. For our use case, the generated code is something that should not be edited since the build process we will put in place will likely overwrite any changes made to generated code. So this would be more than initial scaffolding. The files would already include the necessary updates for default CRUD.

Interesting. How do you envision to apply the necessary updates?

Also if the generated code should be never modified, then what is the advantage of generating the source code files over the approach described in #2036, where the repository & controllers are defined at runtime only?

[elv1s]

Updates should be made to the source of truth (json, yml, oas3.spec) and regenerated.

If we can avoid generating code and use the in-memory approach that might be a good option for us. Your comment in #1889 (comment) seems like a possible approach for this.

[dhmlau]

PR #2245 is in progress, moving it to the "In Progress" pipeline.

[stale]

This issue has been marked stale because it has not seen activity within six months. If you believe this to be in error, please contact one of the code owners, listed in the CODEOWNERS file at the top-level of this repository. This issue will be closed within 30 days of being stale.

[stale]

This issue has been closed due to continued inactivity. Thank you for your understanding. If you believe this to be in error, please contact one of the code owners, listed in the CODEOWNERS file at the top-level of this repository.