[Spike] Allow extensions to contribute custom convention for environment-specific operational configuration

[bajtos]

Timeboxed to 2 weeks; don't spend extra time if done earlier

Different platforms use different ways for configuring operational aspects of application in test/dev/production. LB4 should make it easy to write extensions to support these different platforms.

The goal of this spike is to find out and document what's possible today, identify gaps, propose solutions and create a list of follow-up tasks.

Few examples of operational config:

• configure datasources from environment variables
• use a faster but less-secure hashing algorithm in dev/test (a hashing algorithm is used e.g. to store user passwords)
• apply datasource configuration provided by VCAP_SERVICES env variable in CloudFoundry/IBM Cloud - see [WIP] Spike: todo example on IBM Cloud [DO NOT MERGE] #1574

Possibly related: #983 and #1396

[Edo78]

Is there any news about this issue?
Right now the only way I have found to configure a datasource with environment variables is to instantiate the related class (in application.ts) passing a custom configuration

let dbDataSource = new DbDataSource({
  name: 'db',    
  connector: 'mysql',
  hostname: process.env.DB_HOST,
  port: process.env.DB_PORT,
  user: process.env.DB_USER,
  password: process.env.DB_PASSWORD,
  database: process.env.DB_DATABASE
});

but I'm not sure this could be a best practice.
Am I missing something?

[bajtos]

@Edo78 see the following resources to learn about other options available:

• https://loopback.io/doc/en/lb4/Deploying-to-IBM-Cloud.html#updating-application
• https://itnext.io/loopback-4-database-configuration-8f085399268

TL;DR: if you are using @loopback/boot to load your datasources (as is the default in LB4 applications scaffolded using lb4 CLI tool), then you can bind just the custom datasource configuration.

this.bind('datasources.config.db').to({
  name: 'db',    
  connector: 'mysql',
  hostname: process.env.DB_HOST,
  port: process.env.DB_PORT,
  user: process.env.DB_USER,
  password: process.env.DB_PASSWORD,
  database: process.env.DB_DATABASE
});

[bajtos]

Cross-posting from #1609

#441 (comment) #4

In your proposal, the configuration (loaded from JSON) is hard-coded in the generated (base) class. At the same time, we need to support different configurations for different environments (dev/test/production) and load configuration from environment variables (https://12factor.net).

It was proposed to parse process.env variables inside datasource files. I consider that as an anti-pattern, because it couples high-level datasources with low/system-level configuration code. It makes the code difficult to test, because tests would have to manipulate global process.env. Instead, we should find a way how to inject ENV variables in such way that the actual usage of process.env stays in the top-level index.ts or application.ts file only.

On a similar topic, @raymondfeng proposed a sort of a registry holding datasource configuration for all environments in the app-level Context. I don't think this is a good solution - consider the case where an application is deployed to different geo location and where each geo location requires different connection strings (to use a colocated database instance). In other words, a production config is not a single object, but a set of location-specific objects.

My conclusion is that we should decouple datasource configuration from datasource implementation class and leverage dependency injection to provide arbitrary runtime-specific configuration from the application level. IMO, this addressed both needs 1) have different datasource config depending on dev/test/prod environment 2) build datasource configuration dynamically, either from process.env or perhaps external config providers like https://www.consul.io.

A mock-up app:

    class MyApp extends RestApplication {
      async boot() {
        const dbConfig = 
          // Figure out what config to use.  @loopback/boot can provide
          // helpers to load config from ENV variables
        this.bind('datasources.db$config').to(dbConfig);
        // etc.
      }
    }

A mock-up datasource:

    // default config is used in tests
    const defaultConfig = require('./db.datasource.json');
    
    class DbDataSource extends juggler.DataSource {
      constructor(
        @inject('datasources.db$config')
        config: DataSourceOptions = defaultConfig
      ) {
        super(config);
      }
    }

[adesege]

I was going to create a new issue before seeing this one.

In order not to pollute the codebase with process.env and also to have a central place to manage environment/config variables, we can have a config.json file for example and use dependency injection or decorators to load it for use in a class.

// sample decorator signature
class SampleClass {
 @env('privateid')
 privateId: string;

@env('appSecret')
 appSecret: string;

constructor() {}
}

This can be in a separate package too under @loopback/env.

I am currently working on something similar and I will be glad to raise my first PR if approved.

Let me know what you think 💪

cc: @bajtos

[stale]

This issue has been marked stale because it has not seen activity within six months. If you believe this to be in error, please contact one of the code owners, listed in the CODEOWNERS file at the top-level of this repository. This issue will be closed within 30 days of being stale.

[stale]

This issue has been closed due to continued inactivity. Thank you for your understanding. If you believe this to be in error, please contact one of the code owners, listed in the CODEOWNERS file at the top-level of this repository.

[collaorodrigo7]

I am trying to use environmental variables to load my datasources configuration but I am running into issues when using some of the CLI tools.
I looked at the docs and could not find a recommended way of loading environmental configuration, so I thought I could try dotenv or env-yaml.
So, I am using dotenv and env-yaml and they both require you to add require('env-yaml').config(); as early as possible on your application.
It works pretty well when running the application normally, but when using tools as lb4 repository I can not find a place where to load require('env-yaml').config(); so that the CLI tool will include it.
I was having a similar issue with lb4 discover but after adding the require statement at the top of one of my datasources file the problem was fixed.

Do you guys have any suggestions on how can I solve this?

This issue seems to be closed but there isn't an actual solution yet. So, I would like to propose an idea.
How would you feel about making env-yaml the recommended convention for environemnt-specific configurations? We could have an optional index.js on the source of the repo that can have the require statement, and have the CLI import that index.js if available?.

[collaorodrigo7]

@dhmlau

[OddDev]

Vote to reopen 😄

[achrinza]

Reopened 👍

[gvillenave]

So what is the official recommendation to use environment variables in loopback 4? Is this item still pending?

[grodrigo]

I agree that we need a recomendation or a path, a search in google shows version 3, but in the last release it isn't any more viable.
And here I find a dotenv and env-yaml advice, really thanks for that.
At least leave a comment in the official documentation that we could configure an environment on this way, newbies will find this useful.

[DrMinh]

2024 and still no way to use .env for datasource?