Preserve keyword replacement during export

[phillipuniverse]

Description

Provide a clear and concise description of the issue, including what you expected to happen.

I am using the YAML version of the deploy CLI with variables replaced from a config.json. Here is an example snippet of variable usage in tenant.yaml:

...
resourceServers:
  - name: 'App ##ENVIRONMENT_NAME##'
     identifier: '##API_BASE_URL##'

I then have this config.json:

{
  ...
  "AUTH0_KEYWORD_REPLACE_MAPPINGS": {
    "ENVIRONMENT_NAME": "dev",
    "API_BASE_URL": "https://dev.myapp.com"
  }
}

When I export, variable data in tenant.yaml is overwritten with the real values. Example:

a0deploy dump --format yaml --output_folder . --config_file ./config.json

This modifies tenant.yaml with real values:

...
resourceServers:
  - name: 'App dev'
     identifier: 'https://dev.myapp.com'

Preferably this would update my config.json with the real values being exported.

Reproduction

Use environment variables in tenant.yaml or other files, these variables are incorrectly modified when exporting an Auth0 tenant.

Environment

• Version of this library used: 5.5.3
• Version of the platform or framework used, if applicable: NPM 7.6.0, Node 15.11.0
• Other relevant versions (language, server software, OS, browser): OSX
• Other modules/plugins/libraries that might be involved:

[samiurrehman92]

Hey team, is there any progress on this? I also checked similar feedback submitted here and here.

Correct me if I'm wrong but in the current scenario, nobody can use auth0-cli (more specifically the a0deploy export command) if they have any variable substitution setup in their tenant.yaml file, right?

The export command basically removes all variable substitutions in the tenant.yaml file as documented here and there's hardly any use-case for auth0-cli if it cannot be used for exporting the settings from the server.

Are there any plans to provide a solution for this?

cc @konradsopala @danwoda

[benwozniak]

Would love some feedback on this one. We currently cannot use variable substitution from the CLI, we are manually transforming the different environments.

[DanoThom]

I have to say I was really scratching my head around this whole issue when first testing this flow. Since using variable substitution in the template file seems to be the ENTIRE POINT of using this tool, I just can't understand how this hasn't been addressed anywhere?!

At the moment it seems I have to export, and then manually diff against the file in my source control and hopefully successfully manage to move over and update anything that has changed. What's going on here?

[benwozniak]

Is this project still being maintained? There are no merged PRs in over 2 months.

[willvedd]

Definitely acknowledge that this is a frustrating issue. Unfortunately, no workaround to provide at this time. But I do want to ensure you folks that we are looking closer at this project and at this workflow problem in particular.

Something I wonder is if this setting is too global and should have a more focused configuration. Is anyone here finding themselves replacing a single variable in multiple places?

[benwozniak]

Yes. We have multiple tenants that represent different non-prod/prod environments. In all of our database scripts and Actions, we need to set the base URL of the API that we are calling. With the current tool, we need to go through each database script and action file and update the variable substitution, since they are all wiped out by the tenant export.

edit: Does closing the issue mean that there are no plans to address this?

[willvedd]

Yes. We have multiple tenants that represent different non-prod/prod environments. In all of our database scripts and Actions, we need to set the base URL of the API that we are calling. With the current tool, we need to go through each database script and action file and update the variable substitution, since they are all wiped out by the tenant export.

Understood. That seems like a very reasonable use case to support, and no reason why variable replacement shouldn't accomplish this in theory.

Does closing the issue mean that there are no plans to address this?

I've closed #125 as a duplicate, but this issue is still very much open. We're still fleshing out a roadmap for this project but I can see this being something we prioritize for the near future.

[benwozniak]

Oh apologies I see now that it was just mentioned on a closed issue. Thanks for the update 😄 our team looks forward to hearing where the project is headed!

[vazra]

any updates on this?

[ghego]

Any update on this?

[willvedd]

I want to reassure everyone that this is still on our radar. We have an initiative documented and on our backlog with a tentative slotting for either this upcoming quarter or the next. Not discounting the impact of this ask, but IMO, #70 is a higher-impact feature, so that gets priority. The intention is to have this worked on almost immediately after.

Though I'd like to mention that I'm not sure if preserving variable substitutions is even possible. Sure, you could maybe implement some type of a state file, but that goes against the stateless spirit of this project. You could perform some type of lookup on resource configuration files and dynamically replace keyword values with their keys, but that could unleash a whole set of unsolvable consequences. At best, we would need to apply dynamic replacement policies on a per-resource basis, which is a decent undertaking and still may never be perfect.

All of this to say that this hasn't been forgotten about, we're thinking about solutions and you can expect some type of verdict sometime after #70 gets addressed.

[willvedd]

Update: Our team has been extensively discussing this functionality over the last couple of weeks. We've been able to suss-out some implementation details and questions for the customer around the desired developer experience. The culmination of this research has been published in a keyword preservation proposal. We kindly ask that you review the document and provide feedback; there are a few important technical details that make this work non-trivial and your feedback is crucial for us to strike a better balance of DX.

We appreciate everyone's patience with this issue and understand the burden behind it. Looking forward to providing a suitable solution for you all in the near future!

[willvedd]

Update: This work has been slated for the current quarter and implementation has already begun. Per the proposed implementation in the RFC and the feedback therein, the functionality here won't be a silver bullet but will hopefully alleviate the bulk of the annoyance. The release will come in the next month or two.

[willvedd]

The initial iteration of this feature is complete and released in v7.17.0.

Goal:
Preserve the majority of manually-placed ##KEYWORD_REPLACE## markers in the configuration files when performing subsequent exports. Otherwise, they will be overwritten with the remote values and require toilous step of re-adding.

Usage:
Keyword preservation is an opt-in feature that can be enabled through the AUTH0_PRESERVE_KEYWORDS boolean configuration property (docs).

Prerequisites:
To leverage the keyword preservation feature, the following criteria must be satisfied:

• Presence of local configuration files at the same location as the export target
• Defined keyword replace mappings via the AUTH0_KEYWORD_REPLACE_MAPPINGS configuration property

Limitations:
The keyword preservation functionality will attempt to preserve as many keywords while also maintaining the accuracy of your resource configuration files. And it the majority of cases, it will work without any intervention by the user. However, some key limitations exist:

• In the case of a keyword-replaced configuration field with differing values between local and remote, the local configuration value will always be favored. This will
• Arrays without a specific identifiers are not eligible for preservation. Ex: [ "http://site.com/logout", "localhost:3000/logout", "##LOGOUT_URL##" ]. This is because the ordering of these values are non-deterministic. Alternatively, to preserve these values, it is recommended to leverage the @@ARRAY_REPLACE@@ keyword replace syntax with the entire value.

Final Words:
Despite the above limitations, we believe this feature to reduce the majority of the toil related to managing overwritten keyword replace markers. And best of all, it should be transparent to the developer in most cases. Keep in mind that this is only the first iteration. Despite a plethora of testing, adjustments may need to be made to accommodate real-world usage. Also, the functionality is expected to be iterated on over time to best suit the needs of developers.

Appreciate everyone's patience and feedback on this one, really excited to see how its received!