Provide a way to exclude all of an object during a0deploy import

[Bnewman]

Describe the problem you'd like to have solved

I would like to be able to exclude all of certain configuration objects during import without having to specify individual values. This is especially true of Enterprise connections. Currently, I have to specify the following in my import config to avoid importing them:

  "AUTH0_EXCLUDED_CONNECTIONS": [
        "9395127f-4853-4842-8c2b-2be4135fa89",
        "9563eb0b-accc-489d-b03c-f6373d0ed0a",
        "f0272ae7-9fea-4e0a-b856-9b360250b5a",
        "9cab3ff8-ff15-410a-b8b9-816a7151b4f",
        "614d9405-77c1-4239-bdda-4ca306bd36c",
        "89af4988-4d68-476c-b4c6-c476835088f",
        "test"
   ]

This is error prone and burdensome to maintain.

Describe the ideal solution

I would like the CLI to support wildcards. Here are two examples.

Exclude all connections

  "AUTH0_EXCLUDED_CONNECTIONS": [ "*" ]

Exclude some based on partial name match.

  "AUTH0_EXCLUDED_CONNECTIONS": [ 
    "test*",
    "expermiental*"
 ]

Alternatives and current work-arounds

My only alternative right now is to explicitly list every connection I want to exclude.

[benwozniak]

+1 would love to see this, the "EXCLUDED_PROPS" section unfortunately does not work. I am trying to import my tenant but cannot because of enterprise connections.

[stuarthadfieldglobality]

+1 can anyone get in on this?

[willvedd]

@stuarthadfieldglobality thanks for bumping this. Rest assured, resource exclusion is at the top of mind of this team. The reason why it's taken a while is because the existing methods of excluding resources doesn't adequately scale to all resources; we're carefully looking at ways to employ a sensible and holistic solution that doesn't break backwards compatibility.

I can't make any firm commitments, but this is on our roadmap so stay on the lookout for updates!

[stuarthadfieldglobality]

@willvedd thanks, appreciate it!

I'll add some context as far as the use case for this goes:

Our IT services team configures SSO connections with clients generally on the fly, over a call. They don't have the expertise to manage the rest of the tenant configuration - that lies with one of our engineering teams.

Often they will update something to do with an SSO config on a call, and then some time later our deployment pipeline will execute, and "oopsie daisy" we overwrote some setting and now a client can't log into the platform (BIG problem). Of course there's a cultural problem here of syncing the remote changes and the local changes -- but this is complex to solve / inspire change.

We have many different SSO connections on each of our five tenants - it'd be much easier if our engineering team could just palm off that part of the tenant config - and say "that's the IT services team's baby, we aren't gonna mess with it ever".

In fact, not to derail OP's usecase - but for our usecase it'd be ideal if I could just dictate what entities I wanted to exclude in general, i.e. I could tell the CLI tool not to touch connections at all. Don't make any GET, POST, PATCH, whatever requests - just ignore connections (or whatever other entity, be it actions, rules, applications, etc.) completely.

[stuarthadfieldglobality]

@willvedd, I'm looking through issues, I've noticed:

#427
#397
#394

etc., it's clear this is quite a big pain point for users. I know you've repeatedly stated it's on the roadmap, but can't promise when it'd be delivered: in the interim do you have any advice on workarounds? How could we achieve the same functionality?

Bearing in mind some of our processes are fully automated, it seems to me that the only way I can achieve this is by deleting the folder / entry in the yaml containing the specification for that entity?

This means that the steps we take to import or export code looks like (assuming we're using the directory structure):

Export:

1. Run the export for $tenant
2. rm -rf {undesirable_entity} folder for this tenant (rm -rf removes if it exists only, so that's cool)

Import

1. rm -rf {undesirable_entity} folder for this tenant (rm -rf removes if it exists only, so that's cool)
2. Run the import

As a safeguard, add entries into gitignore for enumerate(tenants)/{undesirable_entity} so that it'd never get checked into source control?

I feel like I shouldn't be tinkering with the files/folders of my code as part of a CI/CD pipeline. In that vein, there any better suggestions? As was mentioned by OP, excluding props doesn't really work either.

[stuarthadfieldglobality]

Hey @willvedd - gonna have one last attempt here (under pressure internally to deliver this functionality), do you have any recommendations in the interim as far as a workaround goes, or am I in unchartered territory? 😄

[willvedd]

@stuarthadfieldglobality Appreciate your interest in the library and for providing your feedback!

I absolutely agree that resource exclusion is a bit of a sore spot with this project. It should be a first-class feature of this utility and until now hasn't gotten the attention it deserved. The three issues you referenced above are just the ones that are open, there are several other closed duplicates.

Speaking candidly here, this repo just recently got more formal resourcing and attention, which means that our team is still finding our bearings in terms of stability and roadmap. Progress is modest but rest assured we're doing our best to identify
the highest impact work, this issue included.

FWIW, I'm looking into a more formal method of exclusion as we speak. I still won't provide a firm date, but I hope that you can hold patient for another week because I'd really like to get your feedback on a proposal I'm drafting.

In terms of workarounds, I regretfully have no suggestions other than adopting the Auth0 Terraform Provider. If you already use Terraform in your development workflows, it should be a frictionless method of managing your config.

[willvedd]

@Bnewman @benwozniak @stuarthadfieldglobality I've created a proposal for resource exclusion that may solve help solve this issue. It would be great to get some feedback on it, I think you folks have a unique perspective that can help shape the direction of this tool.

[willvedd]

As proposed in the resource exclusion RFC, this has been added in the 7.7.0 release. You can exclude all connections by leveraging the AUTH0_EXCLUDED configuration property. Example:

{ 
 "AUTH0_DOMAIN": "<YOUR_DOMAIN>", 
 "AUTH0_CLIENT_ID": "<YOUR_CLIENT_ID>", 
 "AUTH0_CLIENT_SECRET": "<YOUR_CLIENT_SECRET>", 
 "AUTH0_EXCLUDED": [ "connections"] 
 }