Allow skipping export of Organizations

[nathanKramer]

Describe the problem you'd like to have solved

Hi, I've recently updated the CLI to version 7, to allow exporting actions.

However, I'm having an issue where our tenant contains thousands of orgs.

The CLI now wants to export them along with their connections, which can't be done in a timely manner, and is not desired for our use-case.

My preference is to treat these Orgs as data, rather than as config that should be exported.
I'm wondering if there is any support for this.
(or if there are any suggestions for ergonomic work-arounds)

I can't find any evidence that this is supported, but I thought I would ask.

It's not clear to me whether it is safe to try to delete organizations from the resulting export, and then deploy, or if this would delete organizations. (This issue seems relevant: #387)

Describe the ideal solution

Some way to exclude organizations from the export.

e.g, an environment variable, or some kind of manifest that outlines the desired export)

Attempted work arounds

I tried dropping the read:organizations scope, but the export throws:

2022-03-03T03:21:01.811Z - info: Retrieving organizations data from Auth0
2022-03-03T03:21:02.012Z - error: Problem running command export during stage load when processing type organizations
2022-03-03T03:21:02.012Z - error: Insufficient scope, expected any of: read:organizations
2022-03-03T03:21:02.013Z - debug: Forbidden: Insufficient scope, expected any of: read:organizations

[willvedd]

Thanks for raising this. I think this is a duplicate of #387 which I have just merged a relevant fix for. In short, excluding resources from the configuration-as-code files should exclude them from being handled by the deploy CLI. That is, as long as the data doesn't explicitly define those resources as an empty set and rather not defined at all, they should be skipped. I think what you've found is that organizations paired with the YAML format doesn't follow this pattern and that will be fixed in an upcoming release (#434).

But something to remember is that as long as AUTH0_ALLOW_DELETE isn't set to true, you're never at risk of deleting any of your organizations. So as far as workarounds go, until 7.5.0 is released, it's ok to remove organization configuration and have AUTH0_ALLOW_DELETE:false in your config, you'll just see some warnings.

[willvedd]

Good news, 7.5.0 has been released to address this. I encourage you to update and test. I'm closing this for now but please open a new ticket if you're still experiencing this issue.

[nathanKramer]

@willvedd to be clear - a big part of the issue for us is that the export (as opposed to the deploy) takes an extremely long time.

I don't think that has been addressed. Is it worth keeping this issue open?

[willvedd]

Ah ok, my mistake! Resource exclusion on the export side is a commonly requested feature that's on our roadmap. We can certainly keep this open until it gets addressed more formally.

[willvedd]

@nathanKramer I've raised a #451 that may solve this issue. It would be great to receive your feedback since your use case would be enhanced by excluding all organizations.

[willvedd]

As proposed in the resource exclusion RFC, this has been added in the 7.7.0 release. You can bi-directionally exclude all organizations by leveraging the AUTH0_EXCLUDED configuration property. Example:

{ 
 "AUTH0_DOMAIN": "<YOUR_DOMAIN>", 
 "AUTH0_CLIENT_ID": "<YOUR_CLIENT_ID>", 
 "AUTH0_CLIENT_SECRET": "<YOUR_CLIENT_SECRET>", 
 "AUTH0_EXCLUDED": [ "organizations"] 
 }

[nathanKramer]

Thank you @willvedd!! this looks fantastic