[CT-305] Creating dataset with lots of authorised views referring to it in dbt leads to "412 PATCH Precondition check failed" errors

[muscovitebob]

Describe the bug

If you create lots of authorised views referring to a single dataset in one dbt run command, the command will fail and return feedback of the type:

12:12:44  Unhandled error while executing model.project.AuthorisedView
412 PATCH https://bigquery.googleapis.com/bigquery/v2/projects/PROJECTNAME/datasets/DATASETNAME?prettyPrint=false: Precondition check failed.
12:12:44  194 of 211 ERROR creating view model PROJECTNAME.AUTHVIEWNAME [ERROR in 1.68s]

The errored authorised views are not authorised at the end of the run. A few of the authorised views that ran concurrently with the errored ones do get created.

Steps To Reproduce

Create lots of authorised views in one go that all refer to tables in the same dataset and do a dbt run.

Expected behavior

I would expect that dbt creates all the authorised views it is told to create.

System information

dbt 1.0.0 running in the official docker image.

Additional context

This seems like a limitation of how fast you are allowed to modify the dataset objects in BigQuery since authorised views are not truly independent objects but attributes of the dataset object. Probably one way to deal with this is to catch the 412's when we try to make authorised views and do a few retries with exponential backoff.

[muscovitebob]

I think a workaround so to speak is run with just one executor but that kills speed obviously. Any suggestions how to work around this issue until we can the root cause?

[muscovitebob]

A workaround is to force a sequential execution graph just for the authorised views by arbitrarily (and manually) ordering the authorised views and including a ref to each previous authorised view. This way dbt shouldn't ever execute the views concurrently and so we won't ever get a 412 https://docs.getdbt.com/reference/dbt-jinja-functions/ref#forcing-dependencies

[jtcohen6]

@muscovitebob We've run into this issue before: authorized views don't appear to be thread-safe. For reasons of performance, there's fair reason to want dbt to batch up its access grants, to run once at the end of the run: #87

The downside is, for the time between the authorized view being (re)created and the end of the run, it won't have the access it needs to support querying. That delay already exists today, but the gap is much smaller.

[github-actions]

This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please remove the stale label or comment on the issue, or it will be closed in 7 days.