[CT-323] [Bug] Generic test names collide when same args but different config is used

[judahrand]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

If for example there are two different tests which are the same a part from their configs only the first will run due to a naming collision. An example of this might be:

- not_null:
    where: color = 'red'
- not_null:
    where: color = 'blue'

Expected Behavior

Both tests should name and have different names.

Steps To Reproduce

Run the integration tests in #4829 without the commit which fixes the issue.

Relevant log output

No response

Environment

- OS:
- Python:
- dbt:

What database are you using dbt with?

No response

Additional Context

No response

[emmyoop]

Thanks for opening up this issue @judahrand. It's actually a duplicate of #4102. There's an excellent write up of our thinking on this and a workaround for the time being.

I do have one clarifying question from your write up - are you experiencing tests getting silently skipped due to the overlapping names or do you experience an error letting you know about the name collision? As part of that answer, can you share what version of dbt-core you're using?

Unfortunately I won't be able to accept the PR you submitted but I really appreciate you taking the time to dig in to the problem!

[judahrand]

are you experiencing tests getting silently skipped due to the overlapping names

Yes, tests being silently skipped (at least on 0.21.1).

[emmyoop]

Ah, yes, that's expected then. In 1.0 we start to throw exceptions so that the tests no longer silently fail. While it doesn't fix the root of the issue, it does make what's happening more clear so that no tests potentially fail "silently".

I'm going to close this issue as a duplicate for now but I encourage you to contribute to the conversation over in #4102 if you have anything you would like to add!