[Bug] Unit test names aren't required to be unique, and duplicates get swallowed

[joellabes]

Is this a new bug in dbt-core?

• I believe this is a new bug in dbt-core
• I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior

I am making some unit tests and copy-pasted without updating the name

unit_tests:
  - name: ensure_all_pks_in_columns
    model: unit_ensure_all_pks_are_in_column_set
    given: []
    overrides:
      vars:
        primary_key_columns: ['pk1', 'pk2']
        columns: ['pk1', 'pk2', 'column_a', 'column_b']
        
    expect:
      rows:
        - {"col": 'pk1', "row_index": 1}
        - {"col": 'pk2', "row_index": 2}
        - {"col": 'column_a', "row_index": 3}
        - {"col": 'column_b', "row_index": 4}
  
  - name: ensure_all_pks_in_columns_pks_at_end
    model: unit_ensure_all_pks_are_in_column_set
    description: PKs are specified in `columns` so should be at end of list
    given: []
    overrides:
      vars:
        primary_key_columns: ['pk1', 'pk2']
        columns: ['column_a', 'column_b', 'pk1', 'pk2']
        
    expect:
      rows:
        - {"col": 'column_a', "row_index": 1}
        - {"col": 'column_b', "row_index": 2}
        - {"col": 'pk1', "row_index": 3}
        - {"col": 'pk2', "row_index": 4}
      
  - name: ensure_all_pks_in_columns_pks_at_end
    model: unit_ensure_all_pks_are_in_column_set
    description: PK specified in `columns` should be at end of list, missing PK will be added at front
    given: []
    overrides:
      vars:
        primary_key_columns: ['pk1', 'pk2']
        columns: ['column_a', 'column_b', 'pk2']
        
    expect:
      rows:
        - {"col": 'pk1', "row_index": 1}
        - {"col": 'column_a', "row_index": 2}
        - {"col": 'column_b', "row_index": 3}
        - {"col": 'pk2', "row_index": 4}

The second and third tests both have the same name, ensure_all_pks_in_columns_pks_at_end.

When I ran dbt test -s unit_ensure_all_pks_are_in_column_set (the name of the model I'm testing), the output was

03:26:36  Running with dbt=1.8.2
03:26:37  Registered adapter: snowflake=1.8.2
03:26:37  Found 28 models, 2 analyses, 24 seeds, 19 data tests, 606 macros, 24 unit tests
03:26:37  
03:26:39  Concurrency: 8 threads (target='dev')
03:26:39  
03:26:39  1 of 2 START unit_test unit_ensure_all_pks_are_in_column_set::ensure_all_pks_in_columns  [RUN]
03:26:39  2 of 2 START unit_test unit_ensure_all_pks_are_in_column_set::ensure_all_pks_in_columns_pks_at_end  [RUN]
03:26:42  2 of 2 PASS unit_ensure_all_pks_are_in_column_set::ensure_all_pks_in_columns_pks_at_end  [PASS in 3.20s]
03:26:42  1 of 2 PASS unit_ensure_all_pks_are_in_column_set::ensure_all_pks_in_columns ... [PASS in 3.41s]
03:26:42  
03:26:42  Finished running 2 unit tests in 0 hours 0 minutes and 5.12 seconds (5.12s).

From the attached logs, it seems that the first copy of the test is run and the second one is totally ignored.
dbt.log

Expected Behavior

The standard error you get when a node has a nonunuique name:

Compilation Error
  dbt found two models with the name "model_1".
  
  Since these resources have the same name, dbt will be unable to find the correct resource
  when looking for ref("model_1").
  
  To fix this, change the name of one of these resources:
  - model.audit_helper_integration_tests.model_1 (models/model_1.sql)
  - model.audit_helper_integration_tests.model_1 (models/data_tests/model_1.sql)

Steps To Reproduce

Create tests using sample code above

Relevant log output

Attached above

Environment

- OS: macOS
- Python: 
- dbt: 1.8.2

Which database adapter are you using with dbt?

No response

Additional Context

No response

[joellabes]

Update: I just made a similar mistake but spanning across two files, and did get the expected error message.

Compilation Error
  dbt found two unit_tests with the name "compare_classify_identical_tables_event_time_filter".
  
  Since these resources have the same name, dbt will be unable to find the correct resource
  when looking for compare_classify_identical_tables_event_time_filter.
  
  To fix this, change the name of one of these resources:
  - unit_test.audit_helper_integration_tests.unit_compare_classify.compare_classify_identical_tables_event_time_filter (models/unit_test_wrappers/unit_compare_classify.yml)
  - unit_test.audit_helper_integration_tests.unit_compare_classify.compare_classify_identical_tables_event_time_filter (models/unit_test_wrappers/unit_compare_which_columns_differ.yml)

Is it possibly the case that this only happens when both duplicate tests live in the same file, or when attached to the same model?

[dbeatty10]

Thanks for raising this issue @joellabes 🤩

I think this is just the way that our YAML parser works -- any key that is duplicated within the same file just clobbers any earlier instance.

For example, try this out:

models/my_model.sql

select 1 as id

models/_properties.yml

models:
  - name: my_model
    config:
      enabled: false
  - name: my_model
    config:
      enabled: true

dbt list

Then flip the enabled configs and try it again.

While an unfortunate and unwelcome surprise to not to be warned (or err'd) when there is a dupe within a file, I don't think this is a bug, and I'm going to close as "not planned" accordingly.

But if you or anyone else feels very strongly about this, just let me know and we can reconsider as a feature request.

[gshank]

#5114