Differences between CodeClimate Analysis and Qlty CLI analysis results

[nk-ty]

Question

There seems to be a difference between the analysis results of CodeClimate Analysis and Qlty CLI.

Code to be analyzed:

def valid?(params)
  init_parameter(params)

  return false unless @a.valid?(ENV_VARIABLE_A, @params_a)
  return false unless @a.valid?(ENV_VARIABLE_B, @params_b)
  return false unless required_params_present?(params)

  true
end

CodeClimate Quality web results

Method valid? has a Cognitive Complexity of 6 (exceeds 5 allowed). Consider refactoring.

qlty CLI results

qlty smells target_file.rb
     [0/3] 🔍  Analyzing vs. origin/develop... 0.76s
     [1/3] 👀  Checking structure of 1 files...  0.18s
     [2/3] 🤔  Looking for duplication across 1 files...  2.58s
     [3/3] ✨  Reporting..

How to reproduce (optional)

Expected behavior (optional)

No response

CLI Version

qlty 0.463.0 macos-arm64 (e906f76/e906f76 2024-12-10)

Information about Qlty configuration and environment (optional)

.codeclimate.yaml

# exlude pattern is the same
version: "2"
checks:
  argument-count:
    enabled: true
    config:
      threshold: 4
  complex-logic:
    enabled: true
    config:
      threshold: 4
  file-lines:
    enabled: true
    config:
      threshold: 250
  method-complexity:
    enabled: true
    config:
      threshold: 5
  method-count:
    enabled: true
    config:
      threshold: 25
  method-lines:
    enabled: true
    config:
      threshold: 25
  nested-control-flow:
    enabled: true
    config:
      threshold: 4
  return-statements:
    enabled: true
    config:
      threshold: 4
  similar-code:
    enabled: false
    config:
      threshold:
  identical-code:
    enabled: false
    config:
      threshold:

.qlty/qlty.toml

# This file was automatically generated by `qlty init`.
# You can modify it to suit your needs.
# We recommend you to commit this file to your repository.
#
# This configuration is used by both Qlty CLI and Qlty Cloud.
#
#     Qlty CLI -- Code quality toolkit for developers
#     Qlty Cloud -- Fully automated Code Health Platform
#
# Try Qlty Cloud: https://qlty.sh
#
# For a guide to configuration, visit https://qlty.sh/d/config
# Or for a full reference, visit https://qlty.sh/d/qlty-toml
config_version = "0"

exclude_patterns = [
  "*_min.*",
  "*-min.*",
  "*.min.*",
  "**/*.d.ts",
  "**/.yarn/**",
  "**/bower_components/**",
  "**/build/**",
  "**/cache/**",
  "**/config/**",
  "**/db/**",
  "**/deps/**",
  "**/dist/**",
  "**/extern/**",
  "**/external/**",
  "**/generated/**",
  "**/Godeps/**",
  "**/gradlew/**",
  "**/mvnw/**",
  "**/node_modules/**",
  "**/protos/**",
  "**/seed/**",
  "**/target/**",
  "**/testdata/**",
  "**/vendor/**",
  "**/assets/**",
]

test_patterns = [
  "**/test/**",
  "**/spec/**",
  "**/*.test.*",
  "**/*.spec.*",
  "**/*_test.*",
  "**/*_spec.*",
  "**/test_*.*",
  "**/spec_*.*",
]

[[source]]
name = "default"
repository = "https://github.com/qltysh/qlty.git"
tag = "v0.463.0"

[smells.boolean_logic]
threshold = 4
enabled = true

[smells.file_complexity]
threshold = 55
enabled = true

[smells.return_statements]
threshold = 4
enabled = true

[smells.nested_control_flow]
threshold = 4
enabled = true

[smells.function_parameters]
threshold = 4
enabled = true

[smells.function_complexity]
threshold = 5
enabled = true

[smells.duplication]
enabled = false

• OS:

[brynary]

Thanks for this clear question. We will look into this and report back.

[noahd1]

Hi @nk-ty

Thanks so much for this bug report. After some contemplation and investigation (which I will share below), I think that the answer is:

• neither codeclimate nor qlty are returning a cognitive complexity score here that makes sense (qlty: 0; codeclimate: 6)
• while there can be differences in smells between qlty and codeclimate, in this case, there is a "right" answer, which is based on the Cognitive Complexity spec published by Sonar
• we intend to fix qlty where it disagrees with the spec
• we believe the right answer after looking at the spec is that your code sample should have a cognitive complexity of 3

For some more detail about the cognitive complexity here ...

The if (by which I mean to include unless) statement should result in a 1 point penalty to cognitive complexity.

I considered momentarily that the code sample uses a common ruby-ism which is the trailing if statement. However, while the spec does reference some language-specific exceptions, it does not reference this particular one. I believe it's actually just a stylistic choice and doesn't impact the cognitive complexity one way or another. In short, I believe the correct cognitive complexity of this function is 3 -- one for each unless statement.

qlty has 2 bugs that I plan on addressing here:

1. It treats if as having cognitive complexity but not unless
2. It does not count a trailing if statement as adding cognitive complexity

Hope this all makes sense. Let me know if you have follow on questions here.

[noahd1]

FYI, the PR to address this - #1361 - was merged and should be included in the next release of the CLI.