Removing evaluation metrics

[fohrloop]

The default evaluation metrics, config/evaluation/default.yml contain:

metrics:
  # layout metrics

  # Specified symbols shall be located within the leftmost "n" columns of the left hand
  shortcut_keys:
    enabled: false
    weight: 0.35
    normalization:
      type: fixed
      value: 1.0
    params:
      # symbols that shall be considered "shortcut keys"
      shortcut_chars: cvx
      cost: 1.0
      within_n_leftmost_cols: 5
...

If I try to prune the config and use:

metrics:

  shortcut_keys:
    enabled: false

I get an error, as the shortcut_keys is not completely defined. Can I safely just remove the whole shortcut_keys from the metrics, and is it true for every metric? In other words, are any of the metrics "on" by default if I just remove them from the evaluation configuration file?

[dariogoetz]

This is not possible without modifying the source code. The yaml file is parsed according to the definition of the Parameters of all available metrics (those registered in layout_evaluation/src/evaluation.rs). Even if a metric is disabled, the parameters still get parsed. If you want to remove some metric from the config file, you would also need to remove/comment all relevant occurrences of the metric in the evaluation.rs file.

If the section corresponding to a metric is completely removed in the config file, that metric will not be computed.

[fohrloop]

There is for example config/evaluation/klanext.yml with following contents

klanext.yml

metrics:
  # layout metrics

  kla_home_key_words:
    enabled: false
    weight: 500.0
    normalization:
      type: weight_all
      value: 1.0
    params:
      words_filename: "./english-words-30000.csv"
      min_word_length: 3
      home_row_positions:
        - [2,2]
        - [3,2]
        - [4,2]
        - [5,2]
        - [14,2]
        - [15,2]
        - [16,2]
        - [17,2]

  kla_same_finger_words:
    enabled: true
    weight: 17.0
    normalization:
      type: weight_all
      value: 1.0
    params:
      words_filename: "./english-words-30000.csv"
      min_word_length: 3

  # Distance metric that tries to resemble the one on klanext.keyboard-design.com
  kla_distance:
    enabled: true
    weight: 0.44 # 40 weight * 0.0011 penalty * 10 ?
    normalization:
      type: weight_all
      value: 1.0
    params:
      ignore_modifiers: false
      keyup_distance: 10.5263156
      keydown_distance: 10.5263156
      dscoring:
        Left:
          Pinky: 3.67
          Ring: 2.9
          Middle: 2.0
          Index: 1.85
          Thumb: 1.0
        Right:
          Pinky: 3.67
          Ring: 2.9
          Middle: 2.0
          Index: 1.85
          Thumb: 1.0

      hscoring:
        Left: 1.1
        Right: 1.0

  # Finger usage metric that tries to resemble one from an older version of klanext
  kla_finger_usage:
    enabled: false
    weight: 1
    normalization:
      type: weight_all
      value: 1.0
    params:
      ignore_modifiers: false
      fscoring:
        Left:
          Pinky: 3.67
          Ring: 2.9
          Middle: 2.0
          Index: 1.85
          Thumb: 1.0
        Right:
          Pinky: 3.67
          Ring: 2.9
          Middle: 2.0
          Index: 1.85
          Thumb: 1.0

      hscoring:
        Left: 1.1
        Right: 1.0

  # Same finger metric that tries to resemble the one on klanext.keyboard-design.com
  kla_same_finger:
    enabled: true
    weight: 490.2 # 40 weight * 1.23 penalty * 10 ?
    normalization:
      type: weight_all
      value: 1.0
    params:
      ignore_modifiers: false
      fscoring:
        Left:
          Pinky: 3.67
          Ring: 2.9
          Middle: 2.0
          Index: 1.85
          Thumb: 1.0
        Right:
          Pinky: 3.67
          Ring: 2.9
          Middle: 2.0
          Index: 1.85
          Thumb: 1.0

      hscoring:
        Left: 1.1
        Right: 1.0

  # Same hand metric that tries to resemble the one on klanext.keyboard-design.com
  kla_same_hand:
    enabled: true
    weight: 68.2 # 20 weight * 0.341 penalty * 10 * 0.5?
    normalization:
      type: weight_all
      value: 1.0
    params:
      ignore_modifiers: false
      ignore_thumbs: false
      hscoring:
        Left: 1.1
        Right: 1.0


  # trigram metrics

ngrams:
  # Increase the weight of bigrams that have both an absolute weight and relative weight exceed
  # specified thresholds.
  increase_common_ngrams:
    enabled: false
    # Bigrams with a relative weight exceeding this threshold are considered
    critical_fraction: 0.001
    # The weight for bigrams exceeding both thresholds is multiplied by this factor
    factor: 2.0
    # Bigrams with an absolute weight exceeding this threshold are considered
    total_weight_threshold: 20.0


ngram_mapper:
  # Exclude ngrams that contain a line break, followed by a non-line-break character.
  # This encodes a mental pause which usually comes after hitting the "Enter" key, before
  # continuing to write.
  exclude_line_breaks: false

  # Split symbols belonging to higher layers of the layout into combinations involving modifiers
  # required to activate the layer
  split_modifiers:
    enabled: false
    # Multiply the ngram's weight with this factor whenever the resulting ngram involves two
    # modifiers that are required for the same symbol
    same_key_mod_factor: 0.03125

As the klanext.yml does not have following metrics (taken from config/evaluation/default.yml): finger_balance, hand_disbalance, key_costs etc., did I understand correctly that it means that these are still calculated (and possibly affecting the total cost) when using --eval-parameters config/evaluation/klanext.yml? So only way to remove these metrics is to edit the evaluation.rs?

[dariogoetz]

It appears, I was mistaken in the previous answer. It is possible to simply remove complete sections (i.e. metrics) from the config file. These are then NOT computed and DO NOT affect the score.

[fohrloop]

Thanks a lot for the confirmation!