sigma.exceptions.SigmaValueError: Can't merge value lists '<field>' into one item due to different logical linking.

[Res260]

Hello, I tried to convert a Sigma rule but got this weird error when calling .to_dict() on it. I feel like it should work?

from sigma.rule import SigmaRule

sigma_rule = r"""
title: AAAAAAAAAA
id: aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa
status: experimental
level: high
logsource:
  product: AAAA
detection:
  exclusions:
    - D:
        - '5'
        - '6'
      C:
        - '3'
        - '4'

    - D: '2'
      E:
        - '1'
      
  condition: not exclusions
"""


import yaml
rule = SigmaRule.from_dict(yaml.safe_load(sigma_rule))
rule.to_dict()

Here is the error:

Traceback (most recent call last):
  File "REDACTED\scratch_8.py", line 29, in <module>
    rule.to_dict()
  File "REDACTED\pySigma\sigma\rule.py", line 1186, in to_dict
    "detection": self.detection.to_dict(),
                 ^^^^^^^^^^^^^^^^^^^^^^^^
  File "REDACTED\pySigma\sigma\rule.py", line 707, in to_dict
    identifier: detection.to_plain() for identifier, detection in self.detections.items()
                ^^^^^^^^^^^^^^^^^^^^
  File "REDACTED\pySigma\sigma\rule.py", line 602, in to_plain
    raise sigma_exceptions.SigmaValueError(
sigma.exceptions.SigmaValueError: Can't merge value lists 'D' into one item due to different logical linking.

Any idea?

[thomaspatzke]

The to_dict() method tries to merge multiple dict items into one and this is only possible if detection items with the same field have lists as well as plain values as its the case for the field D in your example. Internally lists cause that the logical linking attached to the field is changed to OR. Changing the second occurrence of field D into a single-item list should do the trick here.

The reason here is that there's a possibility to create rules programmatically that can't be expressed in a dict, e.g. because multiple detection items with the same field name are contained in them. The to_dict() code tries to handle such situations and the merging described above is one of the things it does. Normally, a rule originating from YAML code or from_dict() should be convertible with to_dict(). Therefore, I consider this as bug which needs some further investigation.

[thomaspatzke]

Ah, wait, changing the one instance into a list wouldn't help because the problem is that merging to lists of a field-bound detection item would result into a different boolean logic.

[Res260]

Ah, wait, changing the one instance into a list wouldn't help because the problem is that merging to lists of a field-bound detection item would result into a different boolean logic.

Indeed, I tried that and same error.

[Res260]

The reason here is that there's a possibility to create rules programmatically that can't be expressed in a dict, e.g. because multiple detection items with the same field name are contained in them.

I don't really understand this part. YAML and JSON can serialize the same information and they can be converted into one another easily. Also, JSON can be parsed as a dict very easily. If the YAML is valid, why can't it be converted to a dict when the first step of parsing a SigmaRule is to parse it as a dict? This seems contradictory.

From my basic and incomplete understanding, there is no conflict here, because it's just a list of different items, and in a dict (and in JSON and in YAML) you can have lists with identical elements (but you can't have identical keys in an object). What I'm trying to do here is have a list with objects with the same keys but different values in said keys.

[thomaspatzke]

This seems contradictory.

It is 😉 that's the reason it is marked as bug now. Somehow the YAML is parsec into a data structure that can't be converted back or some check is too strict.

[Res260]

As a temporary workaround, I found that simply putting the condition in a separate block doesn't raise the error yet is the same logic:

raises error:

detection:
  exclusions:
    - D:
        - '5'
        - '6'
      C:
        - '3'
        - '4'

    - D: '2'
      E:
        - '1'
  condition: not exclusions

doesn't raise error:

detection:
  exclusion1:
    - D:
        - '5'
        - '6'
      C:
        - '3'
        - '4'
  exclusion2:
    - D: '2'
      E:
        - '1'
      
  condition: not exclusion1 and not exclusion2