Differentiate between missing values and null values

[ntextreme3]

I'm trying to do a search like:

from jmespath import search
data = {
    "grid": [
        {"col1": "a", "col2": None},
        {"col1": "b", "col2": None},
        {"col1": "c", "col2": "x"},
        {"col1": "d", "col2": "y"},
    ]
}
col1_data = search("grid[*].col1", data)  # gets ["a", "b", "c", "d"]
col2_data = search("grid[*].col2", data)  # gets ["x", "y"], expecting [None, None, "x", "y"]

If the col2 element did not exist, I think the current behavior makes sense. However, I think that search by default shouldn't also do filtering.

For reference:

jmespath.py/jmespath/visitor.py

Lines 134 to 138 in 599b0f7

	def visit_field(self, node, value):
	try:
	return value.get(node['value'])
	except AttributeError:
	return None

jmespath.py/jmespath/visitor.py

Lines 279 to 280 in 599b0f7

	if current is not None:
	collected.append(current)

Thoughts ?

[zalmane]

Just came across this myself. Would a PR be welcome?

[springcomp]

@zalmane @ntextreme3 you might be interested to learn that JMESPath Community supports this scenario - albeit currently in an indirect way.

[cdogaru-hrp-ctr]

@zalmane @ntextreme3 you might be interested to learn that JMESPath Community supports this scenario - albeit currently in an indirect way.

That's completely different that what the OP asked...

The questions was how search can support None values as well, currently they are discarded.

[springcomp]

That's completely different that what the OP asked...
The questions was how search can support None values as well, currently they are discarded.

Hum 🤔 then maybe I’m missing something…

search( map(&col2, grid), data ) -> [ null, null, "x", "y" ]

If you need more downstream processing, you can definitely distinguish between missing values vs null values by converting them temporarily for processing:

map(&col2 && col2 || 'value-was-null', grid)-> [ "value-was-null", "value-was-null", "x", "y" ]

Sure, this is kludgey, but that is definitely something that’s possible today.