Skip to content

Commit

Permalink
Fix docs
Browse files Browse the repository at this point in the history
Signed-off-by: Nghia Truong <[email protected]>
  • Loading branch information
ttnghia committed Nov 30, 2023
1 parent 703c6e8 commit b5cbe55
Showing 1 changed file with 7 additions and 6 deletions.
13 changes: 7 additions & 6 deletions docs/compatibility.md
Original file line number Diff line number Diff line change
Expand Up @@ -85,15 +85,16 @@ considered to be a rare occurrence.

### `NaN` vs `NaN`

Apache Spark does not have a consistent way to handle `NaN` comparison. Sometimes, All `NaN` are
considered as one unique value while other times they can be treated as different. The outcome of
Apache Spark does not have a consistent way to handle `NaN` comparison. Sometimes, All `NaN` are
considered as one unique value while other times they can be treated as different. The outcome of
`NaN` comparision can differ in various operations and also changes over time. Due to such complexity,
our plugin cannot guarantee to always match its output with Apache Spark if there are `NaN` values
our plugin cannot guarantee to always match its output with Apache Spark if there are `NaN` values
in the input.

For example, the `inset` operator can treat `NaN` as different values in Spark 3.0.2 and 3.1.2 but
considers them as the same in Spark 3.0.4, 3.1.3, and from Spark 3.2.0 (see the [issue](https://issues.apache.org/jira/browse/SPARK-36792)). On the other hand, our plugin always compares all `NaN` as equal
value.
For example, the `inset` operator can treat `NaN` as different values in Spark 3.0.2 and 3.1.2 but
considers them as the same in Spark 3.0.4, 3.1.3, and from Spark 3.2.0
(see the [issue](https://issues.apache.org/jira/browse/SPARK-36792)).
On the other hand, our plugin always compares all `NaN` as equal value.


## Decimal Support
Expand Down

0 comments on commit b5cbe55

Please sign in to comment.