Skip to content

Commit

Permalink
Change docs
Browse files Browse the repository at this point in the history
Signed-off-by: Nghia Truong <[email protected]>
  • Loading branch information
ttnghia committed Dec 1, 2023
1 parent cbd6112 commit a628087
Showing 1 changed file with 6 additions and 9 deletions.
15 changes: 6 additions & 9 deletions docs/compatibility.md
Original file line number Diff line number Diff line change
Expand Up @@ -85,16 +85,13 @@ considered to be a rare occurrence.

### `NaN` vs `NaN`

Apache Spark does not have a consistent way to handle `NaN` comparison. Sometimes, All `NaN` are
Apache Spark does not have a consistent way to handle `NaN` comparison. Sometimes, all `NaN` are
considered as one unique value while other times they can be treated as different. The outcome of
`NaN` comparision can differ in various operations and also changes over time. Due to such complexity,
our plugin cannot guarantee to always match its output with Apache Spark if there are `NaN` values
in the input.

For example, the `inset` operator can treat `NaN` as different values in Spark 3.0.2 and 3.1.2 but
considers them as the same in Spark 3.0.4, 3.1.3, and from Spark 3.2.0
(see the [issue](https://issues.apache.org/jira/browse/SPARK-36792)).
On the other hand, our plugin always compares all `NaN` as equal value.
`NaN` comparison can differ in various operations and also changed between Spark versions.
Our plugin tries to match its output with Apache Spark except for a few operation(s) listed below:
- `IN` SQL expression: `NaN` can be treated as different values in Spark 3.1.2 but not from
Spark 3.1.3 (see the [issue](https://issues.apache.org/jira/browse/SPARK-36792)).
On the other hand, our plugin always compares all `NaN` as equal value for this operation.


## Decimal Support
Expand Down

0 comments on commit a628087

Please sign in to comment.