Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEA] Sync tools supported ops with plugin #1193

Closed
cindyyuanjiang opened this issue Jul 16, 2024 · 2 comments · Fixed by #1198
Closed

[FEA] Sync tools supported ops with plugin #1193

cindyyuanjiang opened this issue Jul 16, 2024 · 2 comments · Fixed by #1198
Assignees
Labels
core_tools Scope the core module (scala) feature request New feature or request

Comments

@cindyyuanjiang
Copy link
Collaborator

cindyyuanjiang commented Jul 16, 2024

Is your feature request related to a problem? Please describe.

As of Jul 17, 2024:

**supportedDataSource.csv (FROM TOOLS TO PLUGIN)**

**supportedExecs.csv (FROM TOOLS TO PLUGIN)**

**supportedExprs.csv (FROM TOOLS TO PLUGIN)**
Row is added: HiveHash, TNEW, `hive-hash`, None, project, input, S, S, S, S, S, S, S, S, PS, S, NS, S, NS, NS, NS, NS, NS, NS, NS, NS
Row is added: HiveHash, TNEW, `hive-hash`, None, project, result, NA, NA, NA, S, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NS, NS
Row is changed: ShiftLeft, S, `shiftleft`, None, project, value, NA, NA, NA, S, S, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NS, NS
    SQL Func: `shiftleft` -> `<<`; `shiftleft`
Row is changed: ShiftLeft, S, `shiftleft`, None, project, amount, NA, NA, NA, S, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NS, NS
    SQL Func: `shiftleft` -> `<<`; `shiftleft`
Row is changed: ShiftLeft, S, `shiftleft`, None, project, result, NA, NA, NA, S, S, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NS, NS
    SQL Func: `shiftleft` -> `<<`; `shiftleft`
Row is changed: ShiftRight, S, `shiftright`, None, project, value, NA, NA, NA, S, S, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NS, NS
    SQL Func: `shiftright` -> `>>`; `shiftright`
Row is changed: ShiftRight, S, `shiftright`, None, project, amount, NA, NA, NA, S, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NS, NS
    SQL Func: `shiftright` -> `>>`; `shiftright`
Row is changed: ShiftRight, S, `shiftright`, None, project, result, NA, NA, NA, S, S, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NS, NS
    SQL Func: `shiftright` -> `>>`; `shiftright`
Row is changed: ShiftRightUnsigned, S, `shiftrightunsigned`, None, project, value, NA, NA, NA, S, S, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NS, NS
    SQL Func: `shiftrightunsigned` -> `>>>`; `shiftrightunsigned`
Row is changed: ShiftRightUnsigned, S, `shiftrightunsigned`, None, project, amount, NA, NA, NA, S, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NS, NS
    SQL Func: `shiftrightunsigned` -> `>>>`; `shiftrightunsigned`
Row is changed: ShiftRightUnsigned, S, `shiftrightunsigned`, None, project, result, NA, NA, NA, S, S, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NS, NS
    SQL Func: `shiftrightunsigned` -> `>>>`; `shiftrightunsigned`
Row is removed: EphemeralSubstring, S, `substr`; `substring`, None, project, str, NA, NA, NA, NA, NA, NA, NA, NA, NA, S, NA, NA, NS, NA, NA, NA, NA, NA, NS, NS
Row is removed: EphemeralSubstring, S, `substr`; `substring`, None, project, pos, NA, NA, NA, S, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NS, NS
Row is removed: EphemeralSubstring, S, `substr`; `substring`, None, project, len, NA, NA, NA, S, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NS, NS
Row is removed: EphemeralSubstring, S, `substr`; `substring`, None, project, result, NA, NA, NA, NA, NA, NA, NA, NA, NA, S, NA, NA, NS, NA, NA, NA, NA, NA, NS, NS

@amahussein amahussein added the core_tools Scope the core module (scala) label Jul 16, 2024
@amahussein
Copy link
Collaborator

@cindyyuanjiang
Copy link
Collaborator Author

  • Also, it will be interesting why did we miss the changes of the parse_url in the plugin side.

I checked the supported files in tools and plugin. We didn't miss this. The supportedExprs.csv file was not updated in plugin side.

amahussein added a commit to amahussein/spark-rapids-tools that referenced this issue Jul 17, 2024
Signed-off-by: Ahmed Hussein (amahussein) <[email protected]>

Fixes NVIDIA#1193

- HiveHash is added as a new epression. It is unlikely that Q tool
  enconters that expression if it appears deep inside write command for
hash bucketing.
- updated the SQL function of shift operators. Spark-4.0 added shorthand
  for shift operators ">>", "<<", and ">>>"
- update unit tests
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
core_tools Scope the core module (scala) feature request New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants