feat: add some SparkLikeLazyFrame methods

[FBruzzesi]

What type of PR is this? (check all applicable)

• 💾 Refactor
• ✨ Feature
• 🐛 Bug Fix
• 🔧 Optimization
• 📝 Documentation
• ✅ Test
• 🐳 Other

Related issues

Should we start tracking pyspark methods?

Checklist

• Code follows style guide (ruff)
• Tests added
• Documented the changes

If you have comments or can explain your changes, please do so below

It adds:

• drop_nulls
• rename
• unique
• join

And fixes sort

TODO

There is one test failing, because of how I implemented join to coalesce columns. The test is test_left_join_overlapping_column which tries to map a column called d to one called antananarivo, which already exist, and would try to rename such one to antananarivo_right. Yet fails to do so

[FBruzzesi]

This took me some mental sanity to figure out. I was like "does pyspark do not know how sorting works?" for a good 5 minutes

[MarcoGorelli]

cool, thanks!

for tests, maybe after this one we should start adding it to constructor and xfailing things?

[FBruzzesi]

for tests, maybe after this one we should start adding it to constructor and xfailing things?

We have a lot of expressions that would fail 🙃 I would say to wait a little bit more for now

[FBruzzesi]

I am not sure if the failing test should be considered an edge case, or it is worth to create a temporary name mapping layer/step to have it run successfully 🤔 It would definitly add a fair bit of complexity

[MarcoGorelli]

thanks! i think a loud error like

E               pyspark.errors.exceptions.captured.AnalysisException: [COLUMN_ALREADY_EXISTS] The column `antananarivo_right` already exists. Consider to choose another name or rename the existing column.

is fine, i think it's ok to assert that for pyspark it raises

[MarcoGorelli]

thanks @FBruzzesi !