[FEATURE] Test SE with 3.5 and above Pyspark

[asingamaneni]

Is your feature request related to a problem? Please describe.
Test all the features of SE with 3.5 and above, make respective changes for it to work. Update documentation accordingly

Describe the solution you'd like
NA

Describe alternatives you've considered
NA

Additional context
NA

[dannymeijer]

I propose we swap out poetry for hatch and introduce matrix testing. That way we can check with all relevant spark and python versions. For reference: https://github.com/Nike-Inc/koheesio/blob/485ee90f9f624897d27800ecc0510df35a2cb707/.github/workflows/test.yml#L62-L103

[stampthecoder]

I propose we swap out poetry for hatch and introduce matrix testing. That way we can check with all relevant spark and python versions. For reference: https://github.com/Nike-Inc/koheesio/blob/485ee90f9f624897d27800ecc0510df35a2cb707/.github/workflows/test.yml#L62-L103

I agree. This will simplify future testing efforts.