How to support multiple deequ + spark versions?

[SemyonSinchenko]

We should support multiple deequ and spark versions. At the moment we are supporting only deequ 2.0.7 + spark 3.5.2

What would be the best way to do it?

• We can have branches/tags and sync them;
• We can have subprojects + maven profiles;

If we go with subprojects we also have two options:

• copy all the code to subproject;
• create a so called shim-layer + common subprohject;

@zeotuan What do you think about it? @alfonsorr maybe you can also suggest the best way?

[alfonsorr]

Not sure if it will be very complex to move but it could be a good idea to move it to SBT to be similar to other projects of the org.

For the version, the 2 variables have different approaches. For the spark version if the underneath needed to wrap doesn't change (different methods depending on the spark version, different interfaces to call) it can be tested against multiple spark versions and the release will be agnostic to the spark version.

The same can be done with deequ. It depends if the interface needed changes between versions.

In case we need different sources depending on the versions, I found SBT easier to manage them. In Doric we have dependencies for each middle version of spark, and it is quite easy to manage.

[SemyonSinchenko]

I love the idea of moving to sbt, I even tried it once but failed with one place: shading.

At the moment we are using shading rules I copy-pasted from spark itself and without them no spark-connect extension will work.

There are rules: https://github.com/mrpowers-io/tsumugi-spark/blob/main/tsumugi-server/pom.xml#L246-L286

I tried couple of times to implement it in sbt but failed. Another problem is protobuf: we need to generate Java classes from protobuf (otherwise it won't work probably), but the most of sbt / mill plugins are designed to generate scala code (not Java classes) and are very badly supported.

While these two issues are resolved I will be happy to migrate away from this ugly XMLs from 2000-s!

Unfortunately I'm not experience enough with sbt to do it by myself...