UDF dependency handling without "job folder"

[soxofaan]

(I wanted to drop some notes/thoughts on the UDF dependency handling discussion we had earlier.)

Current approach to do automatic UDF dependency handling (initiated from #237) is to do a pip install in the driver and share these dependencies with the executors through a shared "job folder", as a file tree, or as a ZIP (as added with #845).

Important assumption to get this working is this shared "job folder" between driver and executors. However, given recent problems, I'm starting to doubt if it's safe and future-proof to assume that this job folder will/can be shared and timely synced across driver and executors.

Should we consider alternative approaches? such as

• create ZIP on driver and share with executors with explicit S3 link (instead of indirect S3-mounts)
• just drop the driver-executor sharing idea altogether and do the full pip install automatically on each executor separately

Maybe that last option is a good idea to have anyway, as fallback that's not very efficient, but at least always works.

refs:

• udf: add dependency list in comment block? #237
• Optimization of UDF dependency management #788
• Automatic UDF dependency handling in synchronous processing #791
• use zip files for UDF dependencies #845 / zip archive based UDF dependency handling #914
• ZIP based UDF dependency handling: smarter backoff instead of static sleep #924

cc @jdries @EmileSonneveld