Pin all python dependencies

[XN137]

Split out from #280

we should pin all dependencies to avoid random new version releases form affecting our build.

however in the pandas case we relied on pip selecting the right version for us, which needs to be compatible with the (strange) flink dependencies, thus we are now pinning different versions in the respective requirement files.

also noticed that flake8 was in the dev requirements, while its plugins where in the lint requirements, which does not make much sense.
Since we are now using the more restrictive pip dependency resolver (see #280) we should clean up this inconsistency to avoid incompatibilities going forward.

[snazy]

Does it still work fine with the pinned pandas dependency?
I could be wrong, but IIRC there were some issues with pandas before - but could be no longer an issue since we dropped support for Google Colab.

[XN137]

Does it still work fine with the pinned pandas dependency?

define "it" 😅 as you can see the tests are passing...
i can re-test building/running the docker image locally again if you want (I originally tested the whole combination of changes in the other PR)

[snazy]

Ah - should have been more precise...

"it" == mybinder

[XN137]

ok i re-tested everything

i built the new image tag locally:

bash .github/scripts/create_and_push_docker_image.sh projectnessie/nessie-binder-demos $(grep -i 'FROM' binder/Dockerfile | cut -f2 -d":") docker

then locally modified the Dockerfile to use the local image:

-FROM ghcr.io/projectnessie/nessie-binder-demos:982fe3adb1e8b58263c74d0b3fc4b6b9ab97f652
+FROM projectnessie/nessie-binder-demos:982fe3adb1e8b58263c74d0b3fc4b6b9ab97f652

and then ran the repo2docker logic:

jupyter-repo2docker .

and i was able to execute all 4 notebooks without errors.
So i think this is the best we can do to ensure mybinder will continue to work?

[snazy]

I meant to run the notebooks in binder from your branch.

[XN137]

I meant to run the notebooks in binder from your branch.

how do you think its different than running jupyter-repo2docker locally?
i dont see how the binder one would fail due to python dependency changes, while the local build/testing still works (because both are using the same docker image definition?)... so to me this seems like a lot of overhead.

also pretty sure testing on binder would run into #272 still

[XN137]

actually i dont think you can test this branch on binder because the base image tag has changed and is not available yet in the github repo:
https://github.com/projectnessie/nessie-demos/pkgs/container/nessie-binder-demos

it is only published once the workflow on main has run.
which is why local testing with docker+jupyter-repo2docker is the best (only?) option right now imo.

[snazy]

Doesn't it work via e.g. https://mybinder.org/v2/gh/XN137/nessie-demos/pin-all-python-reqs?filepath=notebooks/nessie-iceberg-demo-nba.ipynb ?
Mean, I'd be okay to merge and immediately test on binder and revert if necessary though.

[XN137]

the link runs into the problem of not finding the base image (since as i mentioned, it is not published to ghcr.io yet).

HEAD is now at 94f1bdd run modify_dockerfile.sh
Using DockerBuildPack builder
Step 1/7 : FROM ghcr.io/projectnessie/nessie-binder-demos:982fe3adb1e8b58263c74d0b3fc4b6b9ab97f652
manifest unknownBuilt image, launching...
Failed to connect to event stream

[XN137]

after workflows ran on main i confirmed the binder notebooks still work, so everything went fine i think.