Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support Jupyter Notebook #97

Merged
merged 145 commits into from
Jan 10, 2024
Merged

Conversation

cindytsai
Copy link
Collaborator

@cindytsai cindytsai commented Nov 29, 2023

Support Jupyter Notebook / Lab UI

libyt now supports accessing interactive mode through Jupyter Notebook or Lab. Both serial (compile through g++) and parallel (compile through MPI) are supported. Can run python under MPI environment, gather output to root process, act like normal Python prompt, and has auto-completion feature.

Currently, this feature has only tested on local machines.

API

Use this feature through:

yt_run_JupyterKernel(const char* flag_file_name, bool use_connection_file)

libyt will stop and launch kernel if flag_file_name is found, and find empty port itself if use_connection_file is false.

If you want to specify connection info, you can refer to Connection File and the file name must be libyt_kernel_connection.json.

How to Use?

Compile libyt

  1. Set -DINTERACTIVE_MODE=ON -DJUPYTER_KERNEL=ON

  2. Currently only supports build from CMake (version >= 3.15). It is based on xeus-zmq, which needs other libraries when compiling :

  3. After installing libyt, set JUPYTER_PATH to <libyt-install-dir>/share/jupyter, so that jupyter knows how to find customized kernel. You can put it in .bashrc.

export JUPYTER_PATH=<libyt-install-dir>/share/jupyter:$JUPYTER_PATH

Getting jupyter-client and jupyter_libyt

  1. Get these:

  2. Set LIBYT_KERNEL_INFO_DIR system environment variable when launching jupyter client.

    • jupyter_libyt (or more precisely, the libyt kernel provisioner) uses this path to find libyt kernel and its info.
    • Normally, it is the directory where your application executable is.
export LIBYT_KERNEL_INFO_DIR=<your-application-root-dir>
  1. Launch Jupyter Notebook or Lab:
jupyter-lab # or jupyter notebook

Example

An example workflow of compiling libyt in parallel mode, and launch Jupyter Notebook or Lab:

  1. Compile and install libyt:
cd libyt # root directory of libyt

cmake -S . -B build-jupyter \
-G "Unix Makefiles" \
-DINTERACTIVE_MODE:BOOL=ON \
-DJUPYTER_KERNEL=ON \
-DMPI_PATH=/home/cindytsai/Softwares/openmpi/4.1.4-gnu \
-DPYTHON_PATH=/home/cindytsai/Softwares/python/python3.9 \
-Dxeus-zmq_DIR=/home/cindytsai/Packages/xeus-zmq-1.1.0/build \
-Dnlohmann_json_DIR=/home/cindytsai/Packages/json-3.11.2/build \
-Dxeus_DIR=/home/cindytsai/Packages/xeus-3.1.1/build \
-Dxtl_DIR=/home/cindytsai/Packages/xtl-master/build \
-Dcppzmq_DIR=/home/cindytsai/Packages/cppzmq-4.10.0/build

cmake --build build-jupyter
cmake --install build-jupyter --prefix <install-dir>
  1. Set JUPYTER_PATH:
export JUPYTER_PATH=<install-dir>/share/jupyter
  1. Compile and link your application to libyt.
  2. Run your application. (Remember to put flag file so that libyt will stop and launch the kernel)
  3. Open another terminal and set LIBYT_KERNEL_INFO_DIR to where your application executable is. Then run jupyter-lab or jupyter notebook.

TODO before Merge

  • Doc
  • Test on remote.
  • Make getting dependencies easier by getting and installing them using CMake.
    • Will use other package manager, and put it in different PR.
  • Check support of Makefile.
    • Actually, I want to drop the support for Makefile ... 😿
    • Will drop the support of Makefile in separate PR.
  • Support other publishing format beside text
    • Put in different PR.
  • Support libyt defined command

(murmuring) I really should find some time to familiarize with git command...
Don't know why Xeus does not route inspection calls to this method, it always routes to execute_request.
Still not really sure what this protocol version is for.
Don't know why I should include this line for xeus library rpath. It is weird that python doesn't need this.
@cindytsai cindytsai self-assigned this Jan 10, 2024
@cindytsai cindytsai merged commit a26b9a6 into yt-project:main Jan 10, 2024
19 checks passed
@cindytsai cindytsai deleted the PR-JupyterKernel branch January 10, 2024 10:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
new-feature New feature.
Projects
No open projects
Status: Done
Development

Successfully merging this pull request may close these issues.

1 participant