Kernel state for Cell and Output

[echarles]

This PR introduces a kernel state and let the Output and Cell components use that state via the Kernel Executor.

This allows more control on the ZMQ messages. The Notebook component will have to use that Kernel Executor also.

[echarles]

This PR also support the creation of completely independent Kernels (via a path) as requested by @mattmutt on #281 (comment)

[echarles]

@sok82 I hope I have addressed the requirements of #269 with a new ExecutionPhase state that goes informs when the Cell is completed. You can have a look at these videos and the Output and Cell examples of this PR. (it works even if no output is generated).

Screencast.from.07-30-2024.02.31.54.PM.webm

Screencast.from.07-30-2024.02.34.33.PM.webm

[echarles]

@MarcosVn Once this PR is merged, your PR #280 will need to be revisited.

I have changed a bit how the cell is run in this PR, as well as the state. I have quickly looks the changes you propose, and overall looks good to me. To make your life easier, you can simply close 280 and open a brand new avoiding the merge pain (unless a rebase/merge works for you).

[fcollonval]

Thanks @echarles

[sok82]

@sok82 I hope I have addressed the requirements of #269 with a new ExecutionPhase state that goes informs when the Cell is completed. You can have a look at these videos and the Output and Cell examples of this PR. (it works even if no output is generated).

Screencast.from.07-30-2024.02.31.54.PM.webm
Screencast.from.07-30-2024.02.34.33.PM.webm

Hey @echarles

It's great to see progress on the issue, video looks promising.

Can you please provide few lines of code demonstrating how I can start execution of output in new model and react on changing of execution state (KernelPhase if I got you right)

If i got you right - I need to do smth like this

<Box>
            Kernel Phase: {kernelsStore.getExecutionPhase(kernel.id)}
</Box>

And the thing I need is probably kernelsStore.getExecutionPhase(kernel.id)
So when cell or output is executed this value gives me idea of the actual execution state.
Is it correct?

And what are possible states?
Can I distinguish execution finished with error from execution finished successfully?

[echarles]

Is tried to look at this but see no changes concerning reacting on execution state

{kernelsStore.getExecutionPhase(kernel.id)} is what will return the phases, moving from RUNNING to COMPLETED.

[sok82]

Is tried to look at this but see no changes concerning reacting on execution state

{kernelsStore.getExecutionPhase(kernel.id)} is what will return the phases, moving from RUNNING to COMPLETED.

Great. Already found that.

And what about reacting on execution errors?

How can I check if execution finished with error?

[echarles]

How can I check if execution finished with error?

Great call. For now, in case of error, I explicitly set COMPLETED. Let me change that to COMPLETED_WITH_ERROR

[echarles]

Like this?

Screencast.from.07-30-2024.05.46.04.PM.webm

[sok82]

Hey @echarles ,

I'm experimenting with version 0.17.0 having desired functionality and experiencing a problem

Kernel phase monitoring works fine when execution finishes successfully, but when something is wrong with a code exception is thrown from kernel execution promise and I don't have an idea of how to catch it.

The way I trigger execution is just by increasing executionTrigger for output, so I don't control promise implicitly...

Exception example is

Uncaught Error TypeError: bar() missing 1 required positional argument: 'height'
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In[12], line 1
----> 1 plt.bar(job_satisfaction_rate_train['job_satisfaction_rate'], color='green');

TypeError: bar() missing 1 required positional argument: 'height'
    at reject (/Users/sk/Develop/d20/d20-git-repo/d20-platform-wiki-editor-ui/node_modules/@lumino/coreutils/src/promise.ts:51:1)
    at _onReply (/Users/sk/Develop/src/jupyter/kernel/KernelExecutor.ts:262:1)
    at _handleReply (/Users/sk/Develop/d20/d20-git-repo/d20-platform-wiki-editor-ui/node_modules/@jupyterlab/services/src/kernel/future.ts:237:1)
    at handleMsg (/Users/sk/Develop/d20/d20-git-repo/d20-platform-wiki-editor-ui/node_modules/@jupyterlab/services/src/kernel/future.ts:219:1)
    at _handleMessage (/Users/sk/Develop/d20/d20-git-repo/d20-platform-wiki-editor-ui/node_modules/@jupyterlab/services/src/kernel/default.ts:1430:1)
    at <anonymous> (/Users/sk/Develop/d20/d20-git-repo/d20-platform-wiki-editor-ui/node_modules/@jupyterlab/services/src/kernel/default.ts:1595:1)
    --- Promise.then ---
    at KernelConnection._onWSMessage (/Users/sk/Develop/d20/d20-git-repo/d20-platform-wiki-editor-ui/node_modules/@jupyterlab/services/src/kernel/default.ts:1591:1)

Is there a chance to catch and process this exception?

What is a correct way of doing this?

[echarles]

@sok82 We could add more logic in this code block

jupyter-ui/packages/react/src/jupyter/kernel/KernelExecutor.ts

Lines 216 to 224 in 3ac5b26

	case 'error':
	this._outputs.push(message.content as IStream);
	this._outputsChanged.emit(this._outputs);
	this._model.add(output);
	this._modelChanged.emit(this._model);
	if (this._stopOnError) {
	kernelsStore.getState().setExecutionPhase(this._kernelConnection.id, ExecutionPhase.completed_with_error);
	}
	break;

so the state is notified in case of execution error. Worth opening a dedicated issue to track this feature.

[sok82]

Thanks for quick reply, @echarles !

We definitely need to handle this situation and not just explode with exception which is hard to handle

Opened #289 to fix this.

In my case I just don't need this exception at all (because I have all the details in output which is enough), so for me it's better to suppress it (for example having some flag on output that will control suppression) in JS code and just pass exception description to output

[sok82]

Hey @echarles,
found another issue about controlling execution state - the one connected with output.

Actually what I need - is to get FINAL execution state AND output.

But in current implementation - these are two separate things, which could be obtained only separately.

1. Execution state - from Kernel execution phase
2. Output - from output state handler

And this is a problem for me in my flow of events which is

1. Trigger execution
2. Connect handler to output
3. Check that execution finished
4. Disconnect handler from output (this is important)

Previously I checked that execution is finished by receiving output change event and it worked, but I didn't knew in 100% cases state of execution - success or not

Now I am able to get the state, BUT when I receive new ExecutionPhase = completed I have to disconnect output handler - which is bad, because changing of execution phase can be triggered BEFORE output change is triggered and in this case I do not catch output change event and can't set new output value

And sometimes I see that execution phase change happens AFTER output handler is triggered and in this case I can't be sure that execution finished successfully or not, which makes it useless for me.

I need some way to synchronize these two events, and for now I don't know how...

Can you please modify code of https://github.com/datalayer/jupyter-ui/blob/3ac5b261927f70a526f777b847ee84afc6ad4b3e/packages/react/src/jupyter/kernel/KernelExecutor.ts
in a way that solves this problem?

For me the optimal solution would be changing of execution phase ALWAYS before output change.
In this case I can check execution phase in output handler, knowing that it is final state and process output accordingly

[echarles]

Hi @sok82 would be great to open a separated issue for this. If you have an idea for a fix, an associated PR would be awesome also. Thx again for your reports.