Ipython: Feature request: Be able to receive cell output from a remote kernel after reconnecting

122

Once your browser disconnects from the kernel running on a remote notebook server, there's no way to recover the output once the notebook reconnects to the kernel. My understanding is that even though the output is actually shipped back to the browser, it's just unable to be sent to the right cell for display. If you are using IPython Notebook for long-running computation in a cell, the ability to re-connect and get cell output from the point of reconnection forward would be nice to have.

ptwobrussell picture ptwobrussell  ·  29 Aug 2013

Most helpful comment

27

If you can restrain from just posting +1 that woudl be great. You can or use emojis on the issue, github added them for that, as you can sort issues by number of +1 Emojis. Also if you just like to subscribe to the issue there is the "subscribe" button on the right.

We are aware of the issue and are doing our best. +1 ing won't make things faster. though reviewing open PRs and trying to write simple patches for issue tagged as "Need contributor" can free us some time to actually work on this :-) We'll also be happy to teach you jupyter internals to get new contributors !

Also at some point the issue probably need to be migrated to jupyter/notebook as this track only core IPython issue now.

Thanks !

Carreau picture Carreau  ·  24 Jan 2017

All comments

0

Do you condone +1 comments? (Sorry if not).

michaelaye picture michaelaye  ·  23 Oct 2013
0

+1 (sorry if not...)
This feature will highly improve the ipython notebook interface, particularly for long runs!

eldad-a picture eldad-a  ·  4 Feb 2014
3

There is a "half" of a work-around based on the ipycache extension.
Well, it may not be even a "half" but it is the best I found so far:
A cell can be "cached" such that running it again would load the output instead of executing the code again.

For example:
After executing a cell containing

%load_ext ipycache
%mkdir -vp ipycache
import time

A cell can be "decorated" by the %%cache magic:

%%cache ipycache/test_long 
i = 1
while i<30:
    print '{}, '.format(i),
    time.sleep(1)
    i+=1

The browser can be closed
Running the last cell again will not execute it again but load the cached output (and display it).
However, one has to wait for the execution to complete.
So this is will not help in monitoring the progress of a running code and I am not sure how it deals with errors.

eldad-a picture eldad-a  ·  17 Mar 2014
0

+1

alexbw picture alexbw  ·  27 May 2014
0

+1

kristall picture kristall  ·  8 Oct 2014
0

This would be incredibly useful for creating reports or tests
+1

mhgarci1 picture mhgarci1  ·  17 Nov 2014
0

+1, didn't actually realise this is a limitation of the notebook and lost some output that I incorrectly assumed would be waiting for me when I rejoined my server...

Any chance a core developer could weigh in and give a brief idea as to how feasible this is to fix?

Just to confirm I'm not missing something, whilst this goes unresolved the notebook is not good for running long jobs on remove servers that you can then rejoin to check state.

As a concrete example, a user is at their office workstation and is just finishing up a notebook that will take some hours to run. User leaves the office and on the train later on gets a (sporadic) SSH port forward back to the office workstation. They tweak the notebook and get it finished - they hit run. Connection will bounce in and out for the next few hours - later on the user will rejoin the SSH port forward and have no output waiting for them.

jabooth picture jabooth  ·  17 Dec 2014
0

Any chance a core developer could weigh in and give a brief idea as to how feasible this is to fix?

This is tough. Really. We need to move the M part of MVC to the server and get synchronization with client, so an architecture which is more MC-MVC with the dash being the network.
It's in progress but I wouldn't bet for less than in a year or so (unless we get lots more help, or lots more money)

Carreau picture Carreau  ·  17 Dec 2014
1

So, I guess @jabooth 's work-flow could only be solved today with writing any and all results to disk, correct?

michaelaye picture michaelaye  ·  17 Dec 2014
0

If the kenel still have the value then you can re-display it.
The only problem is if anythong get displayed while no client is connected then the displayed result is just gone.
But the following is still possible.

sleep(5)
a = 1

close browser, wait 10 sec, open browser,

print(a)
# 1

You can probably alo do things with Futures also.

Carreau picture Carreau  ·  17 Dec 2014
0

@Carreau, I am willing to help get this working, as this would be a big help to my coworkers and I who are used to working with screen for remote jobs. You say this is being worked on. Is there a fork or branch I could take a look at? Thanks.

jacobhinkle picture jacobhinkle  ·  20 Jan 2015
0

Thanks a lot for the proposal.

Hum, right now that will be though, but let me make a quick recap of future plan.

  1. get 3.0 out of the door.
  2. get https://github.com/jupyter/jupyter-drive out of the door (need 1)
  3. Refactor IPython to support RT colaboration (technically help with what you ask, 2 will help a lot)
  4. Make an Operational Ttransform server (google RT like) backend for RT communication (independant of 1-3 mostly, but knowing google RT API would help to make things compatible)
  5. Make kernel speek with (4) instead of browser.

Depending on what you want to work on there are plenty to do.
Even just playing with IPython master, testing Pull request, or reviewing code
is of huge help as it free us time to work on more complex stuff.

Even if you are just better at communicating that we are, and know a bit about our future plan,
even proposing a news to be posted on main ipython.org website would be awesome !

I should probably make a blog post that details more what we are doing :-P

Carreau picture Carreau  ·  20 Jan 2015
0

What if the web UI could just show some kind of raw console with the output being sent from the kernel, even if it doesn't know what cell that output should go into? Obviously that's not the full long-term solution, but at least it would alleviate a critical usability issue for those of us using the notebook to monitor long-running jobs.

malmaud picture malmaud  ·  26 Jan 2015
0

https://github.com/ipython/ipython/pull/7087 should at least lay down the basics for that where there are hooks to receive this output. We still don't know exactly what to do with the output, core team is divised half half on "the right thing'" to do.

Carreau picture Carreau  ·  26 Jan 2015
0

This is dirty, DIRTY hack! But if you want to be able at least see the progress output (not cached though), put this code in your custom.js

$([IPython.events]).on('app_initialized.NotebookApp', function(){
      IPython.notebook.events.on('received_unsolicited_message.Kernel', function(evt, data) {
        cells = IPython.notebook.get_cells();
          ncells = cells.length
          idx = ncells-1
          for (var i = ncells-1; i >= 0; i--) {
                if (cells[i].get_text() != ""){
                    idx = i;
                    break;
                }
        }
        cells[idx].output_area.handle_output(data)
        }); 
});

It will append the output from the kernel to the last non-empty code cell. Not nice, but works for me...

jeanpijon picture jeanpijon  ·  14 Aug 2015
0

+1

udibr picture udibr  ·  30 Oct 2015
0

+1

chasewright picture chasewright  ·  11 Dec 2015
0

+1

pasky picture pasky  ·  17 Jan 2016
0

+1

deepakrox picture deepakrox  ·  8 Feb 2016
0

+1

mohamed-ezz picture mohamed-ezz  ·  13 Mar 2016
0

+1

sebastian-schlecht picture sebastian-schlecht  ·  14 Apr 2016
0

+1

zhengbx picture zhengbx  ·  19 Apr 2016
0

+1

andychisholm picture andychisholm  ·  25 May 2016
0

+1

wangz10 picture wangz10  ·  7 Jul 2016
0

+1

tachim picture tachim  ·  9 Oct 2016
0

+1

TashaArvanitis picture TashaArvanitis  ·  17 Oct 2016
0

+1

jtanios picture jtanios  ·  23 Nov 2016
0

+1

paolodoors picture paolodoors  ·  24 Nov 2016
0

+1

mlzxy picture mlzxy  ·  27 Nov 2016
0

I think if the notebook could have another V layer to capture all the progress data. The browser will interact with this layer instead of raw python kernel. The problem could be solved. (Like a long running proxy browser on your server) This doesn't seem to be a small surgery.

But another idea I have is more like frontend point of view. If the jupyter notebook supports server rendering someday using react/angular2.js/etc... then this problem will be automatically solved.

This feature is very important, since jupyter notebook is extremely popular among machine learning development workflow. Some algorithm training programs could last a couple of days.

mlzxy picture mlzxy  ·  27 Nov 2016
0

Yes @BenBBear, you seem to have came up with roughly one of the long-time planned architecture.

The extra question we still haven't solved is how to build real-time collaboration on top of that.

Currently if you used Python you can use Futures to hold intermediate results.

Carreau picture Carreau  ·  27 Nov 2016
0

@Carreau

Great to see RT collaboration is on the road map. Maybe the collaboration could be easier if the proxy layer is websocket-enabled instead of just http? Just my rough idea.

Use Asynchronous computing to hold intermediate result, this is genius... 👍

mlzxy picture mlzxy  ·  27 Nov 2016
0

+1

abrahamneben picture abrahamneben  ·  28 Dec 2016
0

+1

guillaumerosinosky picture guillaumerosinosky  ·  5 Jan 2017
0

+1

mharradon picture mharradon  ·  6 Jan 2017
0

+1

francomarianardini picture francomarianardini  ·  24 Jan 2017
27

If you can restrain from just posting +1 that woudl be great. You can or use emojis on the issue, github added them for that, as you can sort issues by number of +1 Emojis. Also if you just like to subscribe to the issue there is the "subscribe" button on the right.

We are aware of the issue and are doing our best. +1 ing won't make things faster. though reviewing open PRs and trying to write simple patches for issue tagged as "Need contributor" can free us some time to actually work on this :-) We'll also be happy to teach you jupyter internals to get new contributors !

Also at some point the issue probably need to be migrated to jupyter/notebook as this track only core IPython issue now.

Thanks !

Carreau picture Carreau  ·  24 Jan 2017
9

+1 (SCNR)

A connection drop today also made me fall into that trap :-/

I guess even at the moment something on server side must be able to notice that no client is connected anymore. Can't it in such cases modify the ipynb file directly? (Like an auto-saver server side fail-over loop in case no client is reachable.)

Obviously a proper, cool async solution that also allows multi-player edit would be awesome, but after nearly 4 years, maybe a simpler solution first?

joernhees picture joernhees  ·  4 Jun 2017
1

+1

simon19891101 picture simon19891101  ·  11 May 2018
1

+100000000000

wanglovesyang picture wanglovesyang  ·  18 May 2018
0

This actually seems to happen now, though. When I leave the notebook open, put my laptop to sleep while there's output coming (even matplotlib images), then later resume my laptop, re-initiate the connection and reconnect to the kernel, I receive the output that happened while the connection was gone. There's actually two flags controlling this that I had to increase:

## (bytes/sec) Maximum rate at which stream output can be sent on iopub before
#  they are limited.
c.NotebookApp.iopub_data_rate_limit = 100000000

## (msgs/sec) Maximum rate at which messages can be sent on iopub before they are
#  limited.
c.NotebookApp.iopub_msg_rate_limit = 100000

## (sec) Time window used to  check the message and data rate limits.
c.NotebookApp.rate_limit_window = 3
lucasb-eyer picture lucasb-eyer  ·  25 May 2018
0

@lucasb-eyer You mean jupyter notebook or jupyterlab? It seems to be somewhat working in jupyterlab, but not perfectly.

Regrading your settings, could you justify that your settings relevant?

phizaz picture phizaz  ·  25 May 2018
0

Jupyter notebook, I didn't even know about jupyterlab :smile:

Yes, the settings are relevant because when the notebook reconnects, all missed messages come streaming in, which kicks-in these rate-limits. Before adjusting the values as above, I only got the first bunch of missed messages upon reconnect exactly because of these rate-limits. Now they are large enough so that I usually get everything I've missed. That's in my typical workflows and tasks, at least. The notebook actually pointed me to those settings when starting to drop messages.

lucasb-eyer picture lucasb-eyer  ·  25 May 2018
0

Which version of notebook are you using? I couldn't get this working on 4.0.6 version.

AbhinavMehrotra picture AbhinavMehrotra  ·  25 May 2018
0

Uhm, seriously? 4.0.6 is 3 years old! I'm using 5.4.1 in all that I described above.

lucasb-eyer picture lucasb-eyer  ·  25 May 2018
0

Hello Lucas, thanks for your suggestion.

I am using Jupyter Notebook 5.4.0. I put the three options you were
mentioning in the config file.

Unfortunately, I also couldn't get this working. I will update to 5.4.1 to
test the same version you have.

Best regards,

Franco

On Fri, May 25, 2018 at 6:35 PM, Lucas Beyer notifications@github.com
wrote:

Uhm, seriously? 4.0.6 is 3 years old! I'm using 5.4.1 in all that I
described above.


You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/ipython/ipython/issues/4140#issuecomment-392113080,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ADmYKzmINkC0vF6AIHd-We1_5G-_CB0Qks5t2DLAgaJpZM4A8uyX
.

francomarianardini picture francomarianardini  ·  25 May 2018
0

@lucasb-eyer I guess this only works when you reopen the browser, see https://github.com/jupyter/notebook/issues/1647:

What #2871 did was buffer any outputs (and other messages) until the user reconnects -- it will only help you for cases where you're reconnecting the same tab.

So this fix does not really apply for a remote use case where you want to close the browser tab.

CYHSM picture CYHSM  ·  12 Jun 2018
1

@CYHSM Your statement is self-contradicting, I guess you wanted to write

this only works when you don't reopen the browser"

right? Because in that case, I think you are right: I always put my laptop on standby and thus the notebook stays in the same tab.

lucasb-eyer picture lucasb-eyer  ·  13 Jun 2018
7

Any update on this, please?

tanmay-kulkarni picture tanmay-kulkarni  ·  26 Sep 2018
0

+1, this feature would be extremely useful

mjkvaak picture mjkvaak  ·  5 Jul 2019