Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Sagemaker Studio Lab running notebook stops when disconnected from Internet #263

Open
samitChak opened this issue Apr 25, 2024 · 3 comments
Labels
bug Something isn't working

Comments

@samitChak
Copy link

I am new to AWS Sagemaker Studio Lab.

I was testing with a looping program. I saved the program and then started running it. Then I disconnected the internet. After sometime I reconnected and found that the notebook was not running.

Sagemaker is supposed to run on cloud after I start the run and it will not require my local internet connection - right?

Then why did it stop running? Is there any setting that I need to do for the program to run for the entire session duration even if I close my local machine.

@samitChak samitChak added the bug Something isn't working label Apr 25, 2024
@MicheleMonclova
Copy link

Interesting....
A few clarifying questions.
I am assuming you kicked of a training job then closed laptop or something, right?
And when you returned the job never completed? or it completed but there was no connection to receive the results.

@samitChak
Copy link
Author

samitChak commented Apr 25, 2024

Interesting.... A few clarifying questions. I am assuming you kicked of a training job then closed laptop or something, right? And when you returned the job never completed? or it completed but there was no connection to receive the results.

Yes, I started the job (a simple for loop to print something every 10 seconds)...have seen it throw some initial results while running.....then I disconnected from the internet....after a while, I reconnected to internet, I logged in to Studio Lab and saw that the program did not run.....it got interrupted.

@MicheleMonclova
Copy link

MicheleMonclova commented Apr 26, 2024

Hi, after investigating this it looks like this is the expected behavior of JupyterLab. That is, if the internet connection is disconnected it will stop. However, if you are interested in running a training job on Studio Lab, even after the session ends, then you can schedule a job.

See documentation here: https://docs.aws.amazon.com/sagemaker/latest/dg/scheduled-notebook-installation.html

The upside is you can pick different instance types, and have it run as long as you want, even if you are disconnected.
The downside is, it will need to run in your AWS account and you will incur a cost.

We haven't advertised this a lot so let me know if you have questions.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants