Sagemaker - Java Errors while running notebook in a VPC

[gowthamprabhu]

System Information

• Spark or PySpark: Pyspark

Describe the problem

Sagemaker throws the exception as below while running notebook in a VPC
Java gateway process exited before sending its port number.

The same spark code works fine in the notebook running without VPC using conda python3 kernel.

[userkkw]

I have the same issue

[annamykcode]

Same for me

[annamykcode]

I found another way how to work with Spark in Sagemaker
But found the local_pyspark_example.ipynb in Sample Notebooks Sagemaker Processing chapter.
And there they propose to solve the issue as:

1. If you see an exception in running the cell above similar to this - Exception: Java gateway process exited before sending the driver its port number, restart your JupyterServer app to make sure you're on the latest version of Studio.

2. If you are running this notebook in a SageMaker Studio notebook, run the above cell as-is. If you are running on a SageMaker notebook instance, replace com.amazonaws.auth.ContainerCredentialsProvider with com.amazonaws.auth.InstanceProfileCredentialsProvider.