### Get Kernel Start Response Source: https://github.com/jupyter-server/enterprise_gateway/blob/main/docs/source/developers/rest-api.md This is an example of the JSON response received after successfully requesting to start a kernel. It includes the kernel's ID and initial state. ```json { "id": "f88bdc84-04c6-4021-963d-6811a61eca18", "name": "spark_python_yarn_cluster", "last_activity": "2022-02-12T00:40:45.080107Z", "execution_state": "starting", "connections": 0 } ``` -------------------------------- ### Install Enterprise Gateway Kernel Image Files (Independent Image) Source: https://github.com/jupyter-server/enterprise_gateway/blob/main/docs/source/developers/custom-images.md Installs kernel image files using curl for environments where wget is not available. This example also demonstrates setting up a non-root user. ```dockerfile RUN curl -L https://github.com/jupyter-server/enterprise_gateway/releases/download/vVERSION/jupyter_enterprise_gateway_kernel_image_files-VERSION.tar.gz | \ tar -xz -C /usr/local/bin ``` -------------------------------- ### Configure Custom Kernel Session Manager Source: https://github.com/jupyter-server/enterprise_gateway/blob/main/docs/source/operators/config-availability.md Example of starting Enterprise Gateway with a custom kernel session manager class and standalone availability mode. Setting enable_persistence is optional when availability mode is specified. ```bash #!/bin/bash LOG=/var/log/enterprise_gateway.log PIDFILE=/var/run/enterprise_gateway.pid jupyter enterprisegateway --ip=0.0.0.0 --port_retries=0 --log-level=DEBUG \ --EnterpriseGatewayApp.kernel_session_manager_class=custom.package.MyCustomKernelSessionManager \ --MyCustomKernelSessionManager.enable_persistence=True \ --EnterpriseGatewayApp.availability_mode=standalone > $LOG 2>&1 & if [ "$?" -eq 0 ]; then echo $! > $PIDFILE else exit 1 fi ``` -------------------------------- ### GET /hello endpoint Source: https://github.com/jupyter-server/enterprise_gateway/blob/main/enterprise_gateway/tests/resources/kernel_api2.ipynb Prints a simple 'hello world' message, simulating a basic GET request. ```python # GET /hello print(hello_message.format('world')) ``` -------------------------------- ### Install and Run Jupyter Enterprise Gateway Source: https://github.com/jupyter-server/enterprise_gateway/blob/main/README.md Installs the package from PyPI, displays all configuration options, and runs the gateway with default settings. ```bash pip install --upgrade jupyter_enterprise_gateway ``` ```bash jupyter enterprisegateway --help-all ``` ```bash jupyter enterprisegateway ``` -------------------------------- ### Install OS and Python Dependencies Source: https://github.com/jupyter-server/enterprise_gateway/blob/main/docs/source/developers/custom-images.md Installs essential OS packages and Python dependencies required for the kernel wrapper. Ensure these are installed before proceeding. ```dockerfile RUN apt-get update && apt-get install -yq --no-install-recommends \ build-essential \ libsm6 \ libxext-dev \ libxrender1 \ netcat \ python3-dev \ tzdata \ unzip \ && rm -rf /var/lib/apt/lists/* ``` ```dockerfile RUN pip install pycrypto ``` -------------------------------- ### Install Distributed Kernel Specification Source: https://github.com/jupyter-server/enterprise_gateway/blob/main/docs/source/operators/deploy-distributed.md Installs the sample python_distributed kernel specification on a given node. This command must be repeated for each node and kernel specification. ```Bash wget https://github.com/jupyter-server/enterprise_gateway/releases/download/v3.3.0/jupyter_enterprise_gateway_kernelspecs-3.3.0.tar.gz KERNELS_FOLDER=/usr/local/share/jupyter/kernels tar -zxvf jupyter_enterprise_gateway_kernelspecs-3.3.0.tar.gz --strip 1 --directory $KERNELS_FOLDER/python_distributed/ python_distributed/ ``` -------------------------------- ### Install R Kernel (IRkernel) and Packages Source: https://github.com/jupyter-server/enterprise_gateway/blob/main/docs/source/operators/installing-kernels.md Installs R essentials, IRkernel, and argparse using conda. It then creates and executes an R script to install necessary R packages from CRAN and GitHub, and finally installs the IRkernel specification. An optional command to list installed R packages is also provided. ```Bash conda install --yes --quiet -c r r-essentials r-irkernel r-argparse # Create an R-script to run and install packages and update IRkernel cat <<'EOF' > install_packages.R install.packages(c('repr', 'IRdisplay', 'evaluate', 'git2r', 'crayon', 'pbdZMQ', 'devtools', 'uuid', 'digest', 'RCurl', 'curl', 'argparse'), repos='http://cran.rstudio.com/') devtools::install_github('IRkernel/IRkernel@0.8.14') IRkernel::installspec(user = FALSE) EOF # run the package install script $ANACONDA_HOME/bin/Rscript install_packages.R # OPTIONAL: check the installed R packages ls $ANACONDA_HOME/lib/R/library ``` -------------------------------- ### Run Interactive Mode and Start Gateway Manually Source: https://github.com/jupyter-server/enterprise_gateway/blob/main/etc/docker/enterprise-gateway-demo/README.md Run the enterprise-gateway-demo container in interactive mode and manually start the Enterprise Gateway. This allows for direct interaction within the container. ```bash docker run -it --rm -p 8888:8888 -p 8088:8088 -p 8042:8042 --net=jeg elyra/enterprise-gateway-demo /bin/bash ``` ```bash sudo -u jovyan /usr/local/bin/start-enterprise-gateway.sh ``` -------------------------------- ### GET /multi Source: https://github.com/jupyter-server/enterprise_gateway/blob/main/enterprise_gateway/tests/resources/kernel_api3.ipynb Demonstrates handling of multiple GET requests to the same endpoint. ```APIDOC ## GET /multi ### Description Demonstrates handling of multiple GET requests to the same endpoint. This endpoint is called twice, first to initialize 'x' and then to print its value. ### Method GET ### Endpoint /multi ### Response #### Success Response (200) - **message** (string) - Indicates the value of 'x'. ### Response Example { "message": "x is 1" } ``` -------------------------------- ### Example Configuration Option: remote_hosts Source: https://github.com/jupyter-server/enterprise_gateway/blob/main/docs/source/operators/config-file.md This example shows how to configure the `remote_hosts` option within the `jupyter_enterprise_gateway_config.py` file. This option specifies the hosts where DistributedProcessProxy kernels will be launched. Note that environment variables can also be used for configuration. ```python ## Bracketed comma-separated list of hosts on which DistributedProcessProxy # kernels will be launched e.g., ['host1','host2']. # (EG_REMOTE_HOSTS env var - non-bracketed, just comma-separated) # Default: ['localhost'] # c.EnterpriseGatewayConfigMixin.remote_hosts = ['localhost'] ``` -------------------------------- ### Start Enterprise Gateway Stack with Docker Compose Source: https://github.com/jupyter-server/enterprise_gateway/blob/main/docs/source/operators/deploy-docker.md This command starts the Enterprise Gateway stack using Docker Compose for traditional Docker environments. This approach leverages compose but not Swarm's monitoring and restart capabilities. ```bash docker-compose up ``` -------------------------------- ### Install Jupyter Server with pip Source: https://github.com/jupyter-server/enterprise_gateway/blob/main/docs/source/users/installation.md Use this command to install the Jupyter Server package when using pip. ```bash pip install jupyter_server ``` -------------------------------- ### GET /hello Source: https://github.com/jupyter-server/enterprise_gateway/blob/main/enterprise_gateway/tests/resources/kernel_api3.ipynb Retrieves a generic greeting message. ```APIDOC ## GET /hello ### Description Retrieves a generic greeting message. ### Method GET ### Endpoint /hello ### Response #### Success Response (200) - **message** (string) - A greeting message. ### Response Example { "message": "hello world" } ``` -------------------------------- ### Start Enterprise Gateway in Standalone Mode Source: https://github.com/jupyter-server/enterprise_gateway/blob/main/docs/source/operators/config-availability.md Starts Enterprise Gateway with standalone availability enabled. This mode is suitable for single-instance recovery scenarios. ```bash #!/bin/bash LOG=/var/log/enterprise_gateway.log PIDFILE=/var/run/enterprise_gateway.pid jupyter enterprisegateway --ip=0.0.0.0 --port_retries=0 --log-level=DEBUG \ --EnterpriseGatewayApp.availability_mode=standalone > $LOG 2>&1 & if [ "$?" -eq 0 ]; then echo $! > $PIDFILE else exit 1 fi ``` -------------------------------- ### Start a Spark Kernel on YARN Cluster Source: https://github.com/jupyter-server/enterprise_gateway/blob/main/docs/source/developers/rest-api.md Use this command to start a spark_python_yarn_cluster kernel with a specific environment variable set. This is useful for launching distributed Spark jobs. ```console curl -X POST -i 'http://my-gateway-server.com:8888/api/kernels' --data '{ "name": "spark_python_yarn_cluster", "env": { "KERNEL_USERNAME": "jovyan" }}' ``` -------------------------------- ### Install Enterprise Gateway Kernel Image Files (docker-stacks) Source: https://github.com/jupyter-server/enterprise_gateway/blob/main/docs/source/developers/custom-images.md Installs the necessary kernel image files from a tarball and sets permissions. This is typically used when building upon a docker-stacks base image. ```dockerfile RUN wget https://github.com/jupyter-server/enterprise_gateway/releases/download/vVERSION/jupyter_enterprise_gateway_kernel_image_files-VERSION.tar.gz &&\ tar -xvf jupyter_enterprise_gateway_kernel_image_files-VERSION.tar.gz -C /usr/local/bin &&\ rm -f jupyter_enterprise_gateway_kernel_image_files-VERSION.tar.gz &&\ fix-permissions /usr/local/bin ``` -------------------------------- ### Enterprise Gateway Start Script with Logging and Culling Source: https://github.com/jupyter-server/enterprise_gateway/blob/main/docs/source/operators/launching-eg.md This script starts Enterprise Gateway in the background with DEBUG tracing enabled, idle kernel culling configured for 12 hours, and periodic idle checks every 60 seconds. It also manages log and PID files. ```bash #!/bin/bash LOG=/var/log/enterprise_gateway.log PIDFILE=/var/run/enterprise_gateway.pid jupyter enterprisegateway --ip=0.0.0.0 --port_retries=0 --log-level=DEBUG --RemoteKernelManager.cull_idle_timeout=43200 --MappingKernelManager.cull_interval=60 > $LOG 2>&1 & if [ "$?" -eq 0 ]; then echo $! > $PIDFILE else exit 1 fi ``` -------------------------------- ### Extend Existing Kernel Image with Libraries Source: https://github.com/jupyter-server/enterprise_gateway/blob/main/docs/source/developers/custom-images.md Use this Dockerfile to extend an existing kernel image by installing additional libraries. Switch to the root user to perform installations and then back to the notebook user. ```dockerfile FROM elyra/kernel-py:VERSION USER root # switch to root user to perform installation (if necessary) RUN pip install my-libraries USER $NB_UID # switch back to the jovyan user ``` -------------------------------- ### Start Enterprise Gateway in Replication Mode Source: https://github.com/jupyter-server/enterprise_gateway/blob/main/docs/source/operators/config-availability.md Starts Enterprise Gateway with replication availability enabled. This mode is designed for multi-instance high availability and requires a reverse proxy or load balancer. ```bash #!/bin/bash LOG=/var/log/enterprise_gateway.log PIDFILE=/var/run/enterprise_gateway.pid jupyter enterprisegateway --ip=0.0.0.0 --port_retries=0 --log-level=DEBUG \ --EnterpriseGatewayApp.availability_mode=replication > $LOG 2>&1 & if [ "$?" -eq 0 ]; then echo $! > $PIDFILE else exit 1 fi ``` -------------------------------- ### Prepare JSON request for GET /content-type Source: https://github.com/jupyter-server/enterprise_gateway/blob/main/enterprise_gateway/tests/resources/kernel_api2.ipynb Constructs a JSON string with 'headers' including 'Content-Type'. ```python REQUEST = json.dumps({ 'headers' : { 'Content-Type': 'application/json' } }) ``` -------------------------------- ### Display All Help Options Source: https://github.com/jupyter-server/enterprise_gateway/blob/main/docs/source/operators/config-cli.md Use this command to view all available configuration options for the Enterprise Gateway on the command line. ```bash jupyter enterprisegateway --help-all ``` -------------------------------- ### View Gateway Client Help Source: https://github.com/jupyter-server/enterprise_gateway/blob/main/docs/source/users/client-config.md Use these commands to view the available Gateway Client configuration options. The `--help-all` option provides a detailed list, while `--generate-config` can create a configuration file. ```bash jupyter server --help-all ``` ```bash jupyter server --generate-config ``` -------------------------------- ### Set SPARK_HOME for HDP Source: https://github.com/jupyter-server/enterprise_gateway/blob/main/docs/source/operators/deploy-yarn-cluster.md Set the SPARK_HOME environment variable to point to the Apache Spark installation path. This example is for an HDP distribution. ```default SPARK_HOME:/usr/hdp/current/spark2-client # For HDP distribution ``` -------------------------------- ### YARN Client Mode SPARK_HOME Configuration Source: https://github.com/jupyter-server/enterprise_gateway/blob/main/docs/source/operators/deploy-distributed.md Set the SPARK_HOME environment variable to point to your Apache Spark installation path. This example shows a configuration for HDP distribution. ```shell SPARK_HOME:/usr/hdp/current/spark2-client #For HDP distribution ``` -------------------------------- ### Custom Kernel Image Built on Jupyter Image Source: https://github.com/jupyter-server/enterprise_gateway/blob/main/docs/source/developers/custom-images.md Example Dockerfile for a custom Python kernel image based on a Jupyter Docker-stacks image. It installs necessary packages and downloads EG kernel launchers. Replace VERSION with the appropriate value. ```dockerfile # Choose a base image. Preferrably one from https://github.com/jupyter/docker-stacks FROM jupyter/scipy-notebook:61d8aaedaeaf # Switch user to root since, if from docker-stacks, its probably jovyan USER root # Install any packages required for the kernel-wrapper. If the image # does not contain the target kernel (i.e., IPython, IRkernel, etc., # it should be installed as well. RUN pip install pycrypto # Download and extract the enterprise gateway kernel launchers and bootstrap ``` -------------------------------- ### Abstract RemoteProcessProxy confirm_remote_startup Method Source: https://github.com/jupyter-server/enterprise_gateway/blob/main/docs/source/contributors/system-architecture.md Abstract method to be implemented by subclasses of `RemoteProcessProxy`. It is responsible for confirming that a remote kernel has been successfully launched and is ready to accept requests. Requires the kernel command and accepts additional keyword arguments. ```python @abstractmethod def confirm_remote_startup(self, kernel_cmd, **kw): ``` -------------------------------- ### Start a Kernel Source: https://github.com/jupyter-server/enterprise_gateway/blob/main/docs/source/developers/rest-api.md Initiates a new kernel process and returns its unique identifier. Environment variables can be provided to configure the kernel's execution context. ```APIDOC ## POST /api/kernels ### Description Start a kernel and return the uuid. ### Method POST ### Endpoint /api/kernels ### Parameters #### Request Body - **env** (object) - Optional - A dictionary of environment variables and values to include in the kernel process - subject to filtering. - **name** (string) - Optional - Kernel spec name (defaults to default kernel spec for server) ### Response #### Success Response (201 Created) - **Location** (header) - Model for started kernel #### Error Response (403 Forbidden) - The maximum number of kernels have been created. ``` -------------------------------- ### GET /nocontent Response Source: https://github.com/jupyter-server/enterprise_gateway/blob/main/enterprise_gateway/tests/resources/responses_2.ipynb Handles a GET request to /nocontent, which returns no content. ```python # GET /nocontent ``` -------------------------------- ### GET /name Operation Source: https://github.com/jupyter-server/enterprise_gateway/blob/main/enterprise_gateway/tests/resources/simple_api3.ipynb Simulates a GET request to retrieve the value of the 'name' variable. ```python # GET /name print(name) ``` -------------------------------- ### Serve Jekyll Website Locally Source: https://github.com/jupyter-server/enterprise_gateway/blob/main/website/README.md Use this command to build and serve the Jekyll website locally. The `--watch` flag enables live reloading for development. ```bash jekyll serve --watch ``` -------------------------------- ### GET /json Response Source: https://github.com/jupyter-server/enterprise_gateway/blob/main/enterprise_gateway/tests/resources/responses_2.ipynb Generates a simple JSON response for a GET request to /json. ```python # GET /json print '''{ "hello" : "world"}''' ``` -------------------------------- ### Install Jupyter Notebook with conda Source: https://github.com/jupyter-server/enterprise_gateway/blob/main/docs/source/users/installation.md Use this command to install the Jupyter Notebook package from the conda-forge channel. ```bash conda install -c conda-forge notebook ``` -------------------------------- ### Prepare JSON request for /hello/:person Source: https://github.com/jupyter-server/enterprise_gateway/blob/main/enterprise_gateway/tests/resources/kernel_api2.ipynb Constructs a JSON string with path parameters for a dynamic GET request. ```python REQUEST = json.dumps({ 'path' : { 'person' : 'test_person' } }) ``` -------------------------------- ### Install Jupyter Notebook with pip Source: https://github.com/jupyter-server/enterprise_gateway/blob/main/docs/source/users/installation.md Use this command to install the Jupyter Notebook package when using pip. ```bash pip install notebook ``` -------------------------------- ### Run Unit Tests Source: https://github.com/jupyter-server/enterprise_gateway/blob/main/docs/source/contributors/devinstall.md Execute the full unit test suite. To run a subset, pass the 'TEST' argument to the make command. ```bash make test ``` ```bash make test TEST="test_gatewayapp.py" ``` ```bash make test TEST="test_gatewayapp.py::TestGatewayAppConfig" ``` ```bash make test TEST="test_gatewayapp.py::TestGatewayAppConfig::test_config_env_vars_bc" ``` -------------------------------- ### Install Jupyter Server with conda Source: https://github.com/jupyter-server/enterprise_gateway/blob/main/docs/source/users/installation.md Use this command to install the Jupyter Server package from the conda-forge channel. ```bash conda install -c conda-forge jupyter_server ``` -------------------------------- ### Build Documentation Source: https://github.com/jupyter-server/enterprise_gateway/blob/main/docs/source/contributors/devinstall.md Builds the HTML documentation for Enterprise Gateway using Sphinx. This command internally executes 'make requirements html' from the 'docs' sub-directory. ```bash make docs ``` -------------------------------- ### Enable SSL with Configuration File Source: https://github.com/jupyter-server/enterprise_gateway/blob/main/docs/source/operators/config-security.md Set the SSL certificate and key file paths in the Enterprise Gateway configuration file to enable secure communication. This method offers better readability and maintainability compared to command-line arguments. ```python c.EnterpriseGatewayApp.certfile = '/absolute/path/to/your/certificate/fullchain.pem' c.EnterpriseGatewayApp.keyfile = '/absolute/path/to/your/certificate/privatekey.key' ``` -------------------------------- ### GET /etag Response Source: https://github.com/jupyter-server/enterprise_gateway/blob/main/enterprise_gateway/tests/resources/responses_2.ipynb Generates a JSON response for a GET request to /etag, including an ETag header. ```python # GET /etag print '''{ "hello" : "world"}''' ``` -------------------------------- ### Configure Enterprise Gateway Server with SSL Source: https://github.com/jupyter-server/enterprise_gateway/blob/main/docs/source/operators/config-security.md Start the Enterprise Gateway server with SSL enabled by specifying the certfile and keyfile options. This ensures encrypted communication between the server and clients. ```bash jupyter enterprisegateway --ip=0.0.0.0 --port_retries=0 --certfile=mycert.pem --keyfile=mykey.key ``` -------------------------------- ### Handle GET Request for Name Source: https://github.com/jupyter-server/enterprise_gateway/blob/main/enterprise_gateway/tests/resources/simple_api2.ipynb Prints the current value of the 'name' variable when a GET request is made to /name. ```python # GET /name print name ``` -------------------------------- ### Start a Kernel Source: https://github.com/jupyter-server/enterprise_gateway/blob/main/docs/source/developers/rest-api.md Initiates a new kernel by sending a POST request to the /api/kernels endpoint. The request body can specify the kernel name and environment variables. ```APIDOC ## POST /api/kernels ### Description Starts a new kernel. The `name` parameter specifies the kernel to start, and the `env` parameter allows setting environment variables for the kernel. ### Method POST ### Endpoint /api/kernels ### Parameters #### Request Body - **name** (string) - Required - The name of the kernel to start. - **env** (object) - Optional - Environment variables to set for the kernel. - **KERNEL_USERNAME** (string) - Example environment variable. ### Request Example ```json { "name": "spark_python_yarn_cluster", "env": { "KERNEL_USERNAME": "jovyan" } } ``` ### Response #### Success Response (200) - **id** (string) - The unique identifier for the started kernel. - **name** (string) - The name of the started kernel. - **last_activity** (string) - The timestamp of the last activity. - **execution_state** (string) - The current execution state of the kernel. - **connections** (integer) - The number of active connections to the kernel. #### Response Example ```json { "id": "f88bdc84-04c6-4021-963d-6811a61eca18", "name": "spark_python_yarn_cluster", "last_activity": "2022-02-12T00:40:45.080107Z", "execution_state": "starting", "connections": 0 } ``` ``` -------------------------------- ### GET /stderr endpoint simulation Source: https://github.com/jupyter-server/enterprise_gateway/blob/main/enterprise_gateway/tests/resources/kernel_api3.ipynb Simulates a GET request to /stderr, printing messages to both standard output and standard error. ```python # GET /stderr print('I am text on stdout') print('I am text on stderr', file=sys.stderr) ``` -------------------------------- ### Gateway Client Configuration Options Source: https://github.com/jupyter-server/enterprise_gateway/blob/main/docs/source/users/client-config.md This output from `jupyter server --help-all` lists all configurable options for the Gateway Client. Configuration file options are derived by replacing `--` with `c.`. ```default --GatewayClient.auth_scheme= The auth scheme, added as a prefix to the authorization token used in the HTTP headers. (JUPYTER_GATEWAY_AUTH_SCHEME env var) Default: None ``` ```default --GatewayClient.auth_token= The authorization token used in the HTTP headers. The header will be formatted as:: { 'Authorization': '{auth_scheme} {auth_token}' } (JUPYTER_GATEWAY_AUTH_TOKEN env var) Default: None ``` ```default --GatewayClient.ca_certs= The filename of CA certificates or None to use defaults. (JUPYTER_GATEWAY_CA_CERTS env var) Default: None ``` ```default --GatewayClient.client_cert= The filename for client SSL certificate, if any. (JUPYTER_GATEWAY_CLIENT_CERT env var) Default: None ``` ```default --GatewayClient.client_key= The filename for client SSL key, if any. (JUPYTER_GATEWAY_CLIENT_KEY env var) Default: None ``` ```default --GatewayClient.connect_timeout= The time allowed for HTTP connection establishment with the Gateway server. (JUPYTER_GATEWAY_CONNECT_TIMEOUT env var) Default: 40.0 ``` ```default --GatewayClient.env_whitelist= A comma-separated list of environment variable names that will be included, along with their values, in the kernel startup request. The corresponding `env_whitelist` configuration value must also be set on the Gateway server - since that configuration value indicates which environmental values to make available to the kernel. (JUPYTER_GATEWAY_ENV_WHITELIST env var) Default: '' ``` ```default --GatewayClient.gateway_retry_interval= The time allowed for HTTP reconnection with the Gateway server for the first time. Next will be JUPYTER_GATEWAY_RETRY_INTERVAL multiplied by two in factor of numbers of retries but less than JUPYTER_GATEWAY_RETRY_INTERVAL_MAX. (JUPYTER_GATEWAY_RETRY_INTERVAL env var) Default: 1.0 ``` ```default --GatewayClient.gateway_retry_interval_max= The maximum time allowed for HTTP reconnection retry with the Gateway server. (JUPYTER_GATEWAY_RETRY_INTERVAL_MAX env var) Default: 30.0 ``` ```default --GatewayClient.gateway_retry_max= The maximum retries allowed for HTTP reconnection with the Gateway server. (JUPYTER_GATEWAY_RETRY_MAX env var) Default: 5 ``` ```default --GatewayClient.headers= Additional HTTP headers to pass on the request. This value will be converted to a dict. (JUPYTER_GATEWAY_HEADERS env var) Default: '{}' ``` ```default --GatewayClient.http_pwd= The password for HTTP authentication. (JUPYTER_GATEWAY_HTTP_PWD env var) Default: None ``` ```default --GatewayClient.http_user= The username for HTTP authentication. (JUPYTER_GATEWAY_HTTP_USER env var) Default: None ``` ```default --GatewayClient.kernels_endpoint= The gateway API endpoint for accessing kernel resources (JUPYTER_GATEWAY_KERNELS_ENDPOINT env var) Default: '/api/kernels' ``` ```default --GatewayClient.kernelspecs_endpoint= The gateway API endpoint for accessing kernelspecs (JUPYTER_GATEWAY_KERNELSPECS_ENDPOINT env var) Default: '/api/kernelspecs' ``` ```default --GatewayClient.kernelspecs_resource_endpoint= The gateway endpoint for accessing kernelspecs resources (JUPYTER_GATEWAY_KERNELSPECS_RESOURCE_ENDPOINT env var) Default: '/kernelspecs' ``` ```default --GatewayClient.request_timeout= The time allowed for HTTP request completion. (JUPYTER_GATEWAY_REQUEST_TIMEOUT env var) Default: 40.0 ``` ```default --GatewayClient.url= The url of the Kernel or Enterprise Gateway server where kernel specifications are defined and kernel management takes place. If defined, this Notebook server acts as a proxy for all kernel management and kernel specification retrieval. (JUPYTER_GATEWAY_URL env var) Default: None ``` ```default --GatewayClient.validate_cert= For HTTPS requests, determines if server's certificate should be validated or not. (JUPYTER_GATEWAY_VALIDATE_CERT env var) Default: True ``` ```default --GatewayClient.ws_url= The websocket url of the Kernel or Enterprise Gateway server. If not provided, this value will correspond to the value of the Gateway url with 'ws' in place of 'http'. (JUPYTER_GATEWAY_WS_URL env var) Default: None ``` -------------------------------- ### Python Kernel Launcher Invocation Source: https://github.com/jupyter-server/enterprise_gateway/blob/main/docs/source/developers/kernel-launcher.md This snippet illustrates the Python kernel launcher's process: creating a namespace, instantiating IPKernelApp, and starting the kernel. It handles SparkContext initialization if requested. ```python # Create a namespace instance namespace = Namespace() if spark_context_initialization_mode == "exactly-once" or spark_context_initialization_mode == "per-connection": namespace.spark_context = SparkContext.getOrCreate(SparkConf(**spark_conf)) # Instantiate and start the IPKernelApp app = IPKernelApp.instance(namespace=namespace) app.initialize() app.start() ``` -------------------------------- ### Install pycryptodomex with conda Source: https://github.com/jupyter-server/enterprise_gateway/blob/main/docs/source/other/troubleshooting.md Use this command to install or upgrade pycryptodomex to version 3.9.7 or later when using conda. This is required to resolve 'TypeError: Incorrect padding' issues. ```bash [jdoe@node1 ~]$ conda install pycryptodomex ``` -------------------------------- ### Example of KERNEL_EXTRA_SPARK_OPTS usage Source: https://github.com/jupyter-server/enterprise_gateway/blob/main/docs/source/users/kernel-envs.md Demonstrates how to set the KERNEL_EXTRA_SPARK_OPTS environment variable when running a Docker container to pass additional Spark options. Ensure appropriate quoting for space-separated values. ```bash docker run ... -e KERNEL_EXTRA_SPARK_OPTS=\"--conf spark.driver.memory=2g \ --conf spark.executor.memory=2g\" ... jupyterhub/k8s-singleuser-sample ``` -------------------------------- ### Install pycryptodomex with pip Source: https://github.com/jupyter-server/enterprise_gateway/blob/main/docs/source/other/troubleshooting.md Use this command to install or upgrade pycryptodomex to version 3.9.7 or later when using pip. This is required to resolve 'TypeError: Incorrect padding' issues. ```bash [jdoe@node1 ~]$ pip uninstall pycryptodomex [jdoe@node1 ~]$ pip install pycryptodomex ``` -------------------------------- ### Run Integration Tests Source: https://github.com/jupyter-server/enterprise_gateway/blob/main/docs/source/contributors/devinstall.md Execute the integration test suite. This bootstraps a Docker image with Spark and YARN, then tests kernels in various modes. ```bash make itest-yarn ``` -------------------------------- ### Run YARN Master using enterprise-gateway-demo Source: https://github.com/jupyter-server/enterprise_gateway/blob/main/etc/docker/enterprise-gateway-demo/README.md Invoke the elyra/enterprise-gateway-demo image to act as the YARN master. This is an alternative to using elyra/demo-base for the YARN master role in a dual-mode setup. ```bash docker run -d --rm -h yarnmaster --name yarnmaster -p 8088:8088 -p 8042:8042 --net jeg elyra/enterprise-gateway-demo --yarn ``` -------------------------------- ### Generate Enterprise Gateway Configuration File Source: https://github.com/jupyter-server/enterprise_gateway/blob/main/docs/source/operators/config-file.md Run this command to create a template configuration file for the Enterprise Gateway. This file will be located in your `$HOME/.jupyter` directory and contains commented-out options that can be enabled by uncommenting and setting their values. ```bash jupyter enterprisegateway --generate-config ``` -------------------------------- ### Minimal Enterprise Gateway Launch Source: https://github.com/jupyter-server/enterprise_gateway/blob/main/docs/source/operators/launching-eg.md This command starts Enterprise Gateway, exposing it on the public network and ensuring only a single instance is launched. ```bash jupyter enterprisegateway --ip=0.0.0.0 --port_retries=0 ``` -------------------------------- ### Install Enterprise Gateway using pip Source: https://github.com/jupyter-server/enterprise_gateway/blob/main/docs/source/operators/installing-eg.md Use this command to install or upgrade Jupyter Enterprise Gateway from PyPI. Ensure you are using a compatible Python environment due to dependency constraints. ```bash pip install --upgrade jupyter_enterprise_gateway ``` -------------------------------- ### Run Enterprise Gateway Server Source: https://github.com/jupyter-server/enterprise_gateway/blob/main/docs/source/contributors/devinstall.md Starts an instance of the Enterprise Gateway server in jupyter_websocket mode. The server URL will be printed to the console. ```bash make run-dev ``` -------------------------------- ### Kernel Discovery Response Example Source: https://github.com/jupyter-server/enterprise_gateway/blob/main/docs/source/developers/rest-api.md This JSON response details the available kernel specifications, including the default kernel, its arguments, display name, language, and associated resources like logos. ```json { "default": "python3", "kernelspecs": { "python3": { "name": "python3", "spec": { "argv": [ "/usr/bin/env", "/opt/anaconda2/envs/py3/bin/python", "-m", "ipykernel_launcher", "-f", "{connection_file}" ], "display_name": "Python 3", "language": "python", "interrupt_mode": "signal", "metadata": {} }, "resources": { "logo-32x32": "/kernelspecs/python3/logo-32x32.png", "logo-64x64": "/kernelspecs/python3/logo-64x64.png" } }, "ir": { "name": "ir", "spec": { "argv": [ "R", "--slave", "-e", "IRkernel::main()", "--args", "{connection_file}" ], "env": {}, "display_name": "R", "language": "R", "interrupt_mode": "signal", "metadata": {} }, "resources": { "kernel.js": "/kernelspecs/ir/kernel.js", "logo-64x64": "/kernelspecs/ir/logo-64x64.png" } }, "spark_r_yarn_client": { "name": "spark_r_yarn_client", "spec": { "argv": [ "/usr/local/share/jupyter/kernels/spark_R_yarn_client/bin/run.sh", "--RemoteProcessProxy.kernel-id", "{kernel_id}", "--RemoteProcessProxy.response-address", "{response_address}", "--RemoteProcessProxy.public-key", "{public_key}", "--RemoteProcessProxy.port-range", "{port_range}", "--RemoteProcessProxy.spark-context-initialization-mode", "lazy" ], "env": { "SPARK_HOME": "/usr/hdp/current/spark2-client", "SPARK_OPTS": "--master yarn --deploy-mode client --name ${KERNEL_ID:-ERROR__NO__KERNEL_ID} --conf spark.sparkr.r.command=/opt/conda/lib/R/bin/Rscript ${KERNEL_EXTRA_SPARK_OPTS}", "LAUNCH_OPTS": "" }, "display_name": "Spark - R (YARN Client Mode)", "language": "R", "interrupt_mode": "signal", "metadata": { "process_proxy": { "class_name": "enterprise_gateway.services.processproxies.distributed.DistributedProcessProxy" } } }, "resources": { "kernel.js": "/kernelspecs/spark_r_yarn_client/kernel.js", "logo-64x64": "/kernelspecs/spark_r_yarn_client/logo-64x64.png" } }, "spark_r_yarn_cluster": { "name": "spark_r_yarn_cluster", "spec": { "argv": [ "/usr/local/share/jupyter/kernels/spark_R_yarn_cluster/bin/run.sh", "--RemoteProcessProxy.kernel-id", "{kernel_id}", "--RemoteProcessProxy.response-address", "{response_address}", "--RemoteProcessProxy.public-key", "{public_key}", "--RemoteProcessProxy.port-range", "{port_range}", "--RemoteProcessProxy.spark-context-initialization-mode", "eager" ], "env": { "SPARK_HOME": "/usr/hdp/current/spark2-client", "SPARK_OPTS": "--master yarn --deploy-mode cluster --name ${KERNEL_ID:-ERROR__NO__KERNEL_ID} --conf spark.yarn.submit.waitAppCompletion=false --conf spark.yarn.am.waitTime=1d --conf spark.yarn.appMasterEnv.PATH=/opt/conda/bin:$PATH --conf spark.sparkr.r.command=/opt/conda/lib/R/bin/Rscript ${KERNEL_EXTRA_SPARK_OPTS}", "LAUNCH_OPTS": "" }, "display_name": "Spark - R (YARN Cluster Mode)", "language": "R", "interrupt_mode": "signal", "metadata": { "process_proxy": { "class_name": "enterprise_gateway.services.processproxies.yarn.YarnClusterProcessProxy" } } }, "resources": { "kernel.js": "/kernelspecs/spark_r_yarn_cluster/kernel.js", "logo-64x64": "/kernelspecs/spark_r_yarn_cluster/logo-64x64.png" } }, "spark_python_yarn_client": { ``` -------------------------------- ### Install Enterprise Gateway using conda Source: https://github.com/jupyter-server/enterprise_gateway/blob/main/docs/source/operators/installing-eg.md Use this command to install or upgrade Jupyter Enterprise Gateway from conda-forge. This method is recommended for users who prefer conda environments and need to manage dependencies carefully. ```bash conda install -c conda-forge jupyter_enterprise_gateway ``` -------------------------------- ### Run Enterprise Gateway Host in Dual Mode Source: https://github.com/jupyter-server/enterprise_gateway/blob/main/etc/docker/enterprise-gateway-demo/README.md Invoke the elyra/enterprise-gateway-demo image as a dedicated Enterprise Gateway host in a dual-mode setup. It connects to a specified YARN master. ```bash docker run -it --rm -h elyra --name elyra -p 8888:8888 --net jeg -e YARN_HOST=yarnmaster elyra/enterprise-gateway-demo --elyra ``` -------------------------------- ### Process Proxy Launch Process Method Source: https://github.com/jupyter-server/enterprise_gateway/blob/main/docs/source/contributors/system-architecture.md The primary method for launching a kernel process. It takes the kernel command and optional keyword arguments, including environment variables. The process must be running before returning. ```python @abstractmethod def launch_process(self, kernel_cmd, *kw): pass ``` -------------------------------- ### Run Combined YARN/Enterprise Gateway Mode Source: https://github.com/jupyter-server/enterprise_gateway/blob/main/etc/docker/enterprise-gateway-demo/README.md Run the enterprise-gateway-demo image as a combined YARN and Enterprise Gateway instance. This mode runs kernels locally in YARN cluster mode. ```bash docker run -itd --rm -p 8888:8888 -p 8088:8088 -p 8042:8042 --net=jeg elyra/enterprise-gateway-demo --elyra ``` -------------------------------- ### GET /content-type Source: https://github.com/jupyter-server/enterprise_gateway/blob/main/enterprise_gateway/tests/resources/kernel_api3.ipynb Retrieves the Content-Type header from the request. ```APIDOC ## GET /content-type ### Description Retrieves the Content-Type header from the request. ### Method GET ### Endpoint /content-type ### Response #### Success Response (200) - **content_type** (string) - The value of the Content-Type header. ### Response Example { "content_type": "application/json" } ``` -------------------------------- ### GET /people Source: https://github.com/jupyter-server/enterprise_gateway/blob/main/enterprise_gateway/tests/resources/kernel_api3.ipynb Retrieves the current list of people. ```APIDOC ## GET /people ### Description Retrieves the current list of people. ### Method GET ### Endpoint /people ### Response #### Success Response (200) - **people** (array) - A list of names. ### Response Example { "people": ["Corey", "Nitin", "Pete"] } ``` -------------------------------- ### GET /message Source: https://github.com/jupyter-server/enterprise_gateway/blob/main/enterprise_gateway/tests/resources/kernel_api3.ipynb Retrieves the current global message. ```APIDOC ## GET /message ### Description Retrieves the current global message. ### Method GET ### Endpoint /message ### Response #### Success Response (200) - **message** (string) - The current message. ### Response Example { "message": "hello world" } ``` -------------------------------- ### Prepare JSON request for /hello/person Source: https://github.com/jupyter-server/enterprise_gateway/blob/main/enterprise_gateway/tests/resources/kernel_api2.ipynb Constructs a JSON string representing arguments for an API request. ```python REQUEST = json.dumps({ 'args': { 'person' : people } }) ``` -------------------------------- ### Spark-on-Kubernetes Kernel Launch Arguments Source: https://github.com/jupyter-server/enterprise_gateway/blob/main/docs/source/operators/deploy-kubernetes.md Define the 'argv' stanza for launching Spark kernels in Kubernetes. This invokes a shell script that contains the 'spark-submit' invocation, using 'SPARK_OPTS' as a primary parameter and setting the Spark context initialization mode. ```json { "argv": [ "/usr/local/share/jupyter/kernels/spark_python_kubernetes/bin/run.sh", "--RemoteProcessProxy.kernel-id", "{kernel_id}", "--RemoteProcessProxy.response-address", "{response_address}", "--RemoteProcessProxy.public-key", "{public_key}", "--RemoteProcessProxy.spark-context-initialization-mode", "lazy" ] } ``` -------------------------------- ### GET /env_kernel_gateway Source: https://github.com/jupyter-server/enterprise_gateway/blob/main/enterprise_gateway/tests/resources/kernel_api3.ipynb Retrieves the value of the KERNEL_GATEWAY environment variable. ```APIDOC ## GET /env_kernel_gateway ### Description Retrieves the value of the KERNEL_GATEWAY environment variable. ### Method GET ### Endpoint /env_kernel_gateway ### Response #### Success Response (200) - **kernel_gateway_env** (string) - The value of the KERNEL_GATEWAY environment variable. ### Response Example { "kernel_gateway_env": "some_value" } ``` -------------------------------- ### Configure Non-Root User and Kernel Launcher Source: https://github.com/jupyter-server/enterprise_gateway/blob/main/docs/source/developers/custom-images.md Sets up a non-root user (jovyan) with specific UID/GID and configures permissions for the kernel launcher scripts. This is crucial for security and proper execution. ```dockerfile RUN adduser --system --uid 1000 --gid 100 jovyan && \ chown jovyan:users /usr/local/bin/bootstrap-kernel.sh && \ chmod 0755 /usr/local/bin/bootstrap-kernel.sh && \ chown -R jovyan:users /usr/local/bin/kernel-launchers ``` -------------------------------- ### GET /sleep/:time Source: https://github.com/jupyter-server/enterprise_gateway/blob/main/enterprise_gateway/tests/resources/kernel_api3.ipynb Pauses execution for a specified duration. ```APIDOC ## GET /sleep/:time ### Description Pauses execution for a specified duration. ### Method GET ### Endpoint /sleep/:time ### Parameters #### Path Parameters - **time** (integer) - Required - The duration in seconds to sleep. ### Response #### Success Response (200) - **message** (string) - Confirmation of sleep duration. ### Response Example { "message": "Slept for 1 seconds" } ```