### Install Panda-Client Locally Source: https://github.com/pandawms/panda-client/blob/master/README.md Install the Panda-Client using the setup.py script to a specified directory. ```bash python setup.py install --prefix=/path/to/install/dir ``` -------------------------------- ### Install panda-client from Source Source: https://context7.com/pandawms/panda-client/llms.txt Installs the panda-client from its source code repository. This involves cloning the repository, installing it, and setting up the environment. ```bash git clone https://github.com/PanDAWMS/panda-client.git cd panda-client python setup.py install --prefix=/path/to/install/dir source /path/to/install/dir/etc/panda/panda_setup.sh prun -h pathena -h pbook -h ``` -------------------------------- ### Setup PanDA Client in Jupyter Notebook Source: https://context7.com/pandawms/panda-client/llms.txt Run the setup script and import magic commands for using PanDA Client within a Jupyter notebook environment. Ensure the setup script is executed first. ```python # In a Jupyter notebook cell — run setup script first %run panda_setup.py # Then import magic commands from pandaclient import panda_jupyter panda_jupyter.setup() ``` -------------------------------- ### Install panda-client from PyPI Source: https://context7.com/pandawms/panda-client/llms.txt Installs the panda-client package using pip. This is the recommended method for most users. ```bash pip install panda-client ``` -------------------------------- ### Create and Install Panda-Client Tarball Source: https://github.com/pandawms/panda-client/blob/master/README.md Generate a source distribution tarball and install it using pip. ```bash python setup.py sdist pip install dist/panda-client-*.tar.gz ``` -------------------------------- ### Set Up Panda-Client Environment Source: https://github.com/pandawms/panda-client/blob/master/README.md Source the setup script to configure the environment for using Panda-Client tools. This is necessary before running commands like 'prun' or 'pathena'. ```bash source /path/to/install/dir/etc/panda/panda_setup.[c]sh ``` -------------------------------- ### Run Panda Setup Script in Jupyter Source: https://github.com/pandawms/panda-client/wiki/jupyter-tutorial.ipynb Execute the panda_setup.py script to initialize the panda-client API for use in your notebook. This is a prerequisite for interacting with the API. ```python %run panda_setup.py ``` -------------------------------- ### Launch and Use pbook CLI Source: https://context7.com/pandawms/panda-client/llms.txt Launches the interactive pbook session for monitoring and managing tasks. Shows example commands for listing, killing, retrying, finishing, pausing, and resuming tasks. ```bash # Launch interactive pbook session pbook # In the pbook session: # show() -- list recent tasks # show("run") -- list running tasks # show(98765) -- show specific task # kill(98765) -- kill a task # retry(98765) -- retry a failed task # finish(98765, soft=True)-- soft-finish a task # pause(98765) -- pause a task ``` -------------------------------- ### Submit Generic Jobs with prun CLI Source: https://context7.com/pandawms/panda-client/llms.txt Command-line tool for submitting arbitrary executables. Examples show basic echo job, job with build step and output collection, loading options from JSON, and dry run. ```bash # Submit a simple echo job over an input dataset prun \ --exec "echo %IN" \ --inDS user.jdoe.inputAOD \ --outDS user.jdoe.output.v1 \ --nFilesPerJob 1 \ --nJobs 10 ``` ```bash # Submit with a build step and output collection prun \ --exec "myanalysis %IN --output %OUT" \ --bexec "make -j4" \ --athenaTag 22.0.0 \ --inDS mc16_13TeV.364156.Sherpa_input \ --outDS user.jdoe.mc16.analysis.v3 \ --outputs "*.root" \ --nFilesPerJob 5 ``` ```bash # Load all options from a JSON config file prun --loadJson prunConfig.json ``` ```bash # Dry run (no submission) prun --exec "echo test" --outDS user.jdoe.test --dryRun ``` -------------------------------- ### Submit ATLAS Athena Jobs with pathena CLI Source: https://context7.com/pandawms/panda-client/llms.txt Command-line tool for submitting ATLAS Athena framework jobs. Examples cover basic job submission and jobs with configuration options. ```bash # Basic Athena job pathena \ --trf "Reco_trf.py inputAODFile=%IN outputNTUP_PHYSFile=%OUT.NTUP.root" \ --inDS data18_13TeV.00360026.physics_Main.merge.AOD \ --outDS user.jdoe.data18.Ntuple.v1 \ --nFilesPerJob 1 \ --athenaTag Athena,22.0.47 ``` ```bash # With configuration job options pathena \ JetRecConfig.py \ --inDS mc16_13TeV.364156.Sherpa_input \ --outDS user.jdoe.jets.v1 \ --nEventsPerJob 5000 ``` -------------------------------- ### Get Job Descriptions with Client.get_job_descriptions Source: https://context7.com/pandawms/panda-client/llms.txt Fetches job description dictionaries for all jobs or only unsuccessful ones within a task. Useful for debugging failed jobs. ```python from pandaclient import Client jedi_task_id = 98765 # Get all job descriptions status, descriptions = Client.get_job_descriptions(jedi_task_id, unsuccessful_only=False) # Get only failed/cancelled/closed jobs status, failed_desc = Client.get_job_descriptions(jedi_task_id, unsuccessful_only=True) if status == 0: for desc in (failed_desc or []): print("PandaID={}, jobStatus={}, errorDiag={}".format( desc.get("PandaID"), desc.get("jobStatus"), desc.get("jobSubStatus") )) ``` -------------------------------- ### Get Job Metadata with Client.getUserJobMetadata Source: https://context7.com/pandawms/panda-client/llms.txt Returns a list of metadata dictionaries for all analysis jobs in a task. Requires `json` for pretty printing. ```python from pandaclient import Client import json jedi_task_id = 98765 status, metadata_list = Client.getUserJobMetadata(jedi_task_id, verbose=False) if status == 0: print("Got metadata for {} jobs".format(len(metadata_list))) for meta in metadata_list[:2]: print(json.dumps(meta, indent=2, default=str)) # Output: # Got metadata for 42 jobs # {"PandaID": 7654300, "nEvents": 1000, "outputFile": "AOD.pool.root", ...} ``` -------------------------------- ### Get Files in Datasets Source: https://context7.com/pandawms/panda-client/llms.txt Retrieves information about input and pseudo-input files within specified datasets for a given task. Useful for inspecting task inputs before or during execution. ```python status, datasets = Client.get_files_in_datasets( jedi_task_id, dataset_types="input,pseudo_input", verbose=False, ) if status == 0: for ds in datasets: print("Dataset: {}, files: {}".format(ds.get("datasetName"), len(ds.get("files", [])))) for f in ds.get("files", [])[:3]: print(" - {}".format(f.get("lfn"))) ``` -------------------------------- ### Query Full Job Status with Client.getFullJobStatus Source: https://context7.com/pandawms/panda-client/llms.txt Get detailed job specifications for both active and archived jobs. This is useful for post-mortem analysis of job execution. ```python from pandaclient import Client status, job_specs = Client.getFullJobStatus([7654321], verbose=False) if status == 0 and job_specs: job = job_specs[0] print("Job {} final state: {}".format(job.PandaID, job.jobStatus)) print("Pilot error code:", job.pilotErrorCode) print("Transformation:", job.transformation) print("Output files:", job.Files) # Output: # Job 7654321 final state: finished # Pilot error code: 0 # Transformation: Reco_trf.py ``` -------------------------------- ### Get Certificate Attributes with Client.get_cert_attributes Source: https://context7.com/pandawms/panda-client/llms.txt Retrieves VOMS/grid certificate attributes like DN and FQAN from the PanDA server. Ensure the client has access to valid certificate information. ```python from pandaclient import Client status, attrs = Client.get_cert_attributes(verbose=False) if status == 0: for k, v in attrs.items(): print("{}: {}".format(k, v)) ``` -------------------------------- ### Reload Input and Retry Task Source: https://context7.com/pandawms/panda-client/llms.txt Instructs the JEDI system to re-scan input datasets for a task and retry execution. This is useful if input files were added or recovered after the task initially started. ```python from pandaclient import Client task_id = 98765 status, (return_code, message) = Client.reload_input(task_id, verbose=True) if status == 0 and return_code == 0: print("Input reload command registered:", message) # Output: Input reload command registered: command is registered. will be executed in a few minutes ``` -------------------------------- ### Get iDDS Proxy API Object with idds_api.get_api Source: https://context7.com/pandawms/panda-client/llms.txt Obtains a dynamic proxy API object for iDDS that routes attribute access as iDDS command calls through PanDA. Allows calling iDDS commands as methods on the returned object. ```python from pandaclient import idds_api # Get a configured iDDS API handle idds = idds_api.get_api( verbose=False, compress=True, manager=True, json_outputs=True, ) # Call any iDDS command as a method status, result = idds.get_workflow_status(request_id=12345) if status == 0: print("Workflow status:", result) status, result = idds.abort_workflow(request_id=12345) if status == 0: print("Abort result:", result) ``` -------------------------------- ### Get Task Status String with Client.getTaskStatus Source: https://context7.com/pandawms/panda-client/llms.txt Retrieves the current status string for a single JEDI task. Useful for monitoring task progress. ```python from pandaclient import Client jedi_task_id = 98765 status, task_status = Client.getTaskStatus(jedi_task_id, verbose=False) if status == 0: print("Task {} status: {}".format(jedi_task_id, task_status)) # Output: Task 98765 status: running # Possible values: running, done, failed, finished, broken, aborted, paused ``` -------------------------------- ### Get Job IDs for a Task with Client.getPandaIDsWithTaskID Source: https://context7.com/pandawms/panda-client/llms.txt Returns a list of all PanDA job IDs associated with a specific JEDI task. Useful for further job-level analysis. ```python from pandaclient import Client jedi_task_id = 98765 status, panda_ids = Client.getPandaIDsWithTaskID(jedi_task_id, verbose=False) if status == 0: print("Task {} contains {} jobs: {}".format(jedi_task_id, len(panda_ids), panda_ids[:5])) # Output: Task 98765 contains 42 jobs: [7654300, 7654301, 7654302, 7654303, 7654304] ``` -------------------------------- ### Panda-Client Command-Line Tools Source: https://github.com/pandawms/panda-client/blob/master/README.md Basic usage of the 'prun' and 'pathena' command-line tools. Use the -h flag for help. ```bash prun -h pathena -h ``` -------------------------------- ### Initialize PBookCore and Manage Tasks Source: https://context7.com/pandawms/panda-client/llms.txt Initializes PBookCore for task bookkeeping and demonstrates various management operations including showing tasks, killing, finishing, retrying, pausing, resuming, and recovering files. Also covers secrets management and credential generation. ```python from pandaclient.PBookCore import PBookCore pbook = PBookCore(verbose=False) pbook.init() # Validates proxy/token and sets self.username # Show tasks (prints a formatted table) pbook.show(days=7, limit=100, format="standard") pbook.show(days=7, format="long") pbook.show([98765, 98766], format="plain") pbook.show("run") # Only running tasks pbook.show("fin") # Only finished tasks # Task control pbook.kill(98765) pbook.finish(98765, soft=True) pbook.retry(98765, newOpts={"nFilesPerJob": 2}) pbook.killAndRetry(98765) pbook.pause(98765) pbook.resume(98765) pbook.reload_input(98765) # File recovery pbook.recover_lost_files(98765, test_mode=True) # Secrets management pbook.set_secret("MY_TOKEN", "abc123") pbook.set_secret("MY_CERT", "/path/to/cert.pem", is_file=True) pbook.list_secrets(full=False) # Truncates long values # Get job metadata as JSON pbook.getUserJobMetadata(98765, "/tmp/task_metadata.json") # Regenerate credential (OIDC token or grid proxy) pbook.generate_credential() ``` -------------------------------- ### Client.hello Source: https://context7.com/pandawms/panda-client/llms.txt Verifies connectivity to the PanDA server. Returns status 0 and a message string on success, or status 255 on failure. ```APIDOC ## Client.hello — Health Check Verifies connectivity to the PanDA server. Returns status 0 and a message string on success, or status 255 on failure. ```python from pandaclient import Client status, message = Client.hello(verbose=True) if status == 0: print("PanDA server is alive:", message) # Output: PanDA server is alive: OK else: print("Connection failed:", message) # Output: Connection failed: Failed to connect to host. ``` ``` -------------------------------- ### Configure X.509 Grid Proxy Authentication Source: https://context7.com/pandawms/panda-client/llms.txt Sets environment variables for classic X.509 grid proxy authentication. Specifies the location of the user proxy and certificate directory. ```bash # Authentication: X.509 grid proxy (classic) export X509_USER_PROXY="/tmp/x509up_u$(id -u)" export X509_CERT_DIR="/etc/grid-security/certificates" ``` -------------------------------- ### Configure Miscellaneous Client Settings Source: https://context7.com/pandawms/panda-client/llms.txt Sets miscellaneous environment variables for client behavior, such as disabling TLS host verification or specifying configuration directories. ```bash # Miscellaneous export PANDA_VERIFY_HOST="off" # Disable TLS host verification (dev only) export PANDA_CONFIG_ROOT="$HOME/.pathena" # Config and history root directory export PANDA_USE_NATIVE_HTTPLIB=1 # Use Python urllib instead of curl ``` -------------------------------- ### Get Tasks Modified in Time Range with Client.getJobIDsJediTasksInTimeRange Source: https://context7.com/pandawms/panda-client/llms.txt Retrieves task IDs and metadata for tasks modified within a specified time window. Requires `datetime` and `timedelta` for time calculations. ```python from pandaclient import Client from datetime import datetime, timedelta # Get tasks modified in the last 24 hours since = datetime.utcnow() - timedelta(days=1) since_str = since.strftime("%Y-%m-%dT%H:%M:%S") status, tasks = Client.getJobIDsJediTasksInTimeRange( since_str, dn=None, # None = own tasks; or pass a DN string minTaskID=None, verbose=False, task_type="user", # "user" or "prod" ) if status == 0: for task in tasks: print("TaskID={}, status={}".format(task.get("jediTaskID"), task.get("status"))) ``` -------------------------------- ### prun CLI - Submit Generic Jobs Source: https://context7.com/pandawms/panda-client/llms.txt Command-line tool for submitting arbitrary executables to PanDA with automatic sandbox packaging. ```APIDOC ## prun ### Description Submits generic jobs to PanDA. It can package executables and their dependencies into a sandbox for execution. ### Usage ```bash prun [OPTIONS] --exec --inDS --outDS ``` ### Options - **--exec** (str) - Required. The command to execute. - **--inDS** (str) - Required. The input dataset name. - **--outDS** (str) - Required. The output dataset name. - **--nFilesPerJob** (int) - Optional. Number of files per job. - **--nJobs** (int) - Optional. Number of jobs to submit. - **--bexec** (str) - Optional. A command to execute before the main command, typically for building. - **--athenaTag** (str) - Optional. Specifies the Athena software tag. - **--outputs** (str) - Optional. Specifies output file patterns. - **--loadJson** (str) - Optional. Path to a JSON file containing prun options. - **--dryRun** - Optional. Performs a dry run without actual submission. ### Examples ```bash # Submit a simple echo job prun \ --exec "echo %IN" \ --inDS user.jdoe.inputAOD \ --outDS user.jdoe.output.v1 \ --nFilesPerJob 1 \ --nJobs 10 # Submit with build step and output collection prun \ --exec "myanalysis %IN --output %OUT" \ --bexec "make -j4" \ --athenaTag 22.0.0 \ --inDS mc16_13TeV.364156.Sherpa_input \ --outDS user.jdoe.mc16.analysis.v3 \ --outputs "*.root" \ --nFilesPerJob 5 # Load options from a JSON config file prun --loadJson prunConfig.json # Dry run prun --exec "echo test" --outDS user.jdoe.test --dryRun ``` ``` -------------------------------- ### pbook CLI - Interactive Task Manager Source: https://context7.com/pandawms/panda-client/llms.txt Interactive shell for monitoring and managing submitted tasks. Supports all PBookCore operations as commands. ```APIDOC ## pbook CLI ### Description An interactive command-line interface for managing PanDA tasks. It provides a shell environment where `PBookCore` methods can be invoked as commands. ### Usage ```bash pbook ``` ### Commands (within the pbook session) - **show()**: Lists recent tasks. - **show("run")**: Lists only running tasks. - **show(task_id)**: Displays details for a specific task. - **kill(task_id)**: Kills a specified task. - **finish(task_id, soft=True)**: Marks a task as finished (softly if `soft` is True). - **retry(task_id)**: Retries a failed task. - **pause(task_id)**: Pauses a running task. - **resume(task_id)**: Resumes a paused task. ### Example ```bash # Launch interactive pbook session pbook # Inside the pbook session: # > show() # > kill(98765) # > retry(98765) ``` ``` -------------------------------- ### Submit Hyperparameter Optimization with phpo CLI Source: https://context7.com/pandawms/panda-client/llms.txt Command-line tool for submitting hyperparameter optimization jobs using iDDS/PanDA integration. Requires specifying output and training datasets, configuration file, and optimization parameters. ```bash # Submit a hyperparameter optimization task phpo \ --outDS user.jdoe.hpo.v1 \ --trainingDS user.jdoe.trainingData \ --config hpo_config.json \ --nParallelEvaluation 4 \ --minUnevaluatedPoints 2 \ --nPointsPerIteration 4 ``` -------------------------------- ### Upload Panda-Client to Pip Source: https://github.com/pandawms/panda-client/blob/master/README.md Generate a source distribution and upload it to the Python Package Index (PyPI). This is typically done for releasing new versions. ```bash python setup.py sdist upload ``` -------------------------------- ### Initialize PandaAPI and Perform Operations Source: https://context7.com/pandawms/panda-client/llms.txt Initializes the PandaAPI and demonstrates common operations like health checks, submitting prun jobs, listing tasks, and managing tasks (kill, retry, increase attempts). ```python from pandaclient import panda_api api = panda_api.get_api() # Health check status, msg = api.hello() print("Server alive:", status == 0, msg) # Submit a task using prun arguments directly ok, result = api.execute_prun([ "--exec", "echo %IN", "--inDS", "user.jdoe.inputAOD", "--outDS", "user.jdoe.output.v1", "--nFilesPerJob", "1", ]) print("prun submitted:", ok, result.get("jediTaskID")) # List recent tasks (last 7 days, up to 50) tasks = api.get_tasks(limit=50, days=7, status="running") for t in tasks: print("Task {}: {}".format(t.get("jediTaskID"), t.get("status"))) # Kill a task status, (rc, msg) = api.kill_task(98765) print("Kill:", rc, msg) # Retry a task with new parameters status, (rc, msg) = api.retry_task(98765, new_parameters={"site": "CERN-PROD"}) print("Retry:", rc, msg) # Increase attempt numbers status, (rc, msg) = api.increase_attempt_nr(98765, increase=3) print("Increase attempts:", rc, msg) # Get job metadata and save to file api.get_job_metadata(task_id=98765, output_json_filename="/tmp/metadata.json") ``` -------------------------------- ### List Input Files with Client.get_files_in_datasets Source: https://context7.com/pandawms/panda-client/llms.txt Lists files present in the input datasets of a specified task. Can optionally include pseudo-input datasets. ```python from pandaclient import Client jedi_task_id = 98765 ``` -------------------------------- ### Configure OIDC Authentication Source: https://context7.com/pandawms/panda-client/llms.txt Sets environment variables for OIDC token-based authentication. Requires specifying the virtual organization and the ID token or a token file. ```bash # Authentication: OIDC token-based (modern) export PANDA_AUTH="oidc" export PANDA_AUTH_VO="atlas" # Virtual organization export PANDA_AUTH_ID_TOKEN="" # Or use token file: export OIDC_AUTH_TOKEN_FILE="$HOME/.token" ``` -------------------------------- ### Clone Panda-Client Repository Source: https://github.com/pandawms/panda-client/blob/master/README.md Use git to clone the Panda-Client repository from GitHub. ```bash git clone https://github.com/PanDAWMS/panda-client.git cd panda-client ``` -------------------------------- ### Configure PanDA Server URLs Source: https://context7.com/pandawms/panda-client/llms.txt Sets environment variables to specify the URLs for the PanDA server and cache. These can override the default CERN URLs. ```bash # Core server URLs (override defaults: pandaserver.cern.ch) export PANDA_URL="http://pandaserver.cern.ch:25080/server/panda" export PANDA_URL_SSL="https://pandaserver.cern.ch/server/panda" export PANDACACHE_URL="https://pandacache.cern.ch/server/panda" ``` -------------------------------- ### PandaAPI - High-Level Python API Source: https://context7.com/pandawms/panda-client/llms.txt Provides a simplified OOP wrapper around Client and PBookCore for common operations, ideal for scripted pipelines. ```APIDOC ## panda_api.get_api() ### Description Instantiates and returns a high-level Python API object for interacting with PanDA. ### Method ```python api = panda_api.get_api() ``` ## api.hello() ### Description Performs a health check to verify server connectivity. ### Method ```python status, msg = api.hello() ``` ### Response #### Success Response (0) - **status** (int) - 0 indicates success. - **msg** (str) - A message indicating the server is alive. ## api.execute_prun(prun_arguments) ### Description Submits a task to PanDA using `prun` arguments directly. ### Method ```python ok, result = api.execute_prun([ "--exec", "echo %IN", "--inDS", "user.jdoe.inputAOD", "--outDS", "user.jdoe.output.v1", "--nFilesPerJob", "1", ]) ``` ### Parameters #### Request Body - **prun_arguments** (list[str]) - A list of strings representing `prun` command-line arguments. ### Response #### Success Response (True) - **ok** (bool) - True if the submission was successful. - **result** (dict) - A dictionary containing task information, including 'jediTaskID'. ## api.get_tasks(limit, days, status) ### Description Retrieves a list of recent tasks based on specified criteria. ### Method ```python tasks = api.get_tasks(limit=50, days=7, status="running") ``` ### Parameters #### Query Parameters - **limit** (int) - Optional. The maximum number of tasks to return. - **days** (int) - Optional. The number of past days to search for tasks. - **status** (str) - Optional. Filters tasks by their status (e.g., 'running', 'finished'). ### Response #### Success Response - **tasks** (list[dict]) - A list of dictionaries, where each dictionary represents a task. ## api.kill_task(task_id) ### Description Attempts to kill a specified PanDA task. ### Method ```python status, (rc, msg) = api.kill_task(98765) ``` ### Parameters #### Path Parameters - **task_id** (int) - The unique identifier of the task to kill. ### Response #### Success Response - **status** (int) - Indicates the status of the kill operation. - **rc** (int) - Return code of the operation. - **msg** (str) - A message describing the result of the kill operation. ## api.retry_task(task_id, new_parameters) ### Description Retries a specified PanDA task with optional new parameters. ### Method ```python status, (rc, msg) = api.retry_task(98765, new_parameters={"site": "CERN-PROD"}) ``` ### Parameters #### Path Parameters - **task_id** (int) - The unique identifier of the task to retry. #### Request Body - **new_parameters** (dict) - Optional. A dictionary of new parameters to apply for the retry. ### Response #### Success Response - **status** (int) - Indicates the status of the retry operation. - **rc** (int) - Return code of the operation. - **msg** (str) - A message describing the result of the retry operation. ## api.increase_attempt_nr(task_id, increase) ### Description Increases the attempt number for a specified PanDA task. ### Method ```python status, (rc, msg) = api.increase_attempt_nr(98765, increase=3) ``` ### Parameters #### Path Parameters - **task_id** (int) - The unique identifier of the task. #### Query Parameters - **increase** (int) - The number of attempts to increase by. ### Response #### Success Response - **status** (int) - Indicates the status of the operation. - **rc** (int) - Return code of the operation. - **msg** (str) - A message describing the result of the operation. ## api.get_job_metadata(task_id, output_json_filename) ### Description Retrieves job metadata for a given task and saves it to a JSON file. ### Method ```python api.get_job_metadata(task_id=98765, output_json_filename="/tmp/metadata.json") ``` ### Parameters #### Query Parameters - **task_id** (int) - The unique identifier of the task. - **output_json_filename** (str) - The path to the output JSON file. ``` -------------------------------- ### Client.putFile Source: https://context7.com/pandawms/panda-client/llms.txt Uploads a local file to the PanDA cache server. Files named `sources.*` can be up to 768 MB; other files up to 10 MB. It can reuse an identical previous upload if possible. ```APIDOC ## Client.putFile — Upload a Sandbox File ### Description Uploads a local file to the PanDA cache server. Files named `sources.*` can be up to 768 MB; other files up to 10 MB. ### Method POST (implied) ### Endpoint /cache/upload (implied) ### Parameters #### Path Parameters None #### Query Parameters - **verbose** (boolean) - Optional - If True, prints verbose output. - **useCacheSrv** (boolean) - Optional - If True, uses the cache server. - **reuseSandbox** (boolean) - Optional - If True, reuses an identical previous upload if possible. ### Request Body - **file** (file) - Required - The file to upload. ### Request Example ```python from pandaclient import Client import tarfile, os # Create a sandbox archive of your working directory sandbox_name = "sources.{}.tar.gz".format(os.getpid()) with tarfile.open(sandbox_name, "w:gz") as tar: tar.add("MyAnalysis/", arcname="MyAnalysis") # Upload, reusing an identical previous upload if possible status, message = Client.putFile( sandbox_name, verbose=False, useCacheSrv=True, reuseSandbox=True, ) if status == 0: if message.startswith("NewFileName:"): reused_name = message.split(":", 1)[1] print("Reusing existing sandbox:", reused_name) else: print("Upload succeeded:", message) else: print("Upload failed:", message) ``` ### Response #### Success Response (200) - **status** (integer) - 0 for success, non-zero for failure. - **message** (string) - If reusing a sandbox, contains "NewFileName:" followed by the reused name. Otherwise, indicates success or failure. #### Response Example ```json { "status": 0, "message": "NewFileName:sources.12345.tar.gz" } ``` ```json { "status": 0, "message": "True" } ``` ``` -------------------------------- ### Run PanDA Client Command Source: https://context7.com/pandawms/panda-client/llms.txt Execute a PanDA Client command directly from the command line. This is useful for quick checks or simple operations. ```bash pbook --prompt "show(days=3)" ``` -------------------------------- ### pathena CLI - Submit Athena Jobs Source: https://context7.com/pandawms/panda-client/llms.txt Command-line tool for submitting ATLAS Athena framework jobs to PanDA. ```APIDOC ## pathena ### Description Submits ATLAS Athena framework jobs to PanDA. It simplifies the submission of analysis jobs by handling configuration and software environment setup. ### Usage ```bash pathena [OPTIONS] --inDS --outDS ``` ### Parameters - **transformation_script** (str) - The Athena transformation script (e.g., `Reco_trf.py`). ### Options - **--inDS** (str) - Required. The input dataset name. - **--outDS** (str) - Required. The output dataset name. - **--nFilesPerJob** (int) - Optional. Number of files per job. - **--athenaTag** (str) - Optional. Specifies the Athena software tag (e.g., `Athena,22.0.47`). - **--nEventsPerJob** (int) - Optional. Number of events per job. ### Examples ```bash # Basic Athena job pathena \ --trf "Reco_trf.py inputAODFile=%IN outputNTUP_PHYSFile=%OUT.NTUP.root" \ --inDS data18_13TeV.00360026.physics_Main.merge.AOD \ --outDS user.jdoe.data18.Ntuple.v1 \ --nFilesPerJob 1 \ --athenaTag Athena,22.0.47 # With configuration job options pathena \ JetRecConfig.py \ --inDS mc16_13TeV.364156.Sherpa_input \ --outDS user.jdoe.jets.v1 \ --nEventsPerJob 5000 ``` ``` -------------------------------- ### Manage User Secrets with Client Source: https://context7.com/pandawms/panda-client/llms.txt Use Client.get_user_secrets to retrieve all secrets and Client.set_user_secret to delete individual secrets (by setting value to None) or all secrets (by setting key to None). ```python status, (success, secrets) = Client.get_user_secrets(verbose=False) if status == 0 and success: for key, value in secrets.items(): print(" {}: {}...".format(key, value[:20])) ``` ```python status, (success, msg) = Client.set_user_secret("MY_DB_PASSWORD", None, verbose=False) print("Deleted:", success) ``` ```python status, (success, msg) = Client.set_user_secret(None, None, verbose=False) ``` -------------------------------- ### Retrieve Task Parameters with Client.getTaskParamsMap Source: https://context7.com/pandawms/panda-client/llms.txt Fetches the complete task parameter dictionary for a given JEDI task. Requires importing the `json` module for pretty printing. ```python from pandaclient import Client import json jedi_task_id = 98765 status, task_params = Client.getTaskParamsMap(jedi_task_id) if status == 0: print(json.dumps(task_params, indent=2)) # Output: { # "taskName": "user.jdoe.myanalysis.abc123", # "transPath": "Reco_trf.py", # "nFilesPerJob": 1, # ... # } ``` -------------------------------- ### PBookCore - Interactive/Programmatic Task Bookkeeping Source: https://context7.com/pandawms/panda-client/llms.txt Core class for task lifecycle management; used by the `pbook` CLI and embeddable in scripts. ```APIDOC ## PBookCore(verbose) ### Description Initializes the PBookCore class for task lifecycle management. ### Parameters #### Constructor Parameters - **verbose** (bool) - Optional. If True, enables verbose output. ## pbook.init() ### Description Validates the user's proxy/token and sets the username. ## pbook.show(days, limit, format, status_filter) ### Description Displays tasks in a formatted table. Can filter by days, limit, format, and status. ### Method ```python pbook.show(days=7, limit=100, format="standard") pbook.show([98765, 98766], format="plain") pbook.show("run") ``` ### Parameters #### Query Parameters - **days** (int) - Optional. Number of past days to show tasks from. - **limit** (int) - Optional. Maximum number of tasks to display. - **format** (str) - Optional. The display format ('standard', 'long', 'plain'). - **status_filter** (str or list[int]) - Optional. Filters tasks by status (e.g., 'run', 'fin') or by a list of task IDs. ## pbook.kill(task_id) ### Description Kills a specified task. ### Method ```python pbook.kill(98765) ``` ### Parameters #### Path Parameters - **task_id** (int) - The ID of the task to kill. ## pbook.finish(task_id, soft) ### Description Marks a task as finished. ### Method ```python pbook.finish(98765, soft=True) ``` ### Parameters #### Path Parameters - **task_id** (int) - The ID of the task to finish. #### Query Parameters - **soft** (bool) - Optional. If True, performs a soft finish. ## pbook.retry(task_id, newOpts) ### Description Retries a failed task with optional new options. ### Method ```python pbook.retry(98765, newOpts={"nFilesPerJob": 2}) ``` ### Parameters #### Path Parameters - **task_id** (int) - The ID of the task to retry. #### Request Body - **newOpts** (dict) - Optional. Dictionary of new options for the retry. ## pbook.killAndRetry(task_id) ### Description Kills a task and then retries it. ### Method ```python pbook.killAndRetry(98765) ``` ### Parameters #### Path Parameters - **task_id** (int) - The ID of the task to kill and retry. ## pbook.pause(task_id) ### Description Pauses a running task. ### Method ```python pbook.pause(98765) ``` ### Parameters #### Path Parameters - **task_id** (int) - The ID of the task to pause. ## pbook.resume(task_id) ### Description Resumes a paused task. ### Method ```python pbook.resume(98765) ``` ### Parameters #### Path Parameters - **task_id** (int) - The ID of the task to resume. ## pbook.reload_input(task_id) ### Description Reloads the input for a specified task. ### Method ```python pbook.reload_input(98765) ``` ### Parameters #### Path Parameters - **task_id** (int) - The ID of the task to reload input for. ## pbook.recover_lost_files(task_id, test_mode) ### Description Recovers lost files for a specified task. ### Method ```python pbook.recover_lost_files(98765, test_mode=True) ``` ### Parameters #### Path Parameters - **task_id** (int) - The ID of the task. #### Query Parameters - **test_mode** (bool) - Optional. If True, runs in test mode without actual recovery. ## pbook.set_secret(name, value, is_file) ### Description Sets a secret key-value pair or a file path for use in tasks. ### Method ```python pbook.set_secret("MY_TOKEN", "abc123") pbook.set_secret("MY_CERT", "/path/to/cert.pem", is_file=True) ``` ### Parameters #### Query Parameters - **name** (str) - The name of the secret. - **value** (str) - The value of the secret. - **is_file** (bool) - Optional. If True, indicates the value is a file path. ## pbook.list_secrets(full) ### Description Lists all stored secrets. ### Method ```python pbook.list_secrets(full=False) ``` ### Parameters #### Query Parameters - **full** (bool) - Optional. If True, displays full secret values; otherwise, truncates long values. ## pbook.getUserJobMetadata(task_id, output_json_filename) ### Description Retrieves user job metadata for a task and saves it to a JSON file. ### Method ```python pbook.getUserJobMetadata(98765, "/tmp/task_metadata.json") ``` ### Parameters #### Query Parameters - **task_id** (int) - The ID of the task. - **output_json_filename** (str) - The path to the output JSON file. ## pbook.generate_credential() ### Description Regenerates the user's credential (OIDC token or grid proxy). ``` -------------------------------- ### phpo CLI - Hyperparameter Optimization Source: https://context7.com/pandawms/panda-client/llms.txt Submits hyperparameter optimization jobs using iDDS/PanDA integration. ```APIDOC ## phpo ### Description Submits hyperparameter optimization (HPO) tasks to PanDA, leveraging iDDS integration. This tool is designed for machine learning model training and tuning. ### Usage ```bash phpo [OPTIONS] --outDS --trainingDS --config ``` ### Options - **--outDS** (str) - Required. The output dataset name for HPO results. - **--trainingDS** (str) - Required. The dataset containing training data. - **--config** (str) - Required. Path to the HPO configuration JSON file. - **--nParallelEvaluation** (int) - Optional. Number of parallel evaluations to run. - **--minUnevaluatedPoints** (int) - Optional. Minimum number of unevaluated points required. - **--nPointsPerIteration** (int) - Optional. Number of points to evaluate per iteration. ### Example ```bash # Submit a hyperparameter optimization task phpo \ --outDS user.jdoe.hpo.v1 \ --trainingDS user.jdoe.trainingData \ --config hpo_config.json \ --nParallelEvaluation 4 \ --minUnevaluatedPoints 2 \ --nPointsPerIteration 4 ``` ``` -------------------------------- ### Submit JEDI Task Parameters with Client.insertTaskParams Source: https://context7.com/pandawms/panda-client/llms.txt Use this function to submit a full JEDI task parameter dictionary to PanDA/JEDI. This is the primary low-level submission API. Ensure all required task parameters are correctly formatted. ```python import uuid from pandaclient import Client log_ds = "user.jdoe.log.{}".format(uuid.uuid4()) out_ds = "user.jdoe.myanalysis.{}".format(uuid.uuid4()) task_params = { "taskName": "user.jdoe.test_task_{}".format(uuid.uuid4()), "userName": "jdoe", "vo": "atlas", "prodSourceLabel": "user", "taskType": "anal", "taskPriority": 900, "architecture": "x86_64-centos7-gcc8-opt", "transUses": "Atlas-21.2.200", "transHome": "AnalysisBase-21.2.200", "transPath": "Reco_trf.py", "processingType": "reprocessing", "nFilesPerJob": 1, "cloud": "US", "site": "BNL_ATLAS_2", "log": { "dataset": log_ds, "type": "template", "param_type": "log", "token": "local", "destination":"local", "value": "{}.${SN}.log.tgz".format(log_ds), }, "jobParameters": [ {"type": "constant", "value": "--maxEvents=100"}, { "type": "template", "param_type": "output", "token": "local", "destination":"local", "dataset": out_ds, "value": "outputAODFile={}.${SN}.AOD.pool.root".format(out_ds), }, ], } status, result = Client.insertTaskParams(task_params, verbose=False, properErrorCode=True) if status == 0: return_code, message = result print("Task submitted, return_code={}, message={}".format(return_code, message)) # Output: Task submitted, return_code=0, message=jediTaskID=98765 else: print("Submission failed:", result) ``` -------------------------------- ### Client.get_files_in_datasets Source: https://context7.com/pandawms/panda-client/llms.txt Lists files in the input (and optionally pseudo-input) datasets of a task. ```APIDOC ## Client.get_files_in_datasets ### Description Lists files in the input (and optionally pseudo-input) datasets of a task. ### Parameters - **jedi_task_id** (int) - Required - The ID of the JEDI task. ### Return Value - **status** (int) - 0 for success, non-zero for failure. - **files** (list) - A list of dictionaries, where each dictionary contains information about an input file. ``` -------------------------------- ### Client.get_job_descriptions Source: https://context7.com/pandawms/panda-client/llms.txt Fetches job description dictionaries for all jobs (or only failed/cancelled/closed ones) in a task. ```APIDOC ## Client.get_job_descriptions ### Description Fetches job description dictionaries for all jobs (or only failed/cancelled/closed ones) in a task. ### Parameters - **jedi_task_id** (int) - Required - The ID of the JEDI task. - **unsuccessful_only** (bool) - Optional - If True, only returns descriptions for unsuccessful jobs. Defaults to False. ### Return Value - **status** (int) - 0 for success, non-zero for failure. - **descriptions** (list) - A list of dictionaries, where each dictionary contains the description of a job. ``` -------------------------------- ### Client.insertTaskParams Source: https://context7.com/pandawms/panda-client/llms.txt Submits a full JEDI task parameter dictionary to PanDA/JEDI. This is the primary low-level submission API used by prun, pathena, and phpo internally. ```APIDOC ## `Client.insertTaskParams` — Submit a JEDI Task Submits a full JEDI task parameter dictionary to PanDA/JEDI. This is the primary low-level submission API used by `prun`, `pathena`, and `phpo` internally. ```python import uuid from pandaclient import Client log_ds = "user.jdoe.log.{}".format(uuid.uuid4()) out_ds = "user.jdoe.myanalysis.{}".format(uuid.uuid4()) task_params = { "taskName": "user.jdoe.test_task_{}".format(uuid.uuid4()), "userName": "jdoe", "vo": "atlas", "prodSourceLabel": "user", "taskType": "anal", "taskPriority": 900, "architecture": "x86_64-centos7-gcc8-opt", "transUses": "Atlas-21.2.200", "transHome": "AnalysisBase-21.2.200", "transPath": "Reco_trf.py", "processingType": "reprocessing", "nFilesPerJob": 1, "cloud": "US", "site": "BNL_ATLAS_2", "log": { "dataset": log_ds, "type": "template", "param_type": "log", "token": "local", "destination":"local", "value": "{}.${SN}.log.tgz".format(log_ds), }, "jobParameters": [ {"type": "constant", "value": "--maxEvents=100"}, { "type": "template", "param_type": "output", "token": "local", "destination":"local", "dataset": out_ds, "value": "outputAODFile={}.${SN}.AOD.pool.root".format(out_ds), }, ], } status, result = Client.insertTaskParams(task_params, verbose=False, properErrorCode=True) if status == 0: return_code, message = result print("Task submitted, return_code={}, message={}".format(return_code, message)) # Output: Task submitted, return_code=0, message=jediTaskID=98765 else: print("Submission failed:", result) ``` ``` -------------------------------- ### Finish Task Gracefully with Client.finishTask Source: https://context7.com/pandawms/panda-client/llms.txt Use this to signal that a task should be finished after its current running jobs complete. Set `soft=True` for graceful completion. ```python status, (return_code, message) = Client.finishTask(jedi_task_id, soft=True, verbose=False) if status == 0 and return_code == 0: print("Task finish registered:", message) else: print("Failed:", return_code, message) ``` -------------------------------- ### Submit Workflow Request Source: https://context7.com/pandawms/panda-client/llms.txt Submits a workflow request using the Client.send_workflow_request method. Returns status and result. ```python status, result = Client.send_workflow_request(workflow_params, check=False, verbose=False) if status == 0: print("Workflow submitted:", result) ```