### my_qadapter.yaml Example for SLURM Source: https://materialsproject.github.io/jobflow/install_fireworks.html An example my_qadapter.yaml file for the SLURM queuing system. Adjust paths, resource allocations, and pre-execution commands as needed. ```yaml _fw_name: CommonAdapter _fw_q_type: SLURM rocket_launch: rlaunch -w /path/to/fw_config/my_fworker.yaml singleshot nodes: 2 walltime: 00:30:00 account: job_name: my_firework qos: regular pre_rocket: | module load vasp export ATOMATE2_VASP_CMD="srun -N 2 --ntasks-per-node=24" ``` -------------------------------- ### Install Jobflow from Source Source: https://materialsproject.github.io/jobflow/install.html Clone the repository and install the package locally, optionally including extra dependencies or developer mode. ```bash git clone https://github.com/materialsproject/jobflow.git cd jobflow pip install . ``` ```bash pip install .[fireworks] ``` ```bash pip install -e . ``` -------------------------------- ### Log Output Example Source: https://materialsproject.github.io/jobflow/tutorials/2-introduction.html Example log output from running jobs locally, showing job start and finish times, job names, and unique job UUIDs. ```log 2023-12-08 11:15:18,266 INFO Started executing jobs locally 2023-12-08 11:15:18,270 INFO Starting job - time_website (f7831f60-617e-490d-8854-ef3e25a78504) 2023-12-08 11:15:18,873 INFO Finished job - time_website (f7831f60-617e-490d-8854-ef3e25a78504) 2023-12-08 11:15:18,875 INFO Starting job - time_website (993990cd-872e-4c8f-823b-ac4eea8756c6) 2023-12-08 11:15:19,300 INFO Finished job - time_website (993990cd-872e-4c8f-823b-ac4eea8756c6) 2023-12-08 11:15:19,304 INFO Starting job - time_website (3c2d3abd-be33-4857-9943-6526bfb05804) 2023-12-08 11:15:19,774 INFO Finished job - time_website (3c2d3abd-be33-4857-9943-6526bfb05804) 2023-12-08 11:15:19,777 INFO Starting job - sum_numbers (2be44e87-0918-481f-ae77-7a5931074e1e) 2023-12-08 11:15:19,783 INFO Finished job - sum_numbers (2be44e87-0918-481f-ae77-7a5931074e1e) 2023-12-08 11:15:19,785 INFO Finished executing jobs locally ``` -------------------------------- ### Build Documentation Source: https://materialsproject.github.io/jobflow/install.html Install documentation dependencies and generate the site using sphinx-build. ```bash pip install .[docs] ``` ```bash sphinx-build docs/src docs_build ``` -------------------------------- ### Install Jobflow via pip Source: https://materialsproject.github.io/jobflow/index.html Standard installation command for the Jobflow library. ```bash pip install jobflow ``` -------------------------------- ### Example Jobflow Run Outputs Source: https://materialsproject.github.io/jobflow/tutorials/2-introduction.html A concrete example of the 'responses' dictionary returned by run_locally, showing the output for several jobs identified by their UUIDs. ```json { "f7831f60-617e-490d-8854-ef3e25a78504": { "1": { "output": 0.20485366717912257, "detour": null, "addition": null, "replace": null, "stored_data": null, "stop_children": false, "stop_jobflow": false } }, "993990cd-872e-4c8f-823b-ac4eea8756c6": { "1": { "output": 0.06472958298400044, "detour": null, "addition": null, "replace": null, "stored_data": null, "stop_children": false, "stop_jobflow": false } }, "3c2d3abd-be33-4857-9943-6526bfb05804": { "1": { "output": 0.18873387505300343, "detour": null, "addition": null, "replace": null, "stored_data": null, "stop_children": false, "stop_jobflow": false } }, "2be44e87-0918-481f-ae77-7a5931074e1e": { "1": { "output": 0.45831712521612644, "detour": null, "addition": null, "replace": null, "stored_data": null, "stop_children": false, "stop_jobflow": false } } } ``` -------------------------------- ### Define FireWorks Configuration Directories Source: https://materialsproject.github.io/jobflow/tutorials/8-fireworks Example structure for separate FireWorks configuration directories. ```text /path/to/fw_config1 ├── FW_config1.yaml ├── my_fworker1.yaml ├── my_launchpad1.yaml └── my_qadapter1.yaml ``` ```text /path/to/fw_config2 ├── FW_config2.yaml ├── my_fworker2.yaml ├── my_launchpad2.yaml └── my_qadapter2.yaml ``` -------------------------------- ### Install Jobflow via Pip Source: https://materialsproject.github.io/jobflow/install.html Install the core jobflow package or include optional dependencies for specific features. ```bash pip install jobflow ``` ```bash pip install jobflow[fireworks] ``` ```bash pip install jobflow[vis] # Install requirements for visualizing jobs and flows ``` -------------------------------- ### Configure JobStore with GridFSStore Source: https://materialsproject.github.io/jobflow/_modules/jobflow/settings.html Example YAML configuration for a JobStore using MongoStore for documents and GridFSStore for data blobs. ```yaml docs_store: type: MongoStore database: jobflow_unittest collection_name: outputs host: localhost port: 27017 additional_stores: data: type: GridFSStore database: jobflow_unittest collection_name: outputs_blobs host: localhost port: 27017 ``` -------------------------------- ### Run Unit Tests Source: https://materialsproject.github.io/jobflow/install.html Install test dependencies and execute the test suite using pytest. ```bash pip install .[tests] ``` ```bash pytest ``` -------------------------------- ### GET /query Source: https://materialsproject.github.io/jobflow/_modules/jobflow/core/store.html Query the JobStore for multiple documents based on criteria. ```APIDOC ## GET /query ### Description Query the JobStore for documents matching the provided criteria. ### Method GET ### Endpoint /query ### Parameters #### Query Parameters - **criteria** (dict) - Optional - PyMongo filter for documents to search in. - **properties** (dict|list) - Optional - Properties to return in grouped documents. - **sort** (dict) - Optional - Dictionary of sort order for fields. - **skip** (int) - Optional - Number of documents to skip. - **limit** (int) - Optional - Limit on the total number of documents returned. - **load** (load_type) - Optional - Which items to load from additional stores. ### Response #### Success Response (200) - **Iterator[dict]** - The documents matching the query. ``` -------------------------------- ### GET /as_dict Source: https://materialsproject.github.io/jobflow/_modules/jobflow/core/reference.html Serializes the current OutputReference instance into a dictionary format. ```APIDOC ## GET /as_dict ### Description Serialize the reference as a dict. ### Method GET ### Response #### Success Response (200) - **schema** (object) - The output schema associated with the reference. ``` -------------------------------- ### Example: Using Flow Output in Another Flow Source: https://materialsproject.github.io/jobflow/_modules/jobflow/core/flow.html Shows how to use the output of a previously defined Flow as an input for a new job in an outer Flow. This highlights the composability of Flows. ```python >>> add_third = add(flow.output, 5) >>> outer_flow = Flow(jobs=[flow, add_third]) ``` -------------------------------- ### Example: Connected Add Jobs in a Flow Source: https://materialsproject.github.io/jobflow/_modules/jobflow/core/flow.html Illustrates creating a Flow with two connected 'add' jobs. The output of the first job is used as an input for the second, demonstrating data dependency within a Flow. ```python >>> from jobflow import job, Flow >>> @job ... def add(a, b): ... return a + b >>> add_first = add(1, 2) >>> add_second = add(add_first.output, 2) >>> flow = Flow(jobs=[add_first, add_second]) ``` -------------------------------- ### Run Job with JobStore Source: https://materialsproject.github.io/jobflow/_modules/jobflow/core/job.html Executes the job using a provided JobStore. Resolves OutputReferences if configured and logs job start information. Imports necessary modules like `types`, `datetime`, `jobflow`. ```python import types from datetime import datetime from jobflow import CURRENT_JOB from jobflow.core.flow import get_flow from jobflow.core.schemas import JobStoreDocument index_str = f", {self.index}" if self.index != 1 else "" logger.info(f"Starting job - {self.name} ({self.uuid}{index_str})") CURRENT_JOB.job = self if self.config.expose_store: CURRENT_JOB.store = store if self.config.resolve_references: ``` -------------------------------- ### Set FW_CONFIG_FILE Environment Variable Source: https://materialsproject.github.io/jobflow/install_fireworks.html Example of how to set the FW_CONFIG_FILE environment variable in your ~/.bashrc file to point to your FW_config.yaml. ```bash export FW_CONFIG_FILE="/path/to/config/fw_config/FW_config.yaml" ``` -------------------------------- ### Local Execution Log Output Source: https://materialsproject.github.io/jobflow/tutorials/5-dynamic-flows.html Example log output generated during the local execution of fibonacci jobs. ```text 2023-11-23 23:55:13,324 INFO Started executing jobs locally 2023-11-23 23:55:13,325 INFO Starting job - fibonacci (6f7a1e92-577f-4289-85e5-e07a82e542db) 2023-11-23 23:55:13,329 INFO Finished job - fibonacci (6f7a1e92-577f-4289-85e5-e07a82e542db) 2023-11-23 23:55:13,329 INFO Starting job - fibonacci (862ce31a-a963-419a-9862-f419c47c3c9e) 2023-11-23 23:55:13,332 INFO Finished job - fibonacci (862ce31a-a963-419a-9862-f419c47c3c9e) 2023-11-23 23:55:13,333 INFO Starting job - fibonacci (81c28a0a-f1ad-487c-b484-46729f6ca9a6) 2023-11-23 23:55:13,335 INFO Finished job - fibonacci (81c28a0a-f1ad-487c-b484-46729f6ca9a6) 2023-11-23 23:55:13,336 INFO Starting job - fibonacci (084e2022-5a00-4ca8-9edf-46977ca08388) 2023-11-23 23:55:13,338 INFO Finished job - fibonacci (084e2022-5a00-4ca8-9edf-46977ca08388) 2023-11-23 23:55:13,339 INFO Starting job - fibonacci (382f371c-9e92-4185-8dbb-829801ec1afd) 2023-11-23 23:55:13,342 INFO Finished job - fibonacci (382f371c-9e92-4185-8dbb-829801ec1afd) 2023-11-23 23:55:13,342 INFO Starting job - fibonacci (f4521225-993a-4b42-9818-8aaff0da9181) 2023-11-23 23:55:13,345 INFO Finished job - fibonacci (f4521225-993a-4b42-9818-8aaff0da9181) 2023-11-23 23:55:13,346 INFO Starting job - fibonacci (3d475e3e-3774-4015-b056-a23f5cf1da6a) 2023-11-23 23:55:13,349 INFO Finished job - fibonacci (3d475e3e-3774-4015-b056-a23f5cf1da6a) 2023-11-23 23:55:13,351 INFO Starting job - fibonacci (ddc61891-1ce1-4d96-b66e-cb60d52e642a) 2023-11-23 23:55:13,353 INFO Finished job - fibonacci (ddc61891-1ce1-4d96-b66e-cb60d52e642a) 2023-11-23 23:55:13,354 INFO Starting job - fibonacci (499ab638-d8cc-4272-b2b4-f64d246f3676) 2023-11-23 23:55:13,357 INFO Finished job - fibonacci (499ab638-d8cc-4272-b2b4-f64d246f3676) 2023-11-23 23:55:13,357 INFO Starting job - fibonacci (71b7d00f-0109-4a23-8a54-fb29ef143cd2) 2023-11-23 23:55:13,360 INFO Finished job - fibonacci (71b7d00f-0109-4a23-8a54-fb29ef143cd2) 2023-11-23 23:55:13,361 INFO Starting job - fibonacci (d60f3f5d-821a-434a-a8c1-50468b381d5f) 2023-11-23 23:55:13,363 INFO Finished job - fibonacci (d60f3f5d-821a-434a-a8c1-50468b381d5f) 2023-11-23 23:55:13,364 INFO Starting job - fibonacci (7828008e-2f89-4226-aaac-66f25d4ccf0a) 2023-11-23 23:55:13,367 INFO Finished job - fibonacci (7828008e-2f89-4226-aaac-66f25d4ccf0a) 2023-11-23 23:55:13,367 INFO Starting job - fibonacci (062b0cad-2b62-4f42-bd39-ac1b3c90d141) 2023-11-23 23:55:13,370 INFO Finished job - fibonacci (062b0cad-2b62-4f42-bd39-ac1b3c90d141) 2023-11-23 23:55:13,371 INFO Starting job - fibonacci (6c1bf8e4-1808-42b4-a99e-49dd031894bd) 2023-11-23 23:55:13,373 INFO Finished job - fibonacci (6c1bf8e4-1808-42b4-a99e-49dd031894bd) 2023-11-23 23:55:13,374 INFO Starting job - fibonacci (d82e352d-1ca3-40d2-85d1-3f08989d3551) 2023-11-23 23:55:13,376 INFO Finished job - fibonacci (d82e352d-1ca3-40d2-85d1-3f08989d3551) 2023-11-23 23:55:13,377 INFO Finished executing jobs locally ``` -------------------------------- ### Configure JobStore with S3Store Source: https://materialsproject.github.io/jobflow/_modules/jobflow/settings.html Example YAML configuration for a JobStore using S3Store for data blobs, requiring the key field to be set to blob_uuid. ```yaml docs_store: type: MongoStore database: jobflow_unittest collection_name: outputs host: localhost port: 27017 additional_stores: data: type: S3Store bucket: output_blobs key: blob_uuid index: type: MongoStore database: jobflow_unittest collection_name: output_blobs_index host: localhost port: 27017 key: blob_uuid ``` -------------------------------- ### Example Output for Code 1 Execution Source: https://materialsproject.github.io/jobflow/tutorials/7-generalized-makers.html This output shows the execution log when running a flow generated by Code1Maker, demonstrating the independent and code-specific steps. ```log 2023-06-08 09:57:53,829 INFO Started executing jobs locally 2023-06-08 09:57:53,896 INFO Starting job - job1 (b6557183-838d-4c04-b1ab-356b8923016d) DO CODE INDEPENDENT STUFF DO STUFF specific to CODE 1 2023-06-08 09:57:53,896 INFO Finished job - job1 (b6557183-838d-4c04-b1ab-356b8923016d) 2023-06-08 09:57:53,896 INFO Finished executing jobs locally ``` -------------------------------- ### Define Add and Conditional Add Jobs (Output > 10) Source: https://materialsproject.github.io/jobflow/tutorials/5-dynamic-flows.html Similar to the previous example, this defines `add` and `add_with_logic` jobs. However, it sets up a scenario where the initial addition results in a value greater than 10, demonstrating that `Response(addition=...)` will not trigger a new job in this case. Requires `@job` and `Response`. ```python @job def add(a, b): return a + b @job def add_with_logic(a, b): if a < 10: return Response(addition=add(a, b)) return None job1 = add(1, 20) job2 = add_with_logic(job1.output, 20) flow = Flow([job1, job2]) responses = run_locally(flow) ``` -------------------------------- ### Initialize DictMods Class Source: https://materialsproject.github.io/jobflow/_modules/jobflow/utils/dict_mods.html Initializes the DictMods class by populating a dictionary of supported actions based on the class's static methods. This setup is performed once upon instantiation. ```python class DictMods: """ Class to define mongo-like modifications on a dict. Supported keywords include the following Mongo-based keywords, with the usual meanings (refer to Mongo documentation for information): - ``_inc`` - ``_set`` - ``_unset`` - ``_push`` - ``_push_all`` - ``_add_to_set`` (but ``_each`` is not supported) - ``_pop`` - ``_pull`` - ``_pull_all`` - ``_rename`` .. Note:: Note that ``_set`` does not support modification of nested dicts using the mongo ``{"a.b":1}`` notation. This is because mongo does not allow keys with "." to be inserted. Instead, nested dict modification is supported using a special "->" keyword, e.g. ``{"a->b": 1}`` """ def __init__(self): self.supported_actions = {} for i in dir(self): if (not re.match(r"__\w+__", i)) and callable(getattr(self, i)): self.supported_actions["_{}".format(i)] = getattr(self, i) ``` -------------------------------- ### Define a simple job Source: https://materialsproject.github.io/jobflow/jobflow.core.html Defines a basic job function that performs addition. This serves as a starting point for demonstrating configuration updates. ```python >>> from jobflow import job, JobConfig >>> @job ... def add(a, b): ... return a + b >>> add_job = add(1, 2) ``` -------------------------------- ### Example Output for Code 2 Execution Source: https://materialsproject.github.io/jobflow/tutorials/7-generalized-makers.html This output logs the execution of a flow created by Code2Maker, illustrating the sequence of independent and code-specific operations. ```log 2023-06-08 09:57:53,900 INFO Started executing jobs locally 2023-06-08 09:57:53,900 INFO Starting job - job1 (7d744b4f-e0ab-46f0-adb2-72371e0c5070) DO CODE INDEPENDENT STUFF DO STUFF specific to CODE 2 2023-06-08 09:57:53,901 INFO Finished job - job1 (7d744b4f-e0ab-46f0-adb2-72371e0c5070) 2023-06-08 09:57:53,901 INFO Finished executing jobs locally ``` -------------------------------- ### JobStore Configuration from File Source: https://materialsproject.github.io/jobflow/_modules/jobflow/core/store.html Demonstrates how to load a JobStore configuration from a file, which can specify different storage backends for documents and additional data. ```APIDOC ## JobStore Configuration from File ### Description This method loads a JobStore configuration from a file. The file can specify a primary document store (e.g., MongoStore) and additional stores for data (e.g., GridFSStore). ### Method `classmethod` ### Endpoint N/A (Class Method) ### Parameters #### Path Parameters - **db_file** (str) - Required - Path to the file containing the credentials. #### Query Parameters None #### Request Body None ### Request Example None ### Response #### Success Response (JobStore) - **JobStore** - An instance of the JobStore class configured with the specified storage. #### Response Example None ### Notes - The `db_file` can contain a serialized JobStore object or a dictionary specification for the stores. - Additional keyword arguments can be passed to the JobStore constructor. ``` -------------------------------- ### Initialize JobStore from File Source: https://materialsproject.github.io/jobflow/_modules/jobflow/core/store.html Creates a JobStore instance from a JSON or YAML configuration file. ```python @classmethod def from_file(cls, db_file: str | Path, **kwargs) -> Self: """ Create a JobStore from a database file. Two options are supported for the database file. The file should be in json or yaml format. The simplest format is a monty dumped version of the store, generated using: >>> from monty.serialization import dumpfn >>> dumpfn(job_store, "job_store.yaml") Alternatively, the file can contain the keys docs_store, additional_stores and any other keyword arguments supported by the :obj:`JobStore` constructor. """ ``` ```yaml docs_store: ``` -------------------------------- ### JobStore Configuration from Dictionary Specification Source: https://materialsproject.github.io/jobflow/_modules/jobflow/core/store.html Illustrates creating a JobStore instance directly from a dictionary that defines the storage configurations. ```APIDOC ## JobStore Configuration from Dictionary Specification ### Description This class method creates a JobStore from a dictionary specification. The dictionary must include a `docs_store` key and can optionally include `additional_stores` and other keyword arguments for the JobStore constructor. Each store definition requires a `type` key matching a Maggma Store subclass. ### Method `classmethod` ### Endpoint N/A (Class Method) ### Parameters #### Path Parameters None #### Query Parameters None #### Request Body - **spec** (dict) - Required - The dictionary specification for the JobStore configuration. - **docs_store** (dict) - Required - Configuration for the primary document store. - **type** (str) - Required - The type of the store (e.g., `MongoStore`). - **database** (str) - Required - The database name. - **collection_name** (str) - Required - The collection name. - **host** (str) - Required - The host address. - **port** (int) - Required - The port number. - **additional_stores** (dict) - Optional - A dictionary of additional stores. - **[store_name]** (dict) - Required - Configuration for an additional store. - **type** (str) - Required - The type of the store (e.g., `GridFSStore`). - **database** (str) - Required - The database name. - **collection_name** (str) - Required - The collection name. - **host** (str) - Required - The host address. - **port** (int) - Required - The port number. - **kwargs** - Optional - Additional keyword arguments passed to the JobStore constructor. ### Request Example ```yaml docs_store: type: MongoStore database: jobflow_unittest collection_name: outputs host: localhost port: 27017 additional_stores: data: type: GridFSStore database: jobflow_unittest collection_name: outputs_blobs host: localhost port: 27017 ``` ### Response #### Success Response (JobStore) - **JobStore** - An instance of the JobStore class configured with the specified storage. #### Response Example None ### Errors - **ValueError**: If `docs_store` is not present in the `spec` dictionary. ``` -------------------------------- ### Initialize JobStore Source: https://materialsproject.github.io/jobflow/_modules/jobflow/core/store.html The JobStore constructor requires a primary docs_store and accepts optional additional stores, save configurations, and load settings. ```python def __init__( self, docs_store: Store, additional_stores: dict[str, Store] = None, save: save_type = None, load: load_type = False, ): ``` -------------------------------- ### GET /query_one Source: https://materialsproject.github.io/jobflow/_modules/jobflow/core/store.html Query the JobStore for a single document. ```APIDOC ## GET /query_one ### Description Query the Store for a single document matching the criteria. ### Method GET ### Endpoint /query_one ### Parameters #### Query Parameters - **criteria** (dict) - Optional - PyMongo filter for documents to search. - **properties** (dict|list) - Optional - Properties to return in the document. - **sort** (dict) - Optional - Dictionary of sort order for fields. - **load** (load_type) - Optional - Which items to load from additional stores. ### Response #### Success Response (200) - **dict|None** - The document if found, otherwise None. ``` -------------------------------- ### GET /flow/get_flow Source: https://materialsproject.github.io/jobflow/jobflow.core.html Validates dependencies and returns a Flow object. ```APIDOC ## GET /flow/get_flow ### Description Check dependencies and return a flow object. ### Parameters #### Query Parameters - **flow** (Flow | Job | list[jobflow.Job]) - Required - A job, list of jobs, or flow. - **allow_external_references** (bool) - Optional - If False all the references to other outputs should be from other Jobs of the Flow. ### Response #### Success Response (200) - **Flow** (Flow) - A Flow object where connections have been checked. ``` -------------------------------- ### Create a Job with a Standard Library Function Source: https://materialsproject.github.io/jobflow/jobflow.core.html Demonstrates initializing a Job using a function from the Python standard library. ```python >>> import os >>> Job(function=os.path.join, function_args=("folder", "filename.txt")) ``` -------------------------------- ### Initialize JobStore with MemoryStore Source: https://materialsproject.github.io/jobflow/tutorials/2-introduction.html Initialize the JobStore, passing the configured MemoryStore instance. This JobStore will manage the storage and retrieval of job outputs. ```python from jobflow import JobStore store = JobStore(docs_store) ``` -------------------------------- ### Draw a networkx graph Source: https://materialsproject.github.io/jobflow/_modules/jobflow/utils/graph.html Visualizes a graph using matplotlib. Requires matplotlib to be installed. ```python @requires(mpl, "matplotlib must be installed to plot flow graphs.") def draw_graph( graph: nx.DiGraph, layout_function: typing.Callable = None, figsize: tuple[float, float] = (12, 8), ): """ Draw a networkx graph. Parameters ---------- graph A graph object. layout_function A networkx layout function to use as the graph layout. For example, :obj:`.planar_layout`. figsize The figure size as a tuple of ``(width, height)``. Returns ------- matplotlib.pyplot The matplotlib pyplot object. """ import matplotlib.pyplot as plt if layout_function is None: try: pos = nx.nx_pydot.graphviz_layout(graph, prog="dot") except (ImportError, FileNotFoundError): # graphviz not installed pos = nx.planar_layout(graph) else: pos = layout_function(graph) plt.figure(figsize=figsize) nodes = graph.nodes() labels = nx.get_node_attributes(graph, "label") nx.draw_networkx_edges(graph, pos) nx.draw_networkx_nodes( graph, pos, nodelist=nodes, node_color="#B65555", linewidths=1, edgecolors="k" ) nx.draw_networkx_labels(graph, pos, labels=labels) edge_labels = nx.get_edge_attributes(graph, "properties") nx.draw_networkx_edge_labels(graph, pos, edge_labels=edge_labels, rotate=False) plt.gca().axis("off") return plt ``` -------------------------------- ### Connect to Data Sources Source: https://materialsproject.github.io/jobflow/_modules/jobflow/core/store.html Establishes connections to the primary docs_store and all configured additional stores. ```python def connect(self, force_reset: bool = False): """ Connect to the source data. Parameters ---------- force_reset Whether to reset the connection or not. """ self.docs_store.connect(force_reset=force_reset) for additional_store in self.additional_stores.values(): additional_store.connect(force_reset=force_reset) ``` -------------------------------- ### GET /groupby Source: https://materialsproject.github.io/jobflow/_modules/jobflow/core/store.html Groups documents by specified keys with optional filtering, sorting, and pagination. ```APIDOC ## GET /groupby ### Description Group documents by keys and return them as an iterator of (key, documents) tuples. ### Method GET ### Parameters #### Query Parameters - **keys** (list[str] | str) - Required - Fields to group documents by. - **criteria** (dict) - Optional - PyMongo filter for documents to search. - **properties** (dict | list) - Optional - Properties to return in grouped documents. - **sort** (dict) - Optional - Dictionary of sort order for fields. - **skip** (int) - Optional - Number of documents to skip. - **limit** (int) - Optional - Limit on the total number of documents returned. - **load** (any) - Optional - Items to load from additional stores. ### Response #### Success Response (200) - **data** (list) - An iterator of tuples containing the grouping key dictionary and the list of documents. ``` -------------------------------- ### Create a Job with a Custom Function Source: https://materialsproject.github.io/jobflow/jobflow.core.html Demonstrates initializing a Job using a custom, importable function. ```python >>> def add(a, b): ... return a + b >>> add_job = Job(function=add, function_args=(1, 2)) ``` -------------------------------- ### Draw Flow Graph Source: https://materialsproject.github.io/jobflow/_modules/jobflow/core/flow.html Draws the flow graph using matplotlib. Requires matplotlib to be installed. ```APIDOC ## POST /draw_graph ### Description Draws the flow graph using matplotlib. Requires matplotlib to be installed. ### Method POST ### Endpoint /draw_graph ### Parameters #### Query Parameters - **kwargs** (dict) - Keyword arguments passed to `jobflow.utils.graph.draw_graph`. ### Response #### Success Response (200) - **pyplot** (pyplot) - The matplotlib pyplot state object. #### Response Example ```json { "pyplot": "" } ``` ``` -------------------------------- ### Create a Job with a Builtin Function Source: https://materialsproject.github.io/jobflow/jobflow.core.html Demonstrates initializing a Job using a Python builtin function. ```python >>> print_task = Job(function=print, function_args=("I am a job", )) ``` -------------------------------- ### Visualize Flow Graph Source: https://materialsproject.github.io/jobflow/_modules/jobflow/core/flow.html Renders the flow graph using matplotlib. Requires matplotlib to be installed. ```python def draw_graph(self, **kwargs): """ Draw the flow graph using matplotlib. Requires matplotlib to be installed. Parameters ---------- kwargs keyword arguments that are passed to :obj:`jobflow.utils.graph.draw_graph`. Returns ------- pyplot The matplotlib pyplot state object. """ from jobflow.utils.graph import draw_graph return draw_graph(self.graph, **kwargs) ``` -------------------------------- ### POST /from_file Source: https://materialsproject.github.io/jobflow/_modules/jobflow/core/store.html Creates a JobStore instance from a JSON or YAML configuration file. ```APIDOC ## POST /from_file ### Description Create a JobStore from a database file in JSON or YAML format. ### Method POST ### Parameters #### Request Body - **db_file** (str) - Required - Path to the database configuration file. - **kwargs** (dict) - Optional - Additional keyword arguments for the JobStore constructor. ``` -------------------------------- ### Create JobStore from File Source: https://materialsproject.github.io/jobflow/_modules/jobflow/core/store.html Loads a JobStore configuration from a file. If the file contains a serialized JobStore object, it's loaded directly. Otherwise, it's interpreted as a dictionary specification. ```python from monty.serialization import loadfn store_info = loadfn(db_file) if isinstance(store_info, JobStore): return store_info return cls.from_dict_spec(store_info, **kwargs) ``` -------------------------------- ### GET /job/serialize Source: https://materialsproject.github.io/jobflow/_modules/jobflow/core/job.html Serializes the current job state into a dictionary format suitable for storage or transmission. ```APIDOC ## GET /job/serialize ### Description Serializes the job object into a dictionary, ensuring that non-serializable elements like functions or classes are handled via monty/jsanitize. ### Method GET ### Endpoint /job/serialize ### Response #### Success Response (200) - **data** (dict) - The serialized representation of the job. ``` -------------------------------- ### GET /get_output Source: https://materialsproject.github.io/jobflow/_modules/jobflow/core/store.html Retrieves the output of a specific job UUID, automatically resolving any internal references. ```APIDOC ## GET /get_output ### Description Get the output of a job UUID. This function automatically resolves any output references in the job outputs. ### Method GET ### Parameters #### Query Parameters - **uuid** (str) - Required - A job UUID. - **which** (str | int) - Optional - Which index to use: "last" (default), "first", "all", or an integer index. - **load** (bool) - Optional - Whether to load items from additional stores. - **cache** (dict) - Optional - A dictionary cache to use for resolving reference values. - **on_missing** (str) - Optional - Strategy for handling unresolved references. ### Response #### Success Response (200) - **output** (Any) - The output(s) for the job UUID. ``` -------------------------------- ### Get Job Input UUIDs Source: https://materialsproject.github.io/jobflow/_modules/jobflow/core/job.html Extracts the UUIDs from all OutputReference objects found in the job's inputs. ```python return tuple(ref.uuid for ref in self.input_references) ``` -------------------------------- ### Define and execute a simple Jobflow workflow Source: https://materialsproject.github.io/jobflow/index.html Demonstrates how to use the @job decorator to create jobs and combine them into a Flow with automatic dependency resolution. ```python from jobflow import job, Flow @job def add(a, b): return a + b add_first = add(1, 5) add_second = add(add_first.output, 5) flow = Flow([add_first, add_second]) flow.draw_graph().show() ``` -------------------------------- ### my_launchpad.yaml Template for MongoDB Source: https://materialsproject.github.io/jobflow/install_fireworks.html Template for my_launchpad.yaml when using a standard MongoDB connection. Replace placeholders with your database credentials. ```yaml host: port: 27017 name: username: password: logdir: null Istrm_lvl: DEBUG user_indices: [] wf_user_indices: [] ``` -------------------------------- ### GET /get_output Source: https://materialsproject.github.io/jobflow/jobflow.core.html Retrieves the output of a specific job UUID, with options to resolve references and handle multiple runs. ```APIDOC ## GET /get_output ### Description Get the output of a job UUID. This function will automatically try to resolve any output references in the job outputs. ### Method GET ### Parameters #### Query Parameters - **uuid** (str) - Required - A job UUID. - **which** (str|int) - Optional - Which index to use if there are multiple job runs (default: "last"). - **load** (load_type) - Optional - Which items to load from additional stores. - **cache** (dict) - Optional - A dictionary cache to use for resolving of reference values. - **on_missing** (OnMissing) - Optional - What to do if the output contains a reference and the reference cannot be resolved. ### Response #### Success Response (200) - **output** (Any) - The output(s) for the job UUID. ``` -------------------------------- ### Customizing Nested Maker Source: https://materialsproject.github.io/jobflow/_modules/jobflow/core/maker.html Shows how to customize a nested Maker (e.g., 'add_maker') by providing a custom instance when initializing the parent Maker. ```python from dataclasses import dataclass from jobflow import Maker, job @dataclass class AddMaker(Maker): name: str = "add" c: float = 1.5 @job def make(self, a, b): return a + b + self.c @dataclass class SquareThenAddMaker(Maker): name: str = "square then add" add_maker: Maker = AddMaker() @job def make(self, a, b): a_squared = a ** 2 b_squared = b ** 2 add_job = self.add_maker.make(a_squared, b_squared) return Response(detour=add_job) custom_add_maker = AddMaker(c=1000) maker = SquareThenAddMaker(add_maker=custom_add_maker) ``` -------------------------------- ### Create and Visualize a Jobflow Source: https://materialsproject.github.io/jobflow/_modules/jobflow/utils/graph.html Define jobs, chain them, and generate mermaid syntax for visualization. Use mermaid.live to render the graph. ```python from jobflow import job, Flow @job def add(a, b): return a + b add_first = add(1, 2) add_second = add(add_first.output, 2) my_flow = Flow(jobs=[add_first, add_second]) graph_source = to_mermaid(my_flow) ``` -------------------------------- ### Get Job Input References Source: https://materialsproject.github.io/jobflow/_modules/jobflow/core/job.html Retrieves all OutputReference objects from a job's inputs. Requires the `find_and_get_references` function. ```python from jobflow.core.reference import find_and_get_references references: set[jobflow.OutputReference] = set() for arg in tuple(self.function_args) + tuple(self.function_kwargs.values()): references.update(find_and_get_references(arg)) return tuple(references) ``` -------------------------------- ### Annotated Log Message Example Source: https://materialsproject.github.io/jobflow/tutorials/2-introduction.html An annotated log message illustrating the components: timestamp, job name, and job UUID. ```log 2021-06-04 17:36:50,531 INFO Starting job - time_website (d5460743-bb88-4922-8306-0400eb419a91) ↑ ↑ ↑ time job name job uuid ``` -------------------------------- ### Serialize JobStore to file Source: https://materialsproject.github.io/jobflow/jobflow.core.html Use monty to dump a JobStore instance into a YAML file. ```python >>> from monty.serialization import dumpfn >>> dumpfn(job_store, "job_store.yaml") ``` -------------------------------- ### Define and execute a simple workflow Source: https://materialsproject.github.io/jobflow/jobflow.core.html Create a workflow by connecting job outputs and executing it locally. ```python >>> from jobflow import job, Flow >>> @job ... def add(a, b): ... return a + b >>> add_first = add(1, 2) >>> add_second = add(add_first.output, 2) >>> flow = Flow(jobs=[add_first, add_second]) ``` ```python >>> flow = Flow(jobs=[add_first, add_second], output=add_second.output) ``` ```python >>> add_third = add(flow.output, 5) >>> outer_flow = Flow(jobs=[flow, add_third]) ``` ```python >>> from jobflow.managers.local import run_locally >>> response = run_locally(flow) ``` -------------------------------- ### Maker with Instance Variables Source: https://materialsproject.github.io/jobflow/jobflow.core.html Makers can have class instance variables that are used within the make function. This example adds a 'c' value to the sum. ```python @dataclass class AddMaker(Maker): name: str = "add" c: float = 1.5 @job def make(self, a, b): return a + b + self.c ``` ```python maker = AddMaker(c=3.5) add_job = maker.make(1, 2) ``` -------------------------------- ### JobStore Initialization Source: https://materialsproject.github.io/jobflow/_modules/jobflow/core/store.html Initializes a JobStore instance with a primary document store and optional additional stores for large data. It enforces specific keying strategies for efficient data management. ```APIDOC ## JobStore ### Description Store intended to allow pushing and pulling documents into multiple stores. ### Parameters #### Path Parameters None #### Query Parameters None #### Request Body None ### `__init__` Method ```python JobStore( docs_store: Store, additional_stores: dict[str, Store] = None, save: save_type = None, load: load_type = False, ) ``` #### Parameters - **docs_store** (Store) - Required - Store for basic documents. - **additional_stores** (dict[str, Store]) - Optional - Additional stores to use for storing large data/specific objects. Given as a mapping of ``{store name: store}`` where store is a maggma :obj:`.Store` object. - **save** (save_type) - Optional - Which items to save in additional stores when uploading documents. Given as a mapping of ``{store name: store_type}`` where ``store_type`` can be a dictionary key (string or enum), an :obj:`.MSONable` class, or a list of keys/classes. If the list of keys/classes itself contains a list/tuple, this will be treated as the path to the save key. Can be used if a key is duplicate in the output and only a single occurrence shall be put in the additional store. - **load** (load_type) - Optional - Which items to load from additional stores when querying documents. Given as a mapping of ``{store name: store_type}`` where ``store_type`` can be `True``, in which case all saved items are loaded, a dictionary key (string or enum), an :obj:`.MSONable` class, or a list of keys/classes. Alternatively, ``load=True`` will automatically load all items from every additional store. ### Request Example ```python from maggma.stores import MemoryStore from jobflow.core.store import JobStore docs_store = MemoryStore("my_db") additional_store = MemoryStore("my_large_data_store") job_store = JobStore( docs_store=docs_store, additional_stores={"large_data": additional_store}, save={"data": "my_custom_class"}, load=True ) ``` ### Response None (initializes the object) ``` -------------------------------- ### Get Maker Object from Job Source: https://materialsproject.github.io/jobflow/_modules/jobflow/core/job.html Retrieves the host Maker object if the job is configured to run a maker function. Imports `Maker` from `jobflow`. ```python from jobflow import Maker bound = getattr(self.function, "__self__", None) if isinstance(bound, Maker): return bound return None ``` -------------------------------- ### my_launchpad.yaml Template for MongoDB URI Source: https://materialsproject.github.io/jobflow/install_fireworks.html Template for my_launchpad.yaml when connecting via a MongoDB URI, such as with MongoDB Atlas. Set uri_store to true. ```yaml host: port: 27017 name: uri_store: true logdir: null Istrm_lvl: DEBUG user_indices: [] wf_user_indices: [] ``` -------------------------------- ### Initialize MemoryStore for Job Outputs Source: https://materialsproject.github.io/jobflow/tutorials/2-introduction.html Create an instance of MemoryStore to hold job outputs in memory. This is suitable for testing or development; use permanent stores for production. ```python from maggma.stores import MemoryStore docs_store = MemoryStore() ``` -------------------------------- ### Initialize FireWorks Launchpad Source: https://materialsproject.github.io/jobflow/install_fireworks.html Command to reset and initialize the FireWorks launchpad, ensuring a connection to the database. Use with caution as it clears existing data. ```bash lpad reset ``` -------------------------------- ### Get Flow Source: https://materialsproject.github.io/jobflow/_modules/jobflow/core/flow.html Validates dependencies and returns a Flow object. It ensures all job references within the flow are resolvable, either internally or externally if allowed. ```APIDOC ## GET /api/flow ### Description Checks dependencies and returns a flow object. Ensures that all references to other outputs are either from other Jobs within the Flow or, if allowed, from external sources. ### Method GET ### Endpoint /api/flow ### Parameters #### Query Parameters - **flow** (Flow | Job | list[jobflow.Job]) - Required - The job, list of jobs, or flow to be checked. - **allow_external_references** (bool) - Optional - Defaults to False. If False, all references to other outputs must be from other Jobs of the Flow. ### Response #### Success Response (200) - **Flow** (Flow) - A :obj:`Flow` object where connections have been checked. #### Response Example ```json { "flow_object": "..." } ``` ``` -------------------------------- ### Demonstrate Response(detour) in JobFlow Source: https://materialsproject.github.io/jobflow/tutorials/5-dynamic-flows.html Use Response(detour) to switch to a parallel job or flow. This example shows how a conditional detour to an 'add' job is triggered. ```python @job def add(a, b): return a + b @job def add_with_logic(a, b): if a < 10: return Response(detour=add(a, b)) return None job1 = add(1, 2) job2 = add_with_logic(job1.output, 2) flow = Flow([job1, job2]) responses = run_locally(flow) ```