### Quick Start: Basic Job Scheduling Source: https://github.com/atomology/dynamicworkflow.jl/blob/main/README.md Define a job function, start the scheduler, create jobs using the @job macro, and monitor their status or retrieve results. ```julia using DynamicWorkflow # Define a job function function my_add(x, y) x + y end # Start the scheduler start_scheduler() # Create and schedule jobs j1 = @job my_add(1, 2) j2 = @job my_add(3, 2) j3 = @job my_add(j1, j2) # j3 depends on j1 and j2 # Monitor and get results status(j3) # check job status result(j3) # get result (non-blocking call) fetch(j3) # get result (blocking call) ``` -------------------------------- ### Commit Setup File Source: https://github.com/atomology/dynamicworkflow.jl/blob/main/docs/superpowers/plans/2026-04-13-testitemrunner-migration.md Stage and commit the newly created setup.jl file. ```bash git add test/setup.jl git commit -m "test: add shared TestHelpers testsetup" ``` -------------------------------- ### Start Julia REPL with Package Loaded Source: https://github.com/atomology/dynamicworkflow.jl/blob/main/CLAUDE.md Start an interactive Julia REPL session with the DynamicWorkflow package loaded and multithreading enabled. ```julia julia --project -t 2 -e 'using DynamicWorkflow' ``` -------------------------------- ### start_scheduler Source: https://context7.com/atomology/dynamicworkflow.jl/llms.txt Initializes and starts the global job scheduler mainloop. ```APIDOC ## start_scheduler ### Description Initializes and starts the global job scheduler mainloop that processes pending jobs, resolves dependencies, and executes ready jobs using Julia's threading. Must be called before creating any jobs. ### Method Function Call ### Request Example start_scheduler() ``` -------------------------------- ### Initialize the Scheduler Source: https://context7.com/atomology/dynamicworkflow.jl/llms.txt Start the global job scheduler and verify its status before submitting jobs. ```julia using DynamicWorkflow # Start the scheduler (required before creating jobs) start_scheduler() # Output: [ Info: Starting scheduler... # Check if scheduler is alive isalive() # Returns: true # The scheduler is now ready to accept jobs ``` -------------------------------- ### Install DynamicWorkflow.jl Source: https://github.com/atomology/dynamicworkflow.jl/blob/main/README.md Use the Pkg manager to add the DynamicWorkflow package to your Julia environment. ```julia using Pkg Pkg.add("DynamicWorkflow") ``` -------------------------------- ### Update Basic Workflow Example Source: https://github.com/atomology/dynamicworkflow.jl/blob/main/docs/superpowers/plans/2026-04-13-remove-jobcontext-from-user-api.md Modify the `add` and `multiply` functions in `examples/basic_workflow.jl` to remove the `ctx::JobContext` argument, aligning with the project's plain function approach. ```julia function add(ctx::JobContext, x, y) println("Adding $x and $y") x + y end function multiply(ctx::JobContext, x, y) println("Multiplying $x and $y") x * y end ``` ```julia function add(x, y) println("Adding $x and $y") x + y end function multiply(x, y) println("Multiplying $x and $y") x * y end ``` -------------------------------- ### Update Dynamic Workflow Example Source: https://github.com/atomology/dynamicworkflow.jl/blob/main/docs/superpowers/plans/2026-04-13-remove-jobcontext-from-user-api.md Update the `my_add` function in `examples/dynamic_workflow.jl` to remove the `JobContext` argument. The `fibonacci` function is also shown, demonstrating dynamic job creation. ```julia function my_add(x, y) x + y end ``` ```julia function fibonacci(n) if n <= 1 return n end # Create new jobs dynamically j1 = @job fibonacci(n - 1) j2 = @job fibonacci(n - 2) # Wait for both jobs to complete and add their results fetch(j1) + fetch(j2) end ``` -------------------------------- ### Test Job constructor with plain functions Source: https://github.com/atomology/dynamicworkflow.jl/blob/main/docs/superpowers/plans/2026-04-13-remove-jobcontext-from-user-api.md This test suite validates the `Job` constructor's ability to handle plain functions without context injection. It requires the scheduler to be started and stopped using `start_scheduler()` and `stop_scheduler()`. Ensure `fetch` is used to retrieve job outputs. ```julia @testset "Job constructor with plain functions" begin start_scheduler() try plain_add(x, y) = x + y j1 = Job(plain_add, 1, 2) j2 = Job(plain_add, 3, 4) j3 = Job(plain_add, j1.output, j2.output) @test fetch(j1) == 3 @test fetch(j2) == 7 @test fetch(j3) == 10 @test j3.context.curr_id == j3.uuid finally stop_scheduler() end end ``` -------------------------------- ### Update Parallel Processing Example Source: https://github.com/atomology/dynamicworkflow.jl/blob/main/docs/superpowers/plans/2026-04-13-remove-jobcontext-from-user-api.md Refactor the `process_chunk` and `create_parallel_jobs` functions in `examples/parallel_processing.jl` to remove the `ctx::JobContext` argument. ```julia function process_chunk(ctx::JobContext, data_chunk) println("Processing chunk of size ", length(data_chunk)) sleep(0.1) mean(data_chunk) end function create_parallel_jobs(ctx::JobContext, chunks) [@job process_chunk(ctx, chunk) for chunk in chunks] end ``` ```julia function process_chunk(data_chunk) println("Processing chunk of size ", length(data_chunk)) sleep(0.1) mean(data_chunk) end function create_parallel_jobs(chunks) [@job process_chunk(chunk) for chunk in chunks] end ``` -------------------------------- ### Commit Changes for Plain Functions Source: https://github.com/atomology/dynamicworkflow.jl/blob/main/docs/superpowers/plans/2026-04-13-remove-jobcontext-from-user-api.md Bash command to stage and commit the updated test and example files. This commit message reflects the changes made to use plain functions without the JobContext argument. ```bash git add test/runtests.jl git commit -m "update tests to use plain functions without JobContext arg" ``` ```bash git add examples/basic_workflow.jl examples/dynamic_workflow.jl examples/parallel_processing.jl git commit -m "update examples to use plain functions without JobContext arg" ``` -------------------------------- ### Cancel Pending Jobs Source: https://context7.com/atomology/dynamicworkflow.jl/llms.txt Attempt to cancel a job that has not yet started execution. ```julia using DynamicWorkflow start_scheduler() slow_task() = (sleep(10); "done") j = @job slow_task() # Cancel before it starts (if still pending) success = cancel!(j) if success println("Job cancelled") println(status(j)) # Output: CANCELLED else println("Job already started or completed") end stop_scheduler() ``` -------------------------------- ### Test current_context() outside job Source: https://github.com/atomology/dynamicworkflow.jl/blob/main/docs/superpowers/plans/2026-04-13-remove-jobcontext-from-user-api.md This test verifies that `current_context()` returns `nothing` when called outside of an active job execution. It requires no special setup. ```julia @testset "current_context outside job" begin @test current_context() === nothing end ``` -------------------------------- ### Build Documentation Source: https://github.com/atomology/dynamicworkflow.jl/blob/main/CLAUDE.md Build the package documentation. This command must be run from the 'docs/' directory. ```bash cd docs && julia --project=. make.jl ``` -------------------------------- ### Instantiate Project Dependencies Source: https://github.com/atomology/dynamicworkflow.jl/blob/main/CLAUDE.md Instantiate the project dependencies for the first time or after changes to the Project.toml file. ```julia using Pkg; Pkg.instantiate() ``` -------------------------------- ### Run Tests (Test Project) Source: https://github.com/atomology/dynamicworkflow.jl/blob/main/CLAUDE.md Execute the test suite using the separate test project environment. Multithreading is enabled. ```bash julia --project=test -t 2 test/runtests.jl ``` -------------------------------- ### Run project tests Source: https://github.com/atomology/dynamicworkflow.jl/blob/main/docs/superpowers/plans/2026-04-13-job-graph-visualization.md Executes the test suite for the project. ```bash cd /path/to/feature-visualization julia --project=. test/runtests.jl ``` ```bash julia --project=. test/runtests.jl ``` -------------------------------- ### Add Test for Job Context Field in Julia Source: https://github.com/atomology/dynamicworkflow.jl/blob/main/docs/superpowers/plans/2026-04-13-remove-jobcontext-from-user-api.md This test verifies that the `Job` struct correctly includes a `JobContext` field and that its properties are initialized as expected. It requires the scheduler to be started and stopped. ```julia using Test using UUIDs using DynamicWorkflow @testset "job has context field" begin start_scheduler() try my_fn(x, y) = x + y j = @job my_fn(1, 2) @test j.context isa JobContext @test j.context.curr_id == j.uuid @test isnothing(j.context.parent_id) @test isempty(j.context.child_ids) finally stop_scheduler() end end ``` -------------------------------- ### Run workflow and draw graph in Julia Source: https://github.com/atomology/dynamicworkflow.jl/blob/main/docs/superpowers/plans/2026-04-13-job-graph-visualization.md Executes a sample workflow using DynamicWorkflow and GLMakie to visualize job dependencies. ```julia cd("/path/to/feature-visualization/examples") using Pkg; Pkg.activate(".") using DynamicWorkflow, GLMakie function my_add(ctx::JobContext, x, y); x + y; end function spawn_jobs(ctx::JobContext) jobs = Job[] for i in 1:3 j = @job my_add(ctx, 1, i) push!(jobs, j) end return jobs end function add_jobs(ctx::JobContext, a, b, c); a + b + c; end start_scheduler() w = @job spawn_jobs() jobs = fetch(w) j = @job add_jobs(jobs[1], jobs[2], jobs[3]) sleep(1) draw_graph() # should open a window ``` -------------------------------- ### Implement Scheduler Tests Source: https://github.com/atomology/dynamicworkflow.jl/blob/main/docs/superpowers/plans/2026-04-13-testitemrunner-migration.md Define test items for scheduler initialization and lifecycle management. ```julia @testitem "initialization" begin using DynamicWorkflow @test !isqueuealive() @test !allcomplete() end @testitem "scheduler lifecycle" begin using DynamicWorkflow using DynamicWorkflow: Q start_scheduler() sleep(1) t = Q[].mainloop @test istaskstarted(t) stop_scheduler() sleep(1) @test istaskdone(t) end ``` -------------------------------- ### Define Shared Test Helpers Source: https://github.com/atomology/dynamicworkflow.jl/blob/main/docs/superpowers/plans/2026-04-13-testitemrunner-migration.md Create a @testsetup module to provide shared utility functions for test items. ```julia @testsetup module TestHelpers export my_add, my_sleep, bad_job function my_add(x, y) x + y end function my_sleep(n::Int) sleep(n) return n end function bad_job() a = [] return a[1] end end ``` -------------------------------- ### Check Scheduler Status Source: https://context7.com/atomology/dynamicworkflow.jl/llms.txt Use allcomplete to verify if all jobs have finished, and isalive to check if the scheduler loop is active. ```julia using DynamicWorkflow start_scheduler() task(n) = (sleep(0.1); n) # Submit multiple jobs jobs = [@job task(i) for i in 1:5] # Check if all jobs are complete println(allcomplete()) # Output: false (jobs still running) # Wait for all to finish for j in jobs fetch(j) end println(allcomplete()) # Output: true stop_scheduler() ``` ```julia using DynamicWorkflow # Before starting println(isalive()) # Output: false start_scheduler() println(isalive()) # Output: true stop_scheduler() println(isalive()) # Output: false ``` -------------------------------- ### Run Tests Command Source: https://github.com/atomology/dynamicworkflow.jl/blob/main/docs/superpowers/plans/2026-04-13-job-graph-visualization.md Execute all tests in the project using the `julia` command with the project flag. This command is used to confirm the implementation of new features or fixes. ```bash julia --project=. test/runtests.jl ``` -------------------------------- ### Resolve and Instantiate Dependencies Source: https://github.com/atomology/dynamicworkflow.jl/blob/main/CLAUDE.md Resolve and instantiate project dependencies, typically after updating Project.toml or Manifest.toml. ```julia using Pkg; Pkg.resolve(); Pkg.instantiate() ``` -------------------------------- ### Commit Dependency Changes Source: https://github.com/atomology/dynamicworkflow.jl/blob/main/docs/superpowers/plans/2026-04-13-testitemrunner-migration.md Stage and commit the updated Project.toml file. ```bash git add test/Project.toml git commit -m "test: add TestItemRunner dependency" ``` -------------------------------- ### Run Full Package Test Suite Source: https://github.com/atomology/dynamicworkflow.jl/blob/main/CLAUDE.md Execute all tests in the package. Ensure -t 2 is used for multithreading. ```julia julia --project=. -t 2 test/runtests.jl ``` -------------------------------- ### Document Internal Methods Source: https://github.com/atomology/dynamicworkflow.jl/blob/main/docs/src/api.md Lists internal helper methods for the package. ```julia @docs DynamicWorkflow._make_context ``` -------------------------------- ### Commit documentation changes Source: https://github.com/atomology/dynamicworkflow.jl/blob/main/docs/superpowers/plans/2026-04-13-remove-jobcontext-from-user-api.md Git commands to stage and commit updates to the job source file. ```bash git add src/job.jl git commit -m "update docstrings for new context-free API" ``` ```bash git add src/job.jl git commit -m "remove unused dependencies function" ``` -------------------------------- ### @job Source: https://context7.com/atomology/dynamicworkflow.jl/llms.txt Wraps a function call to create a job and submits it to the global scheduler. ```APIDOC ## @job ### Description The @job macro wraps a function call to create a job that is immediately submitted to the global scheduler. When a Job is passed as an argument, it is automatically converted to an OutputRef, establishing a dependency. ### Method Macro ### Request Example @job my_function(arg1, arg2) ``` -------------------------------- ### Update Spawning Jobs Level 2 Testset Source: https://github.com/atomology/dynamicworkflow.jl/blob/main/docs/superpowers/plans/2026-04-13-remove-jobcontext-from-user-api.md Replace the 'spawning jobs level 2' testset with an updated version that uses plain functions for job creation and management. ```julia @testset "spawning jobs level 2" begin t = start_scheduler() try function spawn_jobs1() @job spawn_jobs2() end function spawn_jobs2() jobs = Job[] for i in 1:3 j = @job my_add(1, i) push!(jobs, j) end return jobs end w1 = @job spawn_jobs1() wait(w1) w2 = result(w1) jobs = result(w2) uuids = map(j->j.uuid, jobs) @test w1.uuid == context(w1).curr_id @test w2.uuid == context(w2).curr_id @test isnothing(context(w1).parent_id) @test context(w1).child_ids[1] == w2.uuid @test length(context(w1).child_ids) == 1 @test context(w2).parent_id == w1.uuid @test uuids == context(w2).child_ids @test all([context(j).parent_id == w2.uuid for j in jobs]) add_jobs(a, b, c) = a + b + c j = @job add_jobs(jobs[1], jobs[2], jobs[3]) wait(j) @test fetch(j) == 9 @test allcomplete() catch e throw(e) finally stop_scheduler() end end ``` -------------------------------- ### Verify Test Runner Execution Source: https://github.com/atomology/dynamicworkflow.jl/blob/main/docs/superpowers/plans/2026-04-13-testitemrunner-migration.md Execute the test runner to ensure it is correctly configured before adding individual test files. ```bash julia -t 2 --project=test test/runtests.jl ``` -------------------------------- ### Visualize Workflow Graph and Stop Scheduler Source: https://github.com/atomology/dynamicworkflow.jl/blob/main/README.md Visualize the workflow graph using a Makie backend and then stop the scheduler when done. ```julia # Use any compatible Makie backend like CairoMakie or GLMakie using GLMakie draw_graph() # Exit scheduler stop_scheduler() ``` -------------------------------- ### Verify CI compatibility with Pkg.test Source: https://github.com/atomology/dynamicworkflow.jl/blob/main/docs/superpowers/plans/2026-04-13-testitemrunner-migration.md Ensures the package test suite functions correctly when invoked via Pkg.test. ```bash julia --project -t 2 -e 'using Pkg; Pkg.test()' ``` -------------------------------- ### Dynamic Workflow with Parallel Jobs Source: https://github.com/atomology/dynamicworkflow.jl/blob/main/README.md Create a dynamic workflow where multiple jobs are spawned in parallel based on input data. Fetch the results of these spawned jobs to process them further. ```julia function create_parallel_jobs(v) [@job my_add(e, 1) for e in v] end v = [1,2,3] master_job = @job create_parallel_jobs(v) spawned_jobs = fetch(master_job) new_v = [fetch(j) for j in spawned_jobs] ``` -------------------------------- ### Define basic job tests in Julia Source: https://github.com/atomology/dynamicworkflow.jl/blob/main/docs/superpowers/plans/2026-04-13-testitemrunner-migration.md Tests for job macros and job functions to verify scheduler execution and queue state. ```julia @testitem "basics: job macros" setup=[TestHelpers] begin using DynamicWorkflow using DynamicWorkflow: Q t = start_scheduler() try j1 = @job my_add(1, 2) j2 = @job my_add(3, 2) j3 = @job my_add(j1, j2) @test fetch(j1) == 3 @test fetch(j2) == 5 @test fetch(j3) == 8 sleep(2) @test length(Q[].pending) == 0 @test length(Q[].running) == 0 @test allcomplete() @test length(Q[].completed) == 3 @test nv(Q[].g) == 3 @test ne(Q[].g) == 2 catch e throw(e) finally stop_scheduler() end end @testitem "basics: job function" setup=[TestHelpers] begin using DynamicWorkflow using DynamicWorkflow: Q t = start_scheduler() try j1 = Job(my_add, 1, 2) j2 = Job(my_add, 3, 2) j3 = Job(my_add, j1.output, j2.output) @test fetch(j1) == 3 @test fetch(j2) == 5 @test fetch(j3) == 8 sleep(2) @test length(Q[].pending) == 0 @test length(Q[].running) == 0 @test allcomplete() @test length(Q[].completed) == 3 @test nv(Q[].g) == 3 @test ne(Q[].g) == 2 catch e throw(e) finally stop_scheduler() end end ``` -------------------------------- ### Search for code references Source: https://github.com/atomology/dynamicworkflow.jl/blob/main/docs/superpowers/plans/2026-04-13-remove-jobcontext-from-user-api.md Commands to search for specific strings within the source code to identify dead code or remaining references. ```bash grep -rn "JobContext" src/ ``` ```bash grep -rn "dependencies" src/ test/ ``` -------------------------------- ### Implement Dynamic Child Jobs Source: https://context7.com/atomology/dynamicworkflow.jl/llms.txt Spawn child jobs recursively or sequentially to build a dynamic DAG. ```julia using DynamicWorkflow start_scheduler() my_add(x, y) = x + y # Function that spawns child jobs dynamically function sum_n(n) j = @job my_add(1, 2) if n < 3 return j end for i in 3:n j = @job my_add(j, i) # Each iteration creates a dependent job end return j end # Recursive job spawning (Fibonacci) function fibonacci(n) if n <= 1 return n end j1 = @job fibonacci(n - 1) j2 = @job fibonacci(n - 2) return fetch(j1) + fetch(j2) end # Sequential chain: 1+2+3+4+5 = 15 j1 = sum_n(5) println("Sum of 1 to 5: ", fetch(j1)) # Output: Sum of 1 to 5: 15 # Recursive tree j2 = @job fibonacci(6) println("Fibonacci(6) = ", fetch(j2)) # Output: Fibonacci(6) = 8 stop_scheduler() ``` -------------------------------- ### Run test files via CLI Source: https://github.com/atomology/dynamicworkflow.jl/blob/main/docs/superpowers/plans/2026-04-13-testitemrunner-migration.md Commands to execute specific test files using the Julia test runner. ```bash julia -t 2 --project=test test/runtests.jl test_basics ``` ```bash julia -t 2 --project=test test/runtests.jl test_job_status ``` -------------------------------- ### Commit test files to git Source: https://github.com/atomology/dynamicworkflow.jl/blob/main/docs/superpowers/plans/2026-04-13-testitemrunner-migration.md Git commands to stage and commit the newly created test files. ```bash git add test/test_basics.jl git commit -m "test: add basics testitem file" ``` ```bash git add test/test_job_status.jl git commit -m "test: add job status testitem file" ``` -------------------------------- ### Visualization Utilities Source: https://github.com/atomology/dynamicworkflow.jl/blob/main/docs/src/api.md Functions for rendering and styling workflow graphs. ```APIDOC ## Visualization Utilities ### Description Tools for drawing workflow graphs and customizing node appearance. ### Functions - **draw_graph(workflow)**: Renders the workflow graph. - **hierarchical_layout(graph)**: Applies a hierarchical layout to the graph. - **status_color(status)**: Returns the color associated with a specific status. - **status_text_color(status)**: Returns the text color for a status label. - **node_label(node)**: Generates the label for a graph node. ``` -------------------------------- ### Document Main Functions Source: https://github.com/atomology/dynamicworkflow.jl/blob/main/docs/src/api.md Lists the primary functions for controlling workflow execution and state management. ```julia @docs DynamicWorkflow.@job DynamicWorkflow.start_scheduler DynamicWorkflow.stop_scheduler DynamicWorkflow.status DynamicWorkflow.result DynamicWorkflow.fetch DynamicWorkflow.allcomplete DynamicWorkflow.cancel! DynamicWorkflow.istasksuccess DynamicWorkflow.isalive DynamicWorkflow.current_context DynamicWorkflow.run! DynamicWorkflow.head DynamicWorkflow.task_args DynamicWorkflow.context ``` -------------------------------- ### Test TLS Parent-Child Tracking in Julia Source: https://github.com/atomology/dynamicworkflow.jl/blob/main/docs/superpowers/plans/2026-04-13-remove-jobcontext-from-user-api.md This test case verifies the TLS-based parent-child tracking mechanism for jobs. It spawns multiple child jobs from a parent job and asserts that the context IDs and parent-child relationships are correctly established and reflected in the job contexts. ```julia start_scheduler() try function spawner() jobs = Job[] for i in 1:3 plain_add(x, y) = x + y j = @job plain_add(1, i) push!(jobs, j) end return jobs end w = @job spawner() jobs = fetch(w) sleep(1) uuids = map(j -> j.uuid, jobs) @test w.context.curr_id == w.uuid @test isnothing(w.context.parent_id) @test uuids == w.context.child_ids @test all(j -> j.context.parent_id == w.uuid, jobs) finally stop_scheduler() end ``` -------------------------------- ### Test Dynamic Workflow with While Loop Source: https://github.com/atomology/dynamicworkflow.jl/blob/main/docs/superpowers/plans/2026-04-13-testitemrunner-migration.md Tests a dynamic workflow that utilizes a while loop to iteratively create and execute jobs until a condition is met. Verifies correct job fetching and scheduler state, including graph node and edge counts. ```julia using DynamicWorkflow using DynamicWorkflow: Q t = start_scheduler() try function workflow() j = @job my_add(1, 1) while true j = @job my_add(j.output, 1) if fetch(j) > 3 break end end return fetch(j) end @test workflow() == 4 sleep(1) @test length(Q[].pending) == 0 @test length(Q[].running) == 0 @test allcomplete() @test length(Q[].completed) == 3 @test nv(Q[].g) == 3 @test ne(Q[].g) == 2 catch e throw(e) finally stop_scheduler() end ``` -------------------------------- ### Visualize Workflow Graph Source: https://context7.com/atomology/dynamicworkflow.jl/llms.txt Generate a hierarchical DAG visualization using draw_graph with a Makie backend. ```julia using DynamicWorkflow using GLMakie # or CairoMakie for non-interactive plots start_scheduler() add(x, y) = x + y multiply(x, y) = x * y # Create workflow j1 = @job add(2, 3) j2 = @job add(4, 5) j3 = @job multiply(j1, j2) # Wait for completion fetch(j3) # Draw the workflow graph fig = draw_graph() # Displays interactive figure with: # - Nodes colored by status (green=completed, blue=running, gray=pending) # - Function names and UUID suffixes as labels ``` -------------------------------- ### Run Scheduler Tests Source: https://github.com/atomology/dynamicworkflow.jl/blob/main/docs/superpowers/plans/2026-04-13-testitemrunner-migration.md Execute the specific test file using the runner pattern filter. ```bash julia -t 2 --project=test test/runtests.jl test_scheduler ``` -------------------------------- ### Test Dynamic Workflow with For Loop and Conditional Source: https://github.com/atomology/dynamicworkflow.jl/blob/main/docs/superpowers/plans/2026-04-13-testitemrunner-migration.md Tests a dynamic workflow that includes a for loop to create multiple jobs and a conditional statement to determine the final job execution path. Verifies correct job fetching and scheduler state. ```julia using DynamicWorkflow using DynamicWorkflow: Q t = start_scheduler() try function workflow() j1 = @job my_add(1, 2) jobs = [] for i in 1:4 push!(jobs, @job my_add(j1, i)) end if fetch(jobs[3]) > 4 j2 = @job my_add(jobs[1], jobs[3]) else j2 = @job my_add(jobs[3], jobs[4]) end return fetch(j2) end @test workflow() == 10 sleep(1) @test length(Q[].pending) == 0 @test length(Q[].running) == 0 @test allcomplete() @test length(Q[].completed) == 6 @test nv(Q[].g) == 6 @test ne(Q[].g) == 6 catch e throw(e) finally stop_scheduler() end ``` -------------------------------- ### Document Visualization Functions Source: https://github.com/atomology/dynamicworkflow.jl/blob/main/docs/src/api.md Lists functions used for rendering and styling workflow graphs. ```julia @docs DynamicWorkflow.draw_graph DynamicWorkflow.hierarchical_layout DynamicWorkflow.status_color DynamicWorkflow.status_text_color DynamicWorkflow.node_label DynamicWorkflow.node_width ``` -------------------------------- ### Define @job macro docstring Source: https://github.com/atomology/dynamicworkflow.jl/blob/main/docs/superpowers/plans/2026-04-13-remove-jobcontext-from-user-api.md Documentation for the @job macro, illustrating how to wrap function calls and handle parent-child task relationships. ```julia """ @job f(args...) Wrap a function call and create a job which will be enqueued immediately to the global JobQueue. Functions are plain Julia functions — no special first argument needed. Parent-child relationships are tracked automatically via task-local storage when `@job` is called inside a running job. # Examples ```julia my_add(x, y) = x + y j1 = @job my_add(1, 2) j2 = @job my_add(3, 2) j3 = @job my_add(j1, j2) # j3 depends on j1 and j2 ``` """ ``` -------------------------------- ### Rewrite Job constructor for TLS context Source: https://github.com/atomology/dynamicworkflow.jl/blob/main/docs/superpowers/plans/2026-04-13-remove-jobcontext-from-user-api.md This refactored `Job` constructor utilizes `current_context()` to determine the parent job context via task-local storage. It correctly handles plain functions by not injecting the context as an argument. Ensure `start_scheduler()` has been called prior to job creation. ```julia function Job(f::Function, args...) @debug "[$(now())] creating job with function: $f" uuid = UUIDs.uuid4() @debug "uuid $uuid" if !isassigned(Q) throw("JobQueue not initialized. Use start_scheduler().") end name = string(f) # Check TLS for parent context parent_ctx = current_context() if parent_ctx !== nothing push!(parent_ctx.child_ids, uuid) ctx = JobContext(uuid, parent_ctx.curr_id, UUID[]) else ctx = JobContext(uuid) end args = map(a -> a isa Job ? a.output : a, args) t = WTask(f, args...) output = OutputRef(uuid, Unassigned()) j = Job(name, uuid, ctx, output, PENDING) enqueue!(j, Q[]) return j end ``` -------------------------------- ### Verify Task Success Source: https://context7.com/atomology/dynamicworkflow.jl/llms.txt Use istasksuccess to determine if a job completed without errors. ```julia using DynamicWorkflow start_scheduler() good_task() = 42 bad_task() = error("Something went wrong") j1 = @job good_task() j2 = @job bad_task() # Wait for completion sleep(0.2) println(istasksuccess(j1)) # Output: true println(istasksuccess(j2)) # Output: false println(status(j2)) # Output: FAILED stop_scheduler() ``` -------------------------------- ### Commit test migration Source: https://github.com/atomology/dynamicworkflow.jl/blob/main/docs/superpowers/plans/2026-04-13-testitemrunner-migration.md Stages and commits the updated test directory. ```bash git add test/ git commit -m "test: complete TestItemRunner migration" ``` -------------------------------- ### Run focused tests by pattern Source: https://github.com/atomology/dynamicworkflow.jl/blob/main/docs/superpowers/plans/2026-04-13-testitemrunner-migration.md Filters test execution to specific items matching the provided file pattern. ```bash julia -t 2 --project=test test/runtests.jl test_basics ``` -------------------------------- ### Test Job Status Colors and Text Colors Source: https://github.com/atomology/dynamicworkflow.jl/blob/main/docs/superpowers/plans/2026-04-13-job-graph-visualization.md These tests verify that the `status_color` and `status_text_color` functions return the expected hex codes and color names for various job states. Ensure `DynamicWorkflow` is imported and job states like `PENDING`, `RUNNING`, etc., are defined. ```julia @testset "status_color and status_text_color" begin @test DynamicWorkflow.status_color(PENDING) == "#CCCCCC" @test DynamicWorkflow.status_color(RUNNING) == "#0072B2" @test DynamicWorkflow.status_color(COMPLETED) == "#009E73" @test DynamicWorkflow.status_color(FAILED) == "#D55E00" @test DynamicWorkflow.status_color(CANCELLED) == "#CC79A7" @test DynamicWorkflow.status_text_color(PENDING) == "#333333" @test DynamicWorkflow.status_text_color(RUNNING) == "white" @test DynamicWorkflow.status_text_color(COMPLETED) == "white" @test DynamicWorkflow.status_text_color(FAILED) == "white" @test DynamicWorkflow.status_text_color(CANCELLED) == "white" end ``` -------------------------------- ### Add TestItemRunner Dependency Source: https://github.com/atomology/dynamicworkflow.jl/blob/main/docs/superpowers/plans/2026-04-13-testitemrunner-migration.md Use the Julia package manager to add TestItemRunner to the test project environment. ```bash julia --project=test -e 'using Pkg; Pkg.add("TestItemRunner")' ``` -------------------------------- ### Test Spawning Child Jobs in Dynamic Workflow Source: https://github.com/atomology/dynamicworkflow.jl/blob/main/docs/superpowers/plans/2026-04-13-testitemrunner-migration.md Tests the functionality of spawning and managing child jobs within a dynamic workflow. Ensures correct parent-child ID relationships and job completion. ```julia using DynamicWorkflow using DynamicWorkflow: context, Q t = start_scheduler() try function spawn_jobs() jobs = Job[] for i in 1:3 j = @job my_add(1, i) push!(jobs, j) end return jobs end w = @job spawn_jobs() jobs = fetch(w) sleep(1) uuids = map(j -> j.uuid, jobs) @test w.uuid == context(w).curr_id @test isnothing(context(w).parent_id) @test uuids == context(w).child_ids @test all([context(j).parent_id == w.uuid for j in jobs]) add_jobs(a, b, c) = a + b + c j = @job add_jobs(jobs[1], jobs[2], jobs[3]) sleep(1) @test fetch(j) == 9 @test allcomplete() catch e throw(e) finally stop_scheduler() end ``` -------------------------------- ### Add Files to Git Staging Source: https://github.com/atomology/dynamicworkflow.jl/blob/main/docs/superpowers/plans/2026-04-13-job-graph-visualization.md Stages the modified `src/plot.jl` and `test/runtests.jl` files for the next commit. This command prepares the changes for version control. ```bash git add src/plot.jl test/runtests.jl ``` -------------------------------- ### Commit Scheduler Tests Source: https://github.com/atomology/dynamicworkflow.jl/blob/main/docs/superpowers/plans/2026-04-13-testitemrunner-migration.md Stage and commit the scheduler test file. ```bash git add test/test_scheduler.jl git commit -m "test: add scheduler testitem file" ``` -------------------------------- ### Test @job Macro with Plain Functions Source: https://github.com/atomology/dynamicworkflow.jl/blob/main/docs/superpowers/plans/2026-04-13-remove-jobcontext-from-user-api.md This test case verifies the behavior of the @job macro when used with plain Julia functions. It checks if jobs are created and fetched correctly, and if the JobContext is properly assigned. ```julia start_scheduler() try plain_mul(x, y) = x * y j1 = @job plain_mul(3, 4) j2 = @job plain_mul(5, 6) j3 = @job plain_mul(j1, j2) @test fetch(j1) == 12 @test fetch(j2) == 30 @test fetch(j3) == 360 @test j1.context.curr_id == j1.uuid finally stop_scheduler() end ``` -------------------------------- ### Commit Runner Changes Source: https://github.com/atomology/dynamicworkflow.jl/blob/main/docs/superpowers/plans/2026-04-13-testitemrunner-migration.md Stage and commit the updated runtests.jl file. ```bash git add test/runtests.jl git commit -m "test: switch runtests.jl to TestItemRunner" ``` -------------------------------- ### Implement current_context() TLS helper Source: https://github.com/atomology/dynamicworkflow.jl/blob/main/docs/superpowers/plans/2026-04-13-remove-jobcontext-from-user-api.md This function retrieves the `JobContext` from task-local storage. It returns `nothing` if no job context is found, indicating it's called outside a job. Ensure task-local storage is correctly managed. ```julia """ current_context() -> Union{Nothing, JobContext} Return the `JobContext` of the currently executing job, or `nothing` if called outside a job. Uses task-local storage for implicit context propagation. """ function current_context() return get(task_local_storage(), :job_context, nothing) end ``` -------------------------------- ### Document Core Types Source: https://github.com/atomology/dynamicworkflow.jl/blob/main/docs/src/api.md Lists the primary data structures used for managing jobs and tasks. ```julia @docs DynamicWorkflow.Job DynamicWorkflow.JobContext DynamicWorkflow.JobState DynamicWorkflow.OutputRef DynamicWorkflow.WTask DynamicWorkflow.JobScheduler DynamicWorkflow.SuccessResult DynamicWorkflow.FailResult DynamicWorkflow.Unassigned ``` -------------------------------- ### Define job status tests in Julia Source: https://github.com/atomology/dynamicworkflow.jl/blob/main/docs/superpowers/plans/2026-04-13-testitemrunner-migration.md Tests to verify job state transitions including pending, running, cancelled, and failed statuses. ```julia @testitem "job status" setup=[TestHelpers] begin using DynamicWorkflow using DynamicWorkflow: Q, PENDING, RUNNING, CANCELLED, COMPLETED, FAILED t = start_scheduler() try j1 = @job my_add(1, 2) j2 = @job my_sleep(3) j3 = @job my_add(1, j2) sleep(1) @test status(j1) == COMPLETED @test status(j2) == RUNNING @test status(j3) == PENDING cancel!(j3) @test status(j3) == CANCELLED sleep(3) @test allcomplete() j1 = @job bad_job() sleep(1) @test status(j1) == FAILED @test isqueuealive(Q[]) catch e throw(e) finally stop_scheduler() end end ``` -------------------------------- ### Commit changes for Job constructor rewrite Source: https://github.com/atomology/dynamicworkflow.jl/blob/main/docs/superpowers/plans/2026-04-13-remove-jobcontext-from-user-api.md This bash command stages and commits the modifications to `src/job.jl` related to rewriting the `Job` constructor to use TLS for parent tracking. ```bash git add src/job.jl git commit -m "rewrite Job constructor to use TLS for parent tracking" ``` -------------------------------- ### Run Focused Tests Source: https://github.com/atomology/dynamicworkflow.jl/blob/main/CLAUDE.md Run specific test files within the test suite. Use -t 2 for multithreading, as the scheduler relies on it. ```julia julia --project=. -t 2 test/runtests.jl ``` -------------------------------- ### Hierarchical Layout for Graphs Source: https://github.com/atomology/dynamicworkflow.jl/blob/main/docs/superpowers/plans/2026-04-13-job-graph-visualization.md Implements a hierarchical layout algorithm for directed acyclic graphs (DAGs). This layout is suitable for visualizing job dependencies where layers represent stages of execution. Ensure the graph is a DAG for correct layering. ```julia function hierarchical_layout(g::AbstractGraph) n = nv(g) n == 0 && return Point2f[] layers = zeros(Int, n) for v in reverse(topological_sort_by_dfs(g)) for u in inneighbors(g, v) layers[v] = max(layers[v], layers[u] + 1) end end max_layer = maximum(layers; init=0) layer_nodes = [Int[] for _ in 0:max_layer] for v in 1:n push!(layer_nodes[layers[v] + 1], v) end positions = Vector{Point2f}(undef, n) for (li, nodes) in enumerate(layer_nodes) y = Float32(-(li - 1)) for (i, v) in enumerate(nodes) x = Float32(i) / Float32(length(nodes) + 1) positions[v] = Point2f(x, y) end end return positions end ``` -------------------------------- ### Update Spawning Child Jobs Testset Source: https://github.com/atomology/dynamicworkflow.jl/blob/main/docs/superpowers/plans/2026-04-13-remove-jobcontext-from-user-api.md Replace the existing 'spawning child jobs' testset with an updated version. This new version defines and tests job spawning with plain functions. ```julia @testset "spawning child jobs" begin t = start_scheduler() try function spawn_jobs() jobs = Job[] for i in 1:3 j = @job my_add(1, i) push!(jobs, j) end return jobs end w = @job spawn_jobs() jobs = fetch(w) sleep(1) uuids = map(j->j.uuid, jobs) @test w.uuid == context(w).curr_id @test isnothing(context(w).parent_id) @test uuids == context(w.uuid).child_ids @test all([context(j).parent_id == w.uuid for j in jobs]) add_jobs(a, b, c) = a + b + c j = @job add_jobs(jobs[1], jobs[2], jobs[3]) sleep(1) @test fetch(j) == 9 @test allcomplete() catch e throw(e) finally stop_scheduler() end end ``` -------------------------------- ### Test Nested Child Job Spawning Source: https://github.com/atomology/dynamicworkflow.jl/blob/main/docs/superpowers/plans/2026-04-13-testitemrunner-migration.md Tests multi-level child job spawning, where a job spawns another job which then spawns multiple child jobs. Verifies context and parent-child relationships across levels. ```julia using DynamicWorkflow using DynamicWorkflow: context, Q t = start_scheduler() try function spawn_jobs1() @job spawn_jobs2() end function spawn_jobs2() jobs = Job[] for i in 1:3 j = @job my_add(1, i) push!(jobs, j) end return jobs end w1 = @job spawn_jobs1() wait(w1) w2 = result(w1) jobs = result(w2) uuids = map(j -> j.uuid, jobs) @test w1.uuid == context(w1).curr_id @test w2.uuid == context(w2).curr_id @test isnothing(context(w1).parent_id) @test context(w1).child_ids[1] == w2.uuid @test length(context(w1).child_ids) == 1 @test context(w2).parent_id == w1.uuid @test uuids == context(w2).child_ids @test all([context(j).parent_id == w2.uuid for j in jobs]) add_jobs(a, b, c) = a + b + c j = @job add_jobs(jobs[1], jobs[2], jobs[3]) wait(j) @test fetch(j) == 9 @test allcomplete() catch e throw(e) finally stop_scheduler() end ``` -------------------------------- ### Test Node Width Calculation Source: https://github.com/atomology/dynamicworkflow.jl/blob/main/docs/superpowers/plans/2026-04-13-job-graph-visualization.md This test suite checks the `node_width` function, ensuring it correctly calculates the width for both short labels (hitting a minimum) and long labels (based on character count and a defined character width). Assumes `DynamicWorkflow.NODE_MIN_WIDTH` and `DynamicWorkflow.CHAR_WIDTH` are defined. ```julia @testset "node_width" begin # Short labels hit the minimum width @test DynamicWorkflow.node_width("hi\nab") == DynamicWorkflow.NODE_MIN_WIDTH # Long labels exceed minimum long_label = "a_very_long_function_name\nab12f" expected = max(DynamicWorkflow.NODE_MIN_WIDTH, length("a_very_long_function_name") * DynamicWorkflow.CHAR_WIDTH) @test DynamicWorkflow.node_width(long_label) == expected end ``` -------------------------------- ### Update Job run! to Set TLS Context in Julia Source: https://github.com/atomology/dynamicworkflow.jl/blob/main/docs/superpowers/plans/2026-04-13-remove-jobcontext-from-user-api.md This function is modified to wrap the job's task execution within a `task_local_storage` context. This ensures that the job's context is correctly set in TLS before the actual function is invoked, enabling proper tracking. ```julia function run!(job::Job) ctx = job.context original_f = job.task.f original_args = job.task.args # Wrap execution to set TLS context wrapped() = task_local_storage(:job_context, ctx) do invokelatest(original_f, original_args...) end job.task.task = Task(wrapped) job.task.task.sticky = false schedule(job.task.task) yield() end ``` -------------------------------- ### Define Node Dimensions and Character Width Source: https://github.com/atomology/dynamicworkflow.jl/blob/main/docs/superpowers/plans/2026-04-13-job-graph-visualization.md Constants for node dimensions and character width are defined. `NODE_MIN_WIDTH` sets a minimum width for nodes, and `CHAR_WIDTH` estimates the pixel width of a single monospace character, used for calculating label-based widths. ```julia const NODE_HEIGHT = 30 const NODE_MIN_WIDTH = 60 const CHAR_WIDTH = 7 # approximate pixels per monospace character ``` -------------------------------- ### Execute Parallel Workloads Source: https://context7.com/atomology/dynamicworkflow.jl/llms.txt Distribute data across multiple parallel child jobs using list comprehensions. ```julia using DynamicWorkflow start_scheduler() function process_chunk(data_chunk) println("Processing chunk of size ", length(data_chunk)) sleep(0.1) # Simulate computation return sum(data_chunk) / length(data_chunk) end function create_parallel_jobs(chunks) # Spawn parallel jobs using list comprehension return [@job process_chunk(chunk) for chunk in chunks] end # Create sample data and split into chunks data = rand(1000) chunk_size = 200 chunks = [data[i:min(i + chunk_size - 1, end)] for i in 1:chunk_size:length(data)] # Parent job spawns parallel processing jobs master_job = @job create_parallel_jobs(chunks) chunk_jobs = fetch(master_job) # Returns vector of child jobs # Collect results from all parallel jobs chunk_means = [fetch(j) for j in chunk_jobs] final_mean = sum(chunk_means) / length(chunk_means) println("Parallel result: ", final_mean) println("Expected: ", sum(data) / length(data)) stop_scheduler() ``` -------------------------------- ### Job Status Colors and Labels Source: https://github.com/atomology/dynamicworkflow.jl/blob/main/docs/superpowers/plans/2026-04-13-job-graph-visualization.md Defines color schemes and label generation for job statuses and nodes in the graph visualization. `status_color` and `status_text_color` map job states to display colors. `node_label` creates a label with job name and short UUID. `node_width` calculates the width based on label content. ```julia const STATUS_COLORS = Dict{JobState,Tuple{String,String}}( PENDING => ("#CCCCCC", "#333333"), RUNNING => ("#0072B2", "white"), COMPLETED => ("#009E73", "white"), FAILED => ("#D55E00", "white"), CANCELLED => ("#CC79A7", "white"), ) status_color(s::JobState) = STATUS_COLORS[s][1] status_text_color(s::JobState) = STATUS_COLORS[s][2] ``` ```julia const NODE_HEIGHT = 30 const NODE_MIN_WIDTH = 60 const CHAR_WIDTH = 7 function node_label(j::Job) short_id = string(j.uuid)[end-4:end] return "$(j.name)\n$(short_id)" end function node_width(label::String) max_chars = maximum(length(line) for line in split(label, '\n')) return max(NODE_MIN_WIDTH, max_chars * CHAR_WIDTH) end ``` -------------------------------- ### Check Job Status Source: https://context7.com/atomology/dynamicworkflow.jl/llms.txt Retrieve the current state of a job, such as PENDING, RUNNING, or COMPLETED. ```julia using DynamicWorkflow start_scheduler() slow_task(n) = (sleep(n); n * 2) j = @job slow_task(1) # Check status immediately (likely pending or running) println(status(j)) # Output: PENDING or RUNNING # Wait a moment sleep(1.5) # Check status after completion println(status(j)) # Output: COMPLETED # Status values display with emoji indicators: # 🚧PENDING, šŸƒRUNNING, āœ…COMPLETED, āŒFAILED, ā­•CANCELLED stop_scheduler() ``` -------------------------------- ### Define and Submit Jobs with @job Source: https://context7.com/atomology/dynamicworkflow.jl/llms.txt Use the @job macro to wrap function calls and automatically resolve dependencies when passing Job objects as arguments. ```julia using DynamicWorkflow start_scheduler() # Define simple functions (no special arguments needed) my_add(x, y) = x + y my_multiply(x, y) = x * y # Create independent jobs j1 = @job my_add(2, 3) # Returns Job object immediately j2 = @job my_add(4, 5) # Runs in parallel with j1 # Create dependent job - j3 waits for j1 and j2 to complete j3 = @job my_multiply(j1, j2) # j1 and j2 are auto-converted to OutputRef # Jobs are submitted immediately and run asynchronously println(j1) # Output: Job("my_add", abc12345, status=🚧PENDING) stop_scheduler() ``` -------------------------------- ### Configure TestItemRunner Runner Source: https://github.com/atomology/dynamicworkflow.jl/blob/main/docs/superpowers/plans/2026-04-13-testitemrunner-migration.md Replace the contents of test/runtests.jl to use @run_package_tests with an optional filter for test execution. ```julia using TestItemRunner @run_package_tests filter = ti -> isempty(ARGS) || any(occursin(pat, ti.filename) for pat in ARGS) ``` -------------------------------- ### Define Constants for Node Rendering Source: https://github.com/atomology/dynamicworkflow.jl/blob/main/docs/superpowers/specs/2026-04-13-job-graph-visualization-design.md These constants define the dimensions and character width for rendering rectangular nodes, ensuring consistent sizing based on content. ```julia NODE_HEIGHT = 30 NODE_MIN_WIDTH = 60 CHAR_WIDTH = 7 # approximate px per character in monospace ``` -------------------------------- ### fetch Source: https://context7.com/atomology/dynamicworkflow.jl/llms.txt Blocking call that waits for a job to complete and returns its result. ```APIDOC ## fetch ### Description Blocking call that waits for a job to complete and returns its result value. If the job fails, returns the error. ### Method Function Call ### Parameters #### Path Parameters - **j** (Job) - Required - The job object to wait for. ### Response - **Any** (Value/Error) - The result of the job or the error encountered. ``` -------------------------------- ### Draw Graph with Hierarchical Layout Source: https://github.com/atomology/dynamicworkflow.jl/blob/main/docs/superpowers/specs/2026-04-13-job-graph-visualization-design.md The `draw_graph` function signature remains unchanged, but the default layout is updated to `hierarchical_layout` for improved visualization. ```julia draw_graph(q::JobQueue; layout=hierarchical_layout) draw_graph(; layout=hierarchical_layout) # uses global Q[] ```