### Verify setup: git, docker compose, uv, curl Source: https://schemathesis.readthedocs.io/en/stable/tutorials/cli Commands to verify that Git, Docker Compose, uv, and optionally curl are installed. Run these before starting the tutorial. ```bash git --version docker compose version uv --version curl --version # optional ``` -------------------------------- ### Setup: Install TraceCov, enable hooks, and run Schemathesis Source: https://schemathesis.readthedocs.io/en/stable/guides/coverage Install TraceCov, create a hooks.py file that installs the Schemathesis integration, then run Schemathesis with the SCHEMATHESIS_HOOKS environment variable set to the hooks file. The command generates an HTML report at ./schema-coverage.html after tests complete. ```text uv pip install tracecov ``` ```python # hooks.py import tracecov tracecov.schemathesis.install() ``` ```text SCHEMATHESIS_HOOKS=hooks schemathesis run https://api.example.com/openapi.json ``` -------------------------------- ### Clone and start Booking API Source: https://schemathesis.readthedocs.io/en/stable/tutorials/pytest Clones the Schemathesis repository, navigates to the booking example, and starts the API with Docker Compose. The API will be available at http://localhost:8080/docs. ```bash git clone https://github.com/schemathesis/schemathesis.git cd schemathesis/examples/booking docker compose up -d --build ``` -------------------------------- ### Verify setup versions Source: https://schemathesis.readthedocs.io/en/stable/tutorials/pytest Checks that git, Docker Compose, Python, pytest, and Schemathesis are installed and accessible. ```bash git --version docker compose version python --version pytest --version schemathesis --version ``` -------------------------------- ### Example output from the extension Source: https://schemathesis.readthedocs.io/en/stable/guides/extending-cli Shows the expected output from running the extension with the `--counter-verbose` flag. It includes verbose test start and pass/fail messages, followed by a summary block with total events, test cases, failures, errors, and the output file path. ```text Starting test #1 ✓ Test #1 passed Starting test #2 ✗ Test #2 failed Counter Summary: Total events: 125 Test cases: 2 Failures: 1 Errors: 0 Results written to: results.txt ``` -------------------------------- ### Set example database storage with --generation-database Source: https://schemathesis.readthedocs.io/en/stable/reference/cli Specifies storage for discovered examples, such as memory or a file path. ```bash $ st run openapi.yaml --generation-database ":memory:" $ st run openapi.yaml --generation-database ./schemathesis_examples.db ``` -------------------------------- ### Property-level examples Source: https://schemathesis.readthedocs.io/en/stable/explanations/examples Shows property-level examples for multiple properties in an OpenAPI schema. ```yaml # Property-level example properties: name: type: string example: "John Doe" age: type: integer example: 30 ``` -------------------------------- ### Examples Phase: Schema with examples Source: https://schemathesis.readthedocs.io/en/stable/explanations/data-generation Uses the `example` and `examples` keywords from the schema, filling missing parts with defaults or generated data. The example shows a query parameter `limit` with examples [10, 50, 100], producing three test cases. ```yaml # Schema parameters: - name: limit in: query schema: type: integer examples: [10, 50, 100] # Produces: 3 test cases with limit=10, limit=50, limit=100 ``` -------------------------------- ### Single example with 'example' keyword Source: https://schemathesis.readthedocs.io/en/stable/explanations/examples Defines a single example value for a property using the 'example' keyword in OpenAPI 3.0+. ```yaml # Single example using the 'example' keyword schema: type: object properties: name: type: string example: "John Doe" ``` -------------------------------- ### Customize state machine with setup, teardown, before_call, and after_call Source: https://schemathesis.readthedocs.io/en/stable/guides/stateful-testing Extends the state machine class to add custom behavior: setup runs once at the start of each scenario, teardown at the end, before_call modifies every request, and after_call processes every response. The state machine still handles operation sequencing automatically. ```python import schemathesis schema = schemathesis.openapi.from_url("http://localhost:8000/openapi.json") class APIWorkflow(schema.as_state_machine()): def setup(self): """Run once at the start of each test scenario.""" def teardown(self): """Run once at the end of each test scenario.""" def before_call(self, case): """Modify every request in the sequence.""" def after_call(self, response, case): """Process every response.""" TestAPI = APIWorkflow.TestCase ``` -------------------------------- ### Complete Example with pytest Fixture Source: https://schemathesis.readthedocs.io/en/stable/guides/schema-conformance A complete test workflow using a pytest fixture to load the schema and validate both create and get responses. ```python import pytest import schemathesis @pytest.fixture(scope="session") def api_schema(): return schemathesis.openapi.from_url("http://api.example.com/openapi.json") def test_user_workflow(api_schema): create_response = requests.post("http://api.example.com/users", json={"name": "Test"}) user_id = create_response.json()["id"] api_schema["/users"]["POST"].validate_response(create_response) get_response = requests.get(f"http://api.example.com/users/{user_id}") api_schema["/users/{id}"]["GET"].validate_response(get_response) ``` -------------------------------- ### Data initialization in setup() Source: https://schemathesis.readthedocs.io/en/stable/guides/stateful-testing Overrides the setup() method to create realistic test data, such as a user, before each scenario. The created resource ID is stored for use in operations that require it during before_call(). ```python class APIWorkflow(schema.as_state_machine()): def setup(self): # Create a test user for this scenario case = schema["/users"]["POST"].Case(body={"username": "test_user", "email": "test@example.com"}) response = case.call() self.user_id = response.json()["id"] def before_call(self, case): # Use the created user in operations that need user_id if "user_id" in case.path_parameters: case.path_parameters["user_id"] = self.user_id ``` -------------------------------- ### Install dependencies with uv Source: https://schemathesis.readthedocs.io/en/stable/tutorials/pytest Creates a virtual environment with Python 3.13 and installs pytest and schemathesis. The source command may need adjustment based on your shell. ```bash uv venv -p 3.13 source .venv/bin/activate.fish uv pip install -U pytest schemathesis ``` -------------------------------- ### Async test functions with pytest-asyncio and pytest-trio Source: https://schemathesis.readthedocs.io/en/stable/explanations/pytest Shows how to write asynchronous test functions with Schemathesis. Requires installing pytest-asyncio or pytest-trio; no additional configuration is needed. The example includes both an asyncio-marked test and a trio-marked test. ```python import pytest import schemathesis schema = schemathesis.openapi.from_url("http://127.0.0.1:8080/openapi.json") @pytest.mark.asyncio @schema.parametrize() async def test_api_async(case, client): response = await client.request(case.method, case.formatted_path, headers=case.headers) schema[case.path][case.method].validate_response(response) # Or with trio @pytest.mark.trio @schema.parametrize() async def test_api_trio(case, client): ... ``` -------------------------------- ### Location header learning example Source: https://schemathesis.readthedocs.io/en/stable/explanations/stateful Example showing a POST /users response with a Location header and the learned follow-up operations. ```http POST /users → 201 Created Location: /users/123 # Learns: GET /users/123, PUT /users/123, DELETE /users/123 ``` -------------------------------- ### Configure generation.database Source: https://schemathesis.readthedocs.io/en/stable/reference/configuration Configures storage for discovered examples. Examples show in-memory storage and persistent file storage. ```toml [generation] database = ":memory:" ``` ```toml [generation] database = "./.schemathesis/examples/" ``` -------------------------------- ### Multiple examples strategy with round-robin Source: https://schemathesis.readthedocs.io/en/stable/explanations/examples Schema with multiple examples for 'age' and the round-robin generated test cases using each age value. ```yaml # Schema properties: name: type: string example: "John" age: type: integer examples: [25, 30, 35] ``` ```json {"name": "John", "age": 25, ...} {"name": "John", "age": 30, ...} {"name": "John", "age": 35, ...} ``` -------------------------------- ### Step 4: Generate test scenarios - random workflows Source: https://schemathesis.readthedocs.io/en/stable/explanations/stateful Lists example random workflows that Schemathesis runs, such as create, get, update, delete sequences. ```text Create user -> Get user -> Update user -> Delete user Create user -> Create order for that user -> Get order Create user -> Delete user -> Get user ``` -------------------------------- ### OpenAPI 2.0 multiple examples with x-examples Source: https://schemathesis.readthedocs.io/en/stable/explanations/examples Defines multiple examples in OpenAPI 2.0 using the 'x-examples' extension. ```yaml # OpenAPI 2.0 with multiple examples definitions: User: type: object properties: name: type: string x-examples: - name: "John Doe" - name: "Jane Smith" ``` -------------------------------- ### Manual OpenAPI Links example Source: https://schemathesis.readthedocs.io/en/stable/explanations/stateful Example OpenAPI schema with a link definition to explicitly connect POST /users to getUser operation. ```yaml paths: /users: post: responses: '201': links: GetUser: operationId: getUser parameters: userId: '$response.body#/id' ``` -------------------------------- ### OpenAPI schema example for automatic schema analysis Source: https://schemathesis.readthedocs.io/en/stable/explanations/stateful Example OpenAPI schema showing POST /users and GET /users/{userId} to illustrate how Schemathesis detects relationships and builds sequences. ```yaml paths: /users: post: responses: '201': content: application/json: schema: properties: id: {type: string} email: {type: string} /users/{userId}: get: parameters: - name: userId in: path ``` -------------------------------- ### Example output from demo run Source: https://schemathesis.readthedocs.io/en/stable/quick-start Example output from the demo run showing a server error and a reproduction command. The output includes the failing operation, the 500 response, and a curl command to reproduce the failure. ```text _____________________ POST /improper-input-type-handling _____________________ - Server error [500] Internal Server Error: `{"success":false,"error":"invalid literal for int() with base 10: '\n'"}` Reproduce with: curl -X POST -H 'Content-Type: application/json' \ -d '{"number": "\n\udbcd." }' \ https://example.schemathesis.io/improper-input-type-handling ``` -------------------------------- ### Schema with explicit examples for path parameter Source: https://schemathesis.readthedocs.io/en/stable/guides/coverage Adds explicit examples to a path parameter schema to cover boundary values. The 'examples' map uses keys as labels and 'value' for the actual value. ```yaml /users/{id}: get: parameters: - name: id in: path schema: type: integer minimum: 1 examples: existing: value: 42 boundary: value: 1 ``` -------------------------------- ### Enable fill-missing for examples Source: https://schemathesis.readthedocs.io/en/stable/reference/configuration Enables the generation of examples for API operations that lack explicit schema examples. ```toml [phases.examples] fill-missing = true ``` -------------------------------- ### Separating Schema Examples and Custom Strategies Source: https://schemathesis.readthedocs.io/en/stable/guides/hypothesis-strategies When schema examples exist, avoid parameter mismatches by creating separate test functions for schema examples and property-based testing with custom strategies. ```python from hypothesis import Phase, settings # 1. One for schema examples (without @schema.given()): @schema.parametrize() @settings(phases=[Phase.explicit]) def test_user_endpoints_with_examples(case): case.call_and_validate() # 2. One for property-based testing with your custom strategies: @schema.given(user_id=st.sampled_from(existing_user_ids)) @schema.parametrize() @settings(phases=[Phase.generate]) def test_user_endpoints_with_custom_data(case, user_id): if "user_id" in case.path_parameters: case.path_parameters["user_id"] = user_id case.call_and_validate() ``` -------------------------------- ### Schema example and generated test case Source: https://schemathesis.readthedocs.io/en/stable/explanations/examples Schema with an example for 'name' and the generated test case where 'John' comes from the example and other values are minimal valid values. ```yaml # Schema schema: type: object properties: name: type: string example: "John" age: type: integer address: type: string ``` ```json {"name": "John", "age": 42, "address": "abc"} ``` -------------------------------- ### Authentication flow with login and token headers Source: https://schemathesis.readthedocs.io/en/stable/guides/stateful-testing Overrides setup() to perform a login and store the bearer token, then adds it to every request in before_call() by merging into case.headers. Expects the login endpoint at http://localhost:8000/auth/login. ```python import requests class APIWorkflow(schema.as_state_machine()): def setup(self): # Login and get auth token response = requests.post( "http://localhost:8000/auth/login", json={"username": "test_user", "password": "test_password"} ) token = response.json()["access_token"] self.auth_headers = {"Authorization": f"Bearer {token}"} def before_call(self, case): # Add auth to every request case.headers = {**case.headers, **self.auth_headers} ``` -------------------------------- ### Example output of st replay with fixed and failed cases Source: https://schemathesis.readthedocs.io/en/stable/guides/crash-reproduction Example output from `st replay` showing one fixed and one failed case, with failure details and reproduction commands. ```text Replaying 2 cases from .schemathesis/crashes/ + FIXED GET /users x FAILED POST /orders =================================== FAILURES =================================== _________________________________ POST /orders _________________________________ 1. Test Case ID: Xt9Kp2 - Server error - Undocumented HTTP status code Received: 500 Documented: 200 [500] Internal Server Error: `{"error": "unhandled exception"}` Reproduce with: curl -X POST http://127.0.0.1/orders st replay Xt9Kp2 ========================== 1 fixed, 1 failed in 0.14s ========================== ``` -------------------------------- ### Per-Run Setup with TestCase setUp/tearDown Source: https://schemathesis.readthedocs.io/en/stable/guides/stateful-testing Extend the test class to define per-run setup and teardown. These methods run once for the entire test, regardless of how many scenarios Hypothesis generates. ```python class TestAPI(APIWorkflow.TestCase): def setUp(self): """Create database, start services - runs once per test execution.""" def tearDown(self): """Cleanup resources - runs once per test execution.""" ``` -------------------------------- ### Install Schemathesis with Allure support Source: https://schemathesis.readthedocs.io/en/stable/guides/allure Adds the optional Allure extra to Schemathesis, installing allure-python-commons. Requires the Allure CLI separately to generate and view HTML reports. ```bash uv add 'schemathesis[allure]' ``` -------------------------------- ### Multiple examples with externalValue Source: https://schemathesis.readthedocs.io/en/stable/explanations/examples Uses the 'examples' keyword with an externalValue reference. Schemathesis loads and caches external examples during testing. ```yaml content: application/json: schema: $ref: '#/components/schemas/User' examples: user: summary: "A typical user" # Schemathesis will load and cache external examples during testing. externalValue: 'http://example.com/examples/user.json' ``` -------------------------------- ### Operation-level example Source: https://schemathesis.readthedocs.io/en/stable/explanations/examples Defines an example at the operation level for a request body in OpenAPI 3.0+. ```yaml # Operation-level example requestBody: content: application/json: schema: $ref: '#/components/schemas/User' example: name: "John Doe" age: 30 ``` -------------------------------- ### Setting Up and Cleaning Up Database Resources Source: https://schemathesis.readthedocs.io/en/stable/guides/hypothesis-strategies Create and clean up resources for each Hypothesis example to prevent test pollution. Use a try-finally block to ensure cleanup occurs after each test case. ```python @schema.given( user_data=st.fixed_dictionaries( { "name": st.text(min_size=1, max_size=50), "email": st.emails(), "role": st.sampled_from(["user", "admin"]), } ) ) @schema.parametrize() def test_api_with_db_setup(db, case, user_data): # Create user in database for each Hypothesis example user_id = db.create_user(user_data) try: # Use the created user in API tests if "user_id" in case.path_parameters: case.path_parameters["user_id"] = user_id case.call_and_validate() finally: # Important: cleanup after each example db.cleanup_user(user_id) ``` -------------------------------- ### String-based scalar examples Source: https://schemathesis.readthedocs.io/en/stable/guides/graphql-custom-scalars Examples of registering string-based scalars using email, URL, and regex strategies. ```python schemathesis.graphql.scalar("Email", st.emails().map(nodes.String)) schemathesis.graphql.scalar("URL", st.urls().map(nodes.String)) schemathesis.graphql.scalar("Phone", st.from_regex(r"\+1-\d{3}-\d{3}-\d{4}").map(nodes.String)) ``` -------------------------------- ### GraphQL schema example for automatic schema analysis Source: https://schemathesis.readthedocs.io/en/stable/explanations/stateful Example GraphQL schema with Book type, addBook mutation, deleteBook mutation, and book query to illustrate how Schemathesis builds chains automatically. ```graphql type Book { id: ID! title: String! } type Mutation { addBook(title: String!): Book! # Producer of Book deleteBook(id: ID!): Boolean # Cleanup of Book } type Query { book(id: ID!): Book # Consumer of Book } ``` -------------------------------- ### Numeric scalar examples Source: https://schemathesis.readthedocs.io/en/stable/guides/graphql-custom-scalars Examples of registering numeric scalars using integer and decimal strategies. ```python schemathesis.graphql.scalar("Percentage", st.integers(0, 100).map(nodes.Int)) schemathesis.graphql.scalar("Price", st.decimals(min_value=0, max_value=1000, places=2).map(nodes.Float)) ``` -------------------------------- ### Shrinking example: before and after Source: https://schemathesis.readthedocs.io/en/stable/explanations/data-generation Illustrates how Schemathesis shrinks a failing test case to the minimal example that reproduces the failure. Shrinking is enabled by default; disable with --no-shrink for faster test runs. ```json {"name": "Very long user name", "age": 42, "metadata": {...}} ``` ```json {"name": "a", "age": 42} # Only data that triggers a failure ``` -------------------------------- ### Set up custom auth hooks Source: https://schemathesis.readthedocs.io/en/stable/guides/auth Exports the SCHEMATHESIS_HOOKS environment variable to enable the custom auth hooks, then runs schemathesis. This is the standard setup for custom authentication classes. ```bash export SCHEMATHESIS_HOOKS=auth schemathesis run http://localhost:8000/openapi.json ``` -------------------------------- ### Example output of a partially fixed failure Source: https://schemathesis.readthedocs.io/en/stable/guides/crash-reproduction Example output showing a case that still fails after some checks were fixed; only the fixed checks' files are removed. ```text x FAILED POST /orders + not_a_server_error + status_code_conformance x response_schema_conformance ``` -------------------------------- ### Generate Allure report with --report-allure-path Source: https://schemathesis.readthedocs.io/en/stable/faq Generate Allure-compatible results with Schemathesis and create an Allure report. Requires installing schemathesis[allure] and the Allure CLI. ```bash schemathesis run your_schema.yaml --report-allure-path allure-results allure generate allure-results -o allure-report allure open allure-report ``` -------------------------------- ### Run Schemathesis on demo API with uvx Source: https://schemathesis.readthedocs.io/en/stable/quick-start Runs Schemathesis against the demo API using uvx, which executes in an isolated environment without a permanent install. Replace `uvx schemathesis` with `schemathesis` or `st` if installed directly. ```bash uvx schemathesis run https://example.schemathesis.io/openapi.json ``` -------------------------------- ### Configure generation.maximize Source: https://schemathesis.readthedocs.io/en/stable/reference/configuration Guides input generation to values more likely to expose bugs. Setting to "response_time" focuses on generating inputs that maximize response time. ```toml [generation] maximize = "response_time" ``` -------------------------------- ### Example output of a flaky replay Source: https://schemathesis.readthedocs.io/en/stable/guides/crash-reproduction Example output showing a flaky case where a check passed on some replays and failed on others. ```text ? FLAKY GET /report ? not_a_server_error passed 2 of 3 replays ``` -------------------------------- ### Positive Testing: Valid string examples Source: https://schemathesis.readthedocs.io/en/stable/explanations/data-generation Generates data that should be accepted by the API, valid according to the schema. The example schema `{"type": "string", "minLength": 3}` produces positive examples like "abc", "hello", "test123". ```json # Schema: {"type": "string", "minLength": 3} # Positive examples: "abc", "hello", "test123" ``` -------------------------------- ### Configure max examples Source: https://schemathesis.readthedocs.io/en/stable/reference/configuration Set the maximum number of test cases generated per API operation. ```toml [generation] max-examples = 200 ``` -------------------------------- ### Deeper testing for important API operations Source: https://schemathesis.readthedocs.io/en/stable/configuration Sets a default of 100 examples, but increases to 500 for operations matching the path regex '/(payments|users)/'. ```toml # Default: fast feedback generation.max-examples = 100 # Important operations: more depth [[operations]] include-path-regex = "/(payments|users)/" generation.max-examples = 500 ``` -------------------------------- ### Initialize baseline with schemathesis run and configure path in schemathesis.toml Source: https://schemathesis.readthedocs.io/en/stable/guides/baseline Creates a baseline file with `--baseline` and configures Schemathesis to use it via the `baseline` key in `schemathesis.toml`. The file must not exist yet, otherwise the run would update it instead of writing it. ```shell schemathesis run http://localhost:8080/openapi.json --baseline schemathesis-baseline.json ``` ```toml baseline = "schemathesis-baseline.json" ``` -------------------------------- ### More tests (better coverage, slower) Source: https://schemathesis.readthedocs.io/en/stable/configuration Increases the maximum number of examples to 1000 for better coverage at the cost of speed. ```toml generation.max-examples = 1000 ``` -------------------------------- ### APIStateMachine.setup() Source: https://schemathesis.readthedocs.io/en/stable/reference/python Hook called once at the beginning of each test scenario. ```APIDOC ## APIStateMachine.setup() ### Description Called once at the beginning of each test scenario. ### Method N/A (method on state machine) ### Endpoint N/A ### Parameters None ### Returns None ### Response Example N/A ``` -------------------------------- ### Export Environment Variables for Config Source: https://schemathesis.readthedocs.io/en/stable/guides/auth Export the environment variables referenced in the config file and run Schemathesis. This example sets API_TOKEN and runs the CLI. ```bash export API_TOKEN="your-secret-token" schemathesis run http://localhost:8000/openapi.json ``` -------------------------------- ### Stateful Phase: Schema with links Source: https://schemathesis.readthedocs.io/en/stable/explanations/data-generation Runs when OpenAPI schemas define links between operations, creating sequences where response data feeds into subsequent requests. The example shows a link from POST /users to GET /users/{id}, producing a sequence that extracts the ID and uses it in the GET request. ```yaml # Schema with links: POST /users → GET /users/{id} # Produces: POST /users, extract ID, then GET /users/{extracted_id} ``` -------------------------------- ### Example output showing selected operations Source: https://schemathesis.readthedocs.io/en/stable/guides/cicd Output from a job reporting the number of selected operations out of total. This helps verify that the split groups add up. ```text Operations: 3 selected / 5 total ``` -------------------------------- ### Fewer tests (faster feedback) Source: https://schemathesis.readthedocs.io/en/stable/configuration Decreases the maximum number of examples to 50 for faster feedback. ```toml generation.max-examples = 50 ``` -------------------------------- ### Reusable Authentication Config File Source: https://schemathesis.readthedocs.io/en/stable/guides/auth Use a config file for reusable authentication configuration. The example shows global headers and per-operation headers for specific endpoints, with environment variable placeholders. ```toml # schemathesis.toml headers = { Authorization = "Bearer ${API_TOKEN}" } # Different auth for specific endpoints [[operations]] include-path = "/admin/" headers = { Authorization = "Bearer ${ADMIN_TOKEN}", X-Client-ID = "${CLIENT_ID}" } ``` -------------------------------- ### Control test phases with --phases Source: https://schemathesis.readthedocs.io/en/stable/migration Use --phases to select which test phases to run, e.g., examples and coverage, comma-separated. ```bash schemathesis run --phases=examples,coverage ``` -------------------------------- ### content_type_conformance failure: missing header Source: https://schemathesis.readthedocs.io/en/stable/reference/checks Example failure message when the `Content-Type` header is missing. The documented media types are listed. ```text - Missing Content-Type header The following media types are documented in the schema: - application/json ``` -------------------------------- ### Targeting Specific Operations with Hooks Source: https://schemathesis.readthedocs.io/en/stable/guides/extending Use @schemathesis.hook.apply_to to restrict hook execution to specific paths or operations. The first example skips POST requests for user endpoints, while the second targets a specific GET operation. ```python # Only apply to user endpoints, skip POST requests @schemathesis.hook.apply_to(path_regex=r"/users/").skip_for(method="POST") def map_headers(ctx, headers): headers = headers or {} headers["X-User-Context"] = "test-user" return headers # Only apply to a specific operation @schemathesis.hook.apply_to(name="GET /orders/{order_id}") def map_path_parameters(ctx, path_parameters): path_parameters = path_parameters or {} path_parameters["order_id"] = "order_12345" return path_parameters ``` -------------------------------- ### Serialization example for JSON and XML Source: https://schemathesis.readthedocs.io/en/stable/explanations/data-generation Shows a generated Python object and its serialized forms for application/json and application/xml. If Schemathesis cannot serialize data for a media type, those test cases are skipped. ```python # Generated Python object {"user_id": 123, "name": "test"} # For application/json -> {"user_id": 123, "name": "test"} # For application/xml -> 123test ``` -------------------------------- ### Step 2: Match parameters - link parameters to resources Source: https://schemathesis.readthedocs.io/en/stable/explanations/stateful Shows how parameters are linked to resources, e.g., userId from User resource and orderId from Order resource. ```text GET /users/{userId} → Needs "userId" from User resource GET /orders/{orderId} → Needs "orderId" from Order resource ``` -------------------------------- ### GitLab CI complete workflow example Source: https://schemathesis.readthedocs.io/en/stable/guides/cicd Defines a GitLab CI pipeline stage that runs Schemathesis against an API service. Uses the official Docker image and a JUnit report artifact. ```yaml stages: - test api-tests: stage: test image: name: schemathesis/schemathesis:stable entrypoint: [""] services: - name: your-api:latest alias: api variables: API_TOKEN: "your-secret-token" script: - > schemathesis run http://api:8080/openapi.json --header "Authorization: Bearer $API_TOKEN" --wait-for-schema 60 --report junit artifacts: when: always reports: junit: schemathesis-report/junit-*.xml paths: - schemathesis-report/ ``` -------------------------------- ### Run example-based tests with --phases=examples Source: https://schemathesis.readthedocs.io/en/stable/explanations/examples Runs example-based tests only using the --phases=examples option, skipping other testing phases like coverage, fuzzing, and stateful testing. ```bash $ st run --phases=examples https://example.schemathesis.io/openapi.json ``` -------------------------------- ### Run Schemathesis with recommended settings Source: https://schemathesis.readthedocs.io/en/stable/guides/config-optimization Recommended configuration for maximum bug detection and API coverage. Use when preparing for production releases, conducting security assessments, or when time allows for extensive testing. For fast development feedback, stick with defaults. ```bash schemathesis run \ --max-examples 1000 \ --continue-on-failure ``` -------------------------------- ### Extending State Machine in Fixture Source: https://schemathesis.readthedocs.io/en/stable/guides/stateful-testing Extend the state machine inside a fixture to use fixture dependencies. The setup method obtains a token and before_call adds an Authorization header to each request. ```python @pytest.fixture def state_machine(api_schema, auth_service): class APIWorkflow(api_schema.as_state_machine()): def setup(self): # Use fixture dependencies self.token = auth_service.get_test_token() def before_call(self, case): case.headers["Authorization"] = f"Bearer {self.token}" return APIWorkflow ``` -------------------------------- ### Load custom configuration file with schemathesis.Config.from_path Source: https://schemathesis.readthedocs.io/en/stable/tutorials/pytest Shows how to load a custom TOML configuration file and pass it to the schema loader. ```python import schemathesis config = schemathesis.Config.from_path("path-to-my/config.toml") schema = schemathesis.openapi.from_url("http://127.0.0.1:8080/openapi.json", config=config) ``` -------------------------------- ### Constrained date scalar example Source: https://schemathesis.readthedocs.io/en/stable/guides/graphql-custom-scalars Example of registering a date scalar with a restricted range. ```python # Restricted date range from datetime import date schemathesis.graphql.scalar( "RecentDate", st.dates(min_value=date(2020, 1, 1), max_value=date(2030, 12, 31)).map(str).map(nodes.String) ) ``` -------------------------------- ### Load GraphQL schema from file object or string with schemathesis.graphql.from_file Source: https://schemathesis.readthedocs.io/en/stable/reference/python Loads a GraphQL schema from a file-like object or a raw string containing GraphQL SDL. The example shows both a string and an opened file object. ```python import schemathesis # From GraphQL SDL string schema_sdl = ''' type Query { user(id: ID!): User } type User { id: ID! name: String! } ''' schema = schemathesis.graphql.from_file(schema_sdl) # From file object with open("schema.graphql") as f: schema = schemathesis.graphql.from_file(f) ``` -------------------------------- ### Configure client certificate and key Source: https://schemathesis.readthedocs.io/en/stable/reference/configuration Set the file path for a client certificate and its separate private key. ```toml # Set certificate and separate private key request-cert = "/path/to/client-cert.crt" request-cert-key = "/path/to/private-key.key" ``` -------------------------------- ### Run Schemathesis with recommended benchmark configuration Source: https://schemathesis.readthedocs.io/en/stable/guides/benchmarking Use this command to benchmark Schemathesis. It bounds the run to 3600 seconds, continues after failures, disables shrinking, and suppresses all health checks. These options ensure the run spends its full time budget and includes all operations, producing comparable numbers. ```bash schemathesis run \ --max-time 3600 \ --continue-on-failure \ --no-shrink \ --suppress-health-check all ``` -------------------------------- ### OpenAPI schema usage for custom media types Source: https://schemathesis.readthedocs.io/en/stable/reference/python Schema usage example showing how to reference the custom media types 'application/pdf' and 'application/xml' in an OpenAPI requestBody. The comments indicate which strategy is used for each media type. ```yaml requestBody: content: application/pdf: # Uses your PDF strategy schema: type: string format: binary application/xml: # Uses your XML strategy schema: type: string format: binary ``` -------------------------------- ### Target body fields with dictionaries Source: https://schemathesis.readthedocs.io/en/stable/guides/fuzz-dictionary Use `body.` paths to address request body fields. The example binds `body.ccNumber` and `body.user.email` to dictionaries. The `[*]` syntax substitutes at every array element, as shown with `body.tags[*]` and `body.items[*].name`. ```TOML [dictionaries.card_numbers] values = ["4111-1111-1111-1111", "5500-0000-0000-0004"] [dictionaries.emails] values = ["test@example.com", "admin@example.com"] [parameters] "body.ccNumber" = { dictionary = "card_numbers" } "body.user.email" = { dictionary = "emails" } ``` ```TOML [dictionaries.tag_words] values = ["sale", "featured", "archived"] [parameters] "body.tags[*]" = { dictionary = "tag_words", probability = 0.4 } "body.items[*].name" = { dictionary = "tag_words" } ``` -------------------------------- ### Create schemathesis.toml with generation, timeout, and auth settings Source: https://schemathesis.readthedocs.io/en/stable/configuration Create a `schemathesis.toml` file in your project directory to configure test volume, request timeout, and basic authentication. Environment variables like `${USERNAME}` are read from your environment when Schemathesis loads the config. ```toml # Run more tests for better coverage generation.max-examples = 500 # Increase timeout for slow APIs request-timeout = 10.0 # seconds # Basic authentication [auth] basic = { username = "${USERNAME}", password = "${PASSWORD}" } ``` -------------------------------- ### Run Schemathesis with custom config file Source: https://schemathesis.readthedocs.io/en/stable/tutorials/cli Specifies a custom configuration file using the --config-file option. ```bash uvx schemathesis --config-file path/to/config.toml run http://... ``` -------------------------------- ### Run Schemathesis with --auth-wfc CLI flag Source: https://schemathesis.readthedocs.io/en/stable/guides/auth Runs Schemathesis from the command line, specifying the WFC auth document path with --auth-wfc. The path is required. ```bash $ st run openapi.yaml --auth-wfc auth.json ``` -------------------------------- ### Negative Testing: Invalid string examples Source: https://schemathesis.readthedocs.io/en/stable/explanations/data-generation Generates data that should be rejected by the API, deliberately invalid according to the schema. The example schema `{"type": "string", "minLength": 3}` produces negative examples like 42, [], "", "ab". ```json # Schema: {"type": "string", "minLength": 3} # Negative examples: 42, [], "", "ab" ``` -------------------------------- ### Example pytest output for Schemathesis tests Source: https://schemathesis.readthedocs.io/en/stable/explanations/pytest Example pytest output showing one line per API operation, with each line indicating pass or fail status. ```text test.py::test_api[GET /users] PASSED # One line per API operation test.py::test_api[POST /users] FAILED # Each runs many Hypothesis examples test.py::test_api[DELETE /users/{id}] PASSED ``` -------------------------------- ### Per-Run Setup with pytest Fixture Source: https://schemathesis.readthedocs.io/en/stable/guides/stateful-testing Use a session-scoped pytest fixture for expensive setup that should run once per test execution. The fixture is applied to the test class with @pytest.mark.usefixtures. ```python import pytest @pytest.fixture(scope="session") def database(): # create database yield # drop database @pytest.mark.usefixtures("database") class TestAPI(APIWorkflow.TestCase): pass ``` -------------------------------- ### Step 1: Find resources - identify what the API produces Source: https://schemathesis.readthedocs.io/en/stable/explanations/stateful Shows how Schemathesis identifies resources produced by operations, such as creating a User or Order resource. ```text POST /users → Creates "User" resource POST /orders → Creates "Order" resource ``` -------------------------------- ### Run Schemathesis with Docker (Basic Usage) Source: https://schemathesis.readthedocs.io/en/stable/guides/docker Runs Schemathesis tests from the official Docker image without installing Python. The image uses free-threaded Python (3.14t) and `-w auto` enables parallel execution across all available CPUs. ```bash docker run ghcr.io/schemathesis/schemathesis:stable \ run -w auto https://api.example.com/openapi.json ``` -------------------------------- ### Run Schemathesis with a static schema file Source: https://schemathesis.readthedocs.io/en/stable/guides/cicd Run Schemathesis using a schema file from your repository. The --url flag specifies the API host and port to test against. ```bash schemathesis run ./openapi.json --url http://api-host:port ``` -------------------------------- ### Run Schemathesis with automatic config loading Source: https://schemathesis.readthedocs.io/en/stable/configuration Run Schemathesis; it automatically loads `schemathesis.toml` from the current directory or parent directories up to the repository root. ```bash schemathesis run https://api.example.com/openapi.json ``` -------------------------------- ### Run Schemathesis with existing baseline and show output Source: https://schemathesis.readthedocs.io/en/stable/guides/baseline A plain run loads the baseline file, reports the recorded failures in the header, and exits `0` if there are no new failures; otherwise it exits `1`. The output shows the loaded baseline with `Known failures` count. ```shell schemathesis run http://localhost:8080/openapi.json ``` ```output Baseline: schemathesis-baseline.json / 47 entries ... Baseline: Known failures: 47 =========================== No issues found in 12.03s ========================== ``` -------------------------------- ### operations_with_incoming_links() Source: https://schemathesis.readthedocs.io/en/stable/reference/python Method to get labels of operations that have incoming links. ```APIDOC ## operations_with_incoming_links() ### Description Returns labels of operations that another operation links to; empty when the spec has no links. ### Method N/A (method on schema object) ### Endpoint N/A ### Parameters None ### Returns - **list** - Labels of operations with incoming links. ### Response Example N/A ``` -------------------------------- ### Use the extension with schemathesis run Source: https://schemathesis.readthedocs.io/en/stable/guides/extending-cli Shows how to use the custom extension by setting the `SCHEMATHESIS_HOOKS` environment variable to `cli_extensions` and running `schemathesis run` with the custom options `--counter-initial`, `--counter-output`, and `--counter-verbose`. The command targets an OpenAPI schema at `http://localhost:8000/openapi.json`. ```bash export SCHEMATHESIS_HOOKS=cli_extensions # Use the custom options schemathesis run \ --counter-initial 100 \ --counter-output results.txt \ --counter-verbose \ http://localhost:8000/openapi.json ``` -------------------------------- ### Example JSON report output Source: https://schemathesis.readthedocs.io/en/stable/guides/cicd Example JSON report output showing exit code, stop reason, operation counts, test case counts, phases, failures, errors, and warnings. Used to distinguish between a run that selected zero operations and one with real findings. ```json { "exit_code": 1, "stop_reason": "completed", "operations": { "total": 42, "selected": 40, "tested": 38, "errored": 1, "skipped": 1, "skip_reasons": [] }, "test_cases": { "generated": 3800, "with_failures": 12, "unique_failures": 4, "without_checks": 0 }, "phases": { "fuzzing": { "status": "success", "skip_reason": null } }, "failures": [], "errors": [], "warnings": {} } ``` -------------------------------- ### Configure dictionary in schemathesis.toml Source: https://schemathesis.readthedocs.io/en/stable/guides/fuzz-dictionary Wire the dictionary file into Schemathesis configuration. The [dictionaries.sql] section loads the file, and [generation.dictionaries] applies it to string parameters with a 10% probability. ```toml [dictionaries.sql] from-file = "fuzz/sql_injection.dict" [generation.dictionaries] string = { dictionary = "sql", probability = 0.1 } ``` -------------------------------- ### Example endpoint with performance issue Source: https://schemathesis.readthedocs.io/en/stable/guides/targeted Example API endpoint with a hidden performance issue: inputs with many zeros cause progressively slower responses, and the endpoint fails when the count exceeds 10. This is used to illustrate how targeted testing can find problematic inputs faster than standard testing. ```python async def performance(request): decoded = await request.json() number = str(decoded).count("0") if number > 0: # Progressively slower as zeros increase await asyncio.sleep(0.01 * number) if number > 10: # Fails with too many zeros raise ServerError() return {"result": "success"} ``` -------------------------------- ### Create schemathesis.toml configuration file Source: https://schemathesis.readthedocs.io/en/stable/tutorials/pytest Defines core settings: headers with an environment variable placeholder, disables output sanitization, and sets generation max-examples to 500. ```toml # Core settings from our previous tests headers = { Authorization = "Bearer ${API_TOKEN}" } [output.sanitization] enabled = false [generation] max-examples = 500 ``` -------------------------------- ### Disable specific check Source: https://schemathesis.readthedocs.io/en/stable/reference/configuration Example of disabling the response schema conformance check specifically. ```toml [checks] response_schema_conformance.enabled = false ``` -------------------------------- ### Coverage Phase: Boundary values for string constraints Source: https://schemathesis.readthedocs.io/en/stable/explanations/data-generation Aims to exhaustively cover boundary values for every constraint. The example schema `{"type": "string", "minLength": 2, "maxLength": 10}` produces strings of lengths 1, 2, 3, 9, 10, and 11. ```json # Schema: {"type": "string", "minLength": 2, "maxLength": 10} # Produces: strings of length 1, 2, 3, 9, 10, 11 ``` -------------------------------- ### Static Authentication via CLI Options Source: https://schemathesis.readthedocs.io/en/stable/guides/auth Use CLI options for simple authentication cases: bearer token, basic auth, and API key. The commands set headers or auth credentials directly. ```bash # Bearer token schemathesis run http://localhost:8000/openapi.json \ --header "Authorization: Bearer your-token" # Basic authentication schemathesis run http://localhost:8000/openapi.json \ --auth username:password # API key schemathesis run http://localhost:8000/openapi.json \ --header "X-API-Key: your-api-key" ``` -------------------------------- ### schemathesis.auth Source: https://schemathesis.readthedocs.io/en/stable/reference/python Registers an authentication provider. The provider class must implement get and set methods. ```APIDOC ## schemathesis.auth ### Description Registers an authentication provider. The provider class must implement get and set methods. ### Usage Example ```python @schemathesis.auth() class ContextAwareAuth: def get(self, case, context): # Access operation details if "/admin/" in context.operation.path: return self.get_admin_token() else: return self.get_user_token() def set(self, case, data, context): case.headers = {"Authorization": f"Bearer {data}"} ``` ``` -------------------------------- ### Run Schemathesis with --config-file Source: https://schemathesis.readthedocs.io/en/stable/configuration Use a config file from a different location by passing `--config-file` before the `run` command. ```bash schemathesis --config-file /path/to/config.toml run https://api.example.com/openapi.json ``` -------------------------------- ### Replay crash cases Source: https://schemathesis.readthedocs.io/en/stable/reference/cli Examples of replaying all crashes, a specific case ID, or a specific crash file. ```bash $ st replay $ st replay Xt9Kp2 $ st replay .schemathesis/myapi/cache/crashes/abc123.json ``` -------------------------------- ### Run Schemathesis in Docker with Allure report path Source: https://schemathesis.readthedocs.io/en/stable/guides/allure Runs the Schemathesis Docker image with Allure support, mounting a directory to write raw result files to the host. ```bash docker run \ -v ./allure-results:/app/allure-results \ ghcr.io/schemathesis/schemathesis:stable \ run --report-allure-path /app/allure-results \ https://api.example.com/openapi.json ``` -------------------------------- ### response_headers_conformance failure: missing required header Source: https://schemathesis.readthedocs.io/en/stable/reference/checks Example failure message when a required header is missing from the response. ```text - Missing required headers The following required headers are missing from the response: - `X-RateLimit-Remaining` ``` -------------------------------- ### Generated query for custom scalars Source: https://schemathesis.readthedocs.io/en/stable/guides/graphql-custom-scalars An example of a valid query generated by Schemathesis for the defined custom scalars. ```graphql { getEvents(date: "2023-12-25", id: "550e8400-e29b-41d4-a716-446655440000") } ``` -------------------------------- ### Basic auth configuration Source: https://schemathesis.readthedocs.io/en/stable/configuration Configures basic authentication with username and password from environment variables. ```toml [auth] basic = { username = "${USERNAME}", password = "${PASSWORD}" } ``` -------------------------------- ### Include or exclude operations by expression Source: https://schemathesis.readthedocs.io/en/stable/reference/cli Filters operations using a custom expression starting with a JSON Pointer. ```bash $ st run openapi.yaml --include-by "/tags/0 == 'user'" ``` -------------------------------- ### Set maximum response time Source: https://schemathesis.readthedocs.io/en/stable/reference/configuration Example of setting the maximum allowed API response time to 2.0 seconds. ```toml [checks] max_response_time = 2.0 ``` -------------------------------- ### Load GraphQL schema from file path with schemathesis.graphql.from_path Source: https://schemathesis.readthedocs.io/en/stable/reference/python Loads a GraphQL schema from a filesystem path. The file should be a GraphQL SDL file with .graphql or .gql extension; encoding defaults to 'utf-8'. ```python import schemathesis # Load from GraphQL SDL file schema = schemathesis.graphql.from_path("./schema.graphql") ``` -------------------------------- ### Configure expected status codes Source: https://schemathesis.readthedocs.io/en/stable/reference/configuration Example of customizing the expected status codes for the positive data acceptance check. ```toml [checks] positive_data_acceptance.expected-statuses = [200, 201, 202] ```