TestFlows Handbook

https://testflows.com/handbook
Admin
Your comprehensive handbook for using the TestFlows framework, covering everything from basic setup...
Tokens:44,478
Snippets:368
Trust Score:6.2
Update:1 year ago
Show doc for...
Tokens:
Raw
### Install TestFlows via Pip or Source

Source: https://testflows.com/handbook/index

Install the TestFlows framework using pip for quick setup or clone from source for advanced installations. This covers the primary methods to get the framework running on your system.

```Shell
pip3 install testflows
```

```Shell
git clone https://github.com/testflows/TestFlows.git
cd TestFlows
./build ; ./install
```

--------------------------------

### Multiple Setups and Teardowns

Source: https://testflows.com/handbook/index

Demonstrates performing multiple setup and teardown actions within a single test scenario. The `And` step can be used to chain subsequent setup or teardown actions, making the test procedure more fluid and readable.

```python
@TestScenario def my_scenario(self):
    try:
        with Given("my first setup"):
            do_first_setup()
        with And("my second setup"):
            do_second_setup()
        with When("I perform action"):
            perform_action()
    finally:
        with Finally("first clean up"):
            do_first_cleanup()
        with And("second clean up"):
            do_first_cleanup()
```

--------------------------------

### Python Combinatorial Testing Example: Pressure Switch

Source: https://testflows.com/handbook/index

Provides an example of combinatorial testing setup using a `pressure_switch` function. It defines input parameters and their levels, illustrating how to test different combinations of these parameters.

```python
from testflows.core.test import note

def pressure_switch(pressure, volume):
    """Pressure switch.
    `pressure` levels: 0, 20, 40
    `volume` levels: 0, 100, 200, 300
    """
    if (pressure < 10 or pressure > 30):
        if (volume > 250):
            assert False, "boom!"
        else:
            note("ok, no problem")
    else:
        note("ok")

```

--------------------------------

### Python Scenario with Examples List

Source: https://testflows.com/handbook/index

Demonstrates defining a test scenario with examples provided as a list of tuples. This approach uses the `Scenario` context manager and passes `Examples` directly, suitable for straightforward parameterization.

```python
from testflows.core import *
from testflows.asserts import error  

def add(a, b):
    return a + b  

def check_add(a, b, expected):
    """Check that function add(a, b)
    returns expected result for given `a` and `b` values.
    """
    with When(f"I call add function with {a},{b}"):
        r = add(a, b)
    with Then(f"I expect the result to be {expected}"):
        assert r == expected, error()  

def regression():
    with Scenario("check `add(a, b)` function", examples=Examples("a b expected", [
            (2, 2, 4),
            (-5, 100, 95),
            (-5, -5, -10)
        ])) as scenario:
        for example in scenario.examples:
            with Example(f"check {example.a} + {example.b} == {example.expected}"):
                # `vars(example)` converts example named tuple to a dictionary
                check_add(**vars(example))

if __name__ == "__main__":
    regression()
```

--------------------------------

### Auto Verified Documentation Example

Source: https://testflows.com/handbook/index

Example of a `.tfd` file demonstrating how to embed Python code blocks for auto-verified documentation. Shows variable sharing, dynamic text generation, and f-string substitution.

```Markdown
## This is a heading  This file is written using Markdown where you can have any number of `python:testflows` code blocks that contain executable Python code.  ```python:testflows # This is Python code that will be executed when .tfd document is run.  msg = "Hello TestFlows Texts" ```  The scope is shared between all the code blocks in the same document.  ```python:testflows # so `msg` variable defined above can also be accessed in this # `python:testflows` code block  new_msg = msg + " Thanks for making verifying docs so easy!" ```  The output of executing `.tfd` document using `tfs document run` is the final `.md` file with all the `python:testflows` code blocks removed and replaced with the text added to the document using the `text()` function.  ```python:testflows # Let's use `text()` function to add some text to our document # dynamically in our Python code  text("add this line to the final Markdown document") ```  Any text outside the `python:testflows` code blocks are treated as Python f-strings. This allows you to specify expressions for substitutions. See [Python formatted string literals](https://docs.python.org/3/tutorial/inputoutput.html#formatted-string-literals) for more details.  Here is an example where we will substitute the value of `msg` variable next {msg}. But with Python f-strings you can specify even complex expressions. For example, we can convert our string in `msg` to title case as follows {msg.title()}.  You can double your curly brackets to escape them when substitution expression is not needed using `{{` or `}}`.  By the way, your document can't contain any triple quotes. If you need them then you have to add them using `{triple_quotes}` expression. For example,  ```markdown This text has {triple_quotes} triple quotes. ```  Well, this is pretty much it. With `testflows.texts` you have full power of full featured test framework and Python language at your disposal to make sure your documentation always stays up to date.  
```

--------------------------------

### Basic Test Setup and Teardown

Source: https://testflows.com/handbook/index

Demonstrates explicit setup and teardown using `Given` and `Finally` blocks within a `try...finally` structure. This pattern ensures cleanup actions are executed even if errors occur during the test.

```python
@TestScenario def my_scenario(self):
    try:
        with Given("my setup"):
            do_some_setup()
        with When("I perform action"):
            perform_action()
    finally:
        with Finally("clean up"):
            do_cleanup()
```

--------------------------------

### Install TestFlows Texts Module

Source: https://testflows.com/handbook/index

Command to install or upgrade the testflows.texts Python package using pip.

```Shell
pip3 install --upgrade testflows.texts
```

--------------------------------

### Resource Handling in Teardown

Source: https://testflows.com/handbook/index

Shows how to safely handle resources created during setup. The `Finally` step checks if the resource was successfully created (i.e., not `None`) before attempting cleanup, preventing errors if the setup phase failed.

```python
@TestScenario def my_scenario(self):
    resource = None
    try:
        with Given("my setup"):
            resource = do_some_setup()
        with When("I perform action"):
            perform_action()
    finally:
        with Finally("clean up"):
            if resource is not None:
                do_cleanup()
```

--------------------------------

### Common Setup/Teardown with Feature

Source: https://testflows.com/handbook/index

Illustrates defining common setup and teardown logic at the `Feature` level. This is useful when multiple `Scenario`s within a feature share the same setup and cleanup procedures, promoting code reuse and maintainability.

```python
from testflows.core import *

@TestScenario def my_scenario1(self):
    pass

@TestScenario def my_scenario2(self):
    pass

with Feature("my feature"):
    try:
        with Given("my setup"):
            pass
        Scenario(run=my_scenario1)
        Scenario(run=my_scenario2)
    finally:
        with Finally("clean up"):
            pass
```

--------------------------------

### Python Outline with Examples for Test Steps

Source: https://testflows.com/handbook/index

Shows how to use the `@TestOutline` decorator with `@Examples` to structure tests. This approach is ideal for parameterized test steps, allowing example parameters to be directly passed to the test function, simplifying the loop over examples.

```python
from testflows.core import *
from testflows.asserts import error  

def add(a, b):
    return a + b  

@TestOutline(Scenario)
@Examples("a b expected", [
    (2, 2, 4),
    (-5, 100, 95),
    (-5, -5, -10)
])
def check_add(self, a, b, expected):
    """Check that function add(a, b)
    returns expected result for given `a` and `b` values.
    """
    with When(f"I call add function with {a},{b}"):
        r = add(a, b)
    with Then(f"I expect the result to be {expected}"):
        assert r == expected, error()  

def regression():
    Scenario("check `add(a, b)` function", run=check_add)

if __name__ == "__main__":
    regression()
```

--------------------------------

### Define Background Steps using Decorator

Source: https://testflows.com/handbook/index

Illustrates how to define a complex setup or precondition using the @TestBackground decorator in Python. This approach is suitable for mandatory setup steps that cannot be skipped.

```python
@TestBackground @Name("My complex setup")
def background(self):
    with Given("I need to setup something"):
        pass
    with And("I need to setup something else"):
        pass
```

--------------------------------

### Dynamically generate test examples

Source: https://testflows.com/handbook/index

Shows how to dynamically generate test `Example` steps from a list of test data. This approach makes it easy to add or modify test cases without altering the core test logic.

```python
from testflows.core import *
from testflows.asserts import error

def add(a, b):
    return a + b

def check_add(a, b, expected):
    """Check that function add(a, b)
    returns expected result for given `a` and `b` values.
    """
    with When(f"I call add function with {a},{b}"):
        r = add(a, b)
    with Then(f"I expect the result to be {expected}"):
        assert r == expected, error()

def regression():
    with Scenario("check `add(a, b)` function"):
        examples = [
            (2, 2, 4),
            (-5, 100, 95),
            (-5, -5, -10)
        ]
        for example in examples:
            a, b, expected = example
            with Example(f"check {a} + {b} == {expected}"):
                check_add(a=a, b=b, expected=expected)

if main():
    regression()
```

--------------------------------

### Define Examples for Inline Test

Source: https://testflows.com/handbook/index

Adds examples to an inline test using the 'examples' parameter. This parameter takes an Examples class instance, which defines a header and rows. Examples are accessed via the test.examples attribute.

```python
from testflows.core.test import Test
from testflows.core.examples import Examples
from testflows.core.log import note

with Test("My test", examples=Examples("col0 col1", [("col0_row0", "col1_row0"), ("col0_row1", "col1_row1")])) as test:
    for example in test.examples:
        note(str(example))
```

--------------------------------

### Define Python Test Outline with Examples

Source: https://testflows.com/handbook/index

Demonstrates defining a test outline using the @TestOutline decorator and @Examples for parameterized test cases. It shows how to pass arguments to the outline and execute it, allowing it to iterate over provided example data.

```python
from testflows.core import *

@TestOutline(Scenario)
@Examples("greeting name", [
    ("Hello", "John"),
    ("Goodbye", "Eric")
])
def outline(self, greeting, name):
    note(f"{greeting} {name}!")

outline()
```

--------------------------------

### Define Examples using Decorator

Source: https://testflows.com/handbook/index

Applies examples to a decorated test function using the @Examples decorator. The decorator requires a header and a list of rows. Examples are then iterated through self.examples.

```python
from testflows.core.test import TestScenario
from testflows.core.examples import Examples
from testflows.core.log import note

@TestScenario
@Examples("col0 col1", rows=[
    ("col0_row0", "col1_row0"),
    ("col0_row1", "col1_row1")
])
def scenario(self):
    for example in self.examples:
        note(str(example))
```

--------------------------------

### Python Example: Command Line Pausing

Source: https://testflows.com/handbook/index

Demonstrates how to execute a Python test script with command-line arguments to pause execution at specific tests. This example shows pausing before 'my step 1' and after 'my step 2'.

```python
from testflows.core import *

with Test("my test"):
    with Step("my step 1"):
        note("my step 1")
    with Step("my step 2"):
        note("my step 2")
```

```bash
python3 pause.py --pause-before "/my test/my step 1" --pause-after "/my test/my step 2"
```

--------------------------------

### Example Usage of --only Option

Source: https://testflows.com/handbook/index

Demonstrates how to use the `--only` command-line option with a specific pattern to execute a subset of tests.

```python
test.py --only "/my test/*"
```

--------------------------------

### Define Example Checks Inline within Scenario Context

Source: https://testflows.com/handbook/index

Demonstrates defining 'Example' type checks inline using context managers, typically nested within a 'Scenario'. Examples are often generated automatically from outlines.

```python
with Scenario("My scenario"):
    with Example("Example 1"):
        pass
    with Example("Example 2"):
        pass
```

--------------------------------

### Get Help for Document Writer

Source: https://testflows.com/handbook/index

Demonstrates how to access the help menu for the document writer program by passing the --help argument. This is useful for discovering available options and arguments.

```bash
tfs document run -- --help
```

--------------------------------

### Python: Test structure example

Source: https://testflows.com/handbook/index

Provides an example Python test program structure using testflows.core, including modules, suites, and scenarios, to illustrate test filtering.

```python
from testflows.core import *

@TestScenario
def my_scenario(self):
    with Step("Step A"):
        pass
    with Step("Step B"):
        pass

@TestSuite
def my_suite():
    Scenario("Test A", run=my_scenario)
    Scenario("Test B", run=my_scenario)

with Module("Top Test"):
    Suite("Suite A", run=my_suite)
    Suite("Suite B", run=my_suite)
```

--------------------------------

### Python: Use yield for setup and teardown in Given step

Source: https://testflows.com/handbook/index

Demonstrates using the `yield` statement within a `@TestStep(Given)` decorated function to manage setup and cleanup phases. The `finally` block ensures cleanup code executes after the test step, even if errors occur.

```Python
from testflows.core import *

@TestStep(Given)
def my_setup(self):
    try:
        #do_setup()
        yield
    finally:
        with Finally("clean up"):
            # do_cleanup()
            pass

with Scenario("my scenario"):
    with Given("my setup"):
        my_setup()
```

--------------------------------

### Test `add` function with When/Then steps

Source: https://testflows.com/handbook/index

Illustrates using `Example`, `When`, and `Then` constructs for more detailed test step descriptions. This approach enhances readability and provides better context for test execution and failure reporting.

```python
from testflows.core import *
from testflows.asserts import error

def add(a, b):
    return a + b

def regression():
    with Scenario("check `add(a, b)` function"):
        with Example("check 2 + 2 == 4"):
            with When("I call add function with 2,2"):
                r = add(2, 2)
            with Then("I expect the result to be 4"):
                # error() will generate detailed error message if assertion fails
                assert r == 4, error()

        with Example("check -5 + 100 == -95"):
            with When("I call add function with -5,100"):
                r = add(-5, 100)
            with Then("I expect the result to be -95"):
                assert r == 95, error()

        with Example("check -5 + -5 == -10"):
            with When("I call add function with -5,-5"):
                r = add(-5, -5)
            with Then("I expect the result to be -10"):
                assert r == -10, error()

if main():
    regression()
```

--------------------------------

### Test execution output examples

Source: https://testflows.com/handbook/index

Provides examples of the output generated by the test runner, indicating skipped or passed tests based on the applied filtering options.

```shell
✔ [ Skip ] /Top Test/Suite A/Test A
```

```shell
✔ [ Skip ] /Top Test/Suite A/Test A ✔ [ Skip ] /Top Test/Suite B/Test A
```

```shell
Passing  ✔ [ OK ] /Top Test/Suite A/Test A ✔ [ OK ] /Top Test/Suite A ✔ [ OK ] /Top Test/Suite B ✔ [ OK ] /Top Test
```

--------------------------------

### Python Decorated TestScenario with Examples

Source: https://testflows.com/handbook/index

Illustrates using `@TestScenario` and `@Examples` decorators for defining tests. This method makes the test definition more declarative and integrates examples directly into the test function.

```python
from testflows.core import *
from testflows.asserts import error  

def add(a, b):
    return a + b  

@TestScenario
@Examples("a b expected", [
    (2, 2, 4),
    (-5, 100, 95),
    (-5, -5, -10)
])
def check_add(self):
    """Check that function add(a, b)
    returns expected result for given `a` and `b` values.
    """
    for example in self.examples:
        a, b, expected = example
        with Example(f"check {a} + {b} == {expected}"):
            with When(f"I call add function with {a},{b}"):
                r = add(a, b)
            with Then(f"I expect the result to be {expected}"):
                assert r == expected, error()  

def regression():
    Scenario("check `add(a, b)` function", run=check_add)

if __name__ == "__main__":
    regression()
```

--------------------------------

### Set Attributes for Decorated Tests

Source: https://testflows.com/handbook/index

Provides an example of setting attributes like name, flags, and examples for a decorated test function using specific decorator classes such as `@Name`, `@Flags`, and `@Examples`.

```python
@TestScenario
@Name("check add two numbers")
@Flags(TE)
@Examples("x y result", [
    (1,1,2),
])
def test(self, x, y, result):
    assert sum([x,y]) == result
```

--------------------------------

### Define a Given Step Inline

Source: https://testflows.com/handbook/index

Shows how to define a 'Given' step inline using the Given context manager. This is an alternative to the decorator for defining preconditions or setup steps.

```python
from testflows.core.test import Given

with Given("I have something"): 
    pass
```

--------------------------------

### Raw Output with Grep and Nice Transformation

Source: https://testflows.com/handbook/index

Shows how to combine the 'raw' output with standard Unix tools like 'grep' for filtering specific messages, followed by a 'tfs transform nice' for readability. This example filters for 'RESULT' messages.

```bash
python3 output.py -o raw | grep '{"message_keyword":"RESULT"' | tfs transform nice
```

--------------------------------

### Python Test Scenario with TestSteps

Source: https://testflows.com/handbook/index

Demonstrates defining a test scenario using `testflows.core` decorators like `@TestStep` for 'Given', 'When', 'Then' phases and `TestScenario` context managers for structuring test execution. Includes setup, action, and assertion steps.

```python
from testflows.core import *

@TestStep(Given)
def setup_something(self):
    pass

@TestStep(When)
def do_something(self):
    pass

@TestStep(Then)
def expect_something(self):
    pass

@TestScenario
def my_scenario(self):
    with Given("I setup something",
               description="""detailed description if needed"""):
        setup_something()

    with When("I do something",
              description="""detailed description if needed"""):
        do_something()

    with Then("I expect something",
              description="""detailed description if needed"""):
        expect_something()

if __name__ == "__main__":
    my_scenario()
```

--------------------------------

### Test Execution Output with Note Message

Source: https://testflows.com/handbook/index

Example output from a Testflows execution displaying a message added using the `note()` function.

```text
Nov 15,2021 14:17:21   ⟥  Scenario my scenario                  8ms   ⟥    [note] Hello Your Name!                  8ms   ⟥⟤ OK my scenario, /my scenario
```

--------------------------------

### Hierarchical Requirement Naming Convention

Source: https://testflows.com/handbook/index

Demonstrates using dot-separated names to group requirements, often aligning with document sections, and includes an example for an 'Options' section.

```markdown
### Options

#### RQ.SRS001-CU.LS.Options.ListHiddenEntries version: 1.0
```

--------------------------------

### Python Test Structure with Custom ID Example

Source: https://testflows.com/handbook/index

A Python code structure for a simple test scenario, demonstrating the context where a custom ID would be applied via command line.

```python
from testflows.core import *

with Test("my test"):
    pass
```

--------------------------------

### Configure Custom Argument with Multiple YML Files

Source: https://testflows.com/handbook/index

Example YML files used to demonstrate precedence when applying multiple configurations. The last specified file's values will take effect.

```yaml
config1.yml:
  custom-arg: hello here

config2.yml:
  custom-arg: hello there
```

--------------------------------

### Define Nested And Steps Inline

Source: https://testflows.com/handbook/index

Shows how 'And' steps can be nested within other steps like 'Given' or 'When' to define sequential actions or setup phases.

```python
with Given("I have something"):
    with When("I do some action to setup things"):
        pass
    with And("I do another action to continue the setup"):
        pass
```

--------------------------------

### Manual Test Example

Source: https://testflows.com/handbook/index

Shows how to define a fully manual test scenario in TestFlows. The MANUAL flag is applied at the Scenario level, making all its steps inherit the manual execution behavior and prompting for input.

```python
from testflows.core import *

with Scenario("manual scenario", flags=MANUAL):
    with Given("manual setup"):
        pass
    with When("manual action"):
        pass
```

--------------------------------

### Upgrade TestFlows Framework

Source: https://testflows.com/handbook/index

Upgrade an existing TestFlows installation to the latest version. This command ensures you have the most recent features and bug fixes.

```Shell
pip3 install --upgrade testflows
```

--------------------------------

### Multiple Requirements in Markdown

Source: https://testflows.com/handbook/index

Provides an example of defining multiple distinct requirements within a single Markdown document.

```markdown
### RQ.SRS001-CU.LS version: 1.0

The [ls](#ls) utility SHALL list the contents of a directory.

### RQ.SRS001-CU.LS.ListHiddenEntries version: 1.0

The [ls](#ls) utility SHALL accept `-a, --all` option that SHALL cause the output to list entries starting with `.` that SHALL be considered to be hidden.
```

--------------------------------

### Python Secret Usage Example

Source: https://testflows.com/handbook/index

Shows a basic example of creating and using a secret within a TestFlows scenario. The secret value is masked in logs when accessed via functions like note().

```python
from testflows.core import *

with Scenario("using secrets"):
    secret = Secret("mysecret")("my secret value")
    note(f"my secret is: {secret}")
    note(f"my secret value is: {secret.value}")
```

--------------------------------

### Testflows Step API

Source: https://testflows.com/handbook/index

Provides an overview of core Testflows step definitions for managing test execution flow. These steps are used to define setup, action, and teardown phases within test scenarios.

```APIDOC
Given(description, flags=None)
  - Defines a setup step. By default, has MANDATORY flag set.
  - Parameters:
    - description: A string describing the setup action.
    - flags: Optional flags to modify step behavior (e.g., TE).

When(description, flags=None)
  - Defines the main action step of the test.
  - Parameters:
    - description: A string describing the action.
    - flags: Optional flags.

And(description, flags=None)
  - Used to chain subsequent steps of the same type (e.g., multiple Given or Finally steps).
  - Parameters:
    - description: A string describing the chained action.
    - flags: Optional flags.

Finally(description, flags=None)
  - Defines a teardown or cleanup step. Must be within a `finally` block.
  - By default, has TE (Test Execution) flag implicitly set, ensuring execution.
  - Parameters:
    - description: A string describing the cleanup action.
    - flags: Optional flags.

Scenario(run, name=None, **kwargs)
  - Defines a test scenario to be executed.
  - Parameters:
    - run: A callable function representing the test logic.
    - name: Optional name for the scenario.

Feature(name, description=None, **kwargs)
  - Groups related scenarios under a common feature.
  - Parameters:
    - name: The name of the feature.
    - description: Optional description for the feature.

TestScenario(name=None, **kwargs)
  - Decorator to define a test scenario function.
  - Parameters:
    - name: Optional name for the scenario.

TE flag
  - Ensures a step is executed even if preceding steps fail. Implicitly set for Finally steps.
```

--------------------------------

### Python Specification Object Example

Source: https://testflows.com/handbook/index

Illustrates the generated Python Specification object, which encapsulates the entire requirements document, including its name and description, derived from the Markdown source.

```python
from requirements import RQ_SRS001_CU_LS

# Assuming SRS001_ls_Unix_Command_Utility is defined elsewhere or generated
# SRS001_ls_Unix_Command_Utility = Specification(
#     name='SRS001 `ls` Unix Command Utility',
#     description=None,
#     ...
# )

# Example usage within a test context:
# from testflows.core import Test, note
# with Test("My test", requirements=[RQ_SRS001_CU_LS("1.0")] as test:
#     note(test.requirements)
```

--------------------------------

### Test Execution Output with Custom Argument

Source: https://testflows.com/handbook/index

Example output from a Testflows execution showing the custom argument 'custom-arg' being successfully applied from the YML configuration.

```text
Sep 24,2021 21:40:52   ⟥  Scenario my test                             Attributes                               custom-arg                                 hello there                  3ms   ⟥⟤ OK my test, /my test
```

--------------------------------

### Python ArgumentParser Decorator Example

Source: https://testflows.com/handbook/index

Demonstrates how to use the ArgumentParser decorator to define custom command-line arguments for a test function. The arguments are passed to the decorated function, and help messages are generated.

```python
import argparse

# Assume TestModule and ArgumentParser are defined elsewhere in the framework
# class TestModule:
#     def __init__(self, *args, **kwargs):
#         pass
# 
# def ArgumentParser(parser_func):
#     def decorator(func):
#         # In a real scenario, this would register the parser_func with the test
#         # and pass the parsed arguments to the test function.
#         def wrapper(self, *args, **kwargs):
#             # Placeholder for argument parsing logic
#             print("Simulating argument parsing...")
#             # Example: If arg0 and arg1 were passed via command line
#             # parsed_args = {"arg0": True, "arg1": False}
#             # func(self, **parsed_args)
#             func(self, *args, **kwargs)
#         return wrapper
#     return decorator

def argparser(parser):
    """Custom command line arguments.

    :param parser: an instance of argparse parser.
    """
    parser.add_argument("--arg0", action="store_true",
        help="argument 0")
    parser.add_argument("--arg1", action="store_true",
        help="argument 1")

# @TestModule
# @ArgumentParser(argparser)
# def regression(self, arg0, arg1):
#     note(arg0)
#     note(arg1)

# Example of how the parser function itself might be used:
if __name__ == "__main__":
    parser = argparse.ArgumentParser(description="Test arguments.")
    argparser(parser)
    args = parser.parse_args()
    print(f"--arg0: {args.arg0}")
    print(f"--arg1: {args.arg1}")

```

--------------------------------

### Define Then Steps Inline

Source: https://testflows.com/handbook/index

Provides an example of defining a Then step inline using the 'with Then(...)' context manager. This is used for asserting positive conditions within a test.

```python
with Then("I expect something"):
    assert something
```

--------------------------------

### Test Execution Output with Multiple YMLs

Source: https://testflows.com/handbook/index

Example output showing that the 'custom-arg' value from the last specified YML file ('config2.yml') is applied during test execution.

```text
Sep 24,2021 21:48:47   ⟥  Scenario my test                             Attributes                               custom-arg                                 hello there                  4ms   ⟥⟤ OK my test, /my test
```

--------------------------------

### Python Test Structure Example

Source: https://testflows.com/handbook/index

A basic Python code structure for defining modules and scenarios within Testflows, serving as a base for applying tags, attributes, or custom IDs.

```python
from testflows.core import *

with Module("regression"):
    with Scenario("my test"):
        pass
```

--------------------------------

### Define When Steps Inline

Source: https://testflows.com/handbook/index

Provides an example of defining a When step inline using the 'with When(...)' context manager. This method is used to declare an action within a scenario.

```python
with When("I do some action"):
    do_some_action()
```

--------------------------------

### Define and Run a Simple Inline Scenario

Source: https://testflows.com/handbook/index

Demonstrates how to define a basic 'Hello World!' scenario inline using the Scenario context manager and execute it using the Python interpreter.

```python
from testflows.core import Scenario

with Scenario("Hello World!"):
    pass
```

```shell
python3 ./test.py
```

--------------------------------

### Semi-Automated Test Example

Source: https://testflows.com/handbook/index

Demonstrates how to create a semi-automated test scenario in TestFlows. This involves marking a step with the MANUAL flag, causing the test execution to pause and prompt for user input for that step.

```python
from testflows.core import *

with Scenario("my mixed scenario"):
    with Given("automated setup"):
        pass
    with Step("manual step", flags=MANUAL):
        pass
```

--------------------------------

### Log Output: Test Combination Execution

Source: https://testflows.com/handbook/index

Provides an example of the execution log output generated by TestFlows, detailing the execution of test combinations, their timings, and success status. It shows how each combination is identified and reported.

```Log
```log
Sep 22,2023 17:00:48   ⟥  Scenario check add Sep 22,2023 17:00:48     ⟥  Combination pattern #0                371us     ⟥    [note] 1 + 3 + 5                444us     ⟥⟤ OK pattern #0, /check add/pattern #0 Sep 22,2023 17:00:48     ⟥  Combination pattern #1                216us     ⟥    [note] 1 + 3 + 6                270us     ⟥⟤ OK pattern #1, /check add/pattern #1 Sep 22,2023 17:00:48     ⟥  Combination pattern #2                190us     ⟥    [note] 1 + 4 + 5                238us     ⟥⟤ OK pattern #2, /check add/pattern #2 Sep 22,2023 17:00:48     ⟥  Combination pattern #3                184us     ⟥    [note] 1 + 4 + 6                231us     ⟥⟤ OK pattern #3, /check add/pattern #3 Sep 22,2023 17:00:48     ⟥  Combination pattern #4                174us     ⟥    [note] 2 + 3 + 5                220us     ⟥⟤ OK pattern #4, /check add/pattern #4 Sep 22,2023 17:00:48     ⟥  Combination pattern #5                186us     ⟥    [note] 2 + 3 + 6                234us     ⟥⟤ OK pattern #5, /check add/pattern #5 Sep 22,2023 17:00:48     ⟥  Combination pattern #6                232us     ⟥    [note] 2 + 4 + 5                279us     ⟥⟤ OK pattern #6, /check add/pattern #6 Sep 22,2023 17:00:48     ⟥  Combination pattern #7                168us     ⟥    [note] 2 + 4 + 6                214us     ⟥⟤ OK pattern #7, /check add/pattern #7                  5ms   ⟥⟤ OK check add, /check add
```
```

--------------------------------

### Equivalent to Fail with Specific Type

Source: https://testflows.com/handbook/index

Provides the equivalent code structure for the previous example, showing how a 'Critical' failure can be represented by nesting the fail() call within a 'Critical' context manager.

```python
from testflows.core import *

with Check("Hello World!"):
    with Critical("some critical check"):
        fail("critical fail")
```

--------------------------------

### Define a Given Precondition Step

Source: https://testflows.com/handbook/index

Illustrates defining a 'Given' step, which represents a precondition or setup, using the @TestStep decorator with the Given subtype. This is a common pattern in Behavior-Driven Development (BDD).

```python
from testflows.core.test import TestStep, Given

@TestStep(Given)
def I_have_something(self):
    pass
```

--------------------------------

### Generate Combinations with Covering Array

Source: https://testflows.com/handbook/index

Illustrates using a Covering Array to generate a reduced set of test combinations, significantly lowering the number of tests compared to exhaustive testing. It specifies a strength parameter to control the coverage level, making testing more efficient. The example uses a dictionary to define parameters and their possible values.

```Python
@TestScenario
def check_basic_operations(self):
    """Check basic operations `+`, `-`, `*`, `/`
    with some one digit positive or negative numbers including `0`
    using a covering array of strength `3`.
    """
    for i, combination in enumerate(
        CoveringArray(
            {
                "is_a_negative": [True, False],
                "press_a": [press_0, press_1, press_2],
                "operation": [press_plus, press_minus, press_multiply, press_divide],
                "is_b_negative": [True, False],
                "press_b": [press_0, press_1, press_2],
            },
            strength=3,
        )
    ):
        with Combination(f"pattern #{i}"):
            check_basic_operations_outline(combination=combination.values())
```

--------------------------------

### Define and Run a Basic Test Scenario in Python

Source: https://testflows.com/handbook/index

Demonstrates how to define a test scenario using the TestFlows framework with Given, When, and Then steps. It includes the necessary imports and the main execution block to run the scenario.

```python
from testflows.core import *

@TestScenario
def my_scenario(self):
    with Given("I setup something"):
        pass

    with When("I do something"):
        pass

    with Then("I expect something"):
        pass

if __name__ == "__main__":
    my_scenario()
```

--------------------------------

### Command-line: Basic --only usage

Source: https://testflows.com/handbook/index

Demonstrates the basic syntax for using the --only option to run selected tests based on provided patterns.

```bash
python3 test.py --only "Suite A/*"
```

--------------------------------

### Execute Python Test Outline with Specific Arguments

Source: https://testflows.com/handbook/index

Illustrates executing a test outline with specific, hardcoded arguments, which overrides any examples defined for the outline. This method is useful for running a single, custom test case.

```python
with Scenario("My scenario"):
    outline(greeting="Hey", name="Bob")
```

--------------------------------

### Get Current Test Execution Time

Source: https://testflows.com/handbook/index

Retrieve the current test execution time using the `current_time()` function. It can optionally take a test instance to get time for a specific test.

```python
from testflows.core import *

with Scenario("my test"): 
    with Step("my step") as step:
        note(current_time())
    note(current_time(test=step))
```

--------------------------------

### Running Tests with 'brisk' Output

Source: https://testflows.com/handbook/index

Command to execute Python tests using the 'brisk' output format, which omits test steps for a more focused view of actions.

```bash
python3 output.py -o brisk
```

--------------------------------

### Define Test Execution Order in a Test Module

Source: https://testflows.com/handbook/index

Provides an example of defining the execution order of multiple test scenarios within a TestFlows test module. Scenarios are explicitly listed to control the flow, allowing for easy reordering.

```python
from testflows.core import *

@TestScenario
def testA(self):
    pass

@TestScenario
def testB(self):
    pass

@TestScenario
def testC(self):
    pass

@TestModule
def regression(self):
    Scenario(run=testA)
    Scenario(run=testB)
    Scenario(run=testC)
```

--------------------------------

### Command-line: Select multiple specific tests

Source: https://testflows.com/handbook/index

Demonstrates how to provide multiple patterns to the --only option to run distinct tests from different parts of the test suite.

```bash
python3 test.py --only "/Top Test/Suite A/Test A/*" "/Top Test/Suite B/Test B/*"
```

--------------------------------

### Python @Retries Decorator Usage Example

Source: https://testflows.com/handbook/index

Demonstrates how to apply the @Retries decorator to a Python test class to specify retry counts for different scenarios. This example retries 'my scenario 0' up to 5 times and 'my scenario 1' up to 10 times.

```python
@TestFeature
@Retries({
    "my scenario 0": 5,
    "my scenario 1": 10
}) 
def my_feature(self):
    Scenario(name="my scenario 0", run=my_scenario)
    Scenario(name="my scenario 1", run=my_scenario)
```

--------------------------------

### Basic Testflows Python Structure

Source: https://testflows.com/handbook/index

Demonstrates the basic structure for defining tests using the testflows.core library, including modules, scenarios, and steps with notes.

```python
from testflows.core import *

with Module("regression", flags=TE, attributes=[("name","value")], tags=("tag1", "tag2")):
    with Scenario("my test", description="Test description."):
        with When("I do something"):
            note("do something")
        with Then("I check the result"):
            note("check the result")
```

--------------------------------

### Execute TestFlows Documentation File

Source: https://testflows.com/handbook/index

Shows the command to run a testflows documentation file (.tfd) to generate a Markdown file. This process verifies the embedded code snippets.

```bash
tfs document run -i tutorial.tfd -o tutorial.md
```

--------------------------------

### Run TestFlows Document Generation (Short)

Source: https://testflows.com/handbook/index

Shorthand command to execute the testflows document runner and print the output to stdout.

```Shell
tfs document run -i test.tfd -o -
```

--------------------------------

### Test Execution Output with Debug Message

Source: https://testflows.com/handbook/index

Example output from a Testflows execution showing a message logged using the `debug()` function.

```text
Nov 15,2021 14:19:27   ⟥  Scenario my scenario                  4ms   ⟥    [debug] Hello Your Name!                  4ms   ⟥⟤ OK my scenario, /my scenario
```

--------------------------------

### Running Tests with 'nice' Output

Source: https://testflows.com/handbook/index

Command to execute Python tests using the 'nice' output format, which provides detailed information during development and debugging.

```bash
python3 test.py --output nice
```

--------------------------------

### Run TestFlows Document Generation

Source: https://testflows.com/handbook/index

Command to execute the testflows document runner, converting a `.tfd` source file into a final Markdown document.

```Shell
tfs document run --input my_document.tfd --output my_document.md
```

--------------------------------

### Raw Output with Nice Transformation

Source: https://testflows.com/handbook/index

Demonstrates transforming the 'raw' output format into a more human-readable 'nice' format using the 'tfs transform nice' command. This is useful for advanced message processing.

```bash
python3 test.py -o raw | tfs transfrom nice
```

--------------------------------

### Example: Raising Result Exception Directly

Source: https://testflows.com/handbook/index

Shows how to achieve the same test failure outcome by directly raising a Fail exception, which is what the fail() function internally does.

```python
from testflows.core import *

with Scenario("Hello World!"):
    raise Fail("forcing test fail")
```

--------------------------------

### Set XArgs Decorator

Source: https://testflows.com/handbook/index

Illustrates using the XArgs decorator to set the 'xargs' attribute for a test, providing an example of setting the 'flags' parameter for a test case.

```Python
@TestSuite
@XArgs({
    "my_test": ({"flags": TE},) # set TE flag
})
def suite(self):
    Scenario(run=my_test)

```

--------------------------------

### Raw Output with Custom Test ID

Source: https://testflows.com/handbook/index

Example of raw output from a test run executed with a custom `--id` argument, showing the specified `test_id`.

```json
{"message_keyword":"PROTOCOL",...,"test_id":"/112233445566",...}
```

--------------------------------

### Document 'ls -a' with TestFlows Texts

Source: https://testflows.com/handbook/index

Demonstrates documenting the 'ls -a' command using testflows.texts. It includes creating a hidden file, verifying its listing with 'ls -a', and executing the documentation process.

```python
from subprocess import run

command = "touch .hidden_file"

# add clean up at the end of our document generation
cleanup(run, "rm -rf .hidden_file", shell=True, check=True)

ls_a_command = "ls -a | grep .hidden_file"
cmd = run(ls_a_command, shell=True, capture_output=True, text=True)

assert cmd.returncode == 0, f"returncode {cmd.returncode} is not 0"
assert ".hidden_file" in cmd.stdout, "hidden file '.hidden_file' in not in the outout"
```

--------------------------------

### Raw Output with Default Test ID

Source: https://testflows.com/handbook/index

Example of raw output from a test run executed without a custom ID, showing the automatically generated `test_id`.

```json
{"message_keyword":"PROTOCOL",...,"test_id":"/8a75f8b2-1e52-11ec-8830-cb614fe11752",...}
```

--------------------------------

### Execute Top-Level Test Directly

Source: https://testflows.com/handbook/index

Shows how to run a Python test file directly after making it executable.

```Shell
./test.py
```

--------------------------------

### TestSketch for Basic Calculator Operations

Source: https://testflows.com/handbook/index

Demonstrates using TestSketch to define combinatorial tests for basic calculator operations with positive/negative numbers and zero. It outlines the steps for entering arguments, performing operations, and checking the calculator state.

```python
@TestSketch(Scenario)
def check_basic_operations(self):
    """Check basic operations `+`, `-`, `*`, `/`
    with some one digit positive or negative numbers including `0`.
    """
    try:
        with When("I enter either positive or negative 0,1,2"):
            if either(True, False):
                By(test=press_minus)()
            By(test=either(press_0, press_1, press_2))()

        with And("then I press either +, -, *, / operation"):
            By(test=either(press_plus, press_minus, press_multiply, press_divide))()

        with And("I enter second argument either positive or negative 0,1,2"):
            if either(True, False):
                By(test=press_minus)()
            By(test=either(press_0, press_1, press_2))()

        with And("I press equals to calculate the result"):
            By(test=press_equal)()

        with Then("I check calculator state"):
            By(test=check_state)()
    finally:
        with Finally("I reset calculator back to 0"):
            By(test=reset_calculator)()

```

--------------------------------

### Example: Explicitly Failing a Test

Source: https://testflows.com/handbook/index

Demonstrates how to use the fail() function within a TestFlows scenario to explicitly mark a test as failed. The execution stops at the fail() call.

```python
from testflows.core import *

with Scenario("Hello World!"):
    fail("forcing test fail")
    # this line will not be reached
```
Tokens:
Raw