xDSL

https://github.com/xdslproject/xdsl
Admin
xDSL is a Python-native SSA compiler framework for building custom intermediate representations with...
Tokens:5,643
Snippets:78
Trust Score:8.1
License:Apache-2.0
Update:2 months ago
Show doc for...
Tokens:
Raw
### Install xDSL for development

Source: https://github.com/xdslproject/xdsl/blob/main/README.md

Installs xDSL with development dependencies for testing and contribution.

```bash
pip install xdsl[dev]
```

--------------------------------

### Install and run pre-commit hooks

Source: https://github.com/xdslproject/xdsl/blob/main/CONTRIBUTING.md

Commands to initialize and execute pre-commit hooks for automated code quality checks.

```bash
# Install the pre-commit on your `.git` folder
make precommit-install
# Run the hooks
make precommit
```

--------------------------------

### Install xDSL via pip

Source: https://github.com/xdslproject/xdsl/blob/main/README.md

Standard installation command for using xDSL in a project.

```bash
pip install xdsl
```

--------------------------------

### Verify uv installation

Source: https://github.com/xdslproject/xdsl/blob/main/CONTRIBUTING.md

Check the installed version of the uv dependency manager.

```sh
# Ensure uv is installed
uv --version
```

--------------------------------

### Initialize development environment

Source: https://github.com/xdslproject/xdsl/blob/main/CONTRIBUTING.md

Commands to clone the repository and configure the local development environment, including virtual environment setup, pre-commit hooks, and initial test execution.

```sh
# Clone repo
git clone https://github.com/xdslproject/xdsl.git
cd xdsl

# Set up local environment with all optional and dev dependencies
# Creates a virtual environment called `.venv`
make venv
# Set up pre-commit hook for automatic formatting
make precommit-install
# Run all tests to verify installation was successful
make tests
```

--------------------------------

### High-level Canonicalization Examples

Source: https://github.com/xdslproject/xdsl/blob/main/docs/index.md

Examples of simple algebraic simplifications used in compiler canonicalization.

```txt
x + 0 -> x
2 + 2 -> 4
x * 1 -> x
```

--------------------------------

### Install xDSL subprojects

Source: https://github.com/xdslproject/xdsl/blob/main/README.md

Installs xDSL with specific extra dependencies for subprojects like GUI, JAX, or RISCV support.

```bash
pip install xdsl[gui] # or [jax], [riscv]
```

--------------------------------

### Apply EAFP and walrus operator patterns

Source: https://github.com/xdslproject/xdsl/blob/main/CONTRIBUTING.md

Examples demonstrating preferred lookup patterns and the use of the walrus operator for efficiency.

```python
# Good: single lookup
if (value := mapping.get(key)) is None:
    raise MyException()

# Good: EAFP, when no sentinel is available
try:
    return mapping[key]
except KeyError:
    return default_value

# Bad: LBYL, double lookup
if key not in mapping:
    raise MyException()
return mapping[key]
```

--------------------------------

### Display benchmark help menu

Source: https://github.com/xdslproject/xdsl/blob/main/benchmarks/README.md

View available benchmarks and profilers by invoking a benchmark script with the --help flag.

```text
$ uv run benchmarks/lexer.py --help
usage: lexer.py [-h] [-o OUTPUT] [-q]
                {Lexer.empty_program,Lexer.constant_100,Lexer.constant_1000,Lexer.dense_attr,Lexer.dense_attr_hex,all} [{run,timeit,snakeviz,viztracer,flameprof,pyinstrument,dis}]

positional arguments:
  {Lexer.empty_program,Lexer.constant_100,Lexer.constant_1000,Lexer.dense_attr,Lexer.dense_attr_hex,all}
                        the name of the benchmark to run, `all` to run all benchmarks
  {run,timeit,snakeviz,viztracer,flameprof,pyinstrument,dis}
                        the profiler to use

options:
  -h, --help            show this help message and exit
  -o, --output OUTPUT   the directory into which to write out the profile files
  -q, --quiet           don't show the profiler's UI
```

--------------------------------

### Profiling with Viztracer and cProfile

Source: https://github.com/xdslproject/xdsl/wiki/Performance

Commands to generate performance traces and flame graphs for xDSL tools.

```bash
# Viztracer + perfetto
viztracer xdsl/tools/xdsl_opt.py tests/xdsl_opt/empty_program.mlir
vizviewer result.json
# Alternatively: ui.perfetto.dev + drag and drop result.json

# cProfile + flameprof
python -m cProfile -o my-prof.prof xdsl/tools/xdsl-opt tests/xdsl_opt/empty_program.mlir
flameprof my-prof.prof > my-prof.svg
open my-prof.svg
```

--------------------------------

### Execute benchmark commands

Source: https://github.com/xdslproject/xdsl/blob/main/benchmarks/README.md

Common CLI commands for running, timing, profiling, and disassembling benchmarks.

```bash
uv run benchmarks/lexer.py Lexer.empty_program run
```

```bash
uv run benchmarks/lexer.py all run
```

```bash
uv run benchmarks/lexer.py Lexer.empty_program timeit
```

```bash
uv run benchmarks/lexer.py Lexer.empty_program viztracer
```

```bash
uv run benchmarks/lexer.py Lexer.empty_program dis
```

--------------------------------

### Run formatting and type checking

Source: https://github.com/xdslproject/xdsl/blob/main/CONTRIBUTING.md

Commands to execute Ruff formatting and Pyright type checking via uv or make.

```bash
# Format code
uv run ruff format

# Type check code
uv run pyright
# or via makefile
make pyright
```

--------------------------------

### Configure PATH for MLIR tools

Source: https://github.com/xdslproject/xdsl/blob/main/docs/guides/mlir_interoperation.md

Add the MLIR build directory to the system PATH to ensure mlir-opt, mlir-translate, and clang are accessible.

```bash
export PATH=<insert-your-path>/llvm-project/build/bin:$PATH
```

--------------------------------

### Frontend package structure

Source: https://github.com/xdslproject/xdsl/blob/main/xdsl/frontend/pyast/README.md

Overview of the directory structure and responsibilities of the xDSL frontend components.

```text
.
├── utils
│   ├── block.py              : Mark functions as basic blocks using a decorator
│   ├── const.py              : Mark variables as compile-time constants using a
│   │                           type hint
│   ├── exceptions.py         : Custom exceptions for the frontend and code
│   │                           generation
│   ├── op_inserter.py        : Helper class to add operations from a stack to
│   │                           the end of an operation/region/block
│   ├── python_code_check.py  : Performs two checks for whether the code in the
│   │                           context is supported:
│   │                           1. Structure doesn't have nested block and
│   │                              blocks have explicit terminators
│   │                           2. Guaranteeing and inlining constant values
│   └── type_conversion.py    : Map between Python objects, Python AST nodes,
│                               and xDSL IR for the types and functions
├── code_generation.py        : Walk AST to generate xDSL from nodes. Simple
│                               operands and control flow supported. Explicitly
│                               no support for assignment. Implicitly no support
│                               for complex structures such as classes
├── context.py                : Context manager which parses its inner context
│                               into a provided FrontendProgram, checking its
│                               well-formedness on exit.
├── program.py                : Helper class to store, compile, and print code
└── README.md
```

--------------------------------

### Run Unit Tests

Source: https://github.com/xdslproject/xdsl/blob/main/CONTRIBUTING.md

Execute unit tests using pytest or the project's Makefile.

```bash
# Run unit tests
uv run pytest
# or via makefile
make pytest
```

--------------------------------

### Use filecheckize Tool

Source: https://github.com/xdslproject/xdsl/blob/main/CONTRIBUTING.md

Generate FileCheck directives from IR input using the filecheckize utility.

```sh
echo "%c1 = arith.constant 1 : index
%res = arith.addi %c1, %c1 : index
" | uvx filecheckize
```

```mlir
// CHECK:       %c1 = arith.constant 1 : index
// CHECK-NEXT:  %res = arith.addi %c1, %c1 : index
```

--------------------------------

### MLIR IO Implementation

Source: https://github.com/xdslproject/xdsl/wiki/IO-Example-through-LLVM-stdlib

Demonstrates declaring external C functions and using them within an MLIR module to perform basic IO operations.

```mlir
module{
	// Declare external puts/gets functions. 
	func.func private @puts(!llvm.ptr<i8>) -> i32
	func.func private @gets(!llvm.ptr<i8>) -> !llvm.ptr<i8>
	
	// MLIR's func dialect doesn't handle variadics at this point.
	// To use e.g. printf, one have to use the llvm dialect at the time of writing.
	// It can then be called with llvm.call, which has the same syntax as func.call!
	llvm.func @printf(!llvm.ptr<i8>) -> i32
	
	// Declare your main function
	func.func @main() -> (i32){
		// Allocate some buffer for your data
		%N = arith.constant 1024 : i64
		%buffer = llvm.alloca %N x i8 : (i64) -> !llvm.ptr<i8>
		
		// Just call gets on this buffer
		%ptr = func.call @gets(%buffer) : (!llvm.ptr<i8>) -> !llvm.ptr<i8>
		// Call puts on its returned pointer
		%puts = func.call @puts(%ptr) : (!llvm.ptr<i8>) -> i32
		
		// Return puts's returned value, just for fun
		return %puts : i32
	}
}
```

```mlir
"builtin.module"() ({
  // Declare external puts/gets functions.
  "func.func"() ({
  }) {function_type = (!llvm.ptr<i8>) -> i32, sym_name = "puts", sym_visibility = "private"} : () -> ()
  "func.func"() ({
  }) {function_type = (!llvm.ptr<i8>) -> !llvm.ptr<i8>, sym_name = "gets", sym_visibility = "private"} : () -> ()

  // MLIR's func dialect doesn't handle variadics at this point.
  // To use e.g. printf, one have to use the llvm dialect at the time of writing.
  // It can then be called with llvm.call, which has the same syntax as func.call!
  "llvm.func"() ({
  }) {CConv = #llvm.cconv<ccc>, function_type = !llvm.func<i32 (ptr<i8>)>, linkage = #llvm.linkage<external>, sym_name = "printf"} : () -> ()

  // Declare your main function
  "func.func"() ({
    // Allocate some buffer for your data
    %0 = "arith.constant"() {value = 1024 : i64} : () -> i64
    %1 = "llvm.alloca"(%0) : (i64) -> !llvm.ptr<i8>
		
    // Just call gets on this buffer
    %2 = "func.call"(%1) {callee = @gets} : (!llvm.ptr<i8>) -> !llvm.ptr<i8>
    // Call puts on its returned pointer
    %3 = "func.call"(%2) {callee = @puts} : (!llvm.ptr<i8>) -> i32
		
    // Return puts's returned value, just for fun
    "func.return"(%3) : (i32) -> ()
  }) {function_type = () -> i32, sym_name = "main"} : () -> ()
}) : () -> ()
```

--------------------------------

### Convert MLIR to LLVM IR

Source: https://github.com/xdslproject/xdsl/blob/main/docs/guides/mlir_interoperation.md

Run the conversion pipeline on an MLIR input file and translate the result to LLVM IR.

```bash
mlir-opt --convert-scf-to-cf --convert-cf-to-llvm --convert-func-to-llvm \
         --convert-arith-to-llvm --expand-strided-metadata --normalize-memrefs \
         --memref-expand --fold-memref-alias-ops --finalize-memref-to-llvm \
         --reconcile-unrealized-casts input.mlir | mlir-translate --mlir-to-llvmir > tmp.ll
```

--------------------------------

### Run Canonicalization Test Command

Source: https://github.com/xdslproject/xdsl/blob/main/docs/index.md

Command to execute the RISC-V canonicalization tests using the xdsl-opt tool.

```bash
xdsl-opt --split-input-file -p canonicalize tests/filecheck/backend/riscv/canonicalize.mlir
```

--------------------------------

### Configure Pyright for xDSL

Source: https://github.com/xdslproject/xdsl/blob/main/README.md

Required configuration in pyproject.toml to enable experimental features needed for xDSL type checking.

```toml
[tool.pyright]
enableExperimentalFeatures = true
```

--------------------------------

### Define and parse a simple Python function to MLIR

Source: https://github.com/xdslproject/xdsl/blob/main/xdsl/frontend/pyast/README.md

Initialize a PyASTContext, register mappings for types and operations, and use the @ctx.parse_program decorator to convert a Python function into an xDSL module.

```python
from xdsl.dialects.arith import AddfOp, ConstantOp, MulfOp
from xdsl.dialects.builtin import FloatAttr, f64
from xdsl.frontend.pyast.context import PyASTContext

ctx = PyASTContext()
ctx.register_type(float, f64)
ctx.register_function(float.__add__, AddfOp)
ctx.register_function(float.__mul__, MulfOp)
ctx.register_literal(float, lambda v: ConstantOp(FloatAttr(v, 64)))

@ctx.parse_program
def foo(x: float, y: float) -> float:
    return x + y * 5.0

print(foo.module)
```

--------------------------------

### Run FileCheck Tests

Source: https://github.com/xdslproject/xdsl/blob/main/CONTRIBUTING.md

Execute file-based tests using the lit test runner.

```bash
# Run filecheck tests
uv run lit tests/filecheck
# or via makefile
make filecheck
```

--------------------------------

### Generate Coverage Reports

Source: https://github.com/xdslproject/xdsl/blob/main/CONTRIBUTING.md

Commands to generate and combine coverage data for unit and FileCheck tests.

```bash
# Run coverage for unit tests and filecheck tests, then combine data files
make coverage
# Generate a coverage report
make coverage-report
# Or generate an HTML coverage report
make coverage-report-html
```

--------------------------------

### Run Individual Coverage Tests

Source: https://github.com/xdslproject/xdsl/blob/main/CONTRIBUTING.md

Commands to run coverage analysis for specific test categories.

```bash
# Run coverage for unit tests only
make coverage-tests
# Run coverage for filecheck tests only
make coverage-filecheck-tests
```

--------------------------------

### MLIR output for the toy program

Source: https://github.com/xdslproject/xdsl/blob/main/xdsl/frontend/pyast/README.md

The resulting MLIR representation generated from the Python function.

```mlir
builtin.module {
  func.func @foo(%x: f64, %y: f64) -> f64 {
    %0 = arith.constant 5.000000e+00 : f64
    %1 = arith.mulf %y, %0 : f64
    %2 = arith.addf %x, %1 : f64
    func.return %2 : f64
  }
}
```

--------------------------------

### Register benchmarks for local profiling

Source: https://github.com/xdslproject/xdsl/blob/main/benchmarks/README.md

Associate benchmark functions with a name using the profile call at the end of the script.

```python
from bench_utils import BenchmarkFunction, profile

profile(
    {
        # [snip...]
        "Dialects.newdialect_load": BenchmarkFunction(time_newdialect_load),
        # [snip...]
    }
)
```

--------------------------------

### Define benchmarks as class methods or standalone functions

Source: https://github.com/xdslproject/xdsl/blob/main/benchmarks/README.md

Benchmarks can be implemented as methods within a BenchmarkClass or as standalone functions, using the @safe_to_repeat decorator for independent calls.

```python
import importlib

import xdsl.dialects.newdialect
from benchmarks.bench_utils import BenchmarkClass, safe_to_repeat

class ImportDialects(BenchmarkClass):
    @safe_to_repeat
    def time_newdialect_load(self) -> None:
        """Time loading the `newdialect` dialect."""
        importlib.reload(xdsl.dialects.newdialect)
```

```python
import importlib

import xdsl.dialects.newdialect
from benchmarks.bench_utils import safe_to_repeat

@safe_to_repeat
def time_newdialect_load() -> None:
    """Time loading the `newdialect` dialect."""
    importlib.reload(xdsl.dialects.newdialect)
```
Tokens:
Raw