Portable Hardware Locality (hwloc)

https://www.open-mpi.org/projects/hwloc/doc/v2.12.1
Admin
hwloc is a software project that discovers and presents hardware resources and their locality in...
Tokens:6,411
Snippets:26
Trust Score:9.8
Update:1 year ago
Show doc for...
Tokens:
Raw
### Print Topology Tree Structure

Source: https://www.open-mpi.org/projects/hwloc/doc/v2.12.1/index

This example shows how to print the hardware topology in a tree-like structure. It utilizes a recursive helper function (print_children) starting from the root object obtained via hwloc_get_root_obj.

```C
/* Second example: Walk the topology with a tree style. */
printf("*** Printing overall tree\n");
print_children(topology, hwloc_get_root_obj(topology), 0);
```

--------------------------------

### hwloc API: Print Topology Example

Source: https://www.open-mpi.org/projects/hwloc/doc/v2.12.1/index

A C program demonstrating the hwloc API to discover and print the hardware topology of a machine. It includes functions for iterating through topology objects and displaying their types and attributes.

```c
/* Example hwloc API program.
 *
 * See other examples under doc/examples/ in the source tree
 * for more details.
 *
 * Copyright © 2009-2016 Inria. All rights reserved.
 * Copyright © 2009-2011 Université Bordeaux
 * Copyright © 2009-2010 Cisco Systems, Inc. All rights reserved.
 * See COPYING in top-level directory.
 *
 * hwloc-hello.c
 */

#include "hwloc.h"

#include <errno.h>
#include <stdio.h>
#include <string.h>

static void print_children(hwloc_topology_t topology, hwloc_obj_t obj, int depth)
{
  char type[32], attr[1024];
  unsigned i;

  hwloc_obj_type_snprintf(type, sizeof(type), obj, 0);
  printf("%*s%s", 2*depth, "", type);
  if (obj->os_index != (unsigned) -1)
    printf("#%u", obj->os_index);
  hwloc_obj_attr_snprintf(attr, sizeof(attr), obj, " ", 0);
  if (*attr)
    printf("(%s)", attr);
  printf("\n");

  for (i = 0; i < obj->arity; i++) {
    print_children(topology, obj->children[i], depth + 1);
  }
}

int main(void)
{
  int depth;
  unsigned i, n;
  unsigned long size;
  int levels;
  char string[128];
  int topodepth;
  void *m;
  hwloc_topology_t topology;
  hwloc_cpuset_t cpuset;
  hwloc_obj_t obj;

  /* Allocate and initialize topology object. */
  hwloc_topology_init(&topology);

  /* ... Optionally, put detection configuration here to ignore
     some objects types, define a synthetic topology, etc....

     The defa
```

--------------------------------

### hwloc-hello Example Output

Source: https://www.open-mpi.org/projects/hwloc/doc/v2.12.1/index

Illustrates the hierarchical output of the `hwloc-hello` utility, demonstrating how hwloc represents a machine's topology. The output shows objects at different levels, such as Machine, Package, Core, and Processing Unit (PU).

```shell
$ ./hwloc-hello
*** Objects at level 0
Index 0: Machine
*** Objects at level 1
Index 0: Package#0
Index 1: Package#1
*** Objects at level 2
Index 0: Core#0
Index 1: Core#1
Index 2: Core#3
Index 3: Core#2
*** Objects at level 3
Index 0: PU#0
Index 1: PU#1
Index 2: PU#2
Index 3: PU#3
*** Printing overall tree
Machine
  Package#0
    Core#0
      PU#0
    Core#1
      PU#1
  Package#1
    Core#3
      PU#2
    Core#2
      PU#3
*** 2 package(s) ***
Logical processor 0 has 0 caches totaling 0KB

```

--------------------------------

### Allocate and Bind Memory Example

Source: https://www.open-mpi.org/projects/hwloc/doc/v2.12.1/index

Shows how to allocate memory on a specific NUMA node and bind it using hwloc_alloc_membind. It also demonstrates binding existing memory using hwloc_set_area_membind.

```c
/* Get last node. There's always at least one. */
n = hwloc_get_nbobjs_by_type(topology, HWLOC_OBJ_NUMANODE);
obj = hwloc_get_obj_by_type(topology, HWLOC_OBJ_NUMANODE, n - 1);

size = 1024*1024;
m = hwloc_alloc_membind(topology, size, obj->nodeset, HWLOC_MEMBIND_BIND, HWLOC_MEMBIND_BYNODESET);
hwloc_free(topology, m, size);

m = malloc(size);
hwloc_set_area_membind(topology, m, size, obj->nodeset, HWLOC_MEMBIND_BIND, HWLOC_MEMBIND_BYNODESET);
free(m);
```

--------------------------------

### lstopo Textual Output Example 2 (4-package, 2-core Opteron)

Source: https://www.open-mpi.org/projects/hwloc/doc/v2.12.1/index

Textual representation of lstopo output for a 4-package, 2-core Opteron NUMA machine, showing disallowed cores and memory node sizes.

```text
Machine (32GB total)
  Package L#0
    NUMANode L#0 (P#0 8190MB)
    L2 L#0 (1024KB) + L1 L#0 (64KB) + Core L#0 + PU L#0 (P#0)
    L2 L#1 (1024KB) + L1 L#1 (64KB) + Core L#1 + PU L#1 (P#1)
  Package L#1
    NUMANode L#1 (P#1 8192MB)
    L2 L#2 (1024KB) + L1 L#2 (64KB) + Core L#2 + PU L#2 (P#2)
    L2 L#3 (1024KB) + L1 L#3 (64KB) + Core L#3 + PU L#3 (P#3)
  Package L#2
    NUMANode L#2 (P#2 8192MB)
    L2 L#4 (1024KB) + L1 L#4 (64KB) + Core L#4 + PU L#4 (P#4)
    L2 L#5 (1024KB) + L1 L#5 (64KB) + Core L#5 + PU L#5 (P#5)
  Package L#3
    NUMANode L#3 (P#3 8192MB)
    L2 L#6 (1024KB) + L1 L#6 (64KB) + Core L#6 + PU L#6 (P#6)
    L2 L#7 (1024KB) + L1 L#7 (64KB) + Core L#7 + PU L#7 (P#7)
```

--------------------------------

### Bind to CPU Set Example

Source: https://www.open-mpi.org/projects/hwloc/doc/v2.12.1/index

Demonstrates how to bind the current process to a specific CPU set using hwloc_set_cpubind. It includes error handling and printing the cpuset as a string.

```c
/* And try to bind ourself there. */
if (hwloc_set_cpubind(topology, cpuset, 0)) {
  char *str;
  int error = errno;
  hwloc_bitmap_asprintf(&str, obj->cpuset);
  printf("Couldn't bind to cpuset %s: %s\n", str, strerror(error));
  free(str);
}
/* Free our cpuset copy */
hwloc_bitmap_free(cpuset);
```

--------------------------------

### lstopo Textual Output Example 1 (4-package, 2-core)

Source: https://www.open-mpi.org/projects/hwloc/doc/v2.12.1/index

Textual representation of the lstopo output for a 4-package, 2-core machine with hyper-threading, illustrating the hierarchical structure of CPU cores, caches, and processing units.

```text
Machine
  NUMANode L#0 (P#0)
    Package L#0 + L3 L#0 (4096KB)
      L2 L#0 (1024KB) + L1 L#0 (16KB) + Core L#0
        PU L#0 (P#0)
        PU L#1 (P#8)
      L2 L#1 (1024KB) + L1 L#1 (16KB) + Core L#1
        PU L#2 (P#4)
        PU L#3 (P#12)
  Package L#1 + L3 L#1 (4096KB)
    L2 L#2 (1024KB) + L1 L#2 (16KB) + Core L#2
      PU L#4 (P#1)
      PU L#5 (P#9)
    L2 L#3 (1024KB) + L1 L#3 (16KB) + Core L#3
      PU L#6 (P#5)
      PU L#7 (P#13)
  Package L#2 + L3 L#2 (4096KB)
    L2 L#4 (1024KB) + L1 L#4 (16KB) + Core L#4
      PU L#8 (P#2)
      PU L#9 (P#10)
    L2 L#5 (1024KB) + L1 L#5 (16KB) + Core L#5
      PU L#10 (P#6)
      PU L#11 (P#14)
  Package L#3 + L3 L#3 (4096KB)
    L2 L#6 (1024KB) + L1 L#6 (16KB) + Core L#6
      PU L#12 (P#3)
      PU L#13 (P#11)
    L2 L#7 (1024KB) + L1 L#7 (16KB) + Core L#7
      PU L#14 (P#7)
      PU L#15 (P#15)
```

--------------------------------

### lstopo Textual Output Example 3 (2-package Quad-core Xeon)

Source: https://www.open-mpi.org/projects/hwloc/doc/v2.12.1/index

Textual representation of lstopo output for a 2-package quad-core Xeon processor (pre-Nehalem), detailing dual-core dies within each package.

```text
Machine (total 16GB)
  NUMANode L#0 (P#0 16GB)
  Package L#0
    L2 L#0 (4096KB)
      L1 L#0 (32KB) + Core L#0 + PU L#0 (P#0)
      L1 L#1 (32KB) + Core L#1 + PU L#1 (P#4)
    L2 L#1 (4096KB)
      L1 L#2 (32KB) + Core L#2 + PU L#2 (P#2)
      L1 L#3 (32KB) + Core L#3 + PU L#3 (P#6)
  Package L#1
    L2 L#2 (4096KB)
      L1 L#4 (32KB) + Core L#4 + PU L#4 (P#1)
      L1 L#5 (32KB) + Core L#5 + PU L#5 (P#5)
    L2 L#3 (4096KB)
      L1 L#6 (32KB) + Core L#6 + PU L#6 (P#3)
      L1 L#7 (32KB) + Core L#7 + PU L#7 (P#7)
```

--------------------------------

### Load and Traverse Topology by Depth

Source: https://www.open-mpi.org/projects/hwloc/doc/v2.12.1/index

This example demonstrates loading the hardware topology and then iterating through all objects at each depth level, printing their type and index. It uses functions like hwloc_topology_load, hwloc_topology_get_depth, hwloc_get_nbobjs_by_depth, hwloc_get_obj_by_depth, and hwloc_obj_type_snprintf.

```C
/* Perform the topology detection. */
hwloc_topology_load(topology);

/* Optionally, get some additional topology information in case we need the topology depth later. */
topodepth = hwloc_topology_get_depth(topology);

/* First example: Walk the topology with an array style, from level 0 (always the system level) to the lowest level (always the proc level). */
for (depth = 0; depth < topodepth; depth++) {
  printf("*** Objects at level %d\n", depth);
  for (i = 0; i < hwloc_get_nbobjs_by_depth(topology, depth);
       i++) {
  hwloc_obj_type_snprintf(string, sizeof(string),
  hwloc_get_obj_by_depth(topology, depth, i), 0);
  printf("Index %u: %s\n", i, string);
  }
}
```

--------------------------------

### Count Packages

Source: https://www.open-mpi.org/projects/hwloc/doc/v2.12.1/index

This example demonstrates how to count the number of packages in the system. It first determines the depth of the HWLOC_OBJ_PACKAGE type using hwloc_get_type_depth and then retrieves the count of objects at that depth using hwloc_get_nbobjs_by_depth.

```C
/* Third example: Print the number of packages. */
depth = hwloc_get_type_depth(topology, HWLOC_OBJ_PACKAGE);
if (depth == HWLOC_TYPE_DEPTH_UNKNOWN) {
  printf("*** The number of packages is unknown\n");
} else {
  printf("*** %u package(s)\n",
  hwloc_get_nbobjs_by_depth(topology, depth));
}
```

--------------------------------

### Bind to a Single Thread on the Last Core

Source: https://www.open-mpi.org/projects/hwloc/doc/v2.12.1/index

This example demonstrates how to bind the current process to a single thread of the last core in the system. It finds the last core, gets its cpuset, singlifies it to a single PU, and then applies the binding. It uses functions like hwloc_get_type_or_below_depth, hwloc_get_obj_by_depth, hwloc_get_nbobjs_by_depth, hwloc_bitmap_dup, and hwloc_bitmap_singlify.

```C
/* Fifth example: Bind to only one thread of the last core of the machine. */
/* First find out where cores are, or else smaller sets of CPUs if the OS doesn't have the notion of a "core". */
depth = hwloc_get_type_or_below_depth(topology, HWLOC_OBJ_CORE);

/* Get last core. */
obj = hwloc_get_obj_by_depth(topology, depth,
 hwloc_get_nbobjs_by_depth(topology, depth) - 1);
if (obj) {
  /* Get a copy of its cpuset that we may modify. */
  cpuset = hwloc_bitmap_dup(obj->cpuset);

  /* Get only one logical processor (in case the core is SMT/hyper-threaded). */
  hwloc_bitmap_singlify(cpuset);
```

--------------------------------

### Calculate Cache Size for a Logical Processor

Source: https://www.open-mpi.org/projects/hwloc/doc/v2.12.1/index

This example calculates the total cache size and the number of cache levels above the first logical processor (PU). It iterates through the parent objects of a PU, checking if they are cache types using hwloc_obj_type_is_cache and summing their sizes.

```C
/* Fourth example: Compute the amount of cache that the first logical processor has above it. */
levels = 0;
size = 0;
for (obj = hwloc_get_obj_by_type(topology, HWLOC_OBJ_PU, 0);
     obj;
     obj = obj->parent)
  if (hwloc_obj_type_is_cache(obj->type)) {
    levels++;
    size += obj->attr->cache.size;
  }
printf("*** Logical processor 0 has %d caches totaling %luKB\n",
  levels, size / 1024);
```

--------------------------------

### hwloc Command-Line Tools Overview

Source: https://www.open-mpi.org/projects/hwloc/doc/v2.12.1/index

hwloc offers command-line utilities to display hardware topology in various formats. These tools help users understand the machine's architecture and are essential for debugging and configuration.

```text
lstopo
lstopo-no-graphics
```

--------------------------------

### hwloc Build and Compilation

Source: https://www.open-mpi.org/projects/hwloc/doc/v2.12.1/index

Provides guidance on integrating hwloc into custom projects. It highlights the use of `pkg-config` for obtaining necessary compiler and linker flags, and mentions support for build systems like GNU Make and CMake.

```c
// Example of using pkg-config to get flags:
// gcc my_program.c -o my_program $(pkg-config --cflags --libs hwloc)

// For CMake, you would typically use find_package(hwloc REQUIRED) and target_link_libraries(... hwloc::hwloc)

```

--------------------------------

### hwloc API: Core Functions and Objects

Source: https://www.open-mpi.org/projects/hwloc/doc/v2.12.1/index

Documentation for key hwloc API functions and object types used for topology discovery and manipulation. This includes initialization, attribute retrieval, and type formatting.

```APIDOC
hwloc_topology_t:
  Represents the hardware topology. It is initialized using hwloc_topology_init() and destroyed using hwloc_topology_destroy().

hwloc_obj_t:
  Represents a single object in the hardware topology (e.g., Core, PU, Cache, NUMA Node).
  Attributes:
    os_index: Operating system specific index for the object.
    arity: The number of direct children objects.
    children: An array of pointers to the direct children objects.

hwloc_topology_init(hwloc_topology_t *topologyp):
  Allocates and initializes a topology object. This is the first step in using the hwloc API.

hwloc_obj_type_snprintf(char *type, size_t size, hwloc_obj_t obj, const char *format):
  Formats the type of a topology object into a string. The format string can control the output, e.g., "%%s" for the basic type name.

hwloc_obj_attr_snprintf(char *attr, size_t size, hwloc_obj_t obj, const char *separator, int flags):
  Formats the attributes of a topology object into a string. The separator is used between attribute key-value pairs. Flags can modify the output format.
```

--------------------------------

### hwloc Topology Export Formats

Source: https://www.open-mpi.org/projects/hwloc/doc/v2.12.1/index

hwloc can display the hardware topology in multiple human-readable formats, including graphical and text-based representations. Some formats may require additional support libraries.

```text
Plain text
LaTeX tikzpicture
PDF
PNG
FIG
```

--------------------------------

### hwloc Development/Debugging: Synthetic Topologies

Source: https://www.open-mpi.org/projects/hwloc/doc/v2.12.1/index

For development and debugging, hwloc supports working with 'fake' or synthetic topologies. This includes generating symmetrical resource trees or simulating remote machines via XML files.

```text
Symmetrical tree of resources generated from a list of level arities.
Remote machine simulation through the gathering of topology as XML files.
```

--------------------------------

### hwloc Supported Operating Systems

Source: https://www.open-mpi.org/projects/hwloc/doc/v2.12.1/index

hwloc is designed for portability and supports a wide range of operating systems commonly found in high-performance computing environments.

```text
* Linux
* Solaris
* AIX
* Darwin / OS X
* FreeBSD and its variants
* NetBSD
* HP-UX
* Microsoft Windows
* IBM BlueGene/Q Compute Node Kernel (CNK)
```

--------------------------------

### hwloc Programming Interface Capabilities

Source: https://www.open-mpi.org/projects/hwloc/doc/v2.12.1/index

The hwloc C API allows programmatic manipulation of hardware topologies and objects. It includes a powerful CPU bitmap API for defining object locations and supports binding applications to specific cores or memory nodes.

```c
// Example: Manipulating topologies and objects
// hwloc_topology_t topology;
// hwloc_obj_t root = hwloc_get_obj_by_depth(topology, 0);

// Example: CPU bitmap API
// hwloc_cpuset_t cpuset;
// hwloc_alloc_cpuset_from_nodeset(topology, nodeset, &cpuset);

// Example: Binding applications
// hwloc_set_cpubind(topology, cpuset, HWLOC_CPUBIND_THREAD);
// hwloc_set_membind(topology, nodeset, HWLOC_MEMBIND_FIRSTTOUCH);
```

--------------------------------

### hwloc Topology Depth and Object Retrieval Functions

Source: https://www.open-mpi.org/projects/hwloc/doc/v2.12.1/index

Provides functions to query the depth of objects within the hardware topology, count objects of specific types, and retrieve the root object. These functions are essential for understanding the hierarchical structure of the system.

```APIDOC
hwloc_get_nbobjs_by_type(topology, type)
  Returns the width of level type type.
  Parameters:
    topology: The topology object.
    type: The object type to query.
  Returns: The number of objects of the specified type.

hwloc_get_type_depth(topology, type)
  Returns the depth of objects of type type.
  Parameters:
    topology: The topology object.
    type: The object type to query.
  Returns: The depth of the specified object type, or HWLOC_TYPE_DEPTH_UNKNOWN if no object of that type exists.

hwloc_get_type_or_below_depth(topology, type)
  Returns the depth of objects of type type or below.
  Parameters:
    topology: The topology object.
    type: The object type to query.
  Returns: The depth of the specified object type or the first type below it.

hwloc_topology_get_depth(topology)
  Get the depth of the hierarchical tree of objects.
  Parameters:
    topology: The topology object.
  Returns: The maximum depth of the topology.

hwloc_get_root_obj(topology)
  Returns the top-object of the topology-tree.
  Parameters:
    topology: The topology object.
  Returns: A pointer to the root topology object.

HWLOC_TYPE_DEPTH_UNKNOWN
  Constant indicating that no object of the given type exists in the topology.
```

--------------------------------

### hwloc Memory Allocation and Freeing

Source: https://www.open-mpi.org/projects/hwloc/doc/v2.12.1/index

Provides functions for allocating and freeing memory, specifically designed to work with hwloc's memory binding policies.

```APIDOC
hwloc_free(topology, addr, len)
  Free memory that was previously allocated by hwloc_alloc() or hwloc_alloc_membind().
  Parameters:
    topology: The topology object.
    addr: The address of the memory to free.
    len: The length of the memory block.
  Returns: 0 on success, -1 on error.
```

--------------------------------

### hwloc CPU and Memory Binding Functions

Source: https://www.open-mpi.org/projects/hwloc/doc/v2.12.1/index

Functions for setting CPU and memory binding policies for the current process or thread. This includes allocating memory with specific policies and binding existing memory areas.

```APIDOC
hwloc_set_cpubind(topology, set, flags)
  Bind current process or thread on CPUs given in physical bitmap set.
  Parameters:
    topology: The topology object.
    set: The CPU set to bind to.
    flags: Binding flags.
  Returns: 0 on success, -1 on error.

hwloc_alloc_membind(topology, len, set, policy, flags)
  Allocate some memory on NUMA memory nodes specified by set.
  Parameters:
    topology: The topology object.
    len: The amount of memory to allocate.
    set: The nodeset specifying where to allocate memory.
    policy: The memory binding policy.
    flags: Allocation flags.
  Returns: A pointer to the allocated memory, or NULL on error.

hwloc_set_area_membind(topology, addr, len, set, policy, flags)
  Bind the already-allocated memory identified by (addr, len) to the NUMA node(s) specified by set.
  Parameters:
    topology: The topology object.
    addr: The starting address of the memory area.
    len: The length of the memory area.
    set: The nodeset specifying where to bind the memory.
    policy: The memory binding policy.
    flags: Binding flags.
  Returns: 0 on success, -1 on error.

HWLOC_MEMBIND_BYNODESET
  Constant indicating that the bitmap argument should be considered as a nodeset.

HWLOC_MEMBIND_BIND
  Constant indicating that memory should be allocated on the specified nodes.
```

--------------------------------

### hwloc Object Structure Fields

Source: https://www.open-mpi.org/projects/hwloc/doc/v2.12.1/index

Details common fields within the hwloc object structure, providing insights into hardware topology representation. These fields are fundamental for navigating and understanding the hierarchical structure of computing resources.

```c
typedef struct hwloc_obj {
  unsigned arity;
  hwloc_obj_type_t type;
  union hwloc_obj_attr_u * attr;
  struct hwloc_obj * parent;
  // ... other fields
} hwloc_obj_t;

typedef struct hwloc_obj_attr_u {
  struct hwloc_obj_attr_u::hwloc_cache_attr_s * cache;
  // ... other attribute types
} hwloc_obj_attr_u_t;

typedef struct hwloc_obj_attr_u::hwloc_cache_attr_s {
  hwloc_uint64_t size;
  // ... other cache attributes
} hwloc_cache_attr_s_t;

```

--------------------------------

### hwloc Bitmap Manipulation Functions

Source: https://www.open-mpi.org/projects/hwloc/doc/v2.12.1/index

A set of functions for creating, duplicating, freeing, and manipulating hardware locality bitmaps, which are used to represent sets of CPUs or memory nodes.

```APIDOC
hwloc_bitmap_asprintf(strp, bitmap)
  Stringify a bitmap into a newly allocated string in the default hwloc format.
  Parameters:
    strp: Pointer to a char pointer where the new string will be stored.
    bitmap: The bitmap to stringify.
  Returns: 0 on success, -1 on error.

hwloc_bitmap_free(bitmap)
  Free bitmap bitmap.
  Parameters:
    bitmap: The bitmap to free.

hwloc_bitmap_singlify(bitmap)
  Keep a single index among those set in bitmap bitmap.
  Parameters:
    bitmap: The bitmap to modify.
  Returns: 0 on success, -1 on error.

hwloc_bitmap_dup(bitmap)
  Duplicate bitmap bitmap by allocating a new bitmap and copying bitmap contents.
  Parameters:
    bitmap: The bitmap to duplicate.
  Returns: A pointer to the duplicated bitmap, or NULL on error.
```

--------------------------------

### hwloc Object Attribute and Type Stringification

Source: https://www.open-mpi.org/projects/hwloc/doc/v2.12.1/index

Utility functions for converting topology object attributes and types into human-readable string formats. These are useful for debugging and displaying topology information.

```APIDOC
hwloc_obj_attr_snprintf(string, size, obj, separator, verbose)
  Stringify the attributes of a given topology object into a human-readable form.
  Parameters:
    string: Buffer to store the string.
    size: Size of the buffer.
    obj: The topology object whose attributes to stringify.
    separator: String to use between attributes.
    verbose: Verbosity level.
  Returns: The number of characters written, or a negative value on error.

hwloc_obj_type_snprintf(string, size, obj, verbose)
  Stringify the type of a given topology object into a human-readable form.
  Parameters:
    string: Buffer to store the string.
    size: Size of the buffer.
    obj: The topology object whose type to stringify.
    verbose: Verbosity level.
  Returns: The number of characters written, or a negative value on error.
```

--------------------------------

### hwloc Topology Object Structure

Source: https://www.open-mpi.org/projects/hwloc/doc/v2.12.1/index

Defines the structure of a topology object, which represents a component in the hardware locality hierarchy. It includes fields for children, nodesets, OS indices, and CPU sets.

```APIDOC
hwloc_obj
  Structure of a topology object.

  Fields:
    children: Normal children, children[0 .. arity -1].
    nodeset: NUMA nodes covered by this object or containing this object.
    os_index: OS-provided physical index number. It is not guaranteed unique across the entire machine,...
    cpuset: CPU set covering the processing units included in this object.
```
Tokens:
Raw