### Example of Printing Schema from DataBag in Koda

Source: https://github.com/google/koladata/blob/main/docs/error_message.md

This example demonstrates how to assemble a more readable error message by printing the schema from a DataBag during the Status propagation path, adding valuable context to errors.

```c++
// Example of printing schema from DataBag for error clarification
// ...
std::cout << "Schema: " << data_bag.get_schema() << std::endl;
// ...
```

--------------------------------

### Create an Empty Bag

Source: https://github.com/google/koladata/blob/main/docs/fundamentals.md

Shows how to initialize an empty bag using `kd.bag()`. This is useful when starting with no data or for building a bag incrementally.

```python
kd.bag()  # empty bag
```

--------------------------------

### Koda Functor Tracing Example

Source: https://github.com/google/koladata/blob/main/docs/fundamentals.md

Illustrates the default tracing behavior of `kd.fn` with a simple subtraction operation. This example shows how Koda can generate a computational graph for operations.

```python
# Normally, tracing is enough.
fn = kd.fn(lambda x: x - kd.agg_min(x))
a = kd.slice([[1, 2, 3], [4, 5, 6]])
fn(a)  # [[0, 1, 2], [0, 1, 2]]
```

--------------------------------

### Schema Creation Examples

Source: https://github.com/google/koladata/blob/main/docs/cheatsheet.md

Demonstrates creating explicit schemas using Koda's schema functions for named types, lists, and dictionaries.

```python
# Create a named schema
Point = kd.named_schema('Point', x=kd.INT32, y=kd.FLOAT64)
```

```python
# Attribute 'end' can be anything
Line = kd.named_schema('Line', start=Point, end=kd.OBJECT)
```

```python
# Get the attribute start's schema
Line.start
```

```python
# Check if it is an Entity schema
assert Point.is_entity_schema()
assert Line.is_entity_schema()
```

```python
# List schema
ls1 = kd.list_schema(kd.INT64)
```

```python
# List entity schema is 's1'
ls2 = kd.list_schema(Point)
```

```python
# Get the List item schema
ls2.get_item_schema()
```

```python
# Check if it is a List schema
assert ls2.is_list_schema()
```

--------------------------------

### Work with DataItems and Complex Structures

Source: https://github.com/google/koladata/blob/main/docs/overview.md

Examples of creating and accessing scalar DataItems, lists, and dictionaries.

```python
>>> kd.item(1, schema=kd.FLOAT32)
DataItem(1.0, schema: FLOAT32)
>>> kd.item(1, schema=kd.FLOAT32).to_py()
1.0
>>> kd.list([10, 20, 30, 40])[2]
DataItem(30, schema: INT32, bag_id...)
>>> kd.dict({1: 'a', 2: 'b'})[2]
DataItem('b', schema: STRING, bag_id...)
>>> kd.from_py([{'a': [1, 2, 3], 'b': [4, 5, 6]}, {'a': 3, 'b': 4}])
DataItem(List[Dict{...'a'=List[1, 2, 3]...}, Dict{...'a'=3...}], schema: OBJECT, bag_id...)
>>> d = kd.to_py(kd.from_py({'a': [1, 2, 3], 'b': [4, 5, 6]}))
>>> dict(sorted(d.items()))
{'a': [1, 2, 3], 'b': [4, 5, 6]}
>>> kd.slice([kd.list([1, 2, 3]), kd.list([4, 5])])  # DataSlice of lists
DataSlice([List[1, 2, 3], List[4, 5]], schema: LIST[INT32], ...)
>>> kd.slice([kd.dict({'a':1, 'b':2}), kd.dict({'c':3})])  # DataSlice of dicts
DataSlice([Dict{...'a'=1...}, Dict{'c'=3}], schema: DICT{STRING, INT32}, ...)
```

--------------------------------

### Create a simple DataSlice

Source: https://github.com/google/koladata/blob/main/docs/fundamentals.md

Example of creating a simple DataSlice with irregular inner list lengths.

```python
ds = kd.slice([[1,2,3], [4,5]])
```

--------------------------------

### Get Schema from Root

Source: https://github.com/google/koladata/blob/main/py/koladata/ext/storage/colab/DataSliceManager_basics.ipynb

Retrieve the schema from the root object, which represents the data structure. This is similar to getting the schema from the manager.

```python
print(root.get_schema())
```

--------------------------------

### kd_ext.vis.DescendMode Enum Example

Source: https://github.com/google/koladata/blob/main/docs/api/kd_ext/vis.md

Example demonstrating the usage of an Enum, specifically for creating a collection of name/value pairs. Shows attribute access, value lookup, name lookup, and iteration.

```python
class Color(Enum):
    RED = 1
    BLUE = 2
    GREEN = 3

# Access them by:

# - attribute access:

# >>> Color.RED
# <Color.RED: 1>

# - value lookup:

# >>> Color(1)
# <Color.RED: 1>

# - name lookup:

# >>> Color['RED']
# <Color.RED: 1>

# Enumerations can be iterated over, and know how many members they have:

# >>> len(Color)
# 3

# >>> list(Color)
# [<Color.RED: 1>, <Color.BLUE: 2>, <Color.GREEN: 3>]
```

--------------------------------

### Schema Narrowing Example

Source: https://github.com/google/koladata/blob/main/docs/deep_dive/schema.md

Demonstrates schema narrowing for attribute assignment and list indexing. The input is narrowed before assignment or access.

```python
schema = kd.schema.new_schema(x=kd.INT32)
# The input is narrowed to `kd.item(1, schema=kd.INT32)` before being assigned.
schema.new(x=kd.item(1, schema=kd.OBJECT))  # DataItem(1, schema: INT32)

str_list = kd.list(['foo', 'bar'])
# The index is narrowed to `kd.item(0, schema=kd.INT64)`.
str_list[kd.item(0, schema=kd.OBJECT)]  # DataItem('foo', schema: STRING)

ints = kd.slice([[1, 2], [3]])
# The `ndim` is narrowed to `kd.item(2, schema=kd.INT64)`.
kd.agg_sum(ints, ndim=kd.item(2, schema=kd.OBJECT))  # DataItem(6, schema=INT32)
```

--------------------------------

### Create and manipulate Koladata Objects

Source: https://github.com/google/koladata/blob/main/docs/cheatsheet.md

Examples of creating objects, accessing attributes, and performing updates on object attributes.

```python
>>> o = kd.obj(x=1, y=2)
>>> os = kd.obj(x=kd.slice([1, 2, None]),
...             y=kd.slice([4, None, 6]))

>>> os = kd.slice([kd.obj(x=1),
...                kd.obj(y=2.0),
...                kd.obj(x=1.0, y='a')])

>>> os.get_schema()
DataItem(OBJECT, schema: SCHEMA, ...)
>>> os.get_obj_schema()
DataSlice([
  IMPLICIT_ENTITY(x=INT32),
  IMPLICIT_ENTITY(y=FLOAT32),
  IMPLICIT_ENTITY(x=FLOAT32, y=STRING),
], schema: SCHEMA, ...)
```

```python
>>> itemid = kd.new_itemid()
>>> o1 = kd.obj(x=1, y=2, itemid=itemid)
>>> o2 = kd.obj(x=1, y=2, itemid=itemid)
>>> assert o1.get_itemid() == o2.get_itemid()
```

```python
>>> os1 = kd.slice([kd.obj(x=1), kd.obj(x=1.0, y='a')])

>>> kd.dir(os1)
Traceback (most recent call last):
    ...
ValueError: dir() cannot determine attribute names because objects have different attributes. Please specify intersection= explicitly.

>>> kd.dir(os1, intersection=True)
['x']

>>> kd.dir(os1, intersection=False)
['x', 'y']
```

```python
>>> o.x
DataItem(1, schema: INT32, ...)
>>> o.get_attr('y')
DataItem(2, schema: INT32, ...)
>>> o.maybe('z')
DataItem(None, schema: NONE, ...)
>>> o.get_attr('z', default=0)
DataItem(0, schema: INT32, ...)
>>> os.get_attr('x', default=0)
DataSlice([1.0, 0.0, 1.0], schema: FLOAT32, ...)
```

```python
>>> o = kd.obj(x=1, y=2)

>>> o1 = o.with_attr('x', 3)
>>> o1 = o.with_attr('z', 4)

>>> o1 = o.with_attr('y', 'a')

>>> o1 = o.with_attr('x', None)

>>> o2 = o.with_attrs(z=4, x=None)

>>> o2 = o.with_attrs(z=4, y='a')

>>> upd = kd.attrs(o, z=4, y=10)
>>> o3 = o.updated(upd)

>>> o4 = o.updated(kd.attrs(o, z=4), kd.attrs(o, y=None))

>>> nested = kd.obj(a=kd.obj(c=kd.obj(e=1), d=2), b=3)
>>> nested = nested.updated(kd.attrs(nested.a.c, e=4),
...                         kd.attrs(nested.a, d=5),
...                         kd.attrs(nested, b=6))
```

```python
>>> l = kd.list([1, 2, 3])
>>> l_obj = kd.obj(l)
>>> l_obj[:]
DataSlice([1, 2, 3], schema: INT32, ...)

>>> d = kd.dict({'a': 1, 'b': 2})
>>> d_obj = kd.obj(d)
>>> kd.sort(d_obj.get_keys())
DataSlice(['a', 'b'], schema: STRING, ...)
>>> d_obj['a']
DataItem(1, schema: INT32, ...)

>>> e = kd.new(x=1, y=2)
>>> e_obj = kd.obj(e)

>>> p_obj = kd.obj(1)
>>> p_obj = kd.obj('a')
```

--------------------------------

### Define Protocol Buffer Messages

Source: https://github.com/google/koladata/blob/main/docs/cheatsheet.md

Example proto definitions including extensions and nested messages.

```proto
message Query {
  string query_text = 1;
  float final_ir = 2;
  repeated Doc docs = 3;
  repeated int32 tags = 4;
  map<string, float> term_weight = 5;
  proto2.bridge.MessageSet ms_extensions = 6;

  extensions 1000 to max
}

message QueryExtension {
  extend Query {
    QueryExtension query_extension = 1000;
  }

  extend proto2.bridge.MessageSet {
    QueryExtension ms_extension = 1000;
  }

  int32 extra = 1;
}

message Doc {
  string url = 1;
  string title = 2;
  float score = 3;
  int32 word_count = 4;
  bool spam = 5;

  enum Type {
    UNDEFINED = 0;
    WEB = 1;
    IMAGE = 2;
  }

  Type type = 6;
}
```

--------------------------------

### Create a new entity with attributes

Source: https://github.com/google/koladata/blob/main/docs/deep_dive/data_bag.md

Initializes a new entity with a specified attribute, serving as the basis for attribute lookup examples.

```python
entity = kd.new(a=1)
```

--------------------------------

### Generate a DataSlice with a range of integers

Source: https://github.com/google/koladata/blob/main/docs/api/kd/slices.md

Use `kd.slices.range` to create a DataSlice of INT64s within a specified range [start, end). If `end` is omitted, `start` becomes `end` and `start` defaults to 0. The result has one more dimension than the broadcasted shape of `start` and `end`.

```python
kd.range(5) -> kd.slice([0, 1, 2, 3, 4])
```

```python
kd.range(2, 5) -> kd.slice([2, 3, 4])
```

```python
kd.range(5, 2) -> kd.slice([])  # empty range
```

```python
kd.range(kd.slice([2, 4])) -> kd.slice([[0, 1], [0, 1, 2, 3])
```

```python
kd.range(kd.slice([2, 4]), 6) -> kd.slice([[2, 3, 4, 5], [4, 5])
```

--------------------------------

### Format JSON output

Source: https://github.com/google/koladata/blob/main/docs/api/kd/json.md

Examples of using indent and ensure_ascii arguments to control the formatting of the resulting JSON string.

```python
kd.to_json(kd.list([1, 2, 3]), indent=-1) -> '[1,2,3]'
  kd.to_json(kd.list([1, 2, 3]), indent=2) -> '[
  1,
  2,
  3
]'

  kd.to_json('✨', ensure_ascii=True) -> '"\u2728"'
  kd.to_json('✨', ensure_ascii=False) -> '"✨"'
```

--------------------------------

### Get attribute with default

Source: https://github.com/google/koladata/blob/main/docs/api/data_slice.md

A shortcut for getting an attribute with a default value of None.

```text
A shortcut for kd.get_attr(x, attr_name, default=None).
```

--------------------------------

### Get DataSlice Text in Python

Source: https://github.com/google/koladata/blob/main/py/koladata/ext/storage/colab/DataSliceManager_basics.ipynb

Retrieves and prints the current text content from a DataSlice query.

```python
print(query.text.get_data_slice())
```

--------------------------------

### Initialize and use a named container

Source: https://github.com/google/koladata/blob/main/docs/api/kd.md

Demonstrates how to use kd.named_container to store values and expressions in both eager and tracing modes.

```python
c = kd.named_container()

# 1. Non-tracing mode
# Storing a value:
c.foo = 5
c.foo       # Returns 5

# Storing an expression:
c.x_plus_y = I.x + I.y
c.x_plus_y  # Returns (I.x + I.y).with_name('x_plus_y')

# Listing stored items:
vars(c)  # Returns {'foo': 5, 'x_plus_y': (I.x + I.y).with_name('x_plus_y')}

# 2. Tracing mode
def my_fn(x):
  c = kd.named_container()
  c.a = 2
  c.b = 1
  return c.a * x + c.b

fn = kd.fn(my_fn)
fn.a  # Returns 2 (accessible because it was named by the container)
fn(x=5)  # Returns 11
```

--------------------------------

### Get DataSlice Locale in Python

Source: https://github.com/google/koladata/blob/main/py/koladata/ext/storage/colab/DataSliceManager_basics.ipynb

Retrieves and prints the current locale information from a DataSlice query.

```python
print(query.locale.get_data_slice())
```

--------------------------------

### Get Unique Values from DataSlice

Source: https://context7.com/google/koladata/llms.txt

Extracts the unique values from a DataSlice. This is equivalent to getting the first item from each group when grouping by value.

```python
print(kd.unique(ds).to_py())
```

--------------------------------

### Create and Compare Dictionary Schemas

Source: https://github.com/google/koladata/blob/main/docs/fundamentals.md

Illustrates how to infer schemas for dictionaries from their key-value pairs or create them directly using `kd.dict_schema`. It also demonstrates comparing inferred and directly created dictionary schemas.

```python
kd.dict({'1': 2}).get_schema()  # Dict{STRING, INT32}
kd.dict({'1': 2}).get_schema() == kd.dict({'3': 4}).get_schema()
kd.dict_schema(kd.STRING, kd.INT32) # create a dict schema directly
kd.dict_schema(kd.STRING, kd.INT32) == kd.dict({'1': 2}).get_schema()
```

--------------------------------

### Example: Transform Variables in Functor

Source: https://github.com/google/koladata/blob/main/docs/api/kd/functor.md

This example demonstrates how to use `kd.functor.visit_variables` to transform variables within a functor. It modifies the 'a' attribute of a specific functor.

```python
f = kd.fn(lambda x: kd.V.a * x + kd.V.b).with_attrs(a=42, b=37)

def transform_fn(fn, sub_vars):
  if fn == f:
    return fn.with_attrs(a=12)
  return fn

visitor.visit_variables(f, transform_fn)
# Returns result equivalent to:
# kd.fn(lambda x: kd.V.a * x + kd.V.b).with_attrs(a=12, b=37)
```

--------------------------------

### Creating and Converting Objects

Source: https://github.com/google/koladata/blob/main/docs/fundamentals.md

Illustrates the creation of structured objects using `kd.obj` and conversion of Python data structures (like lists of dictionaries) into Koladata objects using `kd.from_py`.

```python
# Create structured objects directly
kd.obj(x=1, y=2)  # Obj(x=1, y=2)
kd.obj(x=1, z=kd.obj(a=3, b=4))  # Obj(x=1, z=Obj(a=3, b=4))
```

```python
# Convert primitives or entities into objects
kd.obj(1)  # 1, but kd.OBJECT
kd.obj(kd.new(x=1, y=2))  # similar to kd.obj(x=1, y=2), with some differences discussed later
```

```python
# Convert python list of dicts of lists of dicts
# from_py returns an object by default when no schema is provided
x = kd.from_py([{'d': [{'a': 1, 'b': 2}, {'a': 3, 'b': 4}]}, {'d': [{'a': 5, 'b': 6}]}])
x[1]['d'][0]['a']  # 5
x[0]['d'][1].get_values()  # [3, 4]
```

--------------------------------

### Create and Compare List Schemas

Source: https://github.com/google/koladata/blob/main/docs/fundamentals.md

Shows how to infer schemas for lists from their contents or create them directly using `kd.list_schema`. It also demonstrates comparing inferred and directly created list schemas.

```python
kd.list([1, 2]).get_schema()  # LIST[INT32]
kd.list([1, 2]).get_schema() == kd.list([3, 4]).get_schema()  # yes
kd.list_schema(kd.INT32) # create a list schema directly
kd.list([1, 2]).get_schema() == kd.list_schema(kd.INT32) # yes
```

--------------------------------

### Get Representative Values from Grouped Data

Source: https://context7.com/google/koladata/llms.txt

Retrieves the first element from each group in a grouped DataSlice. This is useful for getting a representative value for each unique key.

```python
print(grouped.S[0].to_py())
```

--------------------------------

### Create and Manage Entities and Schemas

Source: https://github.com/google/koladata/blob/main/docs/overview.md

Shows how to create entities with explicit or named schemas and handle nested entity structures.

```python
# kd.new creates new entities and assigns schemas to them
>>> kd.new(x=1, y=2, schema='Point')
DataItem(Entity(x=1, y=2), schema: Point(x=INT32, y=INT32),...)

# Can also explicitly create schema with attributes before using.
>>> my_schema = kd.named_schema('Point', x=kd.INT32, y=kd.INT32)
>>> x = kd.new(x=1, y=2, schema=my_schema)
>>> x.get_schema() == my_schema # Yes, i.e. a present mask.
DataItem(present, schema: MASK)

# When converting from py, can specify schema
>>> kd.from_py({'x': 1, 'y': 2}, schema=my_schema)
DataItem(Entity(x=1, y=2), schema: Point(x=INT32, y=INT32),...)

# It's possible to create nested entities
>>> x = kd.new(a=1, b=kd.new(c=3, schema='Inner'), schema='Outer')
>>> x
DataItem(Entity(a=1, b=Entity(c=3)), schema: Outer(a=INT32, b=Inner(c=INT32)),...)
```

--------------------------------

### DataSlice Boxing Example

Source: https://github.com/google/koladata/blob/main/docs/deep_dive/schema.md

Demonstrates how the schema of a DataSlice is determined by the common schema of its elements. The example shows that the common schema of an INT32 and a FLOAT32 is FLOAT32.

```python
kd.slice([1, 2.0]).get_schema() # -> FLOAT32
```

--------------------------------

### Creating and Slicing Entities with Schemas

Source: https://github.com/google/koladata/blob/main/docs/fundamentals.md

Demonstrates how to create entities with explicit schemas and slice lists of entities, ensuring all entities in the slice share the same schema.

```python
my_schema = kd.named_schema('Point')
kd.new(x=1, y=2, schema=my_schema)  # set explicit schema
kd.new(x=1, y=2, schema='Point').get_schema() == kd.new(x=1, y=2, schema=my_schema).get_schema()  # yes
```

```python
kd.slice([kd.new(x=1, y=2, schema=my_schema), kd.new(x=2, y=3, schema=my_schema)]) # works
kd.slice([kd.new(x=1, y=2, schema='Point'), kd.new(x=2, y=3, schema='Point')]) # works
```

```python
a, b = kd.new(x=1, y=2), kd.new(x=2, y=3)
kd.slice([a.with_schema(my_schema), b.with_schema(my_schema)])  # works
kd.slice([a, b.with_schema(a.get_schema())])  # works
```

--------------------------------

### Get Revision History in Python

Source: https://github.com/google/koladata/blob/main/py/koladata/ext/storage/colab/DataSliceManager_basics.ipynb

Retrieves the revision history of a manager, specifying a timezone. Useful for auditing changes.

```python
manager.get_revision_history(tz=pytz.timezone('Europe/Zurich'))
```

--------------------------------

### Find first occurrence of substring

Source: https://github.com/google/koladata/blob/main/docs/api/kd/strings.md

Finds the starting offset of the first occurrence of a substring within a string or byte sequence. Supports specifying start and end search boundaries.

```python
kd.strings.find(s, substr, start=0, end=None)
```

--------------------------------

### Creating Entity Objects with kd.obj()

Source: https://github.com/google/koladata/blob/main/docs/common_pitfalls.md

Shows how to create objects from entity data using variadic keyword arguments or by passing a Koda entity created with kd.new().

```python
>>> # Entity objects
>>> kd.obj(a=1, b=2)
DataItem(Obj(a=1, b=2), schema: OBJECT,...)
>>> kd.obj(kd.new(a=1, b=2))
DataItem(Obj(a=1, b=2), schema: OBJECT,...)
```

--------------------------------

### Get Schema from Manager

Source: https://github.com/google/koladata/blob/main/py/koladata/ext/storage/colab/DataSliceManager_basics.ipynb

Retrieve the schema of the data managed by the DataSliceManager. This is useful for understanding the structure of the data before loading it.

```python
print(manager.get_schema())
```

--------------------------------

### Initialize Proto Messages

Source: https://github.com/google/koladata/blob/main/docs/cheatsheet.md

Python instantiation of the defined proto messages.

```python
d1 = Doc(
    url='url 1',
    title='title 1',
    word_count=10,
    spam=False,
    type=test_pb2.Doc.Type.WEB,
),
d2 = Doc(
    url='url 2',
    title='title 2',
    score=1.0,
    spam=True,
    type=test_pb2.Doc.Type.IMAGE,
)

q = Query(
    query_text='query 1',
    final_ir=1.0,
    tags=[1, 2, 3],
    term_weight={'a': 1.0, 'b': 2.0},
    docs=[d1, d2],
)

q.Extensions[QueryExtension.query_extension].extra = 1
q.ms_extensions.Extensions[QueryExtension.ms_extension].extra = 2
```

--------------------------------

### Get Data in Parallel in Python

Source: https://github.com/google/koladata/blob/main/py/koladata/ext/storage/colab/DataSliceManager_basics.ipynb

This function demonstrates parallel data loading by requesting multiple features at once. The `root.get(populate=needed_views.values())` call triggers parallel loading from disk and caching.

```python
def get_data_parallel(
    manager: DataSliceManager,
) -> dict[str, kd.types.DataSlice]:
  root = DataSliceManagerView(manager)
  query = root.query[:]
  doc = query.doc[:]

  needed_views = {
      'query_text': query.text,
      'doc_title': doc.title,
      'doc_id': doc.id,
  }
  # Here is the crux of parallel loading: we ask for a DataSlice that is
  # populated with an arbitrary set of features. The needed bags are loaded in
  # parallel from disk and then cached.
  root.get(populate=needed_views.values())
  # Now we just pick up the loaded bags from the cache:
  return {name: view.get() for name, view in needed_views.items()}
```

--------------------------------

### Using sparsity operators

Source: https://github.com/google/koladata/blob/main/docs/fundamentals.md

Demonstrates operators that follow sparsity, shape, or both.

```python
x = kd.slice([[1, None], [None, 3, 4]])
kd.val_like(x, 9)  # [[9, None], [None, 9, 9]]
9 & kd.has(x)  # The same as above
kd.present_shaped_as(x)  # [[present, present], [present, present, present]]
kd.val_shaped_as(x, 9)  # [[9, 9], [9, 9, 9]]
9 & kd.present_shaped_as(x)  # the same as above
```

--------------------------------

### Get Data Sequentially in Python

Source: https://github.com/google/koladata/blob/main/py/koladata/ext/storage/colab/DataSliceManager_basics.ipynb

Use this function to load data slices sequentially. The index structure of queries and docs is cached across calls, optimizing repeated access.

```python
def get_data_sequential(
    manager: DataSliceManager,
) -> dict[str, kd.types.DataSlice]:
  root = DataSliceManagerView(manager)
  query = root.query[:]
  doc = query.doc[:]

  return dict(
      # The calls to get_data_slice() here are run in sequence. However, the
      # index structure of queries and docs is cached across the calls.
      query_text=query.text.get_data_slice(),
      doc_title=doc.title.get_data_slice(),
      doc_id=doc.id.get_data_slice(),
  )


def convert_to_pure_python(data: dict[str, kd.types.DataSlice]):
  return kd.new(**data).flatten().to_py(obj_as_dict=True)
```

--------------------------------

### Calculate Start and End Indices using Split Points

Source: https://github.com/google/koladata/blob/main/docs/deep_dive/jagged_shape.md

Shows how to compute start and end indices for slicing using split points, which represent cumulative sizes. This is more efficient than using sizes directly.

```python
# Our index to use.
i = 2
# The sizes of the relevant dimension.
split_points = [0, 2, 3, 6]

start = split_points[i]  # -> 3
end = split_points[i + 1]  # -> 6
data[start:end]  # -> ['d', 'e', 'f']
```

--------------------------------

### Create entities and objects

Source: https://context7.com/google/koladata/llms.txt

Use kd.new for entities with explicit schemas and kd.obj for objects with embedded schemas, including immutable attribute updates.

```python
from koladata import kd

# Create an entity with kd.new (auto-creates schema)
entity = kd.new(name="Alice", age=30, score=95.5)
print(entity)
# DataItem(Entity(age=30, name='Alice', score=95.5), schema: ENTITY(...), bag_id: ...)

print(entity.name)
# DataItem('Alice', schema: STRING, bag_id: ...)

# Create an object with kd.obj (uses OBJECT schema)
obj = kd.obj(name="Bob", age=25)
print(obj)
# DataItem(Obj(age=25, name='Bob'), schema: OBJECT, bag_id: ...)

# Create multiple entities at once
names = kd.slice(['Alice', 'Bob', 'Carol'])
ages = kd.slice([30, 25, 35])
people = kd.new(name=names, age=ages)
print(people.name.to_py())
# ['Alice', 'Bob', 'Carol']

# Update attributes immutably
updated_entity = entity.with_attrs(score=98.0, grade='A')
print(updated_entity.score)
# DataItem(98.0, schema: FLOAT32, bag_id: ...)
```

--------------------------------

### Calculate Start and End Indices using Sizes

Source: https://github.com/google/koladata/blob/main/docs/deep_dive/jagged_shape.md

Illustrates how to calculate the start and end indices for a slice within flattened data using dimension sizes. This method involves summing prefix sizes.

```python
# Our index to use.
i = 2
# The sizes of the relevant dimension. Since indexing reduces the
# dimensionality, this is the second dimension.
sizes = [2, 1, 3]

start = sum(sizes[:i])  # -> 3
end = start + sizes[i]  # -> 6
data[start:end]  # -> ['d', 'e', 'f']
```

--------------------------------

### Convert Entities to Objects and Vice Versa

Source: https://github.com/google/koladata/blob/main/docs/overview.md

Demonstrates converting Koda entities to objects using `kd.obj` and converting objects back to entities, including applying custom schemas.

```pycon-doctest
>>> x, y = kd.new(a=1), kd.new(b=2)
>>> kd.slice([kd.obj(x), kd.obj(y)])  # convert both entities to objects
DataSlice([Obj(a=1), Obj(b=2)], schema: OBJECT,...)
```

```pycon-doctest
# Objects can be converted to entities
>>> my_schema = kd.named_schema('Point', x=kd.INT32, y=kd.INT32)
>>> a = kd.obj(x=1, y=2).with_schema(my_schema); a
DataItem(Entity(x=1, y=2), schema: Point(x=INT32, y=INT32),...)
```

```pycon-doctest
>>> a2 = kd.from_py({'x': 1, 'y': 2}, dict_as_obj=True).with_schema(my_schema)  # the same as above
>>> kd.testing.assert_equivalent(a, a2)
```

--------------------------------

### Example of Improved Error Message in Koda

Source: https://github.com/google/koladata/blob/main/docs/error_message.md

This example demonstrates an improved error message that includes a user-oriented explanation alongside the technical details. It shows how a `ValueError` related to a missing schema in a `DataItem` can be clarified.

```python
>>> obj = kd.obj(a=1)
>>> obj.with_bag(kd.bag()).a
...

ValueError: object schema is missing for the DataItem whose item is: $0000ShTxCVvMzRgwUPt5cb

  DataItem with the kd.OBJECT schema usually store its schema as an attribute or implicitly hold the type information when it's a primitive type. Perhaps, the OBJECT schema is set by mistake with
  foo.with_schema(kd.OBJECT) when 'foo' does not have stored schema attribute.

The above exception was the direct cause of the following exception:

ValueError: failed to get attribute 'a'
```

--------------------------------

### Verify View Hierarchy

Source: https://github.com/google/koladata/blob/main/py/koladata/ext/storage/colab/DataSliceManager_basics.ipynb

Demonstrates navigating the parent-child relationships within the Koladata schema.

```python
assert query.get_grandparent() == root
```

```python
for child in query.get_children():
  print(child.get_path_from_root())
  assert child.get_parent() == query
```

```python
doc = query.doc[:]

assert doc.title.get_parent() == doc
```

--------------------------------

### Get Schema of Input

Source: https://github.com/google/koladata/blob/main/docs/api/kd/schema.md

Returns the schema of the input `x`.

```python
kd.get_schema(x)
```

--------------------------------

### Get DataSlice Text from Branched Manager After Update in Python

Source: https://github.com/google/koladata/blob/main/py/koladata/ext/storage/colab/DataSliceManager_basics.ipynb

Retrieves and prints the text data slice from a branched manager after it has been updated.

```python
print(root_before_updating_query_text.query[:].text.get_data_slice())
```

--------------------------------

### kd.strings.substr

Source: https://github.com/google/koladata/blob/main/docs/api/kd/strings.md

Extracts substrings from a DataSlice based on start and end indices.

```APIDOC
## kd.strings.substr(x, start=0, end=None)

### Description
Returns a DataSlice of substrings with indices [start, end). Negative indices are computed from the end of the string.

### Parameters
#### Arguments
- **x** (Text or Bytes DataSlice) - Required - The input DataSlice.
- **start** (int) - Optional - The start index (inclusive). Defaults to 0.
- **end** (int) - Optional - The end index (exclusive). Defaults to the length of the string.
```

--------------------------------

### Create a 3-Dimensional DataSlice

Source: https://github.com/google/koladata/blob/main/docs/fundamentals.md

Illustrates the creation of a 3-dimensional DataSlice with nested lists. This example shows how to initialize a DataSlice with irregular partitioning at different dimensions.

```python
# Root
# ├── dim_1:0
# │   ├── dim_2:0
# │   │   ├── dim_3:0 -> 1
# │   │   └── dim_3:1 -> 2
# │   └── dim_2:1
# │       ├── dim_3:0 -> 3
# │       ├── dim_3:1 -> 4
# │       └── dim_3:2 -> 5
# └── dim_1:1
#     ├── dim_2:0
#     │   └── dim_3:0 -> 6
#     ├── dim_2:1 (Empty)
#     └── dim_2:2
#         ├── dim_3:0 -> 7
#         ├── dim_3:1 -> 8
#         ├── dim_3:2 -> 9
#         └── dim_3:3 -> 10
ds = kd.slice([[[1, 2], [3, 4, 5]], [[6], [], [7, 8, 9, 10]]])
```

--------------------------------

### Update DataSlice Text in Python

Source: https://github.com/google/koladata/blob/main/py/koladata/ext/storage/colab/DataSliceManager_basics.ipynb

Updates the text content of a query in a DataSlice. This example shows how to append new text while preserving existing data.

```python
query.text = (
    kd.slice(['How high is the Eiffel tower in Paris', None])
    | query.text.get_data_slice(),
    'Updated the text of the first query about the Eiffel tower',
)
```

--------------------------------

### Get Dictionary Size

Source: https://context7.com/google/koladata/llms.txt

Returns the number of key-value pairs in a dictionary DataItem.

```python
print(d.dict_size())
```

--------------------------------

### Creating Primitive DataSlices with Schemas

Source: https://github.com/google/koladata/blob/main/docs/fundamentals.md

Demonstrates creating primitive DataSlices with default or specified schemas like INT32, INT64, and FLOAT64. Use `kd.slice` or type-specific constructors.

```python
kd.slice([1, 2, 3]) # INT32 is chosen by default when converting from Python
kd.slice([1, 2, 3], schema=kd.INT32) # the same as above
kd.slice([1, 2, 3], schema=kd.INT64) # can specify INT64 schema
kd.int64([1, 2, 3])  # the same as above

kd.slice([1., 2., 3.], schema=kd.FLOAT64) # can specify FLOAT64 schema
kd.float64([1., 2., 3.])  # the same as above
```

--------------------------------

### Get Original DataSlice Text After Branch Update in Python

Source: https://github.com/google/koladata/blob/main/py/koladata/ext/storage/colab/DataSliceManager_basics.ipynb

Retrieves and prints the text data slice from the original manager to show that it remains unaffected by updates to the branch.

```python
print(root.query[:].text.get_data_slice())
```

--------------------------------

### Retrieve object from view with get

Source: https://github.com/google/koladata/blob/main/docs/api/kd_ext/kv/view.md

Returns the underlying object represented by the view.

```python
view('foo').get()
  # 'foo'
  view([[1,2],[3]])[:].get()
  # ([1,2],[3]).
  view([[1,2],[3]])[:][:].get()
  # ((1,2),(3,)).
```

--------------------------------

### Create and Inspect Koda Objects

Source: https://github.com/google/koladata/blob/main/docs/overview.md

Demonstrates the creation of Koda objects using `kd.obj` and `kd.uuobj`, and how to inspect their structure and universally unique IDs. Also shows conversion from Python lists to Koda objects.

```pycon-doctest
>>> kd.obj(x=2, y=kd.obj(z=3))
DataItem(Obj(x=2, y=Obj(z=3)), schema: OBJECT, bag_id:...)
```

```pycon-doctest
>>> x = kd.uuobj(x=2, y=kd.uuobj(z=3))  # universally unique (always the same id)
>>> kd.encode_itemid(x)  # always the same id
DataItem('07...', schema: STRING)
```

```pycon-doctest
>>> x = kd.from_py([{'a': 1, 'b': 2}, {'c': 3, 'd': 4}], dict_as_obj=True)
>>> x[0]
DataItem(Obj(a=1, b=2), schema: OBJECT,...)
```

```pycon-doctest
>>> x[:].maybe('a')
DataSlice([1, None], schema: OBJECT,...)
```

--------------------------------

### Get dimension mapping

Source: https://github.com/google/koladata/blob/main/docs/api/kd/shapes.md

Returns the parent-to-child mapping of a specific dimension in a JaggedShape.

```python
shape = kd.shapes.new([2], [3, 2], [1, 2, 0, 2, 1])
  kd.shapes.dim_mapping(shape, 0) # -> kd.slice([0, 0])
  kd.shapes.dim_mapping(shape, 1) # -> kd.slice([0, 0, 0, 1, 1])
  kd.shapes.dim_mapping(shape, 2) # -> kd.slice([0, 1, 1, 3, 3, 4])
```

--------------------------------

### Entity Creation in Koladata

Source: https://github.com/google/koladata/blob/main/docs/fundamentals.md

Illustrates how to create structured objects (entities) with schemas. Entities can be nested, and schemas can be auto-allocated or explicitly created.

```python
kd.new(x=1, y=2, schema='Point')  # Entity(x=1, y=2)
r = kd.new(x=1, y=2, z=kd.new(a=3, b=4, schema='Data'), schema='PointWithData')  # nested Entity
r.z.a  # 3

# kd.new can also auto-allocate schemas
kd.new(x=1, y=2, schema='Point') == kd.new(x=1, y=2, schema='Point')  # yes
kd.new(x=1, y=2).get_schema() == kd.new(x=1, y=2).get_schema()  # no

# Schemas can also be created explicitly.
# kd.named_schema('Point') creates exactly the same schemas as the one created
```

--------------------------------

### Get Number of Dimensions of DataSlice

Source: https://github.com/google/koladata/blob/main/docs/api/kd/slices.md

Returns the number of dimensions of the input DataSlice `x`.

```python
Returns the number of dimensions of DataSlice `x`.
```

--------------------------------

### Get Dict Values

Source: https://github.com/google/koladata/blob/main/docs/deep_dive/data_slice.md

Use `get_values()` to retrieve all values from a Dict. This operation is vectorized.

```python
dicts.get_values()
```

--------------------------------

### Create and Inspect DataSlices

Source: https://github.com/google/koladata/blob/main/docs/overview.md

Demonstrates creating a 2D DataSlice and inspecting its dimensions and shape.

```python
>>> from koladata import kd
>>> ds = kd.slice([["one", "two", "three"], ["four", "five"]])
>>> ds.get_ndim()
DataItem(2, schema: INT64)
>>> ds.get_shape()
JaggedShape(2, [3, 2])
```

--------------------------------

### Get Dict Keys

Source: https://github.com/google/koladata/blob/main/docs/deep_dive/data_slice.md

Use `get_keys()` to retrieve all keys from a Dict. This operation is vectorized.

```python
dicts.get_keys()
```

--------------------------------

### Get Value Schema of Dict Schema

Source: https://github.com/google/koladata/blob/main/docs/api/kd/schema.md

Returns the value schema of a Dict schema.

```python
kd.schema.get_value_schema(dict_schema)
```

--------------------------------

### Create and inspect DataSlices

Source: https://github.com/google/koladata/blob/main/docs/deep_dive/data_slice.md

Demonstrates creating 2D jagged and 1D mixed-type DataSlices, along with methods to retrieve raw data, schema, and DataBag references.

```python
# 2D jagged DataSlice of integers:
ds = kd.slice([[1, 2], [3]])

ds.flatten().to_py()  # Python list: [1, 2, 3]
ds.get_schema()       # INT32
ds.get_bag()          # None - no DataBag reference.
                      # Values of Koda primitives like INT32s are stored inside
                      # the DataSlice, so no DataBag is needed.

# 1D DataSlice of mixed data:
ds = kd.slice(['a', 2])

ds.to_py()       # raw data: ['a', 2]
ds.get_schema()  # OBJECT - specifies that the actual type of the data is given
                 # by the individual items in the slice, and is not a property
                 # of the entire DataSlice.
ds.get_bag()     # None - STRING and INT are Koda primitives, so this behaves
                 # just like INT32 above.
```

--------------------------------

### Set up Dispatching for Operators

Source: https://github.com/google/koladata/blob/main/docs/creating_operators.md

Configures operator dispatching in my_project.py to handle eager, tracing, and lazy execution modes. This setup ensures operators can be accessed through different paths like my_project.cond, my_project.eager.cond, and my_project.lazy.cond.

```python
import sys as _sys
import types as _py_types
import typing as _typing

from koladata import kd as _kd
from koladata.expr import tracing_mode as _tracing_mode
from koladata.operators import eager_op_utils as _eager_op_utils

from my.namespace import lazy_ops as _lazy_ops


_HAS_DYNAMIC_ATTRIBUTES = True

_tracing_config = {}
_dispatch = lambda eager, tracing: _tracing_mode.configure_tracing(
    _tracing_config, eager=eager, tracing=tracing
)
_eager_only = lambda obj: _tracing_mode.eager_only(_tracing_config, obj)
_same_when_tracing = lambda obj: _tracing_mode.same_when_tracing(
    _tracing_config, obj
)


def _init_ops_and_containers():
  eager_ops = _eager_op_utils.operators_container(
      top_level_arolla_container=_lazy_ops.my_project_ops
  )
  for op_or_container_name in dir(eager_ops):
    globals()[op_or_container_name] = _dispatch(
        eager=getattr(eager_ops, op_or_container_name),
        tracing=getattr(_lazy_ops.my_project_ops, op_or_container_name),
    )


_init_ops_and_containers()


lazy = _eager_only(_lazy_ops.my_project_ops)
eager = _same_when_tracing(_py_types.ModuleType('eager'))


__all__ = [api for api in globals().keys() if not api.startswith('_')]


def __dir__():  # pylint: disable=invalid-name
  return __all__


# `eager` has eager versions of everything, available even in tracing mode.
def _set_up_eager():
  for name in __all__:
    if name != 'eager':
      setattr(eager, name, globals()[name])
  eager.__all__ = [x for x in __all__ if x != 'eager']
  eager.__dir__ = lambda: eager.__all__


_set_up_eager()

# Set up the tracing mode machinery. This must be the last thing in this file.
if not _typing.TYPE_CHECKING:
  _sys.modules[__name__] = _tracing_mode.prepare_module_for_tracing(
      _sys.modules[__name__], _tracing_config
  )
```

--------------------------------

### Get Key Schema of Dict Schema

Source: https://github.com/google/koladata/blob/main/docs/api/kd/schema.md

Returns the key schema of a Dict schema.

```python
kd.schema.get_key_schema(dict_schema)
```

--------------------------------

### Get Item Schema of List Schema

Source: https://github.com/google/koladata/blob/main/docs/api/kd/schema.md

Returns the item schema of a List schema.

```python
kd.schema.get_item_schema(list_schema)
```

--------------------------------

### Create Entities with Deterministic IDs

Source: https://github.com/google/koladata/blob/main/docs/fundamentals.md

Use `kd.uu` and `kd.uuobj` as shortcuts to create entities and objects with deterministically computed IDs and set their attributes. `kd.new` provides equivalent functionality.

```python
kd.uu(x=1, y=2)
```

```python
kd.new(itemid=kd.uuid(x=1, y=2), x=1, y=2)  # the same as above
```

```python
kd.uu(x=1, y=2).get_itemid() == kd.uuid(x=1, y=2)  # yes
```

```python
a = kd.uuobj(x=1, y=2, z=kd.uuobj(a=3, b=4))
a.z == kd.uuobj(a=3, b=4)  # yes
```

--------------------------------

### Get dimension sizes

Source: https://github.com/google/koladata/blob/main/docs/api/kd/shapes.md

Returns the row sizes at the provided dimension in the given shape.

```python
shape = kd.shapes.new([2], [2, 1])
  kd.shapes.dim_sizes(shape, 0)  # -> kd.slice([2])
  kd.shapes.dim_sizes(shape, 1)  # -> kd.slice([2, 1])
```

--------------------------------

### Create Universally Unique Schemas

Source: https://github.com/google/koladata/blob/main/docs/deep_dive/schema.md

Shows how to create schemas that are universally unique rather than dynamically allocated.

```python
assert kd.uu_schema(x=kd.INT32) == kd.uu_schema(x=kd.INT32)
assert kd.uu(x=1).get_schema() == kd.uu(x=1).get_schema()
```

--------------------------------

### Represent lists as attributes

Source: https://github.com/google/koladata/blob/main/docs/overview.md

Example showing how lists are stored as attributes within a DataBag.

```python
c = kd.list([1, 2, 3])

c.get_bag()
# DataBag $10cc:
#  0 Entities/Objects with 0 values in 0 attrs
#  1 non empty Lists with 3 items
#  0 non empty Dicts with 0 key/value entries
#  1 schemas with 1 values
#
# Top attrs:


c.get_bag().contents_repr()
# DataBag $10cc:
# $0FAMhn8RmKvHXJvcKuKJq3[:] => [1, 2, 3]
#
# SchemaBag:
# #7QUhePdHCvsoCyAWAHwtzx.get_item_schema() => INT32
```

--------------------------------

### Represent dicts as attributes

Source: https://github.com/google/koladata/blob/main/docs/overview.md

Example showing how dicts are stored as attributes within a DataBag.

```python
b = kd.dict({'a': 1, 'b': 2})

b.get_bag()
# DataBag $4863:
#   0 Entities/Objects with 0 values in 0 attrs
#   0 non empty Lists with 0 items
#   1 non empty Dicts with 2 key/value entries
#   1 schemas with 2 values
#
# Top attrs:


b.get_bag().contents_repr()
# DataBag $4863:
# $0UGItpaGKCXaPsOxEscFOA['a'] => 1
# $0UGItpaGKCXaPsOxEscFOA['b'] => 2
#
# SchemaBag:
# #7QRy2BAblHHNytHxFmpaGL.get_key_schema() => STRING
# #7QRy2BAblHHNytHxFmpaGL.get_value_schema() => INT32
```

--------------------------------

### Create Koladata Dictionaries

Source: https://github.com/google/koladata/blob/main/docs/cheatsheet.md

Create dictionaries using slices or schemas. These methods allow for flexible dictionary initialization with provided keys, values, or item IDs.

```python
>>> key_ds = kd.slice([['a', 'b'],['c', 'd']])
>>> value_ds = kd.slice([[1, 2],[3, 4]])
>>> d1 = kd.dict_like(x, key_ds, value_ds)

>>> d2 = kd.dict_shaped_as(x, key_ds, value_ds)
>>> d3 = kd.dict_shaped(s, key_ds, value_ds)
>>> kd.testing.assert_equivalent(d2, d3)
```

```python
>>> dict_itemid = kd.new_dictid_shaped_as(x)
>>> d1 = kd.dict_like(x, itemid=dict_itemid)
>>> d2 = kd.dict_shaped_as(x, itemid=dict_itemid)
>>> d3 = kd.dict_shaped(s, itemid=dict_itemid)
>>> kd.testing.assert_equivalent(d2, d3)

>>> schema = kd.dict_schema(kd.STRING, kd.INT32)
>>> d4 = kd.dict_like(x, schema=schema)
```