### Install and Use bump-my-version for Versioning

Source: https://github.com/mobidec/ckanapi_harvesters/blob/main/CONTRIBUTING.md

Demonstrates how to install and use the `bump-my-version` utility to manage project versions. It shows commands for upgrading versions (major, minor, patch, pre-release) and viewing available version changes.

```shell
pip install --upgrade bump-my-version

bump-my-version show-bump

bump-my-version bump <major|minor|patch|pre_l|pre_n>

bump-my-version bump minor

```

--------------------------------

### Install and Use Ruff for Code Formatting

Source: https://github.com/mobidec/ckanapi_harvesters/blob/main/CONTRIBUTING.md

Instructions for installing Ruff and using its commands to check and format Python code according to project standards. Ruff helps ensure code consistency and readability.

```bash
pip install ruff
ruff check --fix
ruff format
```

--------------------------------

### Install and Use Pre-commit for Git Hooks

Source: https://github.com/mobidec/ckanapi_harvesters/blob/main/CONTRIBUTING.md

Steps to install the pre-commit Python package and set up pre-commit hooks. This ensures code is formatted correctly before each commit, passing CI checks.

```bash
pip install pre-commit
pre-commit install
pre-commit run --all-files
```

--------------------------------

### Run Package Builder Example Script (Python)

Source: https://github.com/mobidec/ckanapi_harvesters/blob/main/src/ckanapi_harvesters/builder/example/README.md

This script demonstrates how to run the package builder example. It requires the CKAN URL, an API key file, and the owner organization as command-line arguments. Ensure you have the necessary CKAN client libraries installed.

```python
python example_script.py --ckan-url https://demo.ckan.org/ --apikey-file CKAN_Token.txt --owner-org demo-organization
```

--------------------------------

### Python f-string Usage Example

Source: https://github.com/mobidec/ckanapi_harvesters/blob/main/CONTRIBUTING.md

Demonstrates the recommended way to format strings in Python using f-strings, which is preferred over older methods like % or .format() for clarity and conciseness.

```python
example = f'Text with {a_variable} using f-strings.'
```

--------------------------------

### Install ckanapi_harvesters Python Package

Source: https://github.com/mobidec/ckanapi_harvesters/blob/main/sphinx/notebooks/builder_example_notebook.md

Installs the ckanapi_harvesters package with optional extras. This command is typically run in a shell environment.

```sh
pip install ckanapi_harvesters[extras]
```

--------------------------------

### Compile Sphinx Documentation

Source: https://github.com/mobidec/ckanapi_harvesters/blob/main/CONTRIBUTING.md

These shell commands install the necessary Sphinx packages and then build the HTML documentation. The documentation is built from the `./sphinx` directory and output to the `./public` directory.

```shell
pip install sphinx sphinx-rtd-theme
sphinx-build -b html ./sphinx ./public
```

--------------------------------

### Install and Test ckanapi_harvesters Package Locally

Source: https://github.com/mobidec/ckanapi_harvesters/blob/main/CONTRIBUTING.md

This bash script demonstrates the process of installing and testing the ckanapi_harvesters package locally before merging changes. It involves building the package and then installing the generated wheel file before running pytest.

```bash
pip install build
python -m build
pip install dist/ckanapi_harvesters-0.0.0-py3-none-any.whl
pytest
```

--------------------------------

### Setuptools and Versioning Configuration

Source: https://github.com/mobidec/ckanapi_harvesters/blob/main/CONTRIBUTING.md

Configures setuptools for package discovery and setuptools_scm for version management. It specifies that packages should be found in the 'src' directory and uses 'guess-next-dev' and 'dirty-tag' schemes for versioning.

```toml
[tool.setuptools.packages.find]
where = ["src"]

[tool.setuptools_scm]
version_scheme = "guess-next-dev"
local_scheme = "dirty-tag"

```

--------------------------------

### Import and Initialize CkanApi in Python

Source: https://github.com/mobidec/ckanapi_harvesters/blob/main/README.md

Demonstrates how to import and initialize the CkanApi class from the ckanapi_harvesters package in Python. This is the basic step to start using the package's functionalities.

```python
from ckanapi_harvesters import CkanApi

ckan = CkanApi()
```

--------------------------------

### Initialize CKAN API Connection and Configuration

Source: https://github.com/mobidec/ckanapi_harvesters/blob/main/sphinx/notebooks/builder_example_notebook.ipynb

Initializes a `CkanApi` instance and configures it using metadata from the `BuilderPackage` object. It prompts the user for missing CKAN information, sets request limits and timeouts, and enables verbose output to show script progress.

```python
ckan = CkanApi(None)
# you can specify the CKAN URL, owner organization, API key here or in the Excel workbook
ckan = mdl.init_ckan(ckan)
ckan.input_missing_info(input_args_if_necessary=True, input_owner_org=True, error_not_found=False)  # request user input to configure CKAN
ckan.set_limits(10000)  # reduce if server hangs up
ckan.set_submit_timeout(5)
ckan.set_verbosity(True)  # this displays all the steps performed by the script
```

--------------------------------

### Enable and Load Package Metadata from Excel

Source: https://github.com/mobidec/ckanapi_harvesters/blob/main/sphinx/notebooks/builder_example_notebook.ipynb

This Python code enables external code execution for `BuilderPackage` and then loads package metadata from a specified Excel file (`example_package_xls`). It's crucial to only enable this for trusted sources due to security implications.

```python
BuilderPackage.unlock_external_code_execution()
mdl = BuilderPackage.from_excel(example_package_xls)
```

--------------------------------

### Download CKAN Package Resources

Source: https://github.com/mobidec/ckanapi_harvesters/blob/main/sphinx/notebooks/builder_example_notebook.md

Downloads all resources of a specified CKAN dataset to a local directory. It supports multi-threaded downloading for large datasets and allows skipping existing files or performing a full download. A progress callback can be provided to monitor the download status.

```ipython3
# define the destination directory
example_package_download_dir = os.path.abspath("package_download")
print("Package will be downloaded in: " + example_package_download_dir)
```

```ipython3
threads = 3  # > 1: number of threads to download large datasets
display(f)
mdl.download_request_full(ckan, example_package_download_dir, full_download=True, threads=threads,
                              skip_existing=False, progress_callback=progress_callback)
```

--------------------------------

### Ruff Linting and Formatting Configuration

Source: https://github.com/mobidec/ckanapi_harvesters/blob/main/CONTRIBUTING.md

Sets up the Ruff linter and formatter for the project. It configures the line length, excludes the 'tests' directory from linting, and selects a comprehensive set of linting rules including Pydocstyle with a NumPy convention. It also specifies single quotes and space indentation for formatting.

```toml
[tool.ruff]
line-length = 140
exclude = ["tests"]

[tool.ruff.lint]
extend-select = [
  "UP",
  "E501",
  "I",
  "B",
  "F",
  "E",
  "N",
  "A",
  "PL",
  "D"
]

[tool.ruff.lint.pydocstyle]
convention = "numpy"

[tool.ruff.format]
quote-style = "single"
indent-style = "space"
docstring-code-format = true

```

--------------------------------

### Initiate or Update CKAN Package and Resources

Source: https://github.com/mobidec/ckanapi_harvesters/blob/main/sphinx/notebooks/builder_example_notebook.md

Creates a new CKAN package or updates an existing one based on the metadata from the Excel workbook. It also initializes the package's resources. The `reupload` parameter controls whether all resource data is re-uploaded, which can be resource-intensive for large datasets.

```ipython3
reupload = True  # True: reuploads all documents and resets large datasets to the first document (not recommended if there is a large dataset)
mdl.patch_request_full(ckan, reupload=reupload)
```

--------------------------------

### Set Download Destination Directory

Source: https://github.com/mobidec/ckanapi_harvesters/blob/main/sphinx/notebooks/builder_example_notebook.ipynb

Defines the local directory where the CKAN package resources will be downloaded. This variable is used in the subsequent download function call.

```python
# define the destination directory
example_package_download_dir = os.path.abspath("package_download")
print("Package will be downloaded in: " + example_package_download_dir)
```

--------------------------------

### Initialize CKAN Package Builder from Excel

Source: https://github.com/mobidec/ckanapi_harvesters/blob/main/sphinx/notebooks/builder_example_notebook.md

Initializes the CKAN package builder from an Excel file. It optionally allows for external code execution from the Excel file, which should only be enabled for trusted sources. It then initializes the CKAN API connection and configures it with package metadata.

```ipython3
# optionally, use the ckanapi_harvesters package present in the Git directory
use_git_package = True
if use_git_package:
    import os
    cwd = os.getcwd()
    if not os.path.isdir(os.path.join(cwd, "ckanapi_harvesters")):
        # we assume we are in the examples directory
        cwd = os.path.join(cwd, r"../../src")  # aim for src directory
        assert(os.path.isdir(os.path.join(cwd, "ckanapi_harvesters")))
        os.chdir(cwd)
        print("CWD changed to: " + os.path.abspath(""))
```

```ipython3
import os
from ckanapi_harvesters import CkanApi, BuilderPackage
from ckanapi_harvesters.builder.example import example_package_xls
```

```ipython3
BuilderPackage.unlock_external_code_execution()
mdl = BuilderPackage.from_excel(example_package_xls)
```

```ipython3
ckan = CkanApi(None)
# you can specify the CKAN URL, owner organization, API key here or in the Excel workbook
ckan = mdl.init_ckan(ckan)
ckan.input_missing_info(input_args_if_necessary=True, input_owner_org=True, error_not_found=False)  # request user input to configure CKAN
ckan.set_limits(10000)  # reduce if server hangs up
ckan.set_submit_timeout(5)
ckan.set_verbosity(True)  # this displays all the steps performed by the script
```

--------------------------------

### Display CKAN Package Model DataFrames

Source: https://github.com/mobidec/ckanapi_harvesters/blob/main/sphinx/notebooks/builder_example_notebook.md

Retrieves and displays all DataFrames associated with the CKAN package model. This is useful for inspecting the metadata and structure loaded from the Excel file.

```ipython3
df_dict = mdl.get_all_df()
for tab, df in df_dict.items():
    display(f"Tab {tab}:")
    display(df)
```

--------------------------------

### Import CKAN Harvesters Modules

Source: https://github.com/mobidec/ckanapi_harvesters/blob/main/sphinx/notebooks/builder_example_notebook.ipynb

Imports necessary classes and functions from the ckanapi_harvesters library, including `CkanApi`, `BuilderPackage`, and `example_package_xls`. These are fundamental for interacting with CKAN and building dataset structures from Excel files.

```python
import os
from ckanapi_harvesters import CkanApi, BuilderPackage
from ckanapi_harvesters.builder.example import example_package_xls
```

--------------------------------

### GeoJSON Geometry Example for CKAN DataStore

Source: https://github.com/mobidec/ckanapi_harvesters/blob/main/sphinx/pages/ckan_field_naming_recommendations.md

An example of a GeoJSON geometry structure, specifically a polygon, as recommended for the 'spatial' field in CKAN DataStore for representing complex geometries. This format is recognized by CKAN visualization plugins.

```json
{
  "type":"Polygon",
  "coordinates":[[[2.05827, 49.8625],[2.05827, 55.7447], [-6.41736, 55.7447], [-6.41736, 49.8625], [2.05827, 49.8625]]]
}
```

--------------------------------

### Initialize CKAN API Harvesters Package from Git

Source: https://github.com/mobidec/ckanapi_harvesters/blob/main/sphinx/notebooks/builder_example_notebook.ipynb

This Python code snippet conditionally imports the ckanapi_harvesters package directly from the Git directory. It checks if the package exists in the current working directory and changes the directory if necessary to ensure the package can be imported correctly. This is useful for development or when running examples directly from the source.

```python
# optionally, use the ckanapi_harvesters package present in the Git directory
use_git_package = True
if use_git_package:
    import os
    cwd = os.getcwd()
    if not os.path.isdir(os.path.join(cwd, "ckanapi_harvesters")):
        # we assume we are in the examples directory
        cwd = os.path.join(cwd, r"../../src")  # aim for src directory
        assert(os.path.isdir(os.path.join(cwd, "ckanapi_harvesters")))
        os.chdir(cwd)
        print("CWD changed to: " + os.path.abspath(""))
```

--------------------------------

### Configure Sphinx for Auto Documentation

Source: https://github.com/mobidec/ckanapi_harvesters/blob/main/CONTRIBUTING.md

This reStructuredText snippet configures Sphinx to automatically generate documentation for the ckanapi_harvesters package. It uses `autopackagesummary` to create summaries and `toctree` for navigation.

```rst
auto docstring for package
===========================

   This is my package.

   .. autopackagesummary:: ckanapi_harvesters
      :toctree: ckanapi_harvesters
      :template: autosummary/package.rst

.. toctree::
   :glob:

   ckanapi_harvesters/*

```

--------------------------------

### Add Project Dependencies in pyproject.toml

Source: https://github.com/mobidec/ckanapi_harvesters/blob/main/README.md

The pyproject.toml file manages project dependencies. Dependencies are listed under the 'dependencies' section. This example shows how to add pytest with a specific version.

```toml
dependencies = [
    "pytest == 8.0.1",
    # add necessary dependencies
]
```

--------------------------------

### Make Custom CKAN API Calls

Source: https://context7.com/mobidec/ckanapi_harvesters/llms.txt

Illustrates how to make generic and custom API calls to CKAN using the ckanapi_harvesters library. It includes examples of calling arbitrary API actions with specified HTTP methods and parameters, and retrieving help documentation for any CKAN API action.

```python
from ckanapi_harvesters import CkanApi
from ckanapi_harvesters.auxiliary.ckan_auxiliary import RequestType

ckan = CkanApi(url="https://your-ckan-instance.org", apikey="your-key")

# Generic API action call
response = ckan.api_action_call(
    "organization_list",
    method=RequestType.Get,
    params={"all_fields": True}
)
if response.success:
    for org in response.result:
        print(f"Organization: {org['name']}")

# Get help for any API action
help_text = ckan.api_help_show("package_search", print_output=True)
```

--------------------------------

### Add Production Dependencies to pyproject.toml

Source: https://github.com/mobidec/ckanapi_harvesters/blob/main/CONTRIBUTING.md

This snippet shows how to add dependencies that are only required for the production environment. Dependencies are added to the `[project.optional-dependencies] > production` section of the `pyproject.toml` file.

```toml
[project.optional-dependencies]
production = [
    "dependency-name==1.0.0",
]
```

--------------------------------

### GitLab CI/CD Job Definitions

Source: https://github.com/mobidec/ckanapi_harvesters/blob/main/CONTRIBUTING.md

Defines the jobs for the GitLab CI/CD pipeline, including code formatting checks (`ruff_guardian`), unit testing (`pytest`), documentation generation (`pages`), and package uploading (`py_package_uploader`). These jobs automate the build, test, and deployment process.

```yaml
# Example structure, actual .gitlab-ci.yml would be more detailed

ruff_guardian:
  stage: lint
  script: ["ruff check --fix .", "ruff format ."

pytest:
  stage: test
  script: ["pytest --cov=."]

pages:
  stage: deploy
  script: ["sphinx-build -b html docs public"]

py_package_uploader:
  stage: deploy
  script: ["python -m build", "python -m twine upload --repository nexus dist/*"]

```

--------------------------------

### Project Configuration with pyproject.toml

Source: https://github.com/mobidec/ckanapi_harvesters/blob/main/CONTRIBUTING.md

Defines project metadata, Python version compatibility, and dependencies for the ckanapi_harvesters package. It specifies the project name, version, author information, readme file, required Python version, classifiers, and core dependencies like pytest.

```toml
[project]
name = "ckanapi_harvesters"
version = "0.0.0"
authors = [
  { name = "AUTHOR", email = "author@example.com" }
]
description = "DESCRIPTION"
readme = "README.md"
requires-python = ">=3.10"
classifiers = [
    "Programming Language :: Python :: 3.10",
    "License :: OSI Approved :: MIT License",
    "Operating System :: OS Independent",
]
dependencies = [
    "pytest == 8.0.1",
    # add necessary dependencies here
]

```

--------------------------------

### Upload Large CKAN Datasets

Source: https://github.com/mobidec/ckanapi_harvesters/blob/main/sphinx/notebooks/builder_example_notebook.md

Uploads large datasets to a CKAN package, defined locally as directories containing multiple CSV files. It concatenates these files using `datastore_upsert` in a multi-threaded manner. The number of threads can be adjusted to manage potential HTTP errors.

```ipython3
threads = 3  # > 1: multi-threading mode - reduce if HTTP 502 errors
display(f)
mdl.upload_large_datasets(ckan, threads=threads, progress_callback=progress_callback, only_missing=True)
```

--------------------------------

### Define Progress Bar Callback Function

Source: https://github.com/mobidec/ckanapi_harvesters/blob/main/sphinx/notebooks/builder_example_notebook.md

Defines a callback function to update a progress bar. This function takes the current index and total count to calculate and set the progress bar's value, useful for long-running operations like uploads or downloads.

```ipython3
from ipywidgets import IntProgress
from IPython.display import display
f = IntProgress(min=0,max=100)

def progress_callback(index:int, total:int, **kwargs):
    f.value = int(index/total*100)
```

--------------------------------

### Initialize CkanApi Client and Configure Settings

Source: https://context7.com/mobidec/ckanapi_harvesters/llms.txt

Demonstrates how to initialize the CkanApi client with authentication details (URL, API key, organization) and configure request parameters like limits, timeouts, and verbosity. It also shows an alternative initialization method using environment variables and a connection test.

```python
from ckanapi_harvesters import CkanApi

# Initialize with URL and API key
ckan = CkanApi(
    url="https://your-ckan-instance.org",
    apikey="your-api-key",
    # Or use apikey_file for secure key storage
    # apikey_file="/path/to/apikey.txt",
    owner_org="your-organization"
)

# Alternative: Initialize from environment variables
# Set CKAN_URL, CKAN_API_KEY or CKAN_API_KEY_FILE
ckan = CkanApi()
ckan.init_from_environ()

# Configure request parameters
ckan.set_limits(limit_read=1000)  # Rows per request
ckan.set_requests_timeout(120)    # Request timeout in seconds
ckan.set_verbosity(True)          # Enable verbose output

# Test connection
if ckan.test_ckan_url_reachable():
    ckan.connect()
    print(f"Connected to: {ckan.url}")
```

--------------------------------

### Import and Initialize CkanApi from Sub-module in Python

Source: https://github.com/mobidec/ckanapi_harvesters/blob/main/README.md

Shows how to import and initialize the CkanApi class when it's located in a sub-module of the ckanapi_harvesters package. This is useful for accessing specific module functionalities.

```python
from ckanapi_harvesters.ckan_api import CkanApi

ckan = CkanApi()
```

--------------------------------

### Map CKAN Packages and Resources

Source: https://context7.com/mobidec/ckanapi_harvesters/llms.txt

Illustrates how to map CKAN packages and resources to create a local cache of dataset information. It covers fetching package/resource metadata, including DataStore details, resource views, organization info, and licenses. It also shows how to retrieve specific package and resource information.

```python
from ckanapi_harvesters import CkanApi

ckan = CkanApi(url="https://your-ckan-instance.org", apikey="your-key")

# Map all packages in your organization
ckan_map = ckan.map_resources(
    owner_org="my-organization",
    datastore_info=True,         # Include DataStore metadata
    resource_view_list=True,     # Include resource views
    organization_info=True,      # Include org details
    license_list=True            # Include available licenses
)

# Access mapped data
for pkg_name, pkg_info in ckan.map.packages.items():
    print(f"Package: {pkg_info.name} - {pkg_info.title}")
    for res_name, res_info in pkg_info.package_resources.items():
        print(f"  Resource: {res_info.name} ({res_info.format})")
        if res_info.datastore_info:
            print(f"    DataStore rows: {res_info.datastore_info.row_count}")

# Get specific package info
pkg_info = ckan.package_show("my-dataset-name")
print(f"Package ID: {pkg_info.id}")
print(f"Resources: {len(pkg_info.package_resources)}")

# Get resource info
res_info = ckan.resource_show("resource-uuid-or-name", package_id="my-dataset-name")
print(f"Resource format: {res_info.format}")
print(f"Download URL: {res_info.download_url}")
```

--------------------------------

### Query CKAN DataStore Resources

Source: https://context7.com/mobidec/ckanapi_harvesters/llms.txt

Demonstrates how to query DataStore resources, retrieve data as pandas DataFrames, and handle large datasets efficiently. It covers basic searches, filtering, sorting, selecting specific fields, full-text search, downloading all records, using generators for memory efficiency, and executing SQL queries.

```python
from ckanapi_harvesters import CkanApi
import pandas as pd

ckan = CkanApi(url="https://your-ckan-instance.org", apikey="your-key")
resource_id = "12345-resource-uuid"

# Basic search - get first 100 records
df = ckan.datastore_search(
    resource_id,
    limit=100,
    search_all=False
)
print(df.head())
print(f"Total records available: {df.attrs['total']}")

# Search with filters and sorting
df = ckan.datastore_search(
    resource_id,
    filters={"status": "active", "year": 2024},
    fields=["id", "name", "value", "date"],  # Select specific columns
    sort="date desc, name asc",
    limit=500,
    search_all=False
)

# Full text search
df = ckan.datastore_search(
    resource_id,
    q="search term",
    search_all=True  # Retrieve all matching records
)

# Download entire DataStore
df_full = ckan.datastore_dump(resource_id)  # search_all=True by default
print(f"Downloaded {len(df_full)} records")

# Use generator for large datasets (memory efficient)
for df_chunk in ckan.datastore_search_generator(resource_id, limit=1000, search_all=True):
    process_chunk(df_chunk)

# SQL query (if enabled on server)
df = ckan.datastore_search_sql(
    sql=f'SELECT * FROM "{resource_id}" WHERE "VALUE" > 100 ORDER BY "DATE" DESC',
    search_all=True
)
```

--------------------------------

### Create and Update CKAN DataStore with Schema

Source: https://context7.com/mobidec/ckanapi_harvesters/llms.txt

Demonstrates how to create a CKAN datastore with specified fields, primary keys, and indexes. It also shows how to update field metadata like type, description, and label, with an option to only apply changes if needed.

```python
from ckanapi_harvesters import CkanApi

# Assuming resource_id, fields are defined elsewhere
# ckan = CkanApi(...)

# Create DataStore with schema
ckan.datastore_create(
    resource_id,
    fields=fields,
    primary_key=["id"],
    indexes=["name", "timestamp"],
    aliases=["my_data_alias"],
    records=[
        {"id": 1, "name": "First", "value": 100.0, "is_active": True}
    ]
)

# Update field metadata
update_needed, fields_new, result = ckan.datastore_field_patch(
    resource_id,
    fields_type_override={"value": "numeric"},
    field_description={"name": "User's full name", "value": "Numeric score"},
    fields_label={"name": "Name", "value": "Score"},
    only_if_needed=True
)

# Build field dict programmatically
fields_dict = CkanApi.datastore_field_dict(
    fields=fields,
    fields_type_override={"new_field": "text"},
    fields_description={"new_field": "A new field"},
    return_list=True
)
```

--------------------------------

### Configure Tox for Testing and Linting in Python

Source: https://github.com/mobidec/ckanapi_harvesters/blob/main/README.md

The tox.ini file configures the tox tool to automate testing and linting across multiple Python environments. It specifies Python 3.10, uses the ruff linter, and runs pytest for unit tests. It also supports version management with bump-my-version.

```ini
[tox]
env_list = py310

[testenv]
commands = ruff check . && ruff format . && pytest
python = 3.10

```

--------------------------------

### Automate CKAN Package Management with Excel

Source: https://context7.com/mobidec/ckanapi_harvesters/llms.txt

This Python code automates CKAN package and resource management using an Excel workbook as the definition source. It covers loading package metadata, initializing the CKAN client, creating/updating packages and resources, uploading large datasets efficiently, and downloading entire packages.

```python
from ckanapi_harvesters import CkanApi, BuilderPackage

# Enable external code execution if Excel references Python modules
BuilderPackage.unlock_external_code_execution()

# Load package definition from Excel
builder = BuilderPackage.from_excel("/path/to/package_definition.xlsx")
print(f"Package: {builder.package_name}")
print(f"Resources: {len(builder.resource_builders)}")

# Initialize CKAN client from builder settings
ckan = CkanApi()
ckan = builder.init_ckan(ckan)
ckan.input_missing_info(input_args_if_necessary=True)
ckan.set_verbosity(True)

# Create/update package and resources
builder.patch_request_full(
    ckan,
    reupload=False  # Set True to re-upload all resource files
)

# Upload large datasets (multi-threaded)
def progress_callback(index, total, **kwargs):
    print(f"Progress: {index}/{total}")

builder.upload_large_datasets(
    ckan,
    threads=3,
    progress_callback=progress_callback,
    only_missing=True
)

# Download entire package
builder.download_request_full(
    ckan,
    out_dir="/path/to/downloads",
    full_download=True,
    threads=3,
    skip_existing=False,
    progress_callback=progress_callback
)

# Export builder to JSON for inspection
builder.to_json("/path/to/package_definition.json")
```

--------------------------------

### CKAN Package Creation and Management

Source: https://context7.com/mobidec/ckanapi_harvesters/llms.txt

Create new CKAN packages (datasets) and manage their metadata, including title, description, tags, and custom fields. Supports creating, updating, and deleting packages. Requires `ckanapi_harvesters` and optionally `ckanapi_harvesters.auxiliary.ckan_model.CkanState` for state management.

```python
from ckanapi_harvesters import CkanApi
from ckanapi_harvesters.auxiliary.ckan_model import CkanState

ckan = CkanApi(url="https://your-ckan-instance.org", apikey="your-key", owner_org="my-org")
ckan.full_unlock()

# Create a new package
pkg_info = ckan.package_create(
    package_name="my-new-dataset",
    private=False,
    title="My New Dataset",
    notes="Description of the dataset with **markdown** support",
    tags=["environment", "open-data", "2024"],
    license_id="cc-by",
    version="1.0.0",
    author="Data Team",
    author_email="data@example.org",
    custom_fields={
        "temporal_coverage": "2020-2024",
        "geographic_coverage": "France"
    },
    cancel_if_exists=True,   # Don't error if exists
    update_if_exists=True    # Update metadata if exists
)
print(f"Package created: {pkg_info.id}")
print(f"Newly created: {pkg_info.newly_created}")

# Update package metadata
pkg_info = ckan.package_patch(
    package_id=pkg_info.id,
    title="Updated Dataset Title",
    notes="Updated description",
    tags=["environment", "updated"],
    state=CkanState.Active
)

# Change package visibility
ckan.package_patch(pkg_info.id, private=True)

# Delete package (mark as deleted, recoverable)
ckan.params.enable_admin = True
ckan.package_delete(pkg_info.id, definitive_delete=False)

```

--------------------------------

### CKAN DataStore Schema Creation

Source: https://context7.com/mobidec/ckanapi_harvesters/llms.txt

Define and create CKAN DataStore schemas with specified field types, primary keys, and indexes. Uses a list of dictionaries to define fields and their properties. Requires `ckanapi_harvesters` and optionally `ckanapi_harvesters.auxiliary.ckan_model.CkanField`.

```python
from ckanapi_harvesters import CkanApi
from ckanapi_harvesters.auxiliary.ckan_model import CkanField
import pandas as pd

ckan = CkanApi(url="https://your-ckan-instance.org", apikey="your-key")
ckan.full_unlock()

resource_id = "resource-uuid"

# Define schema with field types
fields = [
    {"id": "id", "type": "int"},
    {"id": "name", "type": "text"},
    {"id": "value", "type": "numeric"},
    {"id": "timestamp", "type": "timestamp"},
    {"id": "is_active", "type": "bool"},
    {"id": "metadata", "type": "json"}
]

# Example of creating a DataStore with this schema (assuming resource_id exists)
# ckan.datastore_create(resource_id, fields=fields, primary_key="id")

```

--------------------------------

### CKAN Resource Creation and Management

Source: https://context7.com/mobidec/ckanapi_harvesters/llms.txt

Create resources within CKAN packages, uploading files, DataFrames, or linking to external URLs. Supports updating and deleting resources. Requires `ckanapi_harvesters` and `pandas` for DataFrame uploads. Can initialize DataStores for resources.

```python
from ckanapi_harvesters import CkanApi
import pandas as pd

ckan = CkanApi(url="https://your-ckan-instance.org", apikey="your-key", owner_org="my-org")
ckan.full_unlock()

package_id = "my-dataset-name"

# Create resource from DataFrame
df = pd.DataFrame({"col1": [1, 2, 3], "col2": ["a", "b", "c"]})
res_info = ckan.resource_create(
    package_id,
    name="my-data-table",
    format="CSV",
    description="Sample data table",
    df=df,
    create_default_view=True,
    datastore_create=True,      # Initialize DataStore
    primary_key="col1",         # Set primary key
    cancel_if_exists=True,
    update_if_exists=True
)
print(f"Resource ID: {res_info.id}")

# Create resource from file
res_info = ckan.resource_create(
    package_id,
    name="documentation",
    format="PDF",
    description="User documentation",
    file_path="/path/to/document.pdf"
)

# Create resource from URL
res_info = ckan.resource_create(
    package_id,
    name="external-data",
    format="JSON",
    description="External API data",
    url="https://api.example.com/data.json"
)

# Update resource file
ckan.resource_patch(
    res_info.id,
    file_path="/path/to/updated_document.pdf"
)

# Delete resource
ckan.params.enable_admin = True
ckan.resource_delete(res_info.id)

```

--------------------------------

### Download CKAN File Resources

Source: https://context7.com/mobidec/ckanapi_harvesters/llms.txt

Provides Python code to download file resources from CKAN. It includes functions to download the raw file content, load CSV data directly into a Pandas DataFrame, and test the accessibility of the resource download URL using a HEAD request.

```python
from ckanapi_harvesters import CkanApi
import os

ckan = CkanApi(url="https://your-ckan-instance.org", apikey="your-key")

resource_id = "file-resource-uuid"

# Download resource file
res_info, response = ckan.resource_download(resource_id)
if response:
    # Save to disk
    filename = res_info.name or "downloaded_file"
    with open(f"/path/to/downloads/{filename}", "wb") as f:
        f.write(response.content)
    print(f"Downloaded: {filename} ({len(response.content)} bytes)")

# Download CSV as DataFrame
res_info, df = ckan.resource_download_df(resource_id)
if df is not None:
    print(f"Loaded DataFrame with {len(df)} rows")

# Test if download URL is accessible
error = ckan.resource_download_test_head(resource_id)
if error is None:
    print("Resource URL is accessible")
else:
    print(f"Error: {error.message}")
```

--------------------------------

### Validate CKAN Data Format Policies

Source: https://context7.com/mobidec/ckanapi_harvesters/llms.txt

This Python code demonstrates how to define and enforce data format policies for CKAN packages and resources. It shows how to load policies from a JSON file or create them programmatically, apply them to the CKAN client, check package compliance, and export policies to JSON.

```python
from ckanapi_harvesters import CkanApi
from ckanapi_harvesters.policies import CkanPackageDataFormatPolicy

# Load policy from JSON file
policy = CkanPackageDataFormatPolicy.from_json("/path/to/policy.json")

# Or create policy programmatically
policy = CkanPackageDataFormatPolicy(
    label="Organization Data Policy",
    description="Enforces metadata standards for all datasets",
    package_mandatory_attributes={"title", "notes", "author", "license_id"},
    resource_mandatory_attributes={"name", "format", "description"},
    datastore_fields_mandatory_attributes={"notes"}
)

# Initialize CkanApi with policy
ckan = CkanApi(
    url="https://your-ckan-instance.org",
    apikey="your-key",
    policy=policy
    # Or: policy_file="/path/to/policy.json"
)

# Check package compliance
errors = ckan.policy_check("my-package-name")
for error in errors:
    print(f"{error.context}: {error.level.name} - {error.message}")

# Export policy to JSON
policy.to_json("/path/to/exported_policy.json")
```

--------------------------------

### CKAN DataStore Write Operations with pandas

Source: https://context7.com/mobidec/ckanapi_harvesters/llms.txt

Perform insert, update, and upsert operations on CKAN DataStores using pandas DataFrames. Requires `ckanapi_harvesters` and `pandas`. Handles data in batches and allows specifying methods like 'upsert'. Can also clear entire DataStores.

```python
from ckanapi_harvesters import CkanApi
import pandas as pd

ckan = CkanApi(url="https://your-ckan-instance.org", apikey="your-key")
ckan.full_unlock()  # Enable write operations

resource_id = "12345-resource-uuid"

# Prepare data
df = pd.DataFrame({
    "id": [1, 2, 3],
    "name": ["Alice", "Bob", "Charlie"],
    "value": [100.5, 200.75, 150.25],
    "active": [True, False, True]
})

# Insert new records
result_df = ckan.datastore_insert(
    df,
    resource_id,
    limit=1000  # Rows per request (auto-batched)
)
print(f"Inserted {len(result_df)} records")

# Upsert (insert or update based on primary key)
df_updates = pd.DataFrame({
    "id": [2, 4],
    "name": ["Bob Updated", "David"],
    "value": [225.00, 175.50],
    "active": [True, True]
})
result_df = ckan.datastore_upsert(
    df_updates,
    resource_id,
    method="upsert"  # "insert", "update", or "upsert"
)

# Update existing records only
ckan.datastore_update(df_updates, resource_id)

# Delete specific rows
ckan.datastore_delete_rows(
    resource_id,
    filters={"active": False}
)

# Clear entire DataStore (requires enable_admin=True)
ckan.params.enable_admin = True
ckan.datastore_clear(resource_id)

```

--------------------------------

### Download URL via CKAN Proxy

Source: https://context7.com/mobidec/ckanapi_harvesters/llms.txt

Downloads content from a given URL using CKAN's proxy settings. This is useful for accessing external data sources through the CKAN instance. It takes the URL as input and an optional authentication flag.

```python
response = ckan.download_url_proxy(
    "https://external-api.example.com/data",
    auth_if_ckan=False
)
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.