### Connector Configuration for Object Storage and Database (JSON)

Source: https://github.com/veeva/vault-direct-data-api-accelerators/blob/main/README.md

Configuration file for connecting to external object storage (AWS S3 example) and data systems (Redshift example). Specifies parameters for file conversion, data extraction, and system-specific connection details.

```json
{
  "convert_to_parquet": false,
  "extract_document_content" : true,
  "retrieve_document_text" : true,
  "direct_data": {
    "start_time": "2000-01-01T00:00Z",
    "stop_time": "2025-04-09T00:00Z",
    "extract_type": "incremental"
  },
  "s3": {
    "iam_role_arn": "arn:aws:iam::123456789:role/Direct-Data-Role",
    "bucket_name": "vault-direct-data-bucket",
    "direct_data_folder": "direct-data",
    "archive_filepath": "direct-data/201287-20250409-0000-F.tar.gz",
    "extract_folder": "201287-20250409-0000-F",
    "document_content_folder": "extracted_doc_content",
    "document_text_folder": "extracted_doc_text"
  },
  "redshift": {
    "host": "direct-data.123GUID.us-east-1.redshift.amazonaws.com",
    "port": "5439",
    "user": "user",
    "password": "password",
    "database": "database",
    "schema": "direct_data",
    "iam_redshift_s3_read": "arn:aws:iam::123456789:role/RedshiftS3Read"
  }
}
```

--------------------------------

### Accelerator Entry Point Script (Python)

Source: https://github.com/veeva/vault-direct-data-api-accelerators/blob/main/README.md

The main Python script (`accelerator.py`) for the Redshift Accelerator. It initializes core services and passes configuration parameters to various data processing functions. It reads configuration files and sets up parameters for direct data extraction, object storage, and database loading.

```python
import sys

from accelerators.redshift.services.redshift_service import RedshiftService

sys.path.append('.')
from common.scripts import (
    direct_data_to_object_storage,
    download_and_unzip_direct_data_files,
    extract_doc_content,
    load_data,
    retrieve_doc_text
)
from common.services.aws_s3_service import AwsS3Service
from common.services.vault_service import VaultService
from common.utilities import read_json_file


def main():
  config_filepath: str = "path/to/connector_config.json"
  vapil_settings_filepath: str = "path/to/vapil_settings.json"

  config_params: dict = read_json_file(config_filepath)
  direct_data_params: dict = config_params['direct_data']
  s3_params: dict = config_params['s3']
  redshift_params: dict = config_params['redshift']

  extract_document_content: bool = config_params.get('extract_document_content')
  retrieve_document_text: bool = config_params.get('retrieve_document_text')

  object_storage_root: str = f's3://{s3_params["bucket_name"]}'

  s3_params['convert_to_parquet'] = config_params['convert_to_parquet']
  redshift_params['convert_to_parquet'] = config_params['convert_to_parquet']

```

--------------------------------

### Initialize Redshift and S3 Services for Direct Data Loading (Python)

Source: https://context7.com/veeva/vault-direct-data-api-accelerators/llms.txt

Initializes AWS S3 and Redshift services with specific parameters for processing Veeva Vault Direct Data. This includes IAM roles, bucket names, folder paths, and conversion settings. It then executes a data loading pipeline that processes manifest files, handles schema evolution, and loads data into Redshift with merge logic.

```Python
s3_service = AwsS3Service(parameters={
    'iam_role_arn': 'arn:aws:iam::123456789:role/RedshiftS3Read',
    'bucket_name': 'vault-data-bucket',
    'direct_data_folder': 'direct-data',
    'archive_filepath': 'archive.tar.gz',
    'extract_folder': '202401-incremental-extract',
    'document_content_folder': 'docs',
    'document_text_folder': 'text',
    'convert_to_parquet': False
})

redshift_service = RedshiftService(parameters={
    'host': 'cluster.abc123.us-east-1.redshift.amazonaws.com',
    'port': '5439',
    'user': 'admin',
    'password': 'Password123',
    'database': 'vaultdata',
    'schema': 'direct_data',
    'iam_redshift_s3_read': 'arn:aws:iam::123456789:role/RedshiftS3Read',
    'convert_to_parquet': False,
    'object_storage_root': 's3://vault-data-bucket'
})

# Execute data loading pipeline
direct_data_params = {'extract_type': 'incremental'}
load_data.run(
    object_storage_service=s3_service,
    database_service=redshift_service,
    direct_data_params=direct_data_params
)
```

--------------------------------

### Download and Unzip Direct Data Files from Object Storage

Source: https://github.com/veeva/vault-direct-data-api-accelerators/blob/main/README.md

Downloads Direct Data files from object storage, unzips the contents, optionally converts CSV files to Parquet, and uploads the processed content back to object storage.

```python
download_and_unzip_direct_data_files.run(object_storage_service=s3_service)
```

--------------------------------

### Download Direct Data to Object Storage

Source: https://github.com/veeva/vault-direct-data-api-accelerators/blob/main/README.md

Downloads a Direct Data file from Vault and uploads the compressed file to an object storage system. This script natively handles multiple file parts.

```python
redshift_params['object_storage_root'] = object_storage_root

s3_service: AwsS3Service = AwsS3Service(s3_params)
redshift_service: RedshiftService = RedshiftService(redshift_params)
vault_service: VaultService = VaultService(vapil_settings_filepath)

direct_data_to_object_storage.run(vault_service=vault_service,
                                    object_storage_service=s3_service,
                                    direct_data_params=direct_data_params)
```

--------------------------------

### Load Data into Snowflake from S3

Source: https://github.com/veeva/vault-direct-data-api-accelerators/blob/main/README.md

Leverages Snowflake's COPY INTO command to load data directly from files in S3 into Snowflake tables. Schema inference is supported and recommended for Full files.

```sql
COPY INTO <table_name>
FROM @<s3_stage_name>
FILE_FORMAT = (TYPE = CSV FIELD_DELIMITER=',' SKIP_HEADER=1)
PATTERN = '.*.csv.gz';

-- For schema inference (typically for Full files):
-- COPY INTO <table_name>
-- FROM @<s3_stage_name>
-- MATCH_BY_COLUMN_NAME;
```

--------------------------------

### Load Direct Data from Object Storage to Database

Source: https://github.com/veeva/vault-direct-data-api-accelerators/blob/main/README.md

Loads direct data extracts from object storage into tables in a target data system, supporting Full, Incremental, and Log file types.

```python
load_data.run(object_storage_service=s3_service,
                database_service=redshift_service,
                direct_data_params=direct_data_params)
```

--------------------------------

### Vault API Settings Configuration (JSON)

Source: https://github.com/veeva/vault-direct-data-api-accelerators/blob/main/README.md

Configuration file for Vault API authentication parameters. Supports Basic username/password or OAuth security policies. Includes parameters for authentication type, credentials, Vault URL, and session validation.

```json
{
  "authenticationType": "BASIC",
  "idpOauthAccessToken": "",
  "idpOauthScope": "openid",
  "idpUsername": "",
  "idpPassword": "",
  "vaultUsername": "integration.user@cholecap.com",
  "vaultPassword": "Password123",
  "vaultDNS": "cholecap.veevavault.com",
  "vaultSessionId": "",
  "vaultClientId": "Cholecap-Vault-",
  "vaultOauthClientId": "",
  "vaultOauthProfileId": "",
  "logApiErrors": true,
  "httpTimeout": null,
  "validateSession": true
}
```

--------------------------------

### Snowflake Data Loading Operations

Source: https://context7.com/veeva/vault-direct-data-api-accelerators/llms.txt

Illustrates how to use the SnowflakeService to connect to Snowflake, check schema existence, create tables from metadata, load full or incremental data, and drop tables. It requires Snowflake connection parameters and pandas for metadata handling.

```python
from accelerators.snowflake.services.snowflake_service import SnowflakeService
import pandas as pd

snowflake_params = {
    'account': 'xyz12345.us-east-1',
    'database': 'VAULT_DATA',
    'warehouse': 'COMPUTE_WH',
    'schema': 'direct_data',
    'username': 'vault_user',
    'role': 'ACCOUNTADMIN',
    'private_key': 'path/to/private_key.p8',
    'private_key_passphrase': 'key_passphrase',
    'stage_name': 'VAULT_STAGE',
    'infer_schema': True,
    'convert_to_parquet': False,
    'object_storage_root': 's3://vault-direct-data-bucket'
}
db_service = SnowflakeService(parameters=snowflake_params)

db_service.db_connection.activate_cursor()

db_service.check_if_schema_exists()

metadata_df = pd.DataFrame({
    'extract': ['Object.documents__sys', 'Object.documents__sys'],
    'column_name': ['id', 'name__v'],
    'type': ['id', 'String'],
    'length': [20, 255]
})
db_service.create_all_tables(
    starting_directory='direct-data/extract',
    metadata_table=metadata_df
)

db_service.load_full_or_log_data(
    table_name='documents__sys',
    object_path='s3://bucket/direct-data/extract/documents__sys.csv',
    headers=['id', 'name__v', 'status__v']
)

db_service.load_incremental_data(
    table_name='documents__sys',
    object_path='s3://bucket/direct-data/extract/documents__sys_updates.csv',
    headers=['id', 'name__v', 'status__v']
)

db_service.drop_tables_in_schema(tables=[('temp_table',), ('staging_table',)])
db_service.db_connection.close()
```

--------------------------------

### S3 Multipart Upload and File Operations

Source: https://context7.com/veeva/vault-direct-data-api-accelerators/llms.txt

Demonstrates how to perform a multipart upload to S3 for large files, including uploading parts and completing the upload. It also shows how to retrieve CSV headers and check for object existence in S3.

```python
multipart_response = s3_service.create_multipart_upload(
    object_path='direct-data/large-file.tar.gz'
)
parts = []
chunk_size = 5 * 1024 * 1024  # 5MB chunks
for part_num, chunk in enumerate(file_chunks, start=1):
    part_info = s3_service.upload_part(
        object_path='direct-data/large-file.tar.gz',
        multipart_upload_response=multipart_response,
        part_number=part_num,
        data=chunk
    )
    parts.append(part_info)
s3_service.complete_multipart_upload(
    object_path='direct-data/large-file.tar.gz',
    multipart_upload_response=multipart_response,
    parts=parts
)

headers = s3_service.get_headers_from_csv_file(
    object_path='direct-data/extracted/document__sys.csv'
)
print(f"CSV columns: {headers}")

s3_service.check_if_object_exists(object_path='direct-data/manifest.csv')
```

--------------------------------

### Extract Document Content from Vault to Object Storage

Source: https://github.com/veeva/vault-direct-data-api-accelerators/blob/main/README.md

Retrieves document version IDs, exports document content to File Staging using the Export Document Versions endpoint, then downloads and uploads the content to object storage.

```python
if extract_document_content:
    extract_doc_content.run(object_storage_service=s3_service,
                            vault_service=vault_service)
```

--------------------------------

### Retrieve Document Text from Vault to Object Storage

Source: https://github.com/veeva/vault-direct-data-api-accelerators/blob/main/README.md

Retrieves document version IDs, calls the Retrieve Document Version Text endpoint, and saves the content to a text file in object storage.

```python
if retrieve_document_text:
    retrieve_doc_text.run(object_storage_service=s3_service,
                          vault_service=vault_service)
```

--------------------------------

### Snowflake Accelerator - Complete Data Pipeline (Python)

Source: https://context7.com/veeva/vault-direct-data-api-accelerators/llms.txt

Implements a full data pipeline to connect Veeva Vault to Snowflake via S3. It loads configuration, initializes Vault, S3, and Snowflake services, and orchestrates the download, unzip, and loading of direct data files. Optionally extracts document content and text.

```Python
from accelerators.snowflake.services.snowflake_service import SnowflakeService
from common.services.aws_s3_service import AwsS3Service
from common.services.vault_service import VaultService
from common.scripts import direct_data_to_object_storage, load_data
from common.utilities import read_json_file

# Load configuration
config = read_json_file('accelerators/snowflake/resources/connector_config.json')
direct_data_params = config['direct_data']
s3_params = config['s3']
snowflake_params = config['snowflake']

# Enrich parameters
s3_params['convert_to_parquet'] = config['convert_to_parquet']
snowflake_params['convert_to_parquet'] = config['convert_to_parquet']
snowflake_params['object_storage_root'] = f"s3://{s3_params['bucket_name']}"

# Initialize services
vault_service = VaultService('accelerators/snowflake/resources/vapil_settings.json')
s3_service = AwsS3Service(s3_params)
snowflake_service = SnowflakeService(snowflake_params)

# Execute pipeline
# Step 1: Download from Vault and upload to S3
direct_data_to_object_storage.run(
    vault_service=vault_service,
    object_storage_service=s3_service,
    direct_data_params=direct_data_params
)

# Step 2: Unzip and optionally convert to Parquet
from common.scripts import download_and_unzip_direct_data_files
download_and_unzip_direct_data_files.run(object_storage_service=s3_service)

# Step 3: Load data into Snowflake
load_data.run(
    object_storage_service=s3_service,
    database_service=snowflake_service,
    direct_data_params=direct_data_params
)

# Step 4: Extract document content (optional)
if config.get('extract_document_content'):
    from common.scripts import extract_doc_content
    extract_doc_content.run(
        object_storage_service=s3_service,
        vault_service=vault_service
    )

# Step 5: Retrieve document text (optional)
if config.get('retrieve_document_text'):
    from common.scripts import retrieve_doc_text
    retrieve_doc_text.run(
        object_storage_service=s3_service,
        vault_service=vault_service
    )
```

--------------------------------

### Snowflake Accelerator Configuration (JSON)

Source: https://context7.com/veeva/vault-direct-data-api-accelerators/llms.txt

This JSON object outlines the complete configuration for the Snowflake accelerator, enabling features like data conversion to Parquet and extraction of document content and text. It specifies parameters for direct data extraction, S3 integration for storage, and detailed Snowflake connection settings including account, database, warehouse, and security credentials.

```json
{
  "convert_to_parquet": true,
  "extract_document_content": true,
  "retrieve_document_text": true,
  "direct_data": {
    "start_time": "2024-01-01T00:00Z",
    "stop_time": "2024-12-31T23:59Z",
    "extract_type": "incremental"
  },
  "s3": {
    "iam_role_arn": "arn:aws:iam::123456789012:role/DirectDataRole",
    "bucket_name": "company-vault-data",
    "direct_data_folder": "direct-data",
    "archive_filepath": "direct-data/202401-extract.tar.gz",
    "extract_folder": "202401-extract",
    "document_content_folder": "document-content",
    "document_text_folder": "document-text"
  },
  "snowflake": {
    "account": "abc12345.us-east-1",
    "database": "VAULT_DATA",
    "warehouse": "COMPUTE_WH",
    "schema": "direct_data",
    "username": "vault_integration",
    "role": "VAULT_DATA_LOADER",
    "private_key": "/path/to/snowflake_key.p8",
    "private_key_passphrase": "keypassphrase",
    "stage_name": "VAULT_DIRECT_DATA_STAGE",
    "infer_schema": true
  }
}
```

--------------------------------

### VaultClient - Authentication and Session Management

Source: https://context7.com/veeva/vault-direct-data-api-accelerators/llms.txt

Provides low-level client for Vault API authentication using various methods, including basic and OAuth, and manages API session details.

```APIDOC
## VaultClient - Authentication and Session Management

### Description
This client handles authentication with the Vault API using multiple authentication types such as Basic authentication and OAuth 2.0 with an access token. It can also authenticate directly from a JSON settings file. Once authenticated, it provides access to session details like Session ID and User ID.

### Method
N/A (This is a client class, not a direct API endpoint)

### Endpoint
N/A

### Parameters
#### Path Parameters
N/A

#### Query Parameters
N/A

#### Request Body
N/A

### Request Example
```python
from common.api.client.vault_client import VaultClient, AuthenticationType

# Basic authentication
vault_client = VaultClient(
    vault_client_id='MyApp-Integration',
    vault_username='integration.user@company.com',
    vault_password='SecurePassword123',
    vault_dns='mycompany.veevavault.com',
    authentication_type=AuthenticationType.BASIC
)
vault_client.authenticate()

# OAuth authentication with access token
vault_client_oauth = VaultClient(
    vault_client_id='MyApp-OAuth',
    vault_dns='mycompany.veevavault.com',
    vault_oauth_profile_id='oauth_profile_id_value',
    idp_oauth_access_token='eyJhbGciOiJSUzI1NiIs...', # Replace with actual token
    authentication_type=AuthenticationType.OAUTH_ACCESS_TOKEN
)
vault_client_oauth.authenticate()

# Authenticate from JSON settings file
vault_client_from_file = VaultClient.authenticate_from_settings_file(
    file_path='config/vapil_settings.json'
)

# Use authenticated client for requests
from common.api.request.direct_data_request import DirectDataRequest
request = vault_client_from_file.new_request(DirectDataRequest)
print(f"Session ID: {vault_client_from_file.authentication_response.sessionId}")
print(f"User ID: {vault_client_from_file.authentication_response.userId}")
```

### Response
#### Success Response (200)
Upon successful authentication, the client instance will have an `authentication_response` attribute containing `sessionId` and `userId`.

#### Response Example
```json
{
  "sessionId": "a1b2c3d4-e5f6-7890-1234-567890abcdef",
  "userId": "user123"
}
```
```

--------------------------------

### Manage AWS S3 Object Storage with AwsS3Service

Source: https://context7.com/veeva/vault-direct-data-api-accelerators/llms.txt

The AwsS3Service class facilitates operations for AWS S3 object storage, including uploading and downloading Direct Data files. It requires AWS credentials and bucket configuration. Dependencies include `common.services.aws_s3_service` and the `boto3` library for S3 interactions.

```python
from common.services.aws_s3_service import AwsS3Service
import io

# Initialize S3 service with configuration
s3_params = {
    'iam_role_arn': 'arn:aws:iam::123456789:role/DirectDataRole', # Replace with your IAM role ARN
    'bucket_name': 'vault-direct-data-bucket', # Replace with your bucket name
    'direct_data_folder': 'direct-data',
    'archive_filepath': 'direct-data/archive.tar.gz',
    'extract_folder': 'extracted-files',
    'document_content_folder': 'doc-content',
    'document_text_folder': 'doc-text',
    'convert_to_parquet': False
}
s3_service = AwsS3Service(parameters=s3_params)

# Upload file to S3
file_data = b"Sample Direct Data file content..."
s3_service.upload_object(
    object_path='direct-data/202401-full.tar.gz',
    data=io.BytesIO(file_data)
)
```

--------------------------------

### Query Loaded Data from SQLite Database

Source: https://context7.com/veeva/vault-direct-data-api-accelerators/llms.txt

This Python script connects to a local SQLite database, queries the total number of documents from the 'documents__sys' table, and prints the count. It utilizes the sqlite3 library for database interaction. The script closes the database connection after execution.

```python
import sqlite3
conn = sqlite3.connect('./data/vault_data.db')
cursor = conn.execute("SELECT COUNT(*) FROM documents__sys")
print(f"Total documents: {cursor.fetchone()[0]}")
conn.close()
```

--------------------------------

### Authenticate Veeva Vault API using VaultClient

Source: https://context7.com/veeva/vault-direct-data-api-accelerators/llms.txt

The VaultClient class provides low-level client functionality for Vault API authentication using multiple authentication types including Basic and OAuth access token. It can also authenticate from a JSON settings file. Dependencies include `common.api.client.vault_client` and `common.api.request.direct_data_request`.

```python
from common.api.client.vault_client import VaultClient, AuthenticationType

# Basic authentication
vault_client = VaultClient(
    vault_client_id='MyApp-Integration',
    vault_username='integration.user@company.com',
    vault_password='SecurePassword123',
    vault_dns='mycompany.veevavault.com',
    authentication_type=AuthenticationType.BASIC
)
vault_client.authenticate()

# OAuth authentication with access token
vault_client_oauth = VaultClient(
    vault_client_id='MyApp-OAuth',
    vault_dns='mycompany.veevavault.com',
    vault_oauth_profile_id='oauth_profile_id_value',
    idp_oauth_access_token='eyJhbGciOiJSUzI1NiIs...', # Replace with actual token
    authentication_type=AuthenticationType.OAUTH_ACCESS_TOKEN
)
vault_client_oauth.authenticate()

# Authenticate from JSON settings file
vault_client_from_file = VaultClient.authenticate_from_settings_file(
    file_path='config/vapil_settings.json'
)

# Use authenticated client for requests
from common.api.request.direct_data_request import DirectDataRequest
request = vault_client_from_file.new_request(DirectDataRequest)
print(f"Session ID: {vault_client_from_file.authentication_response.sessionId}")
print(f"User ID: {vault_client_from_file.authentication_response.userId}")
```

--------------------------------

### Load Extracted Data into Database

Source: https://context7.com/veeva/vault-direct-data-api-accelerators/llms.txt

This script is designed to process manifest files and load CSV or Parquet data into target database tables. It utilizes common scripts for loading data and requires services like RedshiftService or AwsS3Service for database interaction.

```python
from common.scripts import load_data
from accelerators.redshift.services.redshift_service import RedshiftService
from common.services.aws_s3_service import AwsS3Service

```

--------------------------------

### Interact with Veeva Vault Direct Data API using VaultService

Source: https://context7.com/veeva/vault-direct-data-api-accelerators/llms.txt

The VaultService class handles authentication and interaction with Veeva Vault Direct Data API endpoints. It supports retrieving available direct data files, downloading them, and exporting document versions. Dependencies include the `common.services.vault_service` module.

```python
from common.services.vault_service import VaultService

# Initialize service with authentication settings
vault_service = VaultService(vapil_settings_filepath="config/vapil_settings.json")

# List available Direct Data files
response = vault_service.retrieve_available_direct_data_files(
    extract_type="incremental_directdata",
    start_time="2024-01-01T00:00Z",
    stop_time="2024-12-31T23:59Z"
)

# Download a specific Direct Data file
if response.data:
    latest_file = response.data[-1]
    download_response = vault_service.download_direct_data_file(
        name=latest_file.filepart_details[0].name
    )

    # Save binary content
    with open(f"downloads/{latest_file.filename}", "wb") as f:
        f.write(download_response.binary_content)

# Export document versions
document_versions = [
    {"id": "123", "major_version_number": "1", "minor_version_number": "0"},
    {"id": "456", "major_version_number": "2", "minor_version_number": "1"}
]
export_job = vault_service.export_document_versions(document_version_dict=document_versions)
print(f"Export Job ID: {export_job.data.job_id}")
```

--------------------------------

### Direct Data to Object Storage Transfer

Source: https://context7.com/veeva/vault-direct-data-api-accelerators/llms.txt

This script transfers Direct Data archives from Vault to cloud object storage. It initializes VaultService and AwsS3Service, configures extraction parameters like extract type and time range, and then runs the pipeline to download and upload files.

```python
from common.scripts import direct_data_to_object_storage
from common.services.vault_service import VaultService
from common.services.aws_s3_service import AwsS3Service

vault_service = VaultService(vapil_settings_filepath='config/vapil_settings.json')
s3_service = AwsS3Service(parameters={
    'iam_role_arn': 'arn:aws:iam::123456789:role/DirectDataRole',
    'bucket_name': 'vault-data-bucket',
    'direct_data_folder': 'direct-data',
    'archive_filepath': 'archive.tar.gz',
    'extract_folder': 'extracted',
    'document_content_folder': 'docs',
    'document_text_folder': 'text',
    'convert_to_parquet': False
})

direct_data_params = {
    'extract_type': 'incremental',
    'start_time': '2024-01-01T00:00Z',
    'stop_time': '2024-12-31T23:59Z'
}

direct_data_to_object_storage.run(
    vault_service=vault_service,
    object_storage_service=s3_service,
    direct_data_params=direct_data_params
)
```

--------------------------------

### SQLite Accelerator - Local Direct Data Processing (Python)

Source: https://context7.com/veeva/vault-direct-data-api-accelerators/llms.txt

A lightweight accelerator for local development and testing of Veeva Vault Direct Data processing. It bypasses cloud dependencies by using a local SQLite database. The script downloads data from Vault, unzips it, and loads it into a local SQLite database.

```Python
from accelerators.sqlite.services.sqlite_service import SqliteService
from common.services.vault_service import VaultService
from accelerators.sqlite.scripts import download_direct_data_file, unzip_direct_data_file, load_data

# Configuration for local SQLite database
config = {
    'direct_data': {
        'start_time': '2024-01-01T00:00Z',
        'stop_time': '2024-12-31T23:59Z',
        'extract_type': 'full'
    },
    'sqlite': {
        'database_path': './data/vault_data.db',
        'schema': 'main',
        'convert_to_parquet': False
    },
    'local_storage': {
        'download_folder': './downloads',
        'extract_folder': './extracted'
    }
}

# Initialize services
vault_service = VaultService('config/vapil_settings.json')
sqlite_service = SqliteService(parameters=config['sqlite'])

# Download Direct Data file to local storage
download_direct_data_file.run(
    vault_service=vault_service,
    local_storage_params=config['local_storage'],
    direct_data_params=config['direct_data']
)

# Unzip the archive
unzip_direct_data_file.run(
    local_storage_params=config['local_storage']
)

# Load CSV data into SQLite
load_data.run(
    database_service=sqlite_service,
    local_storage_params=config['local_storage'],
    direct_data_params=config['direct_data']
)
```

--------------------------------

### Veeva Vault Authentication Configuration (JSON)

Source: https://context7.com/veeva/vault-direct-data-api-accelerators/llms.txt

This JSON object defines the authentication settings required to connect to a Veeva Vault instance. It includes various authentication methods and parameters such as username, password, and Vault DNS. This configuration is used by the Vault Direct Data API Accelerators to establish a connection.

```json
{
  "authenticationType": "BASIC",
  "idpOauthAccessToken": "",
  "idpOauthScope": "openid",
  "idpUsername": "",
  "idpPassword": "",
  "vaultUsername": "integration.user@company.com",
  "vaultPassword": "SecurePassword123",
  "vaultDNS": "company.veevavault.com",
  "vaultSessionId": "",
  "vaultClientId": "CompanyApp-DirectData-Integration",
  "vaultOauthClientId": "",
  "vaultOauthProfileId": "",
  "logApiErrors": true,
  "httpTimeout": 120,
  "validateSession": true
}
```

--------------------------------

### VaultService - Vault API Integration

Source: https://context7.com/veeva/vault-direct-data-api-accelerators/llms.txt

Handles authentication and interaction with Veeva Vault Direct Data API endpoints for data retrieval and document export.

```APIDOC
## VaultService - Vault API Integration

### Description
This service facilitates authentication and interaction with Veeva Vault Direct Data API endpoints. It allows for retrieving available Direct Data files within a specified time range and downloading specific files. It also supports initiating export jobs for document versions.

### Method
N/A (This is a service class, not a direct API endpoint)

### Endpoint
N/A

### Parameters
#### Path Parameters
N/A

#### Query Parameters
N/A

#### Request Body
N/A

### Request Example
```python
from common.services.vault_service import VaultService

# Initialize service with authentication settings
vault_service = VaultService(vapil_settings_filepath="config/vapil_settings.json")

# List available Direct Data files
response = vault_service.retrieve_available_direct_data_files(
    extract_type="incremental_directdata",
    start_time="2024-01-01T00:00Z",
    stop_time="2024-12-31T23:59Z"
)

# Download a specific Direct Data file
if response.data:
    latest_file = response.data[-1]
    download_response = vault_service.download_direct_data_file(
        name=latest_file.filepart_details[0].name
    )

    # Save binary content
    with open(f"downloads/{latest_file.filename}", "wb") as f:
        f.write(download_response.binary_content)

# Export document versions
document_versions = [
    {"id": "123", "major_version_number": "1", "minor_version_number": "0"},
    {"id": "456", "major_version_number": "2", "minor_version_number": "1"}
]
export_job = vault_service.export_document_versions(document_version_dict=document_versions)
print(f"Export Job ID: {export_job.data.job_id}")
```

### Response
#### Success Response (200)
Responses vary depending on the method called. `retrieve_available_direct_data_files` returns a list of available files. `download_direct_data_file` returns binary content. `export_document_versions` returns job details including a `job_id`.

#### Response Example
(Example for `export_document_versions`)
```json
{
  "data": {
    "job_id": "12345"
  }
}
```
```

--------------------------------

### AwsS3Service - AWS S3 Object Storage Operations

Source: https://context7.com/veeva/vault-direct-data-api-accelerators/llms.txt

Manages Direct Data files in Amazon S3, including uploading, downloading, and organizing files within specified buckets and folders.

```APIDOC
## AwsS3Service - AWS S3 Object Storage Operations

### Description
This service handles operations related to AWS S3 object storage, specifically for managing Direct Data files. It supports uploading files to S3, downloading them, and organizing them into designated folders. Configuration includes parameters for IAM roles, bucket names, folder paths, and conversion options to Parquet format.

### Method
N/A (This is a service class, not a direct API endpoint)

### Endpoint
N/A

### Parameters
#### Path Parameters
N/A

#### Query Parameters
N/A

#### Request Body
N/A

### Request Example
```python
from common.services.aws_s3_service import AwsS3Service
import io

# Initialize S3 service with configuration
s3_params = {
    'iam_role_arn': 'arn:aws:iam::123456789:role/DirectDataRole', # Replace with your ARN
    'bucket_name': 'vault-direct-data-bucket', # Replace with your bucket name
    'direct_data_folder': 'direct-data',
    'archive_filepath': 'direct-data/archive.tar.gz',
    'extract_folder': 'extracted-files',
    'document_content_folder': 'doc-content',
    'document_text_folder': 'doc-text',
    'convert_to_parquet': False
}
s3_service = AwsS3Service(parameters=s3_params)

# Upload file to S3
file_data = b"Sample Direct Data file content..."
s3_service.upload_object(
    object_path='direct-data/202401-full.tar.gz', # Example path in S3
    data=io.BytesIO(file_data)
)
```

### Response
#### Success Response (200)
Operations like `upload_object` typically do not return a specific response body on success, but indicate completion through the execution flow. Errors would be raised as exceptions.

#### Response Example
N/A (Success is indicated by the absence of exceptions.)
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.