### Command-Line Usage Example

Source: https://github.com/chenditc/investment_data/blob/main/_autodocs/api-reference-tushare-daily-update.md

Provides instructions for setting the TUSHARE environment variable, starting the Dolt SQL server, running the update script, and then stopping the server. This is a typical workflow for automated updates.

```bash
export TUSHARE="your_token_here"
dolt sql-server &
python3 tushare/update_a_stock_eod_price_to_latest.py
# ... wait for updates ...
killall dolt
```

--------------------------------

### Multiprocessing Setup

Source: https://github.com/chenditc/investment_data/blob/main/_autodocs/api-reference-qlib-normalize.md

Forces the 'spawn' start method for all multiprocessing. This ensures consistent behavior across Python processes, which is safer and avoids issues with shared resources like database connections.

```python
mp.set_start_method("spawn", force=True)
```

--------------------------------

### Command-Line Usage for Export Script

Source: https://github.com/chenditc/investment_data/blob/main/_autodocs/api-reference-qlib-export.md

Provides examples of how to execute the `dump_all_to_qlib_source.py` script from the command line, including setting the output directory and using the `skip_exists` flag.

```bash
export QLIB_SOURCE_DIR="/home/user/qlib_data"
python3 qlib/dump_all_to_qlib_source.py
python3 qlib/dump_all_to_qlib_source.py --skip_exists=True
```

--------------------------------

### Docker Build Command with Build Argument

Source: https://github.com/chenditc/investment_data/blob/main/_autodocs/configuration.md

Example of how to use the `DOLT_VERSION` build argument when building a Docker image.

```bash
docker build --build-arg DOLT_VERSION=1.88.0 -t investment_data .
```

--------------------------------

### SQL Merge Logic Example

Source: https://github.com/chenditc/investment_data/blob/main/_autodocs/data-flow.md

An example SQL query demonstrating how new stock price data is inserted into the final price table, joining with a link table to apply adjustment ratios.

```sql
INSERT INTO final_a_stock_eod_price (..)
SELECT ts_raw.tradedate, 
       ts_link.w_symbol,
       ts_raw.high, ts_raw.low, ts_raw.open, ts_raw.close,
       ts_raw.volume,
       ts_raw.adjclose / ts_link.adj_ratio as adjclose,
       ts_raw.amount
FROM ts_a_stock_eod_price ts_raw
LEFT JOIN ts_link_table ts_link 
  ON ts_raw.symbol = ts_link.link_symbol
```

--------------------------------

### Command-Line Usage

Source: https://github.com/chenditc/investment_data/blob/main/_autodocs/api-reference-qlib-normalize.md

Example of how to run the normalization process from the command line using `fire` for argument parsing. Specifies source and normalization directories, and the number of parallel workers.

```bash
python3 qlib/normalize.py \
    --source_dir=/data/stocks \
    --normalize_dir=/data/normalized \
    --max_workers=4
```

--------------------------------

### Usage Example: Get Trading Calendar

Source: https://github.com/chenditc/investment_data/blob/main/_autodocs/api-reference-tushare-dump-a-stock.md

Demonstrates how to set up Tushare authentication and retrieve all trading days for the year 2023 using the get_trade_cal function.

```python
import os
import tushare as ts
os.environ["TUSHARE"] = "your_token_here"
ts.set_token(os.environ["TUSHARE"])
pro = ts.pro_api()

# Get all trading days in 2023
trade_cal = get_trade_cal("20230101", "20231231")
print(trade_cal.head())
```

--------------------------------

### Output Message Example

Source: https://github.com/chenditc/investment_data/blob/main/_autodocs/api-reference-tushare-daily-update.md

Illustrates the expected output messages during the data update process, showing the date being downloaded and the number of records inserted.

```text
Downloading 20230221
20230221 Updated: 5234 records
```

--------------------------------

### Example Input Row for Normalization

Source: https://github.com/chenditc/investment_data/blob/main/_autodocs/api-reference-qlib-normalize.md

This is an example of a single row from a CSV file that Qlib processes for normalization. It includes standard trading data fields.

```csv
tradedate,symbol,open,high,low,close,volume,adjclose,amount,vwap
2023-01-01,000001.SZ,10.50,10.75,10.25,10.60,5000000,10.60,53000000.0,106.0
```

--------------------------------

### Usage Example for Exporting Stock Data

Source: https://github.com/chenditc/investment_data/blob/main/_autodocs/api-reference-qlib-export.md

Shows how to set the output directory environment variable and call the `dump_all_to_sqlib_source` function to export stock data. It also illustrates the expected output file structure.

```python
import os
os.environ["QLIB_SOURCE_DIR"] = "/output/qlib_data"

# Export all stock data
dump_all_to_sqlib_source(skip_exists=False)

# Creates files:
# /output/qlib_data/000001.SZ.csv
# /output/qlib_data/000002.SZ.csv
# ... etc for all symbols in database
```

--------------------------------

### Database Configuration Example

Source: https://github.com/chenditc/investment_data/blob/main/_autodocs/api-reference-tushare-daily-update.md

Specifies the connection string for a MySQL database, including the driver, host, port, username, and database name. Assumes MySQL is running on localhost with default credentials.

```python
Database: mysql+pymysql://root:@127.0.0.1/investment_data
```

--------------------------------

### Calculate Start Date from List Date

Source: https://github.com/chenditc/investment_data/blob/main/_autodocs/api-reference-tushare-index-weight.md

When no start date is provided, this code fetches index information and uses its listing date as the starting point for data retrieval.

```python
index_info = pro.index_basic(ts_code=index_name)
list_date = index_info["list_date"][0]  # From first result
list_date_obj = datetime.datetime.strptime(list_date, '%Y%m%d')
index_start_date = list_date_obj
```

--------------------------------

### Usage Example: Generating Index Files

Source: https://github.com/chenditc/investment_data/blob/main/_autodocs/api-reference-qlib-export.md

This Python code snippet shows how to set the `QLIB_INDEX_DIR` environment variable and then call the `dump_all_to_sqlib_source` function to generate the index constituent files. It also lists the expected output files.

```python
import os
os.environ["QLIB_INDEX_DIR"] = "/output/qlib_index"

# Generate index constituent files
dump_all_to_sqlib_source(skip_exists=False)

# Creates files:
# /output/qlib_index/csi300.txt
# /output/qlib_index/csi500.txt
# /output/qlib_index/csi800.txt
# /output/qlib_index/csi1000.txt
# /output/qlib_index/csiall.txt

```

--------------------------------

### Command-Line Usage for A-Stock Data Download

Source: https://github.com/chenditc/investment_data/blob/main/_autodocs/api-reference-tushare-dump-a-stock.md

Shows how to execute the A-stock data download script from the command line using 'fire' for argument parsing. Includes examples with and without explicit skip_exists flag.

```bash
export TUSHARE="your_token_here"
python3 tushare/dump_a_stock_eod_price.py --start_date=20230101 --end_date=20231231
python3 tushare/dump_a_stock_eod_price.py 20230101 20231231  # positional args
python3 tushare/dump_a_stock_eod_price.py 20230101 20231231  --skip_exists=False  # re-download
```

--------------------------------

### Index Constituent Mapping Example

Source: https://github.com/chenditc/investment_data/blob/main/_autodocs/types.md

Provides an example of the tab-separated text format for index constituent mapping, generated by dump_index_weight.py. Each line contains stock prefix, stock code, start date, and end date.

```text
00	000001	2023-01-01	2023-03-31
00	000002	2023-01-01	2023-03-31
00	000003	2023-04-01	2023-06-30
```

--------------------------------

### Python Usage Example with Tushare Token

Source: https://github.com/chenditc/investment_data/blob/main/_autodocs/api-reference-tushare-daily-update.md

Demonstrates how to set the TUSHARE environment variable with your API token and then call the `dump_astock_data` function. Requires a running Dolt SQL server.

```python
import os
os.environ["TUSHARE"] = "your_token_here"

# Must have dolt SQL server running
# Start it with: dolt sql-server &
# Then:

dump_astock_data()

# Typical output:
# Downloading 20230221
# 20230221 Updated: 5234 records
# Downloading 20230222
# 20230222 Updated: 5210 records
```

--------------------------------

### Command-Line Usage for Index Data Dump

Source: https://github.com/chenditc/investment_data/blob/main/_autodocs/api-reference-tushare-dump-index.md

Provides examples for running the index data dump script from the command line, including setting the TUSHARE token and specifying date ranges.

```bash
export TUSHARE="your_token_here"
python3 tushare/dump_index_eod_price.py
python3 tushare/dump_index_eod_price.py --start_date=20230101 --end_date=20231231
```

--------------------------------

### Define Calendar Start Date

Source: https://github.com/chenditc/investment_data/blob/main/_autodocs/api-reference-qlib-normalize.md

Sets the starting date for the trading calendar. Only trading dates from this point forward are considered.

```python
CALENDAR_START_DATE = pd.Timestamp("2000-01-04")
```

--------------------------------

### Python Multiprocessing Configuration

Source: https://github.com/chenditc/investment_data/blob/main/_autodocs/configuration.md

Configures Python's multiprocessing to use the 'spawn' start method globally. This is achieved by creating a `sitecustomize.py` file that is automatically loaded by Python.

```python
RUN printf '%s\n' \
'import multiprocessing as mp' \
'try:' \
'    mp.set_start_method("spawn")' \
'except RuntimeError:' \
'    pass' \
> /usr/local/lib/python3.9/site-packages/sitecustomize.py
```

--------------------------------

### Usage Example for dump_index_data

Source: https://github.com/chenditc/investment_data/blob/main/_autodocs/api-reference-tushare-index-weight.md

Demonstrates how to use the dump_index_data function with and without specifying date ranges. Ensure your TUSHARE API token is set as an environment variable.

```python
import os
os.environ["TUSHARE"] = "your_token_here"

# Download from a specific date forward
dump_index_data(start_date="20230101", end_date="20231231")

# Download from index list dates forward (default)
dump_index_data()

# Files created:
# tushare/index_weight/000905.SH.csv
# tushare/index_weight/399300.SZ.csv
# ... etc
```

--------------------------------

### Dump Index Weight Data CLI

Source: https://github.com/chenditc/investment_data/blob/main/_autodocs/configuration.md

Command-line interface for dumping index weight data. Supports specifying start and end dates, with defaults to use the index list date for start and calculate end date. Can also override skip_exists.

```bash
# Default (from each index's list date forward)
python3 tushare/dump_index_weight.py

# Specific date range
python3 tushare/dump_index_weight.py --start_date=20230101 --end_date=20231231

# Mixed
python3 tushare/dump_index_weight.py --start_date=20230101
```

--------------------------------

### Dolt Version Build Argument

Source: https://github.com/chenditc/investment_data/blob/main/_autodocs/configuration.md

Specifies the version of Dolt (Git-like version control for databases) to install during the Docker build process.

```dockerfile
ARG DOLT_VERSION=1.88.0
```

--------------------------------

### Usage Example: Python Script

Source: https://github.com/chenditc/investment_data/blob/main/_autodocs/api-reference-qlib-normalize.md

Example of how to use the `normalize_crowd_source_data` function within a Python script. Sets environment variables for source and output directories and calls the normalization function with parallel workers.

```python
import os

# Setup paths
os.environ["QLIB_SOURCE_DIR"] = "/data/stocks"  # Input CSVs from dump_all_to_qlib_source
normalize_dir = "/data/normalized"               # Output location

# Normalize with parallel workers
normalize_crowd_source_data(
    source_dir="/data/stocks",
    normalize_dir="/data/normalized",
    max_workers=4,
    interval="1d",
    date_field_name="tradedate",
    symbol_field_name="symbol"
)

# Output: /data/normalized/<symbol>/ directories with Qlib binary format
```

--------------------------------

### Collect Daily Index Prices (Tushare)

Source: https://github.com/chenditc/investment_data/blob/main/_autodocs/project-overview.md

Fetches daily end-of-day prices for stock market indices. Optional start and end dates can be provided.

```bash
python3 tushare/dump_index_eod_price.py [--start_date=YYYYMMDD] [--end_date=YYYYMMDD]
```

--------------------------------

### Usage Example: Fetch Daily Stock Data

Source: https://github.com/chenditc/investment_data/blob/main/_autodocs/api-reference-tushare-dump-a-stock.md

Shows how to fetch daily stock data for a specific trading date using the get_daily function and print the 'ts_code', 'close', and 'adj_close' columns.

```python
# Fetch data for a specific trading date
daily_data = get_daily("20230110")
if daily_data is not None:
    print(daily_data[["ts_code", "close", "adj_close"]].head())
```

--------------------------------

### Update Stock Prices Incrementally

Source: https://github.com/chenditc/investment_data/blob/main/_autodocs/data-flow.md

Starts a Dolt SQL server, fetches incremental daily stock prices and adjustment factors from Tushare, and inserts new data into the price table. This method avoids re-downloading historical data by only fetching missing trading dates.

```bash
dolt sql-server &
sleep 5
python3 tushare/update_a_stock_eod_price_to_latest.py
killall dolt
```

--------------------------------

### Collect Daily Stock Prices (Tushare)

Source: https://github.com/chenditc/investment_data/blob/main/_autodocs/project-overview.md

Use this script to fetch daily end-of-day prices for stocks from Tushare. Specify the start and end dates for the data retrieval.

```bash
python3 tushare/dump_a_stock_eod_price.py --start_date=YYYYMMDD --end_date=YYYYMMDD
```

--------------------------------

### Python Usage Example for dump_astock_data

Source: https://github.com/chenditc/investment_data/blob/main/_autodocs/api-reference-tushare-dump-a-stock.md

Demonstrates how to call the dump_astock_data function to download an entire year's worth of A-stock data. It also shows the expected file naming convention for the generated CSVs.

```python
# Download entire 2023 year of A-stock data
dump_astock_data("20230101", "20231231", skip_exists=True)

# Files created:
# tushare/astock_daily/20230102.csv (first trading day)
# tushare/astock_daily/20230103.csv
# ... etc
```

--------------------------------

### Get Cached Calendar List

Source: https://github.com/chenditc/investment_data/blob/main/_autodocs/api-reference-qlib-normalize.md

Overrides the base method to return a pre-loaded calendar list if available, otherwise falls back to the default Qlib implementation.

```python
def _get_calendar_list(self):
    if self.CALENDAR_LIST is not None:
        return self.CALENDAR_LIST
    return super()._get_calendar_list()
```

--------------------------------

### Get SSE Trading Calendar

Source: https://github.com/chenditc/investment_data/blob/main/_autodocs/api-reference-tushare-dump-a-stock.md

Retrieves the trading calendar for the Shanghai Stock Exchange (SSE) within a specified date range. Requires start and end dates in YYYYMMDD format.

```python
def get_trade_cal(start_date, end_date):
    df = pro.trade_cal(exchange='SSE', is_open='1',
                            start_date=start_date,
                            end_date=end_date,
                            fields='cal_date')
    return df
```

--------------------------------

### Project Configuration and Documentation Files

Source: https://github.com/chenditc/investment_data/blob/main/_autodocs/project-overview.md

Configuration files and documentation for the project, including Python dependencies, Docker build definition, user-facing README files, and the project license.

```dockerfile
Dockerfile
```

```text
requirements.txt
```

```markdown
README.md
```

```markdown
README-ch.md
```

```markdown
final_a_stock_eod_price.md
```

```markdown
final_a_stock_limit.md
```

```text
LICENSE
```

--------------------------------

### Command-Line Usage

Source: https://github.com/chenditc/investment_data/blob/main/_autodocs/api-reference-qlib-export.md

These commands demonstrate how to set the output directory for Qlib index files using an environment variable and then execute the script. An optional flag `--skip_exists=True` can be used to prevent overwriting existing files.

```bash
export QLIB_INDEX_DIR="/home/user/qlib_index"
python3 qlib/dump_index_weight.py
python3 qlib/dump_index_weight.py --skip_exists=True
```

--------------------------------

### Initialize Tushare API

Source: https://github.com/chenditc/investment_data/blob/main/_autodocs/api-reference-tushare-index-weight.md

Sets up the Tushare API client using an environment variable for authentication. The output directory for data is also configured.

```python
ts.set_token(os.environ["TUSHARE"])
pro = ts.pro_api()
file_path = os.path.dirname(os.path.realpath(__file__))
```

--------------------------------

### Set Temporal Calendar Start Date

Source: https://github.com/chenditc/investment_data/blob/main/_autodocs/configuration.md

Defines the earliest date for normalized data. Only trading dates from this point onward are included.

```python
CALENDAR_START_DATE = pd.Timestamp("2000-01-04")
CALENDAR_EXCHANGE = "SSE"
```

--------------------------------

### Collect Index Constituent Weights (Tushare)

Source: https://github.com/chenditc/investment_data/blob/main/_autodocs/project-overview.md

Retrieves the weight information for index constituents. Supports optional start and end dates.

```bash
python3 tushare/dump_index_weight.py [--start_date=YYYYMMDD] [--end_date=YYYYMMDD]
```

--------------------------------

### Define Calendar Exchange

Source: https://github.com/chenditc/investment_data/blob/main/_autodocs/api-reference-qlib-normalize.md

Specifies the stock exchange for which the trading calendar is loaded. This example uses the Shanghai Stock Exchange (SSE).

```python
CALENDAR_EXCHANGE = "SSE"
```

--------------------------------

### Initialize Tushare API Client

Source: https://github.com/chenditc/investment_data/blob/main/_autodocs/api-reference-tushare-dump-a-stock.md

Sets up the Tushare API client by configuring your token and initializing the client object. Ensure your Tushare API token is set as an environment variable named 'TUSHARE'.

```python
import tushare as ts
import os

ts.set_token(os.environ["TUSHARE"])
pro = ts.pro_api()
file_path = os.path.dirname(os.path.realpath(__file__))

```

--------------------------------

### Command-Line Usage for Index Weight Dump

Source: https://github.com/chenditc/investment_data/blob/main/_autodocs/api-reference-tushare-index-weight.md

Shows how to execute the index weight dumping script from the command line, including setting the TUSHARE token and specifying date ranges.

```bash
export TUSHARE="your_token_here"
python3 tushare/dump_index_weight.py
python3 tushare/dump_index_weight.py --start_date=20230101 --end_date=20231231
```

--------------------------------

### Export Stock Data to Qlib Source CLI

Source: https://github.com/chenditc/investment_data/blob/main/_autodocs/configuration.md

Command-line interface for exporting all stock data to the Qlib source directory. Supports skipping export if files already exist. Uses the QLIB_SOURCE_DIR environment variable.

```bash
python3 qlib/dump_all_to_qlib_source.py

python3 qlib/dump_all_to_qlib_source.py --skip_exists=True
```

--------------------------------

### Project Orchestration and Utility Scripts

Source: https://github.com/chenditc/investment_data/blob/main/_autodocs/project-overview.md

Scripts for orchestrating daily updates, generating releases, and uploading them. Includes a shell script for the main daily update process and another for dumping Qlib binary files.

```shell
daily_update.sh
```

```shell
dump_qlib_bin.sh
```

```shell
upload_release.sh
```

--------------------------------

### Export All Data to Qlib Source Format

Source: https://github.com/chenditc/investment_data/blob/main/_autodocs/project-overview.md

Converts all collected data into the Qlib source format. An optional flag can be used to skip existing files.

```bash
python3 qlib/dump_all_to_qlib_source.py [--skip_exists=True]
```

--------------------------------

### Get Latest Trading Day SQL Query

Source: https://github.com/chenditc/investment_data/blob/main/_autodocs/database-schema.md

Retrieves the most recent trading day from the final_a_stock_eod_price table, filtering for days with more than 1000 entries and after a specific date.

```sql
SELECT MAX(tradedate) FROM final_a_stock_eod_price
WHERE tradedate > '2023-05-01'
GROUP BY tradedate HAVING COUNT(*) > 1000
ORDER BY tradedate DESC LIMIT 1
```

--------------------------------

### Initialize Tushare API

Source: https://github.com/chenditc/investment_data/blob/main/_autodocs/api-reference-tushare-daily-update.md

Sets the Tushare API token from an environment variable and initializes the pro API client. Ensure the TUSHARE environment variable is set.

```python
ts.set_token(os.environ["TUSHARE"])
pro = ts.pro_api()
```

--------------------------------

### Run Docker Container for Daily Update and Export

Source: https://github.com/chenditc/investment_data/blob/main/_autodocs/project-overview.md

Executes the Docker container to perform a daily update, export to Qlib binary format, and copy the result to an output volume. Requires TUSHARE environment variable and an output volume mount.

```bash
docker run -e TUSHARE=$TOKEN \
           -v /output:/output \
           -it --rm investment_data \
           bash daily_update.sh && bash dump_qlib_bin.sh && cp qlib_bin.tar.gz /output/
```

--------------------------------

### Initialize Dolt Repository

Source: https://github.com/chenditc/investment_data/blob/main/_autodocs/data-flow.md

Clones the Dolt database if it doesn't exist, fetches the latest changes from the DoltHub remote, and resets the local repository to match the master branch exactly. This ensures you are working with the most up-to-date database state.

```bash
dolt clone chenditc/investment_data
dolt fetch origin master
dolt reset --hard origin/master
```

--------------------------------

### Build Docker Image

Source: https://github.com/chenditc/investment_data/blob/main/_autodocs/project-overview.md

Builds the Docker image for the investment data project with the tag 'investment_data'.

```bash
docker build -t investment_data .
```

--------------------------------

### Get Index Constituents SQL Query

Source: https://github.com/chenditc/investment_data/blob/main/_autodocs/database-schema.md

Retrieves the constituent symbols and their weights for a given index code on a specific trade date, ordered by weight in descending order.

```sql
SELECT ts_code, weight 
FROM ts_index_weight 
WHERE index_code = '399300.SZ' AND trade_date = '2023-12-31'
ORDER BY weight DESC
```

--------------------------------

### Export and Compress Data to Qlib Binary Format

Source: https://github.com/chenditc/investment_data/blob/main/_autodocs/project-overview.md

Runs a bash script to export data to Qlib's binary format and then compress it.

```bash
bash dump_qlib_bin.sh
```

--------------------------------

### dump_index_data()

Source: https://github.com/chenditc/investment_data/blob/main/_autodocs/api-reference-tushare-index-weight.md

Downloads index constituent weights for supported indices. Fetches data in 15-day batches and saves them as CSV files. Supports specifying start and end dates for the data download.

```APIDOC
## dump_index_data()

### Description
Main function to download index constituent weights. It processes data in batches to comply with Tushare API constraints and saves the output to CSV files.

### Parameters

#### Optional Parameters
- **start_date** (str) - Optional - Default: None - Start date in YYYYMMDD format. If None, uses the earliest available date for the index.
- **end_date** (str) - Optional - Default: None - End date in YYYYMMDD format. If None, it's calculated based on the start_date plus 15 days.
- **skip_exists** (bool) - Optional - Default: True - This parameter is currently unused but kept for API compatibility.

### Return Type
None. The function's primary effect is writing CSV files to the `tushare/index_weight/` directory.

### Output Format
CSV files are generated with the naming convention `{index_code}.csv`. The files contain the following columns:
- `trade_date` (str): Date of the index constituent snapshot (YYYYMMDD).
- `ts_code` (str): Stock code in Tushare format (e.g., "000001.SZ").
- `con_code` (str): Constituent stock code, same as `ts_code`.
- `weight` (float): The stock's weighting within the index (e.g., 0.5 for 0.5%).
- `stock_code` (str): Derived stock code.
- Other fields from the Tushare API.

### Usage Example
```python
import os
os.environ["TUSHARE"] = "your_token_here"

# Download data for a specific date range
dump_index_data(start_date="20230101", end_date="20231231")

# Download data from the default start dates
dump_index_data()
```

### Command-Line Usage
```bash
export TUSHARE="your_token_here"
python3 tushare/dump_index_weight.py
python3 tushare/dump_index_weight.py --start_date=20230101 --end_date=20231231
```
```

--------------------------------

### Dump Calendar to Qlib Directory CLI

Source: https://github.com/chenditc/investment_data/blob/main/_autodocs/configuration.md

Command-line interface for dumping calendar data to a specified Qlib directory. Supports skipping existing files.

```bash
python3 tushare/dump_day_calendar.py --qlib_dir=/home/user/.qlib/qlib_data/cn_data

python3 tushare/dump_day_calendar.py /home/user/.qlib/qlib_data/cn_data
```

--------------------------------

### Example Index Constituent Row

Source: https://github.com/chenditc/investment_data/blob/main/_autodocs/api-reference-tushare-index-weight.md

Illustrates the format of a single record representing a stock's membership and weight in an index on a specific date. Use `trade_date` to filter for active constituents.

```text
trade_date=20230101, ts_code=000001.SZ, con_code=000001.SZ, weight=1.23, stock_code=000001.SZ
```

--------------------------------

### Dump Daily Calendar (Tushare)

Source: https://github.com/chenditc/investment_data/blob/main/_autodocs/project-overview.md

Exports the daily trading calendar information. Requires the path to the Qlib directory.

```bash
python3 tushare/dump_day_calendar.py --qlib_dir=<path>
```

--------------------------------

### Export Index Constituent Mappings

Source: https://github.com/chenditc/investment_data/blob/main/_autodocs/data-flow.md

Exports index constituent mappings from the ts_index_weight table. Generates tab-separated files for each index, detailing symbol, start date, and end date for constituent periods.

```bash
python3 qlib/dump_index_weight.py
```

```text
00	000001	2023-01-01	2023-03-31
00	000002	2023-01-01	2023-03-31
00	000003	2023-04-01	2023-06-30
```

--------------------------------

### Get Trading Calendar Dates

Source: https://github.com/chenditc/investment_data/blob/main/_autodocs/api-reference-tushare-dump-index.md

Retrieves a sorted list of trading calendar dates within a specified range. Requires start_date and end_date in YYYYMMDD format. Returns a pandas DataFrame with 'cal_date'.

```python
def get_trade_cal(start_date, end_date):
    df = pro.trade_cal(exchange='SSE', is_open='1',
                            start_date=start_date,
                            end_date=end_date,
                            fields='cal_date')
    df = df.sort_values(by="cal_date").reset_index(drop=True)
    return df
```

--------------------------------

### Normalize Class Initialization

Source: https://github.com/chenditc/investment_data/blob/main/_autodocs/api-reference-qlib-normalize.md

Initializes Qlib's Normalize class with custom parameters for normalization. Specify the normalization class, parallelism level, and field mappings for dates and symbols.

```python
yc = Normalize(
    source_dir=source_dir,
    target_dir=normalize_dir,
    normalize_class=CrowdSourceNormalize,
    max_workers=max_workers,
    date_field_name=date_field_name,
    symbol_field_name=symbol_field_name,
)
```

--------------------------------

### Calculate End Date Incrementally

Source: https://github.com/chenditc/investment_data/blob/main/_autodocs/api-reference-tushare-index-weight.md

If no end date is specified, this logic calculates an end date by adding 15 days to the start date and continues to increment until the end date surpasses the current date.

```python
index_end_date = index_start_date + datetime.timedelta(days=15)
# Keeps shifting forward by 15 days until end_date > now
```

--------------------------------

### Tushare API Token and Initialization

Source: https://github.com/chenditc/investment_data/blob/main/_autodocs/api-reference-tushare-dump-index.md

Sets the Tushare API token from the environment variable 'TUSHARE' and initializes the Tushare API client. Also defines the base output directory for index data.

```python
import os
os.environ["TUSHARE"] = "your_token_here"

# Download all index data from 2023 onwards
dump_index_data(start_date="20230101", end_date="20231231")

# Files created:
# tushare/index/399300.SZ.csv
# tushare/index/000905.SH.csv
# tushare/index/000906.SH.csv
# ... etc
```

```python
ts.set_token(os.environ["TUSHARE"])
pro = ts.pro_api()
file_path = os.path.dirname(os.path.realpath(__file__))
```

--------------------------------

### Export Index Weights to Qlib Format

Source: https://github.com/chenditc/investment_data/blob/main/_autodocs/project-overview.md

Exports index weight data into the Qlib format. Supports an option to skip existing files.

```bash
python3 qlib/dump_index_weight.py [--skip_exists=True]
```

--------------------------------

### Dump A-Stock EOD Price Data CLI

Source: https://github.com/chenditc/investment_data/blob/main/_autodocs/configuration.md

Command-line interface for dumping A-stock end-of-day price data. Supports specifying start and end dates, and overriding the default behavior of skipping existing files.

```bash
# Basic usage with required parameters
python3 tushare/dump_a_stock_eod_price.py --start_date=20230101 --end_date=20231231

# Positional arguments (order matters)
python3 tushare/dump_a_stock_eod_price.py 20230101 20231231

# Override default skip_exists
python3 tushare/dump_a_stock_eod_price.py 20230101 20231231 --skip_exists=False

# Get help
python3 tushare/dump_a_stock_eod_price.py -- --help
```

--------------------------------

### Manual Docker Execution

Source: https://github.com/chenditc/investment_data/blob/main/_autodocs/data-flow.md

Build and run the investment data Docker image for local execution of the daily update and release dump scripts. Ensure the TUSHARE environment variable is set for the update script and the output directory is mounted.

```bash
docker build -t investment_data .
docker run -e TUSHARE=$TOKEN -v /output:/output \
    investment_data bash daily_update.sh
docker run -v /output:/output \
    investment_data bash dump_qlib_bin.sh
```

--------------------------------

### Environment Variables for Qlib Data Output

Source: https://github.com/chenditc/investment_data/blob/main/_autodocs/api-reference-qlib-export.md

Set environment variables to specify output directories for Qlib data. QLIB_SOURCE_DIR is for stock price CSVs, and QLIB_INDEX_DIR is for index constituent files. The default values use the script's directory.

```text
QLIB_SOURCE_DIR | {script_dir}/qlib_source | Output directory for stock price CSV files
QLIB_INDEX_DIR | {script_dir}/qlib_index | Output directory for index constituent files
```

--------------------------------

### Get Stock Time Series SQL Query

Source: https://github.com/chenditc/investment_data/blob/main/_autodocs/database-schema.md

Fetches the historical daily trading data (open, high, low, close, adjusted close, volume) for a specific stock symbol, ordered by trade date.

```sql
SELECT tradedate, open, high, low, close, adjclose, volume 
FROM final_a_stock_eod_price 
WHERE symbol = '000001.SZ' 
ORDER BY tradedate
```

--------------------------------

### Export Index Weights to SQL Source

Source: https://github.com/chenditc/investment_data/blob/main/_autodocs/configuration.md

Use this script to export index weights. The `skip_exists` parameter can prevent overwriting existing files.

```bash
python3 qlib/dump_index_weight.py
```

```bash
python3 qlib/dump_index_weight.py --skip_exists=True
```

--------------------------------

### Execution of Normalization Pipeline

Source: https://github.com/chenditc/investment_data/blob/main/_autodocs/api-reference-qlib-normalize.md

Runs the full normalization pipeline, discovering CSV files, normalizing data using the specified class, and writing the output in Qlib binary format.

```python
yc.normalize()
```

--------------------------------

### Release Upload Schedule

Source: https://github.com/chenditc/investment_data/blob/main/_autodocs/configuration.md

Configures a GitHub Actions workflow to run daily for generating and uploading Qlib binary tarballs.

```yaml
# File: .github/workflows/upload_release.yml

# Runs daily to generate and upload the Qlib binary tarball.

# Trigger: GitHub Actions scheduler (daily)

# Execution: docker run ... bash dump_qlib_bin.sh && bash upload_release.sh

# Environment variables required:
# - GITHUB_PAT: GitHub personal access token for uploading releases
```

--------------------------------

### Initialize Tushare API

Source: https://github.com/chenditc/investment_data/blob/main/_autodocs/configuration.md

Initialize the Tushare API by setting the token from the environment variable and creating a Tushare API object. This code is used by all Tushare modules.

```python
import tushare as ts
import os
ts.set_token(os.environ["TUSHARE"])
pro = ts.pro_api()
```

--------------------------------

### Qlib All Data Dump

Source: https://github.com/chenditc/investment_data/blob/main/_autodocs/project-overview.md

Functions for dumping all data to Qlib source.

```APIDOC
## `dump_all_to_sqlib_source`

### Description
Dumps all available data to the Qlib source directory.

### Parameters
- `skip_exists` (bool) - If True, skips dumping data for entries that already exist. Defaults to False.

### Returns
- `None`
```

--------------------------------

### get_daily()

Source: https://github.com/chenditc/investment_data/blob/main/_autodocs/api-reference-tushare-dump-a-stock.md

Fetches daily price data and adjustment factors for all A-stocks on a specific trading date, then merges them to calculate adjusted closing prices.

```APIDOC
## get_daily()

### Description
Fetches daily price data and adjustment factors for all A-stocks on a specific trading date, then merges them to calculate adjusted closing prices.

### Parameters
#### Path Parameters
- None

#### Query Parameters
- **trade_date** (str) - Optional - '' (current date) - Trading date in YYYYMMDD format

### Request Example
```python
import tushare as ts

daily_data = ts.get_daily(trade_date="20230110")
if daily_data is not None:
    print(daily_data[["ts_code", "close", "adj_close"]].head())
```

### Response
#### Success Response (200)
- **pandas.DataFrame** with columns:
  - `ts_code` (str): Tushare stock code (e.g., "000001.SZ")
  - `open` (float): Opening price
  - `close` (float): Closing price
  - `high` (float): Highest price of the day
  - `low` (float): Lowest price of the day
  - `vol` (int): Trading volume in shares
  - `amount` (float): Trading amount in Yuan
  - `adj_factor` (float): Adjustment factor for splits and dividends
  - `adj_close` (float): Adjusted closing price = close * adj_factor

Returns None if all retry attempts fail.

#### Response Example
```json
{
  "ts_code": ["000001.SZ", "000002.SZ"],
  "open": [10.5, 20.2],
  "close": [10.6, 20.3],
  "high": [10.7, 20.4],
  "low": [10.4, 20.1],
  "vol": [100000, 200000],
  "amount": [1060000.0, 4060000.0],
  "adj_factor": [1.0, 1.0],
  "adj_close": [10.6, 20.3]
}
```
```

--------------------------------

### dump_all_to_sqlib_source()

Source: https://github.com/chenditc/investment_data/blob/main/_autodocs/api-reference-qlib-export.md

Exports normalized OHLCV data from the 'final_a_stock_eod_price' database table to CSV files, with one file per stock symbol. It calculates VWAP and organizes the output for Qlib consumption.

```APIDOC
## dump_all_to_sqlib_source()

### Description
Reads finalized stock price data from the database, calculates VWAP, and exports one CSV file per stock. This function is designed to prepare data for Qlib.

### Method
Python Function

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
None

### Parameters Table
| Parameter | Type | Required | Default | Description |
|---|---|---|---|---|
| skip_exists | bool | No | False | Skip exporting if CSV already exists |

### Return Type
None. Side effect: creates CSV files in output directory.

### Output Format
**Directory:** Controlled by `QLIB_SOURCE_DIR` environment variable, defaults to `{script_dir}/qlib_source/`

**File naming:** `{symbol}.csv` (e.g., `000001.SZ.csv`)

**Columns in each file:**
- `tradedate` (str): Trading date in YYYY-MM-DD format
- `symbol` (str): Stock symbol
- `high` (float): Daily high price
- `low` (float): Daily low price
- `open` (float): Opening price
- `close` (float): Closing price
- `volume` (int): Trading volume in shares
- `adjclose` (float): Adjusted closing price
- `amount` (float): Trading amount in Yuan
- `vwap` (float): Volume-weighted average price = amount / volume * 10

### VWAP Calculation
```python
vwap = amount / volume * 10
```
This formula is specific to Qlib's data format and represents the volume-weighted average price with a scaling factor.

### Database Query
```sql
SELECT *, amount/volume*10 as vwap 
FROM final_a_stock_eod_price
```
Selects all columns from the finalized price table plus calculated VWAP.

### Row Organization
Grouped by `symbol`, so each output file contains:
- All historical dates for one symbol
- Sorted in database query order (typically chronological)
- No index written to CSV

### Throws/Exceptions
- Propagates database connection errors
- Silent skip if `skip_exists=True` and file already exists

### Usage Example
```python
import os
os.environ["QLIB_SOURCE_DIR"] = "/output/qlib_data"

# Export all stock data
dump_all_to_sqlib_source(skip_exists=False)

# Creates files:
# /output/qlib_data/000001.SZ.csv
# /output/qlib_data/000002.SZ.csv
# ... etc for all symbols in database
```

### Command-Line Usage
```bash
export QLIB_SOURCE_DIR="/home/user/qlib_data"
python3 qlib/dump_all_to_qlib_source.py
python3 qlib/dump_all_to_qlib_source.py --skip_exists=True
```
```

--------------------------------

### Daily Update Workflow Script

Source: https://github.com/chenditc/investment_data/blob/main/_autodocs/api-reference-tushare-daily-update.md

A bash script demonstrating the daily update process using Dolt SQL server and a Python script to update stock prices. This workflow is designed for incremental and idempotent updates.

```bash
dolt sql-server &  # Start database server in background
sleep 5 && python3 tushare/update_a_stock_eod_price_to_latest.py  # Update with latest
killall dolt  # Stop server
```

--------------------------------

### Commit and Push Changes

Source: https://github.com/chenditc/investment_data/blob/main/_autodocs/data-flow.md

Stages all changes in the Dolt repository, creates a commit with a specified message, and then force pushes the changes to the origin master branch on DoltHub. This ensures the master branch always reflects the latest state.

```bash
dolt add -A
dolt commit -m "Daily update"
dolt push --force origin master
```

--------------------------------

### Database Connection Configuration

Source: https://github.com/chenditc/investment_data/blob/main/_autodocs/api-reference-qlib-export.md

Configure the database connection string and connection pool recycling time. Ensure the database URL is correctly formatted.

```yaml
Database: mysql+pymysql://root:@127.0.0.1/investment_data
Pool recycle: 3600 seconds
```