### Complete Workflow Example Source: https://context7.com/yehoshuadimarsky/bcpandas/llms.txt Demonstrates the full process from credential setup and DataFrame creation to bulk insertion and verification. ```python import pandas as pd import numpy as np from bcpandas import SqlCreds, to_sql # 1. Setup credentials creds = SqlCreds( server='myserver.database.windows.net', database='mydb', username='admin', password='secretpassword', odbc_kwargs={'Encrypt': 'yes'} ) # 2. Create or load your DataFrame df = pd.DataFrame({ 'product_id': range(1, 10001), 'product_name': [f'Product_{i}' for i in range(1, 10001)], 'price': np.random.uniform(10, 1000, 10000).round(2), 'quantity': np.random.randint(0, 100, 10000), 'is_available': np.random.choice([True, False], 10000) }) # 3. Write to SQL Server with optimal settings to_sql( df=df, table_name='products', creds=creds, index=False, if_exists='replace', schema='sales', batch_size=5000, use_tablock=True, print_output=False # Suppress BCP output ) # 4. Verify the data using pandas native read df_verify = pd.read_sql_query( "SELECT COUNT(*) as cnt FROM sales.products", creds.engine ) print(f"Inserted {df_verify['cnt'].iloc[0]} rows") # 5. Read full table back df_result = pd.read_sql_table('products', creds.engine, schema='sales') print(df_result.head()) ``` -------------------------------- ### Quickstart: Pandas to SQL Server and Back Source: https://github.com/yehoshuadimarsky/bcpandas/blob/master/README.md Demonstrates the basic workflow of creating a pandas DataFrame, writing it to SQL Server using bcpandas, and then reading it back. ```python import pandas as pd import numpy as np from bcpandas import SqlCreds, to_sql ``` ```python creds = SqlCreds( 'my_server', 'my_db', 'my_username', 'my_password' ) ``` ```python df = pd.DataFrame( data=np.ndarray(shape=(10, 6), dtype=int), columns=[f"col_{x}" for x in range(6)] ) ``` ```python df ``` ```python to_sql(df, 'my_test_table', creds, index=False, if_exists='replace') ``` ```python df2 = pd.read_sql_table(table_name='my_test_table', con=creds.engine) ``` ```python df2 ``` -------------------------------- ### Install bcpandas using pip Source: https://github.com/yehoshuadimarsky/bcpandas/blob/master/README.md Install the bcpandas library from PyPI using pip. ```bash pip install bcpandas ``` -------------------------------- ### Install bcpandas using conda Source: https://github.com/yehoshuadimarsky/bcpandas/blob/master/README.md Install the bcpandas library from the conda-forge channel. ```bash conda install -c conda-forge bcpandas ``` -------------------------------- ### Advanced to_sql Configuration Source: https://context7.com/yehoshuadimarsky/bcpandas/llms.txt Configure debugging, custom directories, delimiters, encoding, collation, identity inserts, and BCP paths. ```python import pandas as pd from pathlib import Path from bcpandas import SqlCreds, to_sql creds = SqlCreds('my_server', 'my_db', 'user', 'password') df = pd.DataFrame({ 'name': ['Alice', 'Bob'], 'value': [100, 200] }) # Debug mode - keeps temporary CSV and format files for inspection to_sql( df=df, table_name='debug_table', creds=creds, index=False, if_exists='replace', debug=True, # Temporary files not deleted print_output=True # Print BCP output to console ) ``` ```python # Custom work directory for temporary files to_sql( df=df, table_name='temp_dir_table', creds=creds, index=False, if_exists='replace', work_directory=Path('/tmp/bcpandas_work') ) ``` ```python # Custom delimiter and quote character (use with caution) to_sql( df=df, table_name='custom_delim_table', creds=creds, index=False, if_exists='replace', delimiter='|', quotechar="'" ) ``` ```python # Specify encoding for non-ASCII data to_sql( df=df, table_name='encoded_table', creds=creds, index=False, if_exists='replace', encoding='utf-8' ) ``` ```python # Custom collation for Unicode support to_sql( df=pd.DataFrame({'text': ['Hello', 'World', 'Caf\u00e9']}), table_name='unicode_table', creds=creds, index=False, if_exists='replace', collation='Latin1_General_100_CI_AS_SC_UTF8' ) ``` ```python # Insert values into identity columns to_sql( df=pd.DataFrame({'id': [1, 2, 3], 'name': ['A', 'B', 'C']}), table_name='identity_table', creds=creds, index=False, if_exists='replace', identity_insert=True # Allow explicit identity values ) ``` ```python # Custom BCP executable path to_sql( df=df, table_name='custom_bcp_table', creds=creds, index=False, if_exists='replace', bcp_path='/opt/mssql-tools/bin/bcp' ) ``` -------------------------------- ### Optimize to_sql Performance Source: https://context7.com/yehoshuadimarsky/bcpandas/llms.txt Use batch_size and use_tablock to improve performance when inserting large datasets into SQL Server. ```python to_sql( df=df, table_name='large_table', creds=creds, index=False, if_exists='replace', batch_size=10000 # Insert in batches of 10,000 rows ) ``` ```python to_sql( df=df, table_name='large_table', creds=creds, index=False, if_exists='replace', use_tablock=True # Acquire table-level lock instead of row-level ) ``` ```python to_sql( df=df, table_name='large_table', creds=creds, index=False, if_exists='replace', batch_size=50000, use_tablock=True ) ``` -------------------------------- ### Execute BCP Import Source: https://context7.com/yehoshuadimarsky/bcpandas/llms.txt Perform bulk data imports using the bcp function with various configuration options. ```python bcp( sql_item='my_table', direction='in', flat_file=csv_path, format_file_path=format_path, creds=creds, print_output=True, sql_type='table', schema='dbo' ) ``` ```python bcp( sql_item='my_table', direction='in', flat_file=csv_path, format_file_path=format_path, creds=creds, print_output=True, batch_size=1000, use_tablock=True ) ``` -------------------------------- ### Generate BCP Format Files Source: https://context7.com/yehoshuadimarsky/bcpandas/llms.txt Create non-XML BCP format files for custom column mappings and collation settings. ```python import pandas as pd from bcpandas.utils import build_format_file # Create DataFrame df = pd.DataFrame({ 'name': ['Alice', 'Bob'], 'age': [30, 25], 'city': ['NYC', 'LA'] }) # Generate format file content format_content = build_format_file(df, delimiter=',') print(format_content) # 9.0 # 3 # 1 SQLCHAR 0 0 "," 1 name SQL_Latin1_General_CP1_CI_AS # 2 SQLCHAR 0 0 "," 2 age SQL_Latin1_General_CP1_CI_AS # 3 SQLCHAR 0 0 "\n" 3 city SQL_Latin1_General_CP1_CI_AS ``` ```python # With custom column order mapping (for append operations) db_cols_order = {'age': 1, 'city': 2, 'name': 3} # Database column positions format_content = build_format_file( df, delimiter='|', db_cols_order=db_cols_order ) ``` ```python # With custom collation for Unicode format_content = build_format_file( df, delimiter='\t', collation='Latin1_General_100_CI_AS_SC_UTF8' ) ``` -------------------------------- ### Low-Level BCP Utility Usage Source: https://context7.com/yehoshuadimarsky/bcpandas/llms.txt Directly access the BCP utility for advanced scenarios requiring manual format file generation. ```python from pathlib import Path from bcpandas import SqlCreds from bcpandas.utils import bcp, build_format_file, get_temp_file import pandas as pd creds = SqlCreds('my_server', 'my_db', 'user', 'password') # Prepare DataFrame and save to CSV df = pd.DataFrame({'col1': [1, 2, 3], 'col2': ['a', 'b', 'c']}) csv_path = get_temp_file() df.to_csv(csv_path, sep=',', header=False, index=False) # Build format file format_content = build_format_file(df, delimiter=',') format_path = get_temp_file() with open(format_path, 'w') as f: f.write(format_content) ``` -------------------------------- ### to_sql with Performance Options Source: https://context7.com/yehoshuadimarsky/bcpandas/llms.txt The `to_sql` function provides several options to optimize bulk insert performance including batch sizes and table locking. ```APIDOC ## to_sql with Performance Options ### Description The `to_sql` function provides several options to optimize bulk insert performance including batch sizes and table locking. ### Method `to_sql(df, table_name, creds, index=False, if_exists='fail', schema=None, **kwargs)` ### Parameters #### Path Parameters None #### Query Parameters None #### Request Body - **df** (pd.DataFrame) - The pandas DataFrame to write. - **table_name** (str) - The name of the SQL Server table. - **creds** (SqlCreds) - An instance of the `SqlCreds` class. - **index** (bool) - Whether to include the DataFrame index as a column. Defaults to `False`. - **if_exists** (str) - How to behave if the table already exists. Options: 'fail', 'replace', 'append'. Defaults to 'fail'. - **schema** (str, optional) - The schema to write the table to. Defaults to None. - **kwargs** - Additional keyword arguments passed to the BCP utility, such as `batch_size` and `table_lock`. ### Request Example ```python import pandas as pd from bcpandas import SqlCreds, to_sql creds = SqlCreds('my_server', 'my_db', 'user', 'password') # Create a large DataFrame df = pd.DataFrame({ 'col1': range(100000), 'col2': ['value'] * 100000, 'col3': [1.5] * 100000 }) # Example with performance options (assuming these are valid kwargs for BCP) to_sql(df, 'large_table', creds, index=False, if_exists='replace', batch_size=10000, table_lock=True) ``` ### Response #### Success Response (200) None (This function performs a write operation and does not return a value directly, but it may raise exceptions on failure.) #### Response Example None ``` -------------------------------- ### Create SqlCreds with minimum attributes Source: https://github.com/yehoshuadimarsky/bcpandas/blob/master/README.md Create a SqlCreds object with server, database, username, and password. bcpandas will generate the SQLAlchemy Engine. ```python from bcpandas import SqlCreds creds = SqlCreds('my_server', 'my_db', 'my_username', 'my_password') ``` ```python creds.engine ``` -------------------------------- ### Optimize to_sql Performance Source: https://context7.com/yehoshuadimarsky/bcpandas/llms.txt Configure batch sizes and locking options to optimize bulk insert performance for large DataFrames. ```python import pandas as pd from bcpandas import SqlCreds, to_sql creds = SqlCreds('my_server', 'my_db', 'user', 'password') # Create a large DataFrame df = pd.DataFrame({ 'col1': range(100000), 'col2': ['value'] * 100000, 'col3': [1.5] * 100000 }) ``` -------------------------------- ### Create SqlCreds from SQLAlchemy Engine Source: https://context7.com/yehoshuadimarsky/bcpandas/llms.txt Initialize a SqlCreds object by parsing connection details from an existing SQLAlchemy engine. ```python from bcpandas import SqlCreds import sqlalchemy as sa # Create an engine using SQLAlchemy engine = sa.create_engine( "mssql+pyodbc:///?odbc_connect=" "Driver={ODBC Driver 17 for SQL Server};" "Server=tcp:my_server,1433;" "Database=my_database;" "UID=my_username;" "PWD=my_password" ) # Create SqlCreds from the engine creds = SqlCreds.from_engine(engine) print(creds.server) # 'my_server' print(creds.database) # 'my_database' print(creds.username) # 'my_username' print(creds.engine) # The original engine object ``` -------------------------------- ### SqlCreds.from_engine - Create Credentials from Existing Engine Source: https://context7.com/yehoshuadimarsky/bcpandas/llms.txt The `from_engine` class method creates a `SqlCreds` object from an existing SQLAlchemy engine, parsing the connection string to extract server, database, and authentication details. ```APIDOC ## SqlCreds.from_engine - Create Credentials from Existing Engine ### Description The `from_engine` class method creates a `SqlCreds` object from an existing SQLAlchemy engine, parsing the connection string to extract server, database, and authentication details. ### Method `SqlCreds.from_engine(engine)` ### Parameters #### Path Parameters None #### Query Parameters None #### Request Body None ### Request Example ```python from bcpandas import SqlCreds import sqlalchemy as sa # Create an engine using SQLAlchemy engine = sa.create_engine( "mssql+pyodbc:///?odbc_connect=" "Driver={ODBC Driver 17 for SQL Server};" "Server=tcp:my_server,1433;" "Database=my_database;" "UID=my_username;" "PWD=my_password" ) # Create SqlCreds from the engine creds = SqlCreds.from_engine(engine) print(creds.server) # 'my_server' print(creds.database) # 'my_database' print(creds.username) # 'my_username' print(creds.engine) # The original engine object ``` ### Response #### Success Response (200) None (This is a class method that returns a SqlCreds object) #### Response Example None ``` -------------------------------- ### to_sql - Write DataFrame to SQL Server Source: https://context7.com/yehoshuadimarsky/bcpandas/llms.txt The main function `to_sql` writes a pandas DataFrame to a SQL Server table using BCP for high-performance bulk inserts. It supports creating new tables, replacing existing tables, or appending to existing tables. ```APIDOC ## to_sql - Write DataFrame to SQL Server ### Description The main function `to_sql` writes a pandas DataFrame to a SQL Server table using BCP for high-performance bulk inserts. It supports creating new tables, replacing existing tables, or appending to existing tables. ### Method `to_sql(df, table_name, creds, index=False, if_exists='fail', schema=None, **kwargs)` ### Parameters #### Path Parameters None #### Query Parameters None #### Request Body - **df** (pd.DataFrame) - The pandas DataFrame to write. - **table_name** (str) - The name of the SQL Server table. - **creds** (SqlCreds) - An instance of the `SqlCreds` class. - **index** (bool) - Whether to include the DataFrame index as a column. Defaults to `False`. - **if_exists** (str) - How to behave if the table already exists. Options: 'fail', 'replace', 'append'. Defaults to 'fail'. - **schema** (str, optional) - The schema to write the table to. Defaults to None. - **kwargs** - Additional keyword arguments passed to the BCP utility. ### Request Example ```python import pandas as pd import numpy as np from bcpandas import SqlCreds, to_sql # Create credentials creds = SqlCreds( server='my_server', database='my_database', username='my_username', password='my_password' ) # Create sample DataFrame df = pd.DataFrame({ 'id': [1, 2, 3, 4, 5], 'name': ['Alice', 'Bob', 'Charlie', 'Diana', 'Eve'], 'score': [95.5, 87.3, 92.1, 88.9, 91.7], 'active': [True, True, False, True, False] }) # Basic usage - create new table (fails if exists) to_sql(df, 'students', creds, index=False, if_exists='fail') # Replace existing table to_sql(df, 'students', creds, index=False, if_exists='replace') # Append to existing table to_sql(df, 'students', creds, index=False, if_exists='append') # Include DataFrame index as a column to_sql(df, 'students', creds, index=True, if_exists='replace') # Specify schema to_sql(df, 'students', creds, index=False, if_exists='replace', schema='analytics') # Read data back using pandas native method df_result = pd.read_sql_table('students', creds.engine) print(df_result) ``` ### Response #### Success Response (200) None (This function performs a write operation and does not return a value directly, but it may raise exceptions on failure.) #### Response Example None ``` -------------------------------- ### Specify MSSQL Docker Image for Pytest Source: https://github.com/yehoshuadimarsky/bcpandas/blob/master/README.md When running pytest, you can specify a particular Docker image for the MSSQL server using the --mssql-docker-image option. ```bash pytest bcpandas/tests --mssql-docker-image mcr.microsoft.com/mssql/server:2019-latest ``` -------------------------------- ### Manage Database Credentials with SqlCreds Source: https://context7.com/yehoshuadimarsky/bcpandas/llms.txt Use SqlCreds to handle authentication for SQL Server, including username/password, Kerberos, and Microsoft Entra ID. The class automatically generates a SQLAlchemy engine for read operations. ```python from bcpandas import SqlCreds # Basic authentication with username and password creds = SqlCreds( server='my_server.database.windows.net', database='my_database', username='my_username', password='my_password' ) # Access the SQLAlchemy engine for read operations print(creds.engine) # Engine(mssql+pyodbc:///?odbc_connect=Driver={ODBC Driver 17 for SQL Server};Server=tcp:my_server.database.windows.net,1433;Database=my_database;UID=my_username;PWD=my_password) # Windows/Kerberos authentication (no username/password) creds_kerberos = SqlCreds( server='my_server', database='my_database' ) print(creds_kerberos.with_krb_auth) # True # Custom port and ODBC options creds_custom = SqlCreds( server='my_server', database='my_database', username='my_username', password='my_password', port=1434, driver_version=17, odbc_kwargs={'Encrypt': 'yes', 'TrustServerCertificate': 'no'} ) # Microsoft Entra ID authentication creds_entra = SqlCreds( server='my_server.database.windows.net', database='my_database', entra_id_token='your_entra_id_token' ) ``` -------------------------------- ### Handle BCPandas Exceptions Source: https://context7.com/yehoshuadimarsky/bcpandas/llms.txt Manage errors during bulk operations such as table existence conflicts, duplicate columns, or delimiter issues. ```python import pandas as pd from bcpandas import SqlCreds, to_sql from bcpandas.constants import BCPandasValueError, BCPandasException creds = SqlCreds('my_server', 'my_db', 'user', 'password') # Handle table already exists error df = pd.DataFrame({'col1': [1, 2, 3]}) try: to_sql(df, 'existing_table', creds, index=False, if_exists='fail') except BCPandasValueError as e: print(f"Table operation failed: {e}") # Handle duplicate column names df_bad = pd.DataFrame([[1, 2, 3]], columns=['a', 'b', 'a']) try: to_sql(df_bad, 'test_table', creds, index=False, if_exists='replace') except BCPandasValueError as e: print(f"Validation error: {e}") # Handle all delimiter characters in data df_delims = pd.DataFrame({ 'text': [',', '|', '\t'] # Contains all possible delimiters }) try: to_sql(df_delims, 'delim_table', creds, index=False, if_exists='replace') except BCPandasValueError as e: print(f"Delimiter error: {e}") # Solution: Replace one delimiter character in your data df_delims['text'] = df_delims['text'].str.replace('|', '/') to_sql(df_delims, 'delim_table', creds, index=False, if_exists='replace') # Handle BCP execution errors try: to_sql(df, 'test_table', creds, index=False, if_exists='replace') except BCPandasException as e: print(f"BCP execution failed: {e}") print(f"Error details: {e.details}") ``` -------------------------------- ### SqlCreds - Database Credentials Source: https://context7.com/yehoshuadimarsky/bcpandas/llms.txt The SqlCreds class manages database credentials and automatically creates a SQLAlchemy engine for connecting to Microsoft SQL Server. It supports various authentication methods. ```APIDOC ## SqlCreds - Database Credentials ### Description The `SqlCreds` class manages database credentials and automatically creates a SQLAlchemy engine for connecting to Microsoft SQL Server. It supports username/password authentication, Kerberos authentication, and Microsoft Entra ID token-based authentication. ### Method `SqlCreds(...)` ### Parameters #### Path Parameters None #### Query Parameters None #### Request Body None ### Request Example ```python from bcpandas import SqlCreds # Basic authentication with username and password creds = SqlCreds( server='my_server.database.windows.net', database='my_database', username='my_username', password='my_password' ) # Windows/Kerberos authentication (no username/password) creds_kerberos = SqlCreds( server='my_server', database='my_database' ) # Custom port and ODBC options creds_custom = SqlCreds( server='my_server', database='my_database', username='my_username', password='my_password', port=1434, driver_version=17, odbc_kwargs={'Encrypt': 'yes', 'TrustServerCertificate': 'no'} ) # Microsoft Entra ID authentication creds_entra = SqlCreds( server='my_server.database.windows.net', database='my_database', entra_id_token='your_entra_id_token' ) ``` ### Response #### Success Response (200) None (This is a constructor) #### Response Example None ``` -------------------------------- ### Write DataFrame to SQL Server with to_sql Source: https://context7.com/yehoshuadimarsky/bcpandas/llms.txt Perform high-performance bulk inserts from a pandas DataFrame into a SQL Server table. Supports various table handling modes like fail, replace, and append. ```python import pandas as pd import numpy as np from bcpandas import SqlCreds, to_sql # Create credentials creds = SqlCreds( server='my_server', database='my_database', username='my_username', password='my_password' ) # Create sample DataFrame df = pd.DataFrame({ 'id': [1, 2, 3, 4, 5], 'name': ['Alice', 'Bob', 'Charlie', 'Diana', 'Eve'], 'score': [95.5, 87.3, 92.1, 88.9, 91.7], 'active': [True, True, False, True, False] }) # Basic usage - create new table (fails if exists) to_sql(df, 'students', creds, index=False, if_exists='fail') # Replace existing table to_sql(df, 'students', creds, index=False, if_exists='replace') # Append to existing table to_sql(df, 'students', creds, index=False, if_exists='append') # Include DataFrame index as a column to_sql(df, 'students', creds, index=True, if_exists='replace') # Specify schema to_sql(df, 'students', creds, index=False, if_exists='replace', schema='analytics') # Read data back using pandas native method df_result = pd.read_sql_table('students', creds.engine) print(df_result) ``` -------------------------------- ### Specify Column Types with to_sql Source: https://context7.com/yehoshuadimarsky/bcpandas/llms.txt Use the dtype parameter to map pandas DataFrames to specific SQL Server column types using SQLAlchemy types. ```python import pandas as pd import sqlalchemy from datetime import date from bcpandas import SqlCreds, to_sql creds = SqlCreds('my_server', 'my_db', 'user', 'password') df = pd.DataFrame({ 'id': [1, 2, 3, 4], 'price': [19.99, 29.99, 39.99, 49.99], 'name': ['Widget', 'Gadget', 'Gizmo', 'Thing'], 'created': [date(2024, 1, 1), date(2024, 1, 2), date(2024, 1, 3), date(2024, 1, 4)], 'is_active': [True, True, False, True] }) # Specify exact SQL Server column types to_sql( df=df, table_name='products', creds=creds, index=False, if_exists='replace', dtype={ 'id': sqlalchemy.types.INTEGER(), 'price': sqlalchemy.types.DECIMAL(10, 2), 'name': sqlalchemy.types.NVARCHAR(length=100), 'created': sqlalchemy.types.DATE(), 'is_active': sqlalchemy.dialects.mssql.BIT() } ) ``` ```python # Verify column types result = pd.read_sql_query(""" SELECT COLUMN_NAME, DATA_TYPE, CHARACTER_MAXIMUM_LENGTH FROM INFORMATION_SCHEMA.COLUMNS WHERE TABLE_NAME = 'products' """, creds.engine) print(result) ``` -------------------------------- ### Check for Delimiter Character Occurrences Source: https://github.com/yehoshuadimarsky/bcpandas/blob/master/README.md Count the occurrences of a delimiter character (e.g., '|') in a DataFrame column. This is used to confirm the successful removal of the delimiter before BCP import. ```python my_df['some_text_column'].str.contains('Â\|').sum() ``` -------------------------------- ### Check for Quote Character Occurrences Source: https://github.com/yehoshuadimarsky/bcpandas/blob/master/README.md Verify the number of occurrences of a specific quote character within a DataFrame column before and after replacement. This helps confirm that the character has been successfully removed. ```python my_df['some_text_column'].str.contains('~').sum() ``` -------------------------------- ### Replace Quote Characters in DataFrame Source: https://github.com/yehoshuadimarsky/bcpandas/blob/master/README.md Use this code to replace a specific quote character (e.g., '~') with another character (e.g., '-') in a DataFrame column. This is useful when all possible quote characters are present in the data, preventing BCP import. ```python my_df['some_text_column'] = my_df['some_text_column'].str.replace('~','-') ``` -------------------------------- ### Replace Delimiter Characters in DataFrame Source: https://github.com/yehoshuadimarsky/bcpandas/blob/master/README.md Replace a delimiter character (e.g., '|') with another character (e.g., '/') in a DataFrame column. Note that the pipe character needs to be escaped with a backslash in regular expressions. ```python my_df['some_text_column'] = my_df['some_text_column'].str.replace('Â\|','/') ``` -------------------------------- ### Replace Spaces in Column Names Source: https://github.com/yehoshuadimarsky/bcpandas/blob/master/README.md Replace any space characters in DataFrame column names with underscores. This is a common solution for errors encountered when writing to databases due to spaces in column names. ```python my_df.columns = my_df.columns.str.replace(' ','_') ``` === COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.