### Analyze SQL File with SQLLineage CLI Source: https://github.com/reata/sqllineage/blob/master/docs/first_steps/getting_started.rst Demonstrates using the sqllineage command-line tool with the -f option to analyze SQL queries contained within a specified file. It shows how to get source and target tables from the file's content. ```Bash $ sqllineage -f foo.sql Statements(#): 1 Source Tables: .table_bar .table_baz Target Tables: .table_foo ``` -------------------------------- ### Analyze SQL String with SQLLineage CLI Source: https://github.com/reata/sqllineage/blob/master/docs/first_steps/getting_started.rst Demonstrates using the sqllineage command-line tool with the -e option to analyze a SQL query provided directly as a quoted string. It shows how to get source and target tables from the query. ```Bash $ sqllineage -e "insert into table_foo select * from table_bar union select * from table_baz" Statements(#): 1 Source Tables: .table_bar .table_baz Target Tables: .table_foo ``` -------------------------------- ### Install SQLLineage using Pip - Bash Source: https://github.com/reata/sqllineage/blob/master/README.md This command installs the SQLLineage library using the Python package installer, pip. It is the standard way to get the tool ready for use. ```Bash pip install sqllineage ``` -------------------------------- ### Install Pre-commit Hook - Bash Source: https://github.com/reata/sqllineage/blob/master/CONTRIBUTING.md Installs the pre-commit hook into the local Git repository. Once installed, the hook will automatically run configured checks (like black, ruff, mypy) before each commit. ```Bash pre-commit install ``` -------------------------------- ### Install Pre-commit - Bash Source: https://github.com/reata/sqllineage/blob/master/CONTRIBUTING.md Installs the pre-commit tool using pip. Pre-commit is used to run checks on client-side before commits are made. ```Bash pip install pre-commit ``` -------------------------------- ### Generating Lineage Visualization Graph with sqllineage CLI Source: https://github.com/reata/sqllineage/blob/master/README.md Shows how to use the sqllineage command-line tool with the -g (graph visualization) and -f (file) options to start a web server that displays the lineage result as a DAG in a browser. ```Shell sqllineage -g -f foo.sql ``` -------------------------------- ### Create SQLite Table for Metadata Example (Bash) Source: https://github.com/reata/sqllineage/blob/master/docs/first_steps/advanced_usage.rst Provides a bash command using `sqlite3` to create a sample table named `baz` in a database file `db.db`. This table creation is part of an example demonstrating how providing metadata can improve column-level lineage results. ```bash sqlite3 db.db 'CREATE TABLE IF NOT EXISTS baz (bar_id int, col1 int, col4 int)'; ``` -------------------------------- ### Using SQLAlchemyMetaDataProvider with Snowflake URL in Python Source: https://github.com/reata/sqllineage/blob/master/docs/gear_up/metadata.rst Provides an example of how to instantiate SQLAlchemyMetaDataProvider using a Snowflake connection URL string, demonstrating the format required after installing the appropriate driver. ```python >>> use, password, account = "", "", "" >>> provider = SQLAlchemyMetaDataProvider(f"snowflake://{user}:{password}@{account}/") ``` -------------------------------- ### Example SQL for Column-Level Lineage Source: https://github.com/reata/sqllineage/blob/master/docs/first_steps/advanced_usage.rst Provides a multi-statement SQL script used as an example input for demonstrating column-level lineage analysis with `sqllineage`. It involves joins, subqueries, and different column selection methods. ```sql INSERT INTO foo SELECT a.col1, b.col1 AS col2, c.col3_sum AS col3, col4, d.* FROM bar a JOIN baz b ON a.id = b.bar_id LEFT JOIN (SELECT bar_id, sum(col3) AS col3_sum FROM qux GROUP BY bar_id) c ON a.id = sq.bar_id CROSS JOIN quux d; INSERT INTO corge SELECT a.col1, a.col2 + b.col2 AS col2 FROM foo a LEFT JOIN grault b ON a.col1 = b.col1; ``` -------------------------------- ### Install Snowflake SQLAlchemy Driver via pip Source: https://github.com/reata/sqllineage/blob/master/docs/gear_up/metadata.rst Shows the bash command to install the necessary SQLAlchemy driver for connecting to Snowflake, which is required for SQLAlchemyMetaDataProvider to work with Snowflake databases. ```bash pip install snowflake-sqlalchemy ``` -------------------------------- ### Run SQLLineage with Graph Visualization Source: https://github.com/reata/sqllineage/blob/master/docs/first_steps/advanced_usage.rst Executes the `sqllineage` command-line tool on the 'test.sql' file with the graph visualization option (`-g`). This command starts a local web server that displays the lineage result as a directed acyclic graph (DAG) in a web browser. ```bash sqllineage -g -f test.sql ``` -------------------------------- ### Analyze SQL with Hive Dialect (Bash) Source: https://github.com/reata/sqllineage/blob/master/docs/first_steps/advanced_usage.rst Shows using the `sqllineage` command with the `--dialect=hive` option to analyze a SQL statement. This example demonstrates that even with the Hive dialect, a syntax error occurs because `MAP` is a reserved keyword in Hive. ```bash $ sqllineage -e "insert overwrite table map select * from foo" --dialect=hive ``` -------------------------------- ### Get Verbose Lineage Result per Statement (Bash) Source: https://github.com/reata/sqllineage/blob/master/docs/first_steps/advanced_usage.rst Shows how to use the `-v` (verbose) option with the `sqllineage` command-line tool to display the lineage result for each individual SQL statement within a multi-statement input string, in addition to the overall summary. ```bash $ sqllineage -v -e "insert into db1.table1 select * from db2.table2; insert into db3.table3 select * from db1.table1;" ``` -------------------------------- ### SQL Lateral Column Alias Reference Example Source: https://github.com/reata/sqllineage/blob/master/docs/gear_up/configuration.rst Illustrates the SQL syntax feature where a column alias defined earlier in the SELECT list (probability) is referenced by a subsequent expression (round(100 * probability, 1)). This feature's correct resolution in SQLLineage depends on the LATERAL_COLUMN_ALIAS_REFERENCE config and metadata provider. ```sql SELECT clicks / impressions as probability, round(100 * probability, 1) as percentage FROM raw_data ``` -------------------------------- ### Using DummyMetaDataProvider in Python Source: https://github.com/reata/sqllineage/blob/master/docs/gear_up/metadata.rst Demonstrates how to use DummyMetaDataProvider by providing a dictionary of table metadata. Shows how it handles `select *` based on provided metadata and its limitation when metadata is missing for a table. ```python >>> from sqllineage.core.metadata.dummy import DummyMetaDataProvider >>> from sqllineage.runner import LineageRunner >>> sql1 = "insert into main.foo select * from main.bar" >>> metadata = {"main.bar": ["col1", "col2"]} >>> provider = DummyMetaDataProvider(metadata) >>> LineageRunner(sql1, metadata_provider=provider).print_column_lineage() main.foo.col1 <- main.bar.col1 main.foo.col2 <- main.bar.col2 >>> sql2 = "insert into main.foo select * from main.baz" main.foo.* <- main.baz.* ``` -------------------------------- ### Creating Sample SQLite Tables for Metadata-Aware Lineage Source: https://github.com/reata/sqllineage/blob/master/README.md Provides shell commands using sqlite3 to create two tables, baz and quux, with specific columns in a database file 'db.db', used as metadata context for sqllineage. ```Shell sqlite3 db.db 'CREATE TABLE IF NOT EXISTS baz (bar_id int, col1 int, col4 int)'; sqlite3 db.db 'CREATE TABLE IF NOT EXISTS quux (quux_id int, col5 int, col6 int)'; ``` -------------------------------- ### Run sqllineage from command line with dialect Source: https://github.com/reata/sqllineage/blob/master/docs/behind_the_scene/dialect-awareness_lineage_design.rst Demonstrates how to execute the sqllineage command-line tool, specifying an input SQL file (`test.sql`) and the desired SQL dialect (`ansi`) using the `--dialect` option. This allows users to leverage dialect-specific parsing capabilities. ```bash sqllineage -f test.sql --dialect=ansi ``` -------------------------------- ### Analyzing Metadata-Aware Column Lineage with sqllineage CLI Source: https://github.com/reata/sqllineage/blob/master/README.md Demonstrates how to use sqllineage with the --sqlalchemy_url option and SQLLINEAGE_DEFAULT_SCHEMA environment variable to perform column lineage analysis using metadata from a SQLite database. ```Shell SQLLINEAGE_DEFAULT_SCHEMA=main sqllineage -f test.sql -l column --sqlalchemy_url=sqlite:///db.db ``` -------------------------------- ### Running SQL Lineage with Python API Source: https://github.com/reata/sqllineage/blob/master/docs/first_steps/beyond_command_line.rst This snippet illustrates the fundamental usage of the sqllineage LineageRunner class. It shows how to import the class, pass SQL statements to it, obtain a result object, and access the parsed source and target tables. It also mentions the draw() method for visualization. ```python from sqllineage.runner import LineageRunner sql = "insert into db1.table11 select * from db2.table21 union select * from db2.table22;" sql += "insert into db3.table3 select * from db1.table11 join db1.table12;" result = LineageRunner(sql) # To show lineage summary # print(result) # To parse all the source tables for tbl in result.source_tables: print(tbl) # likewise for target tables for tbl in result.target_tables: print(tbl) # To pop up a webserver for visualization # result.draw() ``` -------------------------------- ### Analyzing SQL Lineage with Hive Dialect using sqllineage CLI Source: https://github.com/reata/sqllineage/blob/master/README.md Shows how sqllineage analyzes a SQL statement using the Hive dialect, which also reports a syntax error due to 'map' being a reserved keyword in Hive. ```Shell sqllineage -e "INSERT OVERWRITE TABLE map SELECT * FROM foo" --dialect=hive ``` -------------------------------- ### Extracting Column-Level Lineage from File using sqllineage CLI Source: https://github.com/reata/sqllineage/blob/master/README.md Shows how to use the sqllineage command-line tool with the -f (file) and -l column options to analyze a SQL script stored in 'test.sql' and output column-level lineage paths. ```Shell sqllineage -f test.sql -l column ``` -------------------------------- ### Setting SQLLineage Configuration at Runtime with Python Source: https://github.com/reata/sqllineage/blob/master/docs/gear_up/configuration.rst Demonstrates how to use the SQLLineageConfig context manager in Python to set configuration options like DEFAULT_SCHEMA locally for a specific block of code, overriding global settings. ```python >>> from sqllineage.config import SQLLineageConfig >>> from sqllineage.runner import LineageRunner >>> with SQLLineageConfig(DEFAULT_SCHEMA="ods"): >>> print(LineageRunner("select * from test").source_tables) [Table: ods.test] >>> with SQLLineageConfig(DEFAULT_SCHEMA="dwd"): >>> print(LineageRunner("select * from test").source_tables) [Table: dwd.test] ``` -------------------------------- ### Combine Lineage from Multiple SQL Statements (Bash) Source: https://github.com/reata/sqllineage/blob/master/docs/first_steps/advanced_usage.rst Demonstrates how to use the `sqllineage` command-line tool to analyze the lineage of multiple SQL statements provided as a single string. It shows how intermediate tables are identified in the combined lineage result. ```bash $ sqllineage -e "insert into db1.table1 select * from db2.table2; insert into db3.table3 select * from db1.table1;" ``` -------------------------------- ### Analyzing SQL Lineage with SparkSQL Dialect using sqllineage CLI Source: https://github.com/reata/sqllineage/blob/master/README.md Illustrates how sqllineage correctly analyzes the SQL statement using the SparkSQL dialect, which supports INSERT OVERWRITE and does not treat 'map' as a reserved keyword. ```Shell sqllineage -e "INSERT OVERWRITE TABLE map SELECT * FROM foo" --dialect=sparksql ``` -------------------------------- ### Using SQLAlchemyMetaDataProvider with SQLite in Python Source: https://github.com/reata/sqllineage/blob/master/docs/gear_up/metadata.rst Illustrates initializing SQLAlchemyMetaDataProvider with a database connection URL (SQLite in this case). SQLLineage will query the database for metadata as needed during lineage analysis. ```python >>> from sqllineage.core.metadata.sqlalchemy import SQLAlchemyMetaDataProvider >>> from sqllineage.runner import LineageRunner >>> sql1 = "insert into main.foo select * from main.bar" >>> url = "sqlite:///db.db" >>> provider = SQLAlchemyMetaDataProvider(url) >>> LineageRunner(sql1, metadata_provider=provider).print_column_lineage() ``` -------------------------------- ### Analyzing SQL Lineage with ANSI Dialect using sqllineage CLI Source: https://github.com/reata/sqllineage/blob/master/README.md Demonstrates how sqllineage analyzes a SQL statement using the default ANSI dialect, showing a syntax error for non-standard syntax like INSERT OVERWRITE. ```Shell sqllineage -e "INSERT OVERWRITE TABLE map SELECT * FROM foo" ``` -------------------------------- ### Using SQLAlchemyMetaDataProvider with BigQuery and engine_kwargs in Python Source: https://github.com/reata/sqllineage/blob/master/docs/gear_up/metadata.rst Demonstrates how to pass extra connection arguments, such as location for BigQuery, to the underlying SQLAlchemy engine creation via the `engine_kwargs` parameter of SQLAlchemyMetaDataProvider. ```python >>> provider = SQLAlchemyMetaDataProvider('bigquery://project', engine_kwargs={"location": "asia-northeast1"}) ``` -------------------------------- ### Create SQLite Table via Bash Source: https://github.com/reata/sqllineage/blob/master/docs/first_steps/advanced_usage.rst Executes a SQL command using the `sqlite3` command-line tool to create a table named 'quux' in the 'db.db' file if it does not already exist. The table includes columns 'quux_id', 'col5', and 'col6'. ```bash sqlite3 db.db 'CREATE TABLE IF NOT EXISTS quux (quux_id int, col5 int, col6 int)'; ``` -------------------------------- ### Analyze Column-Level Lineage from File (Bash) Source: https://github.com/reata/sqllineage/blob/master/docs/first_steps/advanced_usage.rst Shows how to use the `sqllineage` command with the `-f` option to read SQL from a file (`test.sql`) and the `-l column` option to perform and display column-level lineage analysis. ```bash $ sqllineage -f test.sql -l column ``` -------------------------------- ### Allowing All User Agents - robots.txt Source: https://github.com/reata/sqllineage/blob/master/sqllineagejs/public/robots.txt This snippet defines rules for web crawlers. The `User-agent: *` directive applies the following rules to all bots. The `Disallow:` directive with an empty value means that no paths are disallowed, effectively allowing crawlers to access all content on the site. ```robots.txt User-agent: * Disallow: ``` -------------------------------- ### Run sqllineage using Python API with dialect Source: https://github.com/reata/sqllineage/blob/master/docs/behind_the_scene/dialect-awareness_lineage_design.rst Shows how to use the `LineageRunner` class from the sqllineage Python API to analyze SQL lineage. It initializes the runner with a SQL string and specifies the SQL dialect (`ansi`) via the `dialect` parameter, enabling dialect-aware analysis programmatically. ```python from sqllineage.runner import LineageRunner sql = "select * from dual" result = LineageRunner(sql, dialect="ansi") ``` -------------------------------- ### Run Tox for Multiple Python Versions - Bash Source: https://github.com/reata/sqllineage/blob/master/CONTRIBUTING.md Executes the tox command without specifying a Python version. This triggers tests against all Python versions defined in the project's pyproject.toml file, useful for multi-version compatibility testing. ```Bash tox ``` -------------------------------- ### Analyze SQL with SparkSQL Dialect (Bash) Source: https://github.com/reata/sqllineage/blob/master/docs/first_steps/advanced_usage.rst Demonstrates using the `sqllineage` command with the `--dialect=sparksql` option to correctly analyze a SQL statement (`INSERT OVERWRITE`) that is valid in SparkSQL, showing the source and target tables. ```bash $ sqllineage -e "insert overwrite table map select * from foo" --dialect=sparksql ``` -------------------------------- ### Multi-Statement SQL Query for Column Lineage Analysis Source: https://github.com/reata/sqllineage/blob/master/README.md Provides a complex SQL script with multiple INSERT statements, joins, and a subquery, designed as input for sqllineage to demonstrate column-level lineage tracking. ```SQL INSERT INTO foo SELECT a.col1, b.col1 AS col2, c.col3_sum AS col3, col4, d.* FROM bar a JOIN baz b ON a.id = b.bar_id LEFT JOIN (SELECT bar_id, sum(col3) AS col3_sum FROM qux GROUP BY bar_id) c ON a.id = sq.bar_id CROSS JOIN quux d; INSERT INTO corge SELECT a.col1, a.col2 + b.col2 AS col2 FROM foo a LEFT JOIN grault b ON a.col1 = b.col1; ``` -------------------------------- ### Parse Multiple SQL Statements with SQLLineage - Bash Source: https://github.com/reata/sqllineage/blob/master/README.md This command demonstrates parsing multiple SQL statements separated by semicolons using the `-e` option. SQLLineage analyzes the combined lineage and identifies source, target, and intermediate tables across the statements. ```Bash sqllineage -e "insert into db1.table1 select * from db2.table2; insert into db3.table3 select * from db1.table1;" ``` -------------------------------- ### Parse SQL File with SQLLineage - Bash Source: https://github.com/reata/sqllineage/blob/master/README.md This command uses the SQLLineage command-line interface (`sqllineage`) with the `-f` option to parse SQL statements contained within a specified file (`foo.sql`). It outputs the identified source and target tables from the file's content. ```Bash sqllineage -f foo.sql ``` -------------------------------- ### Run SQLLineage Column Lineage with SQLite Metadata Source: https://github.com/reata/sqllineage/blob/master/docs/first_steps/advanced_usage.rst Executes the `sqllineage` command-line tool on the 'test.sql' file to perform column-level lineage analysis (`-l column`). It uses a SQLite database specified by the SQLAlchemy URL (`--sqlalchemy_url=sqlite:///db.db`) for metadata lookup, which helps resolve unqualified table names. The `SQLLINEAGE_DEFAULT_SCHEMA=main` environment variable is set to specify the default schema name for unqualified tables in SQLite. The output shows the resolved column lineage paths. ```bash $ SQLLINEAGE_DEFAULT_SCHEMA=main sqllineage -f test.sql -l column --sqlalchemy_url=sqlite:///db.db main.corge.col1 <- main.foo.col1 <- main.bar.col1 main.corge.col2 <- main.foo.col2 <- main.bar.col1 main.corge.col2 <- main.grault.col2 main.foo.col3 <- c.col3_sum <- main.qux.col3 main.foo.col4 <- main.baz.col4 main.foo.col5 <- main.quux.col5 main.foo.col6 <- main.quux.col6 ``` -------------------------------- ### Run Tox for Specific Python Version - Bash Source: https://github.com/reata/sqllineage/blob/master/CONTRIBUTING.md Executes the tox command to run CI checks specifically for Python version 3.9. This is useful for testing against a particular environment locally. ```Bash tox -e py39 ``` -------------------------------- ### Parse Multiple SQL Statements Verbose Output - Bash Source: https://github.com/reata/sqllineage/blob/master/README.md This command uses the `-v` (verbose) option along with `-e` to parse multiple SQL statements. The verbose output shows the lineage analysis results for each individual statement before providing the overall summary of source, target, and intermediate tables. ```Bash sqllineage -v -e "insert into db1.table1 select * from db2.table2; insert into db3.table3 select * from db1.table1;" ``` -------------------------------- ### Parse SQL Query String with SQLLineage - Bash Source: https://github.com/reata/sqllineage/blob/master/README.md This command uses the SQLLineage command-line interface (`sqllineage`) with the `-e` option to parse a SQL query provided as a quoted string. It outputs the identified source and target tables. ```Bash sqllineage -e "insert into db1.table1 select * from db2.table2" ``` -------------------------------- ### Handling SELECT * in SQL Lineage - SQL Source: https://github.com/reata/sqllineage/blob/master/docs/behind_the_scene/column-level_lineage_design.rst This SQL snippet illustrates the use of `SELECT *` in an `INSERT OVERWRITE` statement, which presents a challenge for column-level lineage tracking as the specific columns being selected from `tab2` are not explicitly named. ```sql INSERT OVERWRITE tab1 SELECT * FROM tab2; ``` -------------------------------- ### Analyze SQL with Default ANSI Dialect (Bash) Source: https://github.com/reata/sqllineage/blob/master/docs/first_steps/advanced_usage.rst Illustrates using the `sqllineage` command with the default `ansi` dialect to analyze a SQL statement (`INSERT OVERWRITE`) that is not standard ANSI SQL, resulting in a syntax error. ```bash $ sqllineage -e "insert overwrite table map select * from foo" ``` -------------------------------- ### Handling Ambiguous Columns in JOIN - SQL Source: https://github.com/reata/sqllineage/blob/master/docs/behind_the_scene/column-level_lineage_design.rst This SQL snippet demonstrates a join operation where a selected column (`col2`) lacks a table prefix. This ambiguity makes it difficult to determine whether `col2` originates from `tab2` or `tab3` for column-level lineage tracking. ```sql INSERT OVERWRITE tab1 SELECT col2 FROM tab2 JOIN tab3 ON tab2.col1 = tab3.col1 ``` === COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.