### Set Up PyHive Local Testing Environment Source: https://github.com/dropbox/pyhive/wiki/Home Detailed steps to configure an Ubuntu VM for PyHive development and testing. This includes setting timezone, adding Java PPA, installing essential tools like git, openssh-server, python-pip, virtualenv, cloning the PyHive repository, installing dependencies, and running tests. ```bash echo Etc/UTC | sudo tee /etc/timezone sudo dpkg-reconfigure --frontend noninteractive tzdata sudo add-apt-repository -y ppa:webupd8team/java sudo apt-get install -y git openssh-server python-pip sudo pip install virtualenv wheel virtualenv --no-site-packages ~/env source ~/env/bin/activate git clone https://github.com/dropbox/PyHive.git cd PyHive CDH=cdh5 CDH_VERSION=5 PRESTO=RELEASE SQLALCHEMY=sqlalchemy scripts/travis-install.sh py.test -v ``` -------------------------------- ### Install LDAP Server for HiveServer2 Authentication Source: https://github.com/dropbox/pyhive/wiki/Home Commands to install OpenLDAP server, utilities, and phpLDAPadmin on an Ubuntu system. This setup is typically a prerequisite for configuring HiveServer2 to use LDAP for user authentication. ```bash sudo apt-get install sldap ldap-utils phpldapadmin ``` -------------------------------- ### Run PyHive Test Suite Source: https://github.com/dropbox/pyhive/blob/master/README.rst These commands provide a sequence for setting up a virtual environment, installing PyHive and its development dependencies, and then executing the test suite. It's critical that a Hive/Presto environment is accessible for these tests to run successfully. ```Shell ./scripts/make_test_tables.sh virtualenv --no-site-packages env source env/bin/activate pip install -e . pip install -r dev_requirements.txt py.test ``` -------------------------------- ### Start Docker Compose Services Source: https://github.com/dropbox/pyhive/blob/master/README.rst This shell command starts all services defined in the `docker-compose.yaml` file. The `-d` flag runs the containers in detached mode, allowing them to run in the background. ```Shell docker-compose up -d ``` -------------------------------- ### Release PyHive Distribution Source: https://github.com/dropbox/pyhive/wiki/Home Commands to build a source distribution, sign it with GPG, and upload it to PyPI using Twine. This ensures the package is properly prepared and secured for public release. ```bash python setup.py sdist gpg --detach-sign --armor *.tar.gz twine upload PyHive* ``` -------------------------------- ### Connect and Query with PyHive DB-API (Synchronous) Source: https://github.com/dropbox/pyhive/blob/master/README.rst Demonstrates basic synchronous connection and query execution using PyHive's DB-API for Presto, Hive, or Trino. It shows how to establish a connection, execute a simple query, and fetch results. ```Python from pyhive import presto # or import hive or import trino cursor = presto.connect('localhost').cursor() # or use hive.connect or use trino.connect cursor.execute('SELECT * FROM my_awesome_data LIMIT 10') print cursor.fetchone() print cursor.fetchall() ``` -------------------------------- ### Add Trino Service to Docker Compose Source: https://github.com/dropbox/pyhive/blob/master/README.rst This YAML snippet demonstrates how to add a Trino service definition to an existing `docker-compose.yaml` file. It configures the Trino image, maps the necessary port, and mounts a volume for Trino configuration, enabling Trino to run alongside other services in a Dockerized test environment. ```YAML trino: image: trinodb/trino:351 ports: - "18080:18080" volumes: - ./trino:/etc/trino ``` -------------------------------- ### Connect and Query with PyHive SQLAlchemy Source: https://github.com/dropbox/pyhive/blob/master/README.rst Illustrates how to integrate PyHive with SQLAlchemy for connecting to Presto, Trino, and Hive databases. It covers creating engine connections, autoloading tables, and executing queries for both SQLAlchemy versions prior to 2.0 and 2.0+ syntax, including secure connections with HTTPS and LDAP. ```Python from sqlalchemy import * from sqlalchemy.engine import create_engine from sqlalchemy.schema import * # Presto engine = create_engine('presto://localhost:8080/hive/default') # Trino engine = create_engine('trino+pyhive://localhost:8080/hive/default') # Hive engine = create_engine('hive://localhost:10000/default') # SQLAlchemy < 2.0 logs = Table('my_awesome_data', MetaData(bind=engine), autoload=True) print select([func.count('*')], from_obj=logs).scalar() # Hive + HTTPS + LDAP or basic Auth engine = create_engine('hive+https://username:password@localhost:10000/') logs = Table('my_awesome_data', MetaData(bind=engine), autoload=True) print select([func.count('*')], from_obj=logs).scalar() # SQLAlchemy >= 2.0 metadata_obj = MetaData() books = Table("books", metadata_obj, Column("id", Integer), Column("title", String), Column("primary_author", String)) metadata_obj.create_all(engine) inspector = inspect(engine) inspector.get_columns('books') with engine.connect() as con: data = [{ "id": 1, "title": "The Hobbit", "primary_author": "Tolkien" }, { "id": 2, "title": "The Silmarillion", "primary_author": "Tolkien" }] con.execute(books.insert(), data[0]) result = con.execute(text("select * from books")) print(result.fetchall()) ``` -------------------------------- ### Configure Session Properties for PyHive Connections Source: https://github.com/dropbox/pyhive/blob/master/README.rst Demonstrates how to pass session-specific configurations to PyHive connections for both DB-API and SQLAlchemy interfaces. This includes setting properties like 'hive.exec.reducers.max' or 'query_max_run_time', and handling authentication methods like LDAP during connection establishment. ```Python # DB-API hive.connect('localhost', configuration={'hive.exec.reducers.max': '123'}) presto.connect('localhost', session_props={'query_max_run_time': '1234m'}) trino.connect('localhost', session_props={'query_max_run_time': '1234m'}) # SQLAlchemy create_engine( 'presto://user@host:443/hive', connect_args={'protocol': 'https', 'session_props': {'query_max_run_time': '1234m'}} ) create_engine( 'trino+pyhive://user@host:443/hive', connect_args={'protocol': 'https', 'session_props': {'query_max_run_time': '1234m'}} ) create_engine( 'hive://user@host:10000/database', connect_args={'configuration': {'hive.exec.reducers.max': '123'}} ) # SQLAlchemy with LDAP create_engine( 'hive://user:password@host:10000/database', connect_args={'auth': 'LDAP'} ) ``` -------------------------------- ### Restart HiveServer2 Service Source: https://github.com/dropbox/pyhive/wiki/Home Command to restart the HiveServer2 service. This step is necessary after making any configuration changes, such as enabling LDAP authentication, for the changes to take effect. ```bash sudo service hive-server2 restart ``` -------------------------------- ### Test PyHive LDAP Authentication with Python Source: https://github.com/dropbox/pyhive/wiki/Home Python code demonstrating how to connect to HiveServer2 using PyHive with LDAP authentication. It includes a successful connection attempt with valid credentials and a deliberate failed connection to verify error handling for invalid credentials, specifically checking for `TTransportException`. ```python from pyhive.hive import * from thrift.transport.TTransport import * c = connect('localhost', username='cn=admin,dc=test,dc=com', password='a') cur = c.cursor() cur.execute('show tables') print(cur.fetchall()) # should fail try: connect('localhost', username='cn=admin,dc=test,dc=com', password='ab') assert False except TTransportException as e: assert 'Error validating the login' in e.message print(e) ``` -------------------------------- ### Update TCLIService Module Source: https://github.com/dropbox/pyhive/blob/master/README.rst This command executes the `generate.py` script to update the autogenerated TCLIService module. It takes a URL to a `TCLIService.thrift` file as an argument. If the URL is omitted, the script defaults to downloading the version for Hive 2.3. ```Shell python generate.py ``` -------------------------------- ### PyHive Core Runtime Python Dependencies Source: https://github.com/dropbox/pyhive/blob/master/dev_requirements.txt Lists the essential Python packages required for PyHive's runtime functionality. These dependencies specify minimum versions, allowing for greater flexibility. Notably, `thrift_sasl` is sourced directly from a GitHub repository to incorporate specific Python 3 SASL patches. ```Python requests>=1.0.0 requests_kerberos>=0.12.0 sasl>=0.2.1 pure-sasl>=0.6.2 kerberos>=1.3.0 thrift>=0.10.0 git+https://github.com/cloudera/thrift_sasl ``` -------------------------------- ### Configure HiveServer2 for LDAP Authentication Source: https://github.com/dropbox/pyhive/wiki/Home XML configuration snippet to enable LDAP authentication in HiveServer2. This involves setting the authentication type to LDAP and specifying the LDAP server URL within the `hive-site.xml` configuration file. ```xml hive.server2.authentication LDAP hive.server2.authentication.ldap.url ldap://localhost:389 ``` -------------------------------- ### Execute Asynchronous Queries with PyHive DB-API Source: https://github.com/dropbox/pyhive/blob/master/README.rst Shows how to perform asynchronous queries using PyHive's DB-API, specifically with Hive. This includes polling for operation status, fetching logs during execution, and the option to cancel ongoing queries. It also addresses the 'async' keyword conflict in Python 3.7+. ```Python from pyhive import hive from TCLIService.ttypes import TOperationState cursor = hive.connect('localhost').cursor() cursor.execute('SELECT * FROM my_awesome_data LIMIT 10', async=True) status = cursor.poll().operationState while status in (TOperationState.INITIALIZED_STATE, TOperationState.RUNNING_STATE): logs = cursor.fetch_logs() for message in logs: print message # If needed, an asynchronous query can be cancelled at any time with: # cursor.cancel() status = cursor.poll().operationState print cursor.fetchall() ``` ```Python cursor.execute('SELECT * FROM my_awesome_data LIMIT 10', async_=True) ``` -------------------------------- ### PyHive Test-Only Python Dependencies Source: https://github.com/dropbox/pyhive/blob/master/dev_requirements.txt Specifies Python packages exclusively used for testing the PyHive project. All packages are pinned to exact versions to ensure a stable and reproducible test environment, minimizing the risk of breaking changes from dependency updates. ```Python flake8==3.4.1 mock==2.0.0 pycodestyle==2.3.1 pytest==3.2.1 pytest-cov==2.5.1 pytest-flake8==0.8.1 pytest-random==0.2 pytest-timeout==1.2.0 ``` === COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.