### RelStorage Configuration Example
Source: https://relstorage.readthedocs.io/en/latest/_sources/configure-application.rst.txt
Example configuration for setting up RelStorage with a MySQL database. Ensure the correct versions of RelStorage and database drivers are installed.
```ini
rel-storage =
type mysql
db plone
user plone
passwd PASSWORD
```
--------------------------------
### Install RelStorage with Database Adapters
Source: https://relstorage.readthedocs.io/en/latest/install.html
Install RelStorage with specific database adapter extras. This is the easiest way to get the recommended and tested database adapter for your platform and database. Ensure you have the necessary database client software and development libraries installed.
```bash
pip install "RelStorage[mysql]"
```
```bash
pip install "RelStorage[postgresql]"
```
```bash
pip install "RelStorage[oracle]"
```
```bash
pip install "RelStorage[sqlite3]"
```
--------------------------------
### Start SYS Session in Oracle
Source: https://relstorage.readthedocs.io/en/latest/developing/running.html
Initiates a SQL*Plus session with SYS privileges for Oracle database setup. Requires 'oracle' user privileges.
```bash
$ su - oracle
$ sqlplus / as sysdba
```
--------------------------------
### PostgreSQL and MySQL URI Examples
Source: https://relstorage.readthedocs.io/en/latest/zodburi.html
Examples of postgres and mysql URIs, including user, password, host, port, database name, and query string arguments like connection_cache_size.
```text
postgres://someuser:somepass@somehost:5432/somedb?connection_cache_size=20000
mysql://someuser:somepass@somehost:5432/somedb?connection_cache_size=20000
```
--------------------------------
### Install RelStorage with Oracle adapter
Source: https://relstorage.readthedocs.io/en/latest/_sources/install.rst.txt
Install RelStorage with the Oracle database adapter using pip extras. Ensure database client software and development libraries are installed.
```bash
pip install "RelStorage[oracle]"
```
--------------------------------
### PostgreSQL Test Database Setup (psql)
Source: https://relstorage.readthedocs.io/en/latest/_sources/developing/running.rst.txt
Shell commands to set up PostgreSQL databases and users for testing.
```shell
.. literalinclude:: ../../.travis/postgres.sh
:language: shell
```
--------------------------------
### Install RelStorage with SQLite3 adapter
Source: https://relstorage.readthedocs.io/en/latest/_sources/install.rst.txt
Install RelStorage with the SQLite3 database adapter using pip extras. Ensure database client software and development libraries are installed.
```bash
pip install "RelStorage[sqlite3]"
```
--------------------------------
### Install RelStorage with Test and MySQL Dependencies
Source: https://relstorage.readthedocs.io/en/latest/_sources/developing/running.rst.txt
Install RelStorage in editable mode with necessary test and MySQL driver dependencies.
```bash
pip install -e ".[test,mysql]"
```
--------------------------------
### Install RelStorage with MySQL adapter
Source: https://relstorage.readthedocs.io/en/latest/_sources/install.rst.txt
Install RelStorage with the MySQL database adapter using pip extras. Ensure database client software and development libraries are installed.
```bash
pip install "RelStorage[mysql]"
```
--------------------------------
### PostgreSQL URI Example
Source: https://relstorage.readthedocs.io/en/latest/_sources/zodburi.rst.txt
Example of a PostgreSQL URI with user, password, host, port, database name, and connection cache size.
```sql
postgres://someuser:somepass@somehost:5432/somedb?connection_cache_size=20000
```
--------------------------------
### Example Zlib Storage Configuration
Source: https://relstorage.readthedocs.io/en/latest/zodbconvert.html
This configuration shows a zlib storage wrapper around a file storage.
```ini
path /zope/var/Data.fs
```
--------------------------------
### MySQL URI Example
Source: https://relstorage.readthedocs.io/en/latest/_sources/zodburi.rst.txt
Example of a MySQL URI with user, password, host, port, database name, and connection cache size.
```sql
mysql://someuser:somepass@somehost:5432/somedb?connection_cache_size=20000
```
--------------------------------
### SQLite URI Example
Source: https://relstorage.readthedocs.io/en/latest/_sources/zodburi.rst.txt
Example of a SQLite URI, which only requires the path to the data directory.
```sql
sqlite://?data_dir=/the/path/to/database
```
--------------------------------
### MySQL Test Setup Script
Source: https://relstorage.readthedocs.io/en/latest/_sources/developing/running.rst.txt
Shell commands to set up MySQL databases and users for testing.
```shell
.. literalinclude:: ../../.travis/mysql.sh
:language: shell
```
--------------------------------
### Install RelStorage using pip
Source: https://relstorage.readthedocs.io/en/latest/_sources/install.rst.txt
Install the core RelStorage package using pip. This command installs the latest compatible version.
```bash
pip install RelStorage
```
--------------------------------
### SQLite URI Example
Source: https://relstorage.readthedocs.io/en/latest/zodburi.html
Example of a SQLite URI, specifying the directory for database files using the data_dir query argument.
```text
sqlite://?data_dir=/the/path/to/database
```
--------------------------------
### Oracle URI Example
Source: https://relstorage.readthedocs.io/en/latest/_sources/zodburi.rst.txt
Example of an Oracle URI, which only requires the DSN.
```sql
oracle://?dsn="HERE GOES THE DSN"
```
--------------------------------
### Install RelStorage with PostgreSQL adapter
Source: https://relstorage.readthedocs.io/en/latest/_sources/install.rst.txt
Install RelStorage with the PostgreSQL database adapter using pip extras. This attempts to install from source and may require database client software and development libraries.
```bash
pip install "RelStorage[postgresql]"
```
--------------------------------
### Example zodbpack Configuration File
Source: https://relstorage.readthedocs.io/en/latest/zodbpack.html
An example configuration file in ZConfig format for the zodbpack script, specifying a RelStorage with MySQL backend and garbage collection enabled.
```zconfig
pack-gc true
db zodb
```
--------------------------------
### PostgreSQL URI with Query String Arguments
Source: https://relstorage.readthedocs.io/en/latest/zodburi.html
An example demonstrating a PostgreSQL URI that includes connection details and query string arguments for cache size.
```text
postgres://someuser:somepass@somehost:5432/somedb?connection_cache_size=20000
```
--------------------------------
### Install Oracle Package
Source: https://relstorage.readthedocs.io/en/latest/_modules/relstorage/adapters/oracle/schema.html
Installs or updates the 'relstorage_op' package. It checks the current version and installs the appropriate package code (history-preserving or history-free).
```python
def install_package(self, cursor):
"""Install the package containing stored procedures"""
if self.keep_history:
code = oracle_history_preserving_package
else:
code = oracle_history_free_package
for stmt in code.split('\n/\n'):
if stmt.strip():
cursor.execute(stmt)
```
--------------------------------
### Schema Installer Interface: Prepare Schema
Source: https://relstorage.readthedocs.io/en/latest/_modules/relstorage/adapters/interfaces.html
Ensures the database schema exists and is up-to-date. It creates the schema if it doesn't exist, performs necessary migration steps, and then calls `verify`.
```python
def prepare():
"""
Create the database schema if it does not already exist.
Perform any migration steps needed, and call :meth:`verify`
before returning.
"""
```
--------------------------------
### Configure Oracle with RelStorage
Source: https://relstorage.readthedocs.io/en/latest/_sources/configure-application.rst.txt
Configure RelStorage for Oracle (10g XE example) using ZConfig.
```xml
user username
password pass
dsn XE
```
--------------------------------
### Schema Installer Interface: Verify Schema
Source: https://relstorage.readthedocs.io/en/latest/_modules/relstorage/adapters/interfaces.html
Checks if the currently installed database schema is compatible with this RelStorage configuration. Raises an exception if there's a mismatch, such as a history-preserving schema with a history-free configuration.
```python
def verify():
"""
Ensure that the schema that's installed can be used by this
RelStorage.
If it cannot, for example it's history-preserving and we were configured
to be history-free, raise an exception.
"""
```
--------------------------------
### Schema Installer Interface: Zap All Data
Source: https://relstorage.readthedocs.io/en/latest/_modules/relstorage/adapters/interfaces.html
Clears all data from the database, leaving the schema intact.
```python
def zap_all():
"""Clear all data out of the database."""
```
--------------------------------
### PostgreSQL Adapter Initialization
Source: https://relstorage.readthedocs.io/en/latest/_modules/relstorage/adapters/postgresql/adapter.html
Initializes the PostgreSQL adapter, setting up essential components like the connection manager, locker, schema installer, mover, and object allocator. This is the primary entry point for configuring the PostgreSQL backend.
```python
@implementer(IRelStorageAdapter)
class PostgreSQLAdapter(AbstractAdapter):
"""PostgreSQL adapter for RelStorage."""
# pylint:disable=too-many-instance-attributes
driver_options = drivers
def __init__(self, dsn='', options=None, oidallocator=None,
locker=None,
mover=None,
connmanager=None
):
# options is a relstorage.options.Options or None
self._dsn = dsn
self.oidallocator = oidallocator
self.locker = locker
self.mover = mover
self.connmanager = connmanager
super().__init__(options)
def _create(self):
driver = self.driver
options = self.options
dsn = self._dsn
self.version_detector = PostgreSQLVersionDetector()
if self.connmanager is None:
self.connmanager = Psycopg2ConnectionManager(
driver,
dsn=dsn,
options=options,
)
self.runner = ScriptRunner()
if self.locker is None:
self.locker = PostgreSQLLocker(
options,
driver,
PostgreSQLRowBatcher,
)
self.schema = PostgreSQLSchemaInstaller(
options=options,
connmanager=self.connmanager,
runner=self.runner,
locker=self.locker,
)
if self.mover is None:
self.mover = PostgreSQLObjectMover(
driver,
options=options,
runner=self.runner,
version_detector=self.version_detector,
batcher_factory=PostgreSQLRowBatcher,
)
if self.oidallocator is None:
self.oidallocator = PostgreSQLOIDAllocator()
self.poller = PGPoller(
self.driver,
keep_history=self.keep_history,
runner=self.runner,
revert_when_stale=options.revert_when_stale,
transactions_may_go_backwards=(
self.connmanager.replica_selector is not None
or self.connmanager.ro_replica_selector is not None
)
)
self.txncontrol = PostgreSQLTransactionControl(
connmanager=self.connmanager,
poller=self.poller,
keep_history=self.keep_history,
Binary=driver.Binary,
)
```
--------------------------------
### SQLite Configuration with PRAGMAs
Source: https://relstorage.readthedocs.io/en/latest/_sources/sqlite3/options.rst.txt
Example of configuring SQLite with specific PRAGMAs for maximum performance, potentially at the expense of safety. This includes settings like synchronous, fullfsync, and foreign_keys.
```ini
data-dir /dev/shm
synchronous off
checkpoint_fullfsysnc off
defer_foreign_keys on
fullfsync off
ignore_check_constraints on
foreign_keys off
```
--------------------------------
### zodbconvert Configuration File Example
Source: https://relstorage.readthedocs.io/en/latest/_sources/zodbconvert.rst.txt
This configuration defines a FileStorage as the source and a MySQL-based RelStorage as the destination for data conversion using zodbconvert.
```ini
path /zope/var/Data.fs
db zodb
```
--------------------------------
### Example ZConfig Configuration for RelStorage
Source: https://relstorage.readthedocs.io/en/latest/_sources/zodbpack.rst.txt
This configuration specifies a RelStorage with MySQL backend and enables garbage collection during packing.
```zconfig
pack-gc true
db zodb
```
--------------------------------
### Initialize Temporary Tables on Store Open
Source: https://relstorage.readthedocs.io/en/latest/_modules/relstorage/adapters/sqlite/mover.html
The `on_store_opened` method initializes temporary tables required for blob chunk storage and general temporary data. It uses `Sqlite3ScriptRunner` to execute the setup script and ensures the connection is not in a transaction before execution.
```python
@metricmethod_sampled
def on_store_opened(self, cursor, restart=False):
"""Create the temporary table for storing objects"""
# Don't use cursor.executescript, it commits. Remember our store
# connection is usually meant to be in auto-commit mode and shouldn't take
# exclusive locks until tpc_vote time. Prior to Python 3.6, if we were
# in a transaction this would commit it. but we shouldn't be in a transaction.
if not restart:
assert not cursor.connection.in_transaction, cursor
Sqlite3ScriptRunner().run_script(cursor, self.__on_store_opened_script)
assert not cursor.connection.in_transaction
cursor.connection.register_before_commit_cleanup(self._before_commit)
super().on_store_opened(cursor, restart)
```
--------------------------------
### Opening ZODB Database from Configuration String
Source: https://relstorage.readthedocs.io/en/latest/configure-application.html
Open a ZODB database directly from a configuration string using ZODB.config.databaseFromString(). This example uses MySQL configuration.
```python
conf = """
%%import relstorage
db %s
user %s
passwd %s
""" % (dbname, user, passwd)
db = ZODB.config.databaseFromString(conf)
conn = db.open()
...
```
--------------------------------
### Instantiate RelStorage with Options
Source: https://relstorage.readthedocs.io/en/latest/relstorage.options.html
Demonstrates how to instantiate RelStorage with packing options. These parameters can be provided as keyword options in the RelStorage constructor.
```python
storage = RelStorage(adapter, pack_gc=True, pack_prepack_only=True)
```
--------------------------------
### Install PostgreSQL Procedures
Source: https://relstorage.readthedocs.io/en/latest/_modules/relstorage/adapters/postgresql/schema.html
Installs necessary stored procedures if they are not already installed. It first checks for existing procedures and then proceeds with installation if required. It also disables function body checking during installation to handle interdependencies.
```python
def create_procedures(self, cursor):
if not self.__all_procedures_installed(cursor):
self.__install_procedures(cursor)
if not self.__all_procedures_installed(cursor):
raise AssertionError(
"Could not get version information after "
"installing the stored procedures.")
```
```python
def __install_procedures(self, cursor):
"""Install the stored procedures"""
self.__install_languages(cursor)
# PostgreSQL procedures in the SQL language
# do lots of validation at compile time; in particular,
# they check that the functions they use in SELECT statements
# actually exist. When we have procedures that call each other,
# that means there's an order they have to be created in.
# Rather than try to figure out what that order is, or encode it
# in names somehow, we'll disable that validation for the duration
# of this transaction (just like pg_dump.)
cursor.execute('SET LOCAL check_function_bodies = off')
```
--------------------------------
### Create MySQL User and Database
Source: https://relstorage.readthedocs.io/en/latest/_sources/mysql/setup.rst.txt
Sample SQL statements to create a dedicated user, database, and grant necessary privileges for RelStorage.
```sql
CREATE USER 'zodbuser'@'localhost' IDENTIFIED BY 'mypassword';
CREATE DATABASE zodb;
GRANT ALL ON zodb.* TO 'zodbuser'@'localhost';
FLUSH PRIVILEGES;
```
--------------------------------
### Check All Procedures Installed
Source: https://relstorage.readthedocs.io/en/latest/_modules/relstorage/adapters/postgresql/schema.html
Verifies if all required stored procedures are installed and up-to-date in the database. It compares the expected set of procedures with those currently installed.
```python
def __all_procedures_installed(self, cursor):
"""
Check whether all required stored procedures are installed.
Returns True only if all required procedures are installed and
up to date.
"""
expected = self.procedures
installed = self.list_procedures(cursor)
# If the database evolves over tiem, there could be
# extra procs still there that we don't care about.
installed = {k: v for k, v in installed.items() if k in expected}
if installed != expected:
logger.info(
"Procedures incorrect, will reinstall. "
"Expected: %s."
"Actual: %s",
expected, installed
)
return False
return True
```
--------------------------------
### Show cache_simul.py Help
Source: https://relstorage.readthedocs.io/en/latest/_sources/cache-tracing.rst.txt
Displays the command-line syntax for the cache_simul.py tool. This tool simulates ZEO client cache behavior based on trace file events.
```bash
python -m ZEO.scripts.cache_simul --help
```
--------------------------------
### Set up MySQL Test Environment
Source: https://relstorage.readthedocs.io/en/latest/developing/running.html
Bash script to create a MySQL user, databases, and grant necessary privileges for RelStorage testing. It handles dynamic database names and connection details.
```bash
#!/bin/bash
# To be able to successfully test against a MySQL on a different host,
# it's best to set up a proxy: socat tcp-listen:3306,reuseaddr,fork tcp:192.168.99.100:3306
# This can arise when running mysql in docker on macOS, which doesn't
# correctly handle the `docker run` `-p` option to forward ports to the local host;
# specify the address of the virtual machine.
# docker run --publish 3306:3306 --rm --name mysqld -e MYSQL_ALLOW_EMPTY_PASSWORD=yes mysql:8.0
HOST="-h ${RS_DB_HOST-localhost}"
DBNAME=${RELSTORAGETEST_DBNAME:-relstoragetest}
DBNAME2=${DBNAME}2
DBNAME_HF=${DBNAME}_hf
DBNAME2_HF=${DBNAME}2_hf
PW=$RELSTORAGETEST_MY_PW
echo $DBNAME_hf
echo $DBNAME2_hf
mysql -uroot $PW $HOST -e "CREATE USER 'relstoragetest' IDENTIFIED BY 'relstoragetest';"
mysql -uroot $PW $HOST -e "CREATE DATABASE $DBNAME;"
mysql -uroot $PW $HOST -e "GRANT ALL ON $DBNAME.* TO 'relstoragetest';"
mysql -uroot $PW $HOST -e "CREATE DATABASE $DBNAME2;"
mysql -uroot $PW $HOST -e "GRANT ALL ON $DBNAME2.* TO 'relstoragetest';"
mysql -uroot $PW $HOST -e "CREATE DATABASE $DBNAME_HF;"
mysql -uroot $PW $HOST -e "GRANT ALL ON $DBNAME_HF.* TO 'relstoragetest';"
mysql -uroot $PW $HOST -e "CREATE DATABASE $DBNAME2_HF;"
mysql -uroot $PW $HOST -e "GRANT ALL ON $DBNAME2_HF.* TO 'relstoragetest';"
mysql -uroot $PW $HOST -e "GRANT SELECT ON performance_schema.* TO 'relstoragetest'"
mysql -uroot $PW $HOST -e "GRANT SELECT ON sys.* TO 'relstoragetest'"
mysql -uroot $PW $HOST -e "GRANT PROCESS ON *.* TO 'relstoragetest'"
mysql -uroot $PW $HOST -e "SELECT version()"
mysql -uroot $PW $HOST -e "FLUSH PRIVILEGES;"
```
--------------------------------
### Create RelStorage Options Instance
Source: https://relstorage.readthedocs.io/en/latest/relstorage.options.html
Shows how to create an Options instance and pass it to the RelStorage constructor. This is an alternative to passing keyword options directly.
```python
options = Options(pack_gc=True, pack_prepack_only=True)
storage = RelStorage(adapter, options=options)
```
--------------------------------
### Handling Object References in Pre-Pack
Source: https://relstorage.readthedocs.io/en/latest/_modules/relstorage/adapters/packundo.html
Shows how to retrieve references from an object's state and insert them into the 'object_ref' table. Includes error handling for unpickling issues.
```Python
if state:
try:
to_oids = get_references(state)
except:
logger.exception(
"pre_pack: can't unpickle "
"object %d in transaction %d; state length = %d",
from_oid, tid, len(state)
)
raise
for to_oid in to_oids:
row = (from_oid, tid, to_oid)
num_refs_found += 1
store_batcher.insert_into(
'object_ref (zoid, tid, to_zoid)',
object_ref_schema,
row,
row,
size=3
)
```
--------------------------------
### Oracle URI Example
Source: https://relstorage.readthedocs.io/en/latest/zodburi.html
Example of an Oracle URI, which primarily uses a Data Source Name (DSN) for connection information.
```text
oracle://?dsn="HERE GOES THE DSN"
```
--------------------------------
### Sqlite3Adapter Initialization
Source: https://relstorage.readthedocs.io/en/latest/_modules/relstorage/adapters/sqlite/adapter.html
Illustrates the initialization of the Sqlite3Adapter, setting up essential components for SQLite database operations. This includes defining the data directory, pragmas, and optional components like oidallocator, locker, mover, and connmanager.
```python
# -*- coding: utf-8 -*-
##############################################################################
#
# Copyright (c) 2019 Zope Foundation and Contributors.
# All Rights Reserved.
#
# This software is subject to the provisions of the Zope Public License,
# Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution.
# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
# FOR A PARTICULAR PURPOSE.
#
##############################################################################
"""sqlite3 adapter for RelStorage."""
from __future__ import absolute_import
from __future__ import division
from __future__ import print_function
import os.path
from zope.interface import implementer
from ..adapter import AbstractAdapter
from ..dbiter import HistoryFreeDatabaseIterator
from ..dbiter import HistoryPreservingDatabaseIterator
from ..interfaces import IRelStorageAdapter
from ..packundo import HistoryFreePackUndo
from ..packundo import HistoryPreservingPackUndo
from . import drivers
from .batch import Sqlite3RowBatcher
from .connmanager import Sqlite3ConnectionManager
from .locker import Sqlite3Locker
from .mover import Sqlite3ObjectMover
from .oidallocator import Sqlite3OIDAllocator
from .schema import Sqlite3SchemaInstaller
from .stats import Sqlite3Stats
from .txncontrol import Sqlite3TransactionControl
from .poller import Sqlite3Poller
from .scriptrunner import Sqlite3ScriptRunner
@implementer(IRelStorageAdapter)
class Sqlite3Adapter(AbstractAdapter): # pylint:disable=too-many-instance-attributes
"""
Adapter for sqlite.
"""
driver_options = drivers
WRITING_REQUIRES_EXCLUSIVE_LOCK = True
def __init__(self, data_dir, pragmas,
options=None, oidallocator=None,
locker=None,
mover=None,
connmanager=None):
self.data_dir = os.path.abspath(data_dir)
self.pragmas = pragmas
self.oidallocator = oidallocator
self.locker = locker
self.mover = mover
self.connmanager = connmanager
super().__init__(options)
def _create(self):
driver = self.driver
options = self.options
if self.connmanager is None:
self.connmanager = Sqlite3ConnectionManager(
driver,
path=os.path.join(self.data_dir, 'main.sqlite3'),
pragmas=self.pragmas,
options=options
)
if self.mover is None:
self.mover = Sqlite3ObjectMover(
driver,
options=options,
)
if self.locker is None:
self.locker = Sqlite3Locker(
options,
driver,
batcher_factory=Sqlite3RowBatcher)
if not self.oidallocator:
self.oidallocator = Sqlite3OIDAllocator(
os.path.join(self.data_dir, 'oids.sqlite3'),
# No switching during OID allocation. It holds an exclusive
# lock anyway.
driver=drivers.Sqlite3Driver()
)
self.runner = Sqlite3ScriptRunner()
self.poller = Sqlite3Poller(
self.driver,
keep_history=self.keep_history,
runner=self.runner,
revert_when_stale=options.revert_when_stale,
transactions_may_go_backwards=False
)
self.txncontrol = Sqlite3TransactionControl(
connmanager=self.connmanager,
poller=self.poller,
keep_history=self.keep_history,
Binary=driver.Binary,
)
self.schema = Sqlite3SchemaInstaller(
driver=driver,
oid_allocator=self.oidallocator,
connmanager=self.connmanager,
runner=self.runner,
keep_history=self.keep_history
)
self.stats = Sqlite3Stats(
self.connmanager,
self.keep_history
)
if self.keep_history:
self.packundo = HistoryPreservingPackUndo(
driver,
connmanager=self.connmanager,
runner=self.runner,
locker=self.locker,
options=options,
)
self.dbiter = HistoryPreservingDatabaseIterator(
driver,
)
else:
self.packundo = HistoryFreePackUndo(
driver,
connmanager=self.connmanager,
runner=self.runner,
locker=self.locker,
options=options,
)
self.dbiter = HistoryFreeDatabaseIterator(
driver,
)
```
--------------------------------
### Initialize PostgreSQL Adapter
Source: https://relstorage.readthedocs.io/en/latest/_modules/relstorage/adapters/postgresql/adapter.html
Initializes the PostgreSQL adapter, setting up packundo and dbiter based on whether history preservation is enabled. It also initializes statistics.
```python
if self.keep_history:
self.packundo = HistoryPreservingPackUndo(
driver,
connmanager=self.connmanager,
runner=self.runner,
locker=self.locker,
options=options,
)
self.dbiter = HistoryPreservingDatabaseIterator(
driver,
)
else:
self.packundo = HistoryFreePackUndo(
driver,
connmanager=self.connmanager,
runner=self.runner,
locker=self.locker,
options=options,
)
self.dbiter = HistoryFreeDatabaseIterator(
driver,
)
self.stats = PostgreSQLStats(
connmanager=self.connmanager,
keep_history=self.keep_history
)
```
--------------------------------
### Install PostgreSQL Triggers
Source: https://relstorage.readthedocs.io/en/latest/_modules/relstorage/adapters/postgresql/schema.html
Installs the 'blob_chunk_delete' trigger if it is not already present in the database. It first lists existing triggers to check for its presence.
```python
def create_triggers(self, cursor):
triggers = self.list_triggers(cursor)
__traceback_info__ = triggers, self.list_tables(cursor), self.get_database_name(cursor)
if 'blob_chunk_delete' not in triggers:
self.__install_triggers(cursor)
```
--------------------------------
### AbstractConnectionManager Initialization
Source: https://relstorage.readthedocs.io/en/latest/_modules/relstorage/adapters/connmanager.html
Initializes the AbstractConnectionManager with options and a driver, setting up replica selectors and ignored exceptions.
```python
def __init__(self, options, driver):
"""
:param options: A :class:`relstorage.options.Options`.
:param driver: A :class:`relstorage.adapters.interfaces.IDBDriver`,
which we use for its exceptions.
"""
self.keep_history = options.keep_history
self.driver = driver
self.options = options
self._ignored_exceptions = tuple(set(
driver.close_exceptions
+ driver.disconnected_exceptions
+ (ReplicaClosedException,)
))
if options.replica_conf:
self.replica_selector = ReplicaSelector(
options.replica_conf, options.replica_timeout)
if options.ro_replica_conf:
self.ro_replica_selector = ReplicaSelector(
options.ro_replica_conf, options.replica_timeout)
else:
self.ro_replica_selector = self.replica_selector
```
--------------------------------
### Install PL/pgSQL Language
Source: https://relstorage.readthedocs.io/en/latest/_modules/relstorage/adapters/postgresql/schema.html
Installs the 'plpgsql' language if it is not already available in the PostgreSQL database. This is a prerequisite for using stored procedures written in PL/pgSQL.
```python
def __install_languages(self, cursor):
if 'plpgsql' not in self.list_languages(cursor):
cursor.execute("CREATE LANGUAGE plpgsql")
```
--------------------------------
### Create New Adapter Instance
Source: https://relstorage.readthedocs.io/en/latest/_modules/relstorage/adapters/postgresql/adapter.html
Creates and returns a new instance of the adapter, copying configuration from the current instance.
```python
def new_instance(self):
inst = type(self)(
dsn=self._dsn,
options=self.options,
oidallocator=self.oidallocator.new_instance(),
locker=self.locker,
mover=self.mover,
connmanager=self.connmanager,
)
return inst
```
--------------------------------
### List Oracle Installed Packages
Source: https://relstorage.readthedocs.io/en/latest/_modules/relstorage/adapters/oracle/schema.html
Lists installed stored procedure packages and their versions. It queries user_objects and user_source to find package names and extract version information.
```python
def list_packages(self, cursor):
"""List installed stored procedure packages.
Returns {package name: version}. version may be None.
"""
stmt = """
SELECT object_name
FROM user_objects
WHERE object_type = 'PACKAGE'
"""
cursor.execute(stmt)
names = [name for (name,) in cursor.fetchall()]
res = {}
for name in names:
version = None
stmt = """
SELECT TEXT FROM USER_SOURCE
WHERE TYPE='PACKAGE BODY'
AND NAME=:1
"""
cursor.execute(stmt, (name,))
for (text,) in cursor:
match = re.search(r'Version:\s*([0-9a-zA-Z.]+)', text)
if match is not None:
version = match.group(1)
break
res[name.lower()] = version
return res
```
--------------------------------
### ISchemaInstaller
Source: https://relstorage.readthedocs.io/en/latest/_modules/relstorage/adapters/interfaces.html
Interface for installing, verifying, clearing, and dropping the database schema.
```APIDOC
## prepare
### Description
Create the database schema if it does not already exist. Perform any migration steps needed, and call :meth:`verify` before returning.
## verify
### Description
Ensure that the schema that's installed can be used by this RelStorage. If it cannot, for example it's history-preserving and we were configured to be history-free, raise an exception.
## zap_all
### Description
Clear all data out of the database.
## drop_all
### Description
Drop all tables and sequences.
```
--------------------------------
### iter_transactions_range
Source: https://relstorage.readthedocs.io/en/latest/_modules/relstorage/adapters/dbiter.html
Iterates over a range of transactions specified by start and stop TID values.
```APIDOC
## iter_transactions_range
### Description
Iterates over a range of transactions, from `start` to `stop` TID. If `start` or `stop` are not provided, defaults are used (0 for start, MAX_TID for stop).
### Method
`iter_transactions_range(cursor, start=None, stop=None)`
### Parameters
#### Path Parameters
None
#### Query Parameters
None
#### Request Body
None
### Request Example
None
### Response
#### Success Response (200)
Returns an iterable of transaction records within the specified range.
#### Response Example
None
```
--------------------------------
### StoreConnection Begin Transaction
Source: https://relstorage.readthedocs.io/en/latest/_modules/relstorage/adapters/connections.html
Initiates a transaction on the store connection, opening it if necessary.
```Python
def begin(self):
self.connmanager.begin(*self.open_if_needed())
```
--------------------------------
### Reset Object ID Sequence
Source: https://relstorage.readthedocs.io/en/latest/_modules/relstorage/adapters/interfaces.html
Resets the OID allocation sequence to begin from the start.
```python
def reset_oid(cursor):
"""
Cause the sequence of OIDs to begin again from the beginning.
"""
```
--------------------------------
### Getting Pooled Connection Count
Source: https://relstorage.readthedocs.io/en/latest/_modules/relstorage/adapters/connections.html
Returns the current number of connections available in the pool.
```python
@property
def pooled_connection_count(self):
return len(self._connections)
```
--------------------------------
### Configure PostgreSQL pg_hba.conf
Source: https://relstorage.readthedocs.io/en/latest/developing/running.html
Add these lines to the top of your pg_hba.conf file to allow local connections for RelStorage test databases. Ensure they are not overridden by parameters lower in the file.
```text
local relstoragetest relstoragetest md5
local relstoragetest2 relstoragetest md5
local relstoragetest_hf relstoragetest md5
local relstoragetest2_hf relstoragetest md5
host relstoragetest relstoragetest 127.0.0.1/32 md5
host relstoragetest_hf relstoragetest 127.0.0.1/32 md5
```
--------------------------------
### List PostgreSQL Languages
Source: https://relstorage.readthedocs.io/en/latest/_modules/relstorage/adapters/postgresql/schema.html
Retrieves a list of all installed languages in the PostgreSQL database by querying 'pg_catalog.pg_language'.
```sql
SELECT lanname FROM pg_catalog.pg_language
```
```python
def list_languages(self, cursor):
cursor.execute("SELECT lanname FROM pg_catalog.pg_language")
return self.__native_names_only(cursor)
```
--------------------------------
### Control Packing Process Options
Source: https://relstorage.readthedocs.io/en/latest/changelog.html
Use `--prepack` to analyze data for packing separately from the actual packing phase. Use `--use-prepack-state` to apply a previously generated pre-pack state.
```bash
zodbpack --prepack
zodbpack --use-prepack-state
```
--------------------------------
### Run zodbpack with prepack-only option
Source: https://relstorage.readthedocs.io/en/latest/_sources/zodbpack.rst.txt
The --prepack option instructs the storage to only perform the pre-pack phase without deleting any data. This is useful for analysis before actual packing.
```bash
zodbpack --prepack
```
--------------------------------
### Schema Installer Interface: Drop All Tables
Source: https://relstorage.readthedocs.io/en/latest/_modules/relstorage/adapters/interfaces.html
Removes all tables and sequences associated with the RelStorage schema from the database.
```python
def drop_all():
"""Drop all tables and sequences."""
```
--------------------------------
### Create ZODB DB from repoze.zodbconn URI
Source: https://relstorage.readthedocs.io/en/latest/_sources/configure-application.rst.txt
Create a ZODB DB using repoze.zodbconn.db_from_uri with a ZConfig URI.
```python
from repoze.zodbconn.uri import db_from_uri
db = db_from_uri('zconfig://path/to/configuration#main')
```
--------------------------------
### Initialize OracleAdapter
Source: https://relstorage.readthedocs.io/en/latest/_modules/relstorage/adapters/oracle/adapter.html
Initializes the OracleAdapter with connection details and configuration options. The user, password, and dsn parameters are passed to cx_Oracle. The twophase option enables Oracle-level two-phase commit.
```python
def __init__(self,
user='relstoragetest',
password='relstoragetest',
dsn='192.168.1.131/orcl',
commit_lock_id=0,
twophase=False, options=None,
locker=None,
mover=None,
connmanager=None,
):
"""Create an Oracle adapter.
The user, password, and dsn parameters are provided to
cx_Oracle at connection time.
If twophase is true, all commits go through an Oracle-level
two-phase commit process. This is disabled by default; it causes
deadlock detection to become slow and based on a timeout. Even
when this option is disabled, the ZODB two-phase commit is
still in effect.
"""
# pylint:disable=unused-argument
self._user = user
self._password = password
self._dsn = dsn
self._twophase = twophase
self.locker = locker
self.mover = mover
self.connmanager = connmanager
super().__init__(options)
```
--------------------------------
### Getting Instance Count
Source: https://relstorage.readthedocs.io/en/latest/_modules/relstorage/adapters/connections.html
Returns the number of active RelStorage instances, which influences the desired pool size.
```python
@property
def instance_count(self):
return self._count
```
--------------------------------
### Connect to MySQL
Source: https://relstorage.readthedocs.io/en/latest/_sources/mysql/setup.rst.txt
Use the mysql utility to connect to the MySQL server. The -p option is usually required if the account needs a password.
```shell
$ mysql -u root -p
```
--------------------------------
### Install Blob Chunk Delete Trigger
Source: https://relstorage.readthedocs.io/en/latest/_modules/relstorage/adapters/postgresql/schema.html
Creates a BEFORE DELETE trigger on the 'blob_chunk' table that executes the 'blob_chunk_delete_trigger' procedure.
```sql
CREATE TRIGGER blob_chunk_delete
BEFORE DELETE ON blob_chunk
FOR EACH ROW
EXECUTE PROCEDURE blob_chunk_delete_trigger()
```
--------------------------------
### Connect to MySQL
Source: https://relstorage.readthedocs.io/en/latest/mysql/setup.html
Use this command to connect to the MySQL server as the root user. The `-p` option prompts for a password.
```bash
$ mysql -u root -p
```
--------------------------------
### Oracle SQL*Plus SYS Session
Source: https://relstorage.readthedocs.io/en/latest/_sources/developing/running.rst.txt
Commands to start an Oracle SQL*Plus session with SYS privileges.
```shell
$ su - oracle
$ sqlplus / as sysdba
```
--------------------------------
### Create PostgreSQL Test User and Databases
Source: https://relstorage.readthedocs.io/en/latest/developing/running.html
SQL commands to create a test user and multiple databases for RelStorage testing in PostgreSQL. Adapt the SQL statements as needed for your environment.
```bash
RS_PG_UNAME=${RELSTORAGETEST_PG_UNAME:-postgres}
psql -U $RS_PG_UNAME $RELSTORAGETEST_PG_DBNAME -c "CREATE USER relstoragetest WITH PASSWORD 'relstoragetest';"
psql -U $RS_PG_UNAME $RELSTORAGETEST_PG_DBNAME -c "CREATE DATABASE relstoragetest OWNER relstoragetest;"
psql -U $RS_PG_UNAME $RELSTORAGETEST_PG_DBNAME -c "CREATE DATABASE relstoragetest2 OWNER relstoragetest;"
psql -U $RS_PG_UNAME $RELSTORAGETEST_PG_DBNAME -c "CREATE DATABASE relstoragetest_hf OWNER relstoragetest;"
psql -U $RS_PG_UNAME $RELSTORAGETEST_PG_DBNAME -c "CREATE DATABASE relstoragetest2_hf OWNER relstoragetest;"
psql -U $RS_PG_UNAME $RELSTORAGETEST_PG_DBNAME -c "SELECT version();"
```
--------------------------------
### Get Oracle Database Name
Source: https://relstorage.readthedocs.io/en/latest/_modules/relstorage/adapters/oracle/schema.html
Retrieves the current Oracle database name using a simple SQL query.
```python
def get_database_name(self, cursor):
cursor.execute("SELECT ora_database_name FROM DUAL")
for (name,) in cursor:
return name
```
--------------------------------
### Get Database Messages
Source: https://relstorage.readthedocs.io/en/latest/_modules/relstorage/adapters/drivers.html
A placeholder method to retrieve messages from the database connection. It currently returns an empty tuple.
```python
def get_messages(self, conn): # pragma: no cover pylint:disable=unused-argument
return ()
```
--------------------------------
### Opening Connection for Pre-pack Phase
Source: https://relstorage.readthedocs.io/en/latest/relstorage.adapters.sqlite.connmanager.html
Opens a specific database connection intended for the pre-pack phase of operations. Returns the connection and cursor objects.
```python
open_for_pre_pack()
```
--------------------------------
### SQLite Too Old Error
Source: https://relstorage.readthedocs.io/en/latest/_modules/relstorage/cache/persistence.html
Custom exception raised when the installed sqlite3 module version is below the minimum required version (3.8.3).
```python
class Sqlite3TooOldError(ValueError):
"""Raised if the sqlite3 module is too old."""
def __init__(self, *_args):
super().__init__(
"Unable to use sqlite; minimum version is %s but this version is %s" % (
"3.8.3", sqlite3.sqlite_version_info
))
```
--------------------------------
### Display zodbpack help information
Source: https://relstorage.readthedocs.io/en/latest/_sources/zodbpack.rst.txt
Use the --help option to display all available command-line options for the zodbpack script.
```bash
zodbpack --help
```
--------------------------------
### Reset Zoid Sequence
Source: https://relstorage.readthedocs.io/en/latest/_modules/relstorage/adapters/postgresql/schema.html
Resets the 'zoid_seq' sequence to start with a value of 1. This is typically used to re-initialize the OID generation.
```sql
def _reset_oid(self, cursor):
stmt = "ALTER SEQUENCE zoid_seq RESTART WITH 1;"
self.runner.run_script(cursor, stmt)
```
--------------------------------
### Create PostgreSQL User and Database
Source: https://relstorage.readthedocs.io/en/latest/_sources/postgresql/setup.rst.txt
Use these shell commands to switch to the postgres user, create a new database user with a prompt for the password, and then create a database owned by that user.
```shell
$ sudo su - postgres
$ createuser --pwprompt zodbuser
$ createdb -O zodbuser zodb
```
--------------------------------
### Get current object tids
Source: https://relstorage.readthedocs.io/en/latest/relstorage.adapters.mover.html
Returns a dictionary mapping object IDs to their current transaction IDs for the specified objects.
```python
def current_object_tids(self, _cursor, _oids, _timeout=None):
"""
Returns the current {oid: tid} for specified object ids.
"""
pass
```
--------------------------------
### Pre-Pack Garbage Collection Entry Point
Source: https://relstorage.readthedocs.io/en/latest/_modules/relstorage/adapters/packundo.html
The main entry point for the pre-pack garbage collection process. It conditionally enables garbage collection based on options and manages connection lifecycles.
```Python
@metricmethod
def pre_pack(self, pack_tid, get_references):
"""
Decide what the garbage collector should delete.
Objects created or modified after pack_tid will not be garbage
collected.
get_references is a function that accepts a pickled state and
returns a set of OIDs that state refers to.
The self.options.pack_gc flag indicates whether to run garbage
collection. If pack_gc is false, this method does nothing.
"""
if not self.options.pack_gc:
logger.warning("pre_pack: garbage collection is disabled on a "
"history-free storage, so doing nothing")
return
load_connection = LoadConnection(self.connmanager)
store_connection = PrePackConnection(self.connmanager)
try:
try:
self._pre_pack_main(load_connection, store_connection,
pack_tid, get_references)
except:
logger.exception("pre_pack: failed")
store_connection.rollback_quietly()
raise
store_connection.commit()
logger.info("pre_pack: finished successfully")
finally:
load_connection.drop()
store_connection.drop()
```
--------------------------------
### IReplicaSelector
Source: https://relstorage.readthedocs.io/en/latest/_modules/relstorage/adapters/interfaces.html
Interface for selecting a database replica. It provides methods to get the current replica and to retrieve the next replica to try.
```APIDOC
## IReplicaSelector
### Description
Selects a database replica.
### Methods
- **current()**: Gets the current replica. Returns a string representing the replica (e.g., host:port for PostgreSQL/MySQL, DSN for Oracle).
- **next()**: Returns the next replica to try. Returns None if there are no more replicas defined.
```
--------------------------------
### Create Tables
Source: https://relstorage.readthedocs.io/en/latest/_modules/relstorage/adapters/schema.html
Public method to create all necessary database tables, optionally considering existing tables.
```python
def create_tables(self, cursor, existing_tables=()):
"""Create the database tables."""
_, existing_tables = self._create_schema_objects(cursor, self.all_tables, existing_tables)
```
--------------------------------
### Get Complete Since TID
Source: https://relstorage.readthedocs.io/en/latest/_modules/relstorage/cache/mvcc.html
Retrieves the complete since tid from the object index, or -1 if the object index is not available.
```python
@property
def complete_since_tid(self):
return -1 if self.object_index is None else self.object_index.complete_since_tid
```
--------------------------------
### Get Checkpoints from Database
Source: https://relstorage.readthedocs.io/en/latest/_modules/relstorage/cache/local_database.html
Retrieves checkpoint information (max_hvt and complete_since) from the 'checkpoints' table. Returns None if no checkpoints exist.
```python
self.cursor.execute("SELECT max_hvt, complete_since FROM checkpoints")
return self.cursor.fetchone()
```
--------------------------------
### Initialize PackUndo with Database Parameters
Source: https://relstorage.readthedocs.io/en/latest/_modules/relstorage/adapters/packundo.html
Initializes the PackUndo class with database driver, connection manager, runner, locker, and options. Configuration parameters like cursor array size and store batch size are derived from environment variables.
```python
class PackUndo(DatabaseHelpersMixin):
"""Abstract base class for pack/undo"""
_choose_pack_transaction_query = None
driver = None
connmanager = None
runner = None
locker = None
options = None
###
# Parameters. Some are used by only history-free or history-preserving
# implementations.
###
cursor_arraysize = get_positive_integer_from_environ('RS_PACK_CURSOR_ARRAYSIZE',
4096,
logger=logger)
# These are generally very small rows we're uploading (a few integers).
# Be generous.
store_batch_size = get_positive_integer_from_environ('RS_PACK_STORE_BATCH_SIZE', 4096,
logger=logger)
# How often, in seconds, to commit work in progress.
# This is a variable here for testing.
fill_object_refs_commit_frequency = get_duration_from_environ('RS_PACK_COMMIT_FREQUENCY', 120,
logger=logger)
# How many object states to find references in at any one time.
# This is a control on the amount of memory used by the Python
# process during packing, especially if the database driver
# doesn't use server-side cursors.
fill_object_refs_batch_size = get_positive_integer_from_environ('RS_PACK_DOWNLOAD_BATCH_SIZE',
1024,
logger=logger)
def __init__(self, database_driver, connmanager, runner, locker, options):
self.driver = database_driver
self.connmanager = connmanager
self.runner = runner
self.locker = locker
self.options = options
```
--------------------------------
### Get MySQL Full Version
Source: https://relstorage.readthedocs.io/en/latest/_modules/relstorage/adapters/mysql/schema.html
Retrieves the full version string of the MySQL server. This is used for detailed compatibility checks.
```python
self._setup(cursor)
return self._version
```
--------------------------------
### Get Row Schema for Insert
Source: https://relstorage.readthedocs.io/en/latest/_modules/relstorage/adapters/batch.html
Retrieves or creates a row schema string based on the parameter count for insert operations.
```Python
def row_schema_of_length(self, param_count):
# Use as the *row_schema* parameter to insert_into
key = (self.insert_placeholder, param_count)
try:
return _row_schemas[key]
except KeyError:
p = [self.insert_placeholder] * param_count
p = ', '.join(p)
_row_schemas[key] = p
return p
```
--------------------------------
### Configure RelStorage with repoze.zodbconn
Source: https://relstorage.readthedocs.io/en/latest/_sources/configure-application.rst.txt
Use RelStorage with repoze.zodbconn by creating a ZConfig file with a element.
```zconfig
%import relstorage
db zodb
cache-size 100000
```
--------------------------------
### AbstractLocker _on_store_opened_set_row_lock_timeout
Source: https://relstorage.readthedocs.io/en/latest/_modules/relstorage/adapters/locker.html
Installs a per-session row lock timeout, typically set to the commit_lock_timeout. This affects locks acquired during the tpc_vote phase.
```python
def _on_store_opened_set_row_lock_timeout(self, cursor, restart=False):
"""
Install a per-session row lock timeout.
This should be set to the :attr:`commit_lock_timeout`.
This applies to the locks taken by :meth:`lock_current_objects` during the
``tpc_vote`` phase of a transaction.
"""
```
--------------------------------
### Run zodbpack with use-prepack-state option
Source: https://relstorage.readthedocs.io/en/latest/_sources/zodbpack.rst.txt
The --use-prepack-state option skips the pre-pack analysis and only performs the deletion (packing) phase. This is useful when pre-packing has already been done.
```bash
zodbpack --use-prepack-state
```
--------------------------------
### Setting Transaction Isolation Level in MySQL
Source: https://relstorage.readthedocs.io/en/latest/_modules/relstorage/adapters/mysql/connmanager.html
Sets the session transaction isolation level for MySQL. This must be done before starting a transaction.
```python
self._db_driver.set_autocommit(conn, True)
# Transaction isolation cannot be changed inside a
# transaction. 'SET SESSION' changes it for all
# upcoming transactions.
stmt = "SET SESSION TRANSACTION %s" % isolation
__traceback_info__ = stmt
cursor.execute(stmt)
self._db_driver.set_autocommit(conn, False)
conn.replica = replica
conn.readonly = read_only
return conn, cursor
```
--------------------------------
### Simulate Default Cache Size (20MB)
Source: https://relstorage.readthedocs.io/en/latest/_sources/cache-tracing.rst.txt
Run the cache simulation script using the default cache size (20MB) to evaluate its performance.
```bash
python -m ZEO.scripts.cache_simul.py /tmp/cachetrace.log
```
--------------------------------
### Poller Interface: Get Current Transaction ID
Source: https://relstorage.readthedocs.io/en/latest/_modules/relstorage/adapters/interfaces.html
Retrieves the highest transaction ID visible to a cursor. Returns 0 if there are no transactions.
```python
def get_current_tid(cursor):
"""
Returns the highest transaction ID visible to the cursor.
If there are no transactions, returns 0.
"""
```
--------------------------------
### Implement Database Driver Options
Source: https://relstorage.readthedocs.io/en/latest/_modules/relstorage/adapters/drivers.html
A helper function to register database drivers and make them available through an interface, including driver selection logic.
```python
def implement_db_driver_options(name, *driver_modules):
"""
Helper function to be called at a module scope to
make it implement ``IDBDriverOptions``.
:param str name: The value of ``__name__``.
:param driver_modules: Each of these names a module that has
one or more implementations of ``IDBDriver`` in it,
as named in their ``__all__`` attribute.
"""
module = sys.modules[name]
driver_factories = set()
for driver_module in driver_modules:
driver_module = importlib.import_module('.' + driver_module,
name)
for factory in driver_module.__all__:
factory = getattr(driver_module, factory)
if IDBDriver.implementedBy(factory): # pylint:disable=no-value-for-parameter
driver_factories.add(_ClassDriverFactory(factory))
module.known_driver_factories = lambda: sorted(
driver_factories,
key=lambda factory: factory.PRIORITY if not PYPY else factory.PRIORITY_PYPY,
)
directlyProvides(module, IDBDriverOptions)
module.select_driver = lambda driver_name=None: _select_driver_by_name(driver_name,
sys.modules[name])
```
--------------------------------
### Get Next Replica to Try
Source: https://relstorage.readthedocs.io/en/latest/_modules/relstorage/adapters/interfaces.html
Returns the next available database replica to attempt a connection with. Returns None if no more replicas are defined.
```Python
def next():
"""Return the next replica to try.
Return None if there are no more replicas defined.
"""
```
--------------------------------
### TempStoreCopyBuffer Length and Next Row
Source: https://relstorage.readthedocs.io/en/latest/_modules/relstorage/adapters/postgresql/mover.html
Defines the length of the buffer based on the iterable and provides a method to get the next row from the iterator.
```python
def __len__(self):
return len(self.state_oid_tid_iterable)
def _next_row(self):
return next(self._iter)
```