### Initializing pgsync Sync Python Source: https://github.com/toluaina/pgsync/blob/main/docs/usage.rst This snippet shows the standard way to start the `pgsync` synchronization process from Python code. It imports the `sync` module and calls its `main()` function, which typically reads configuration and initiates the data transfer from a source database to a target database. This requires the `pgsync` package to be installed in the environment. ```python from pgsync import sync sync.main() ``` -------------------------------- ### Installing PGSync via Pip Source: https://github.com/toluaina/pgsync/blob/main/README.md This standard Python command installs the PGSync package and its dependencies from the Python Package Index (PyPI) using the pip package installer, making the `pgsync` and `bootstrap` commands available. ```Shell $ pip install pgsync ``` -------------------------------- ### Bootstrapping PGSync Database Schema Source: https://github.com/toluaina/pgsync/blob/main/README.md This command initializes the database for use with PGSync, including setting up triggers and potentially other required database objects based on the provided JSON schema definition file. This is typically a one-time setup step. ```Shell bootstrap --config schema.json ``` -------------------------------- ### Starting PGSync services with Docker Source: https://github.com/toluaina/pgsync/blob/main/README.md This command uses docker-compose to start all necessary services (Postgres, Redis, Elasticsearch/OpenSearch) defined in the docker-compose.yml file for running PGSync in a containerized environment. ```Shell docker-compose up ``` -------------------------------- ### Running PGSync Process Source: https://github.com/toluaina/pgsync/blob/main/README.md This command starts the main PGSync process, which reads the configuration from the specified schema file and begins monitoring the PostgreSQL database for changes, synchronizing them to Elasticsearch/OpenSearch accordingly. ```Shell pgsync --config schema.json ``` -------------------------------- ### Setting Up Development Environment - Shell Source: https://github.com/toluaina/pgsync/blob/main/CONTRIBUTING.rst These commands set up a virtual environment named 'pgsync', navigate into the project directory, and install the project's dependencies in development mode using `python setup.py develop`. This allows you to work on the code directly within the virtual environment. ```shell $ mkvirtualenv pgsync $ cd pgsync/ $ python setup.py develop ``` -------------------------------- ### Example Output Document Structure (JSON) Source: https://github.com/toluaina/pgsync/blob/main/README.md This JSON snippet provides examples of the document structure generated by PGSync after synchronizing data from the relational database based on the specified schema. It shows how information from the 'book' table is combined with aggregated author names from the related 'author' table. ```json [ { "isbn": "9785811243570", "title": "Charlie and the chocolate factory", "description": "Willy Wonka’s famous chocolate factory is opening at last!", "authors": ["Roald Dahl"] }, { "isbn": "9788374950978", "title": "Kafka on the Shore", "description": "Kafka on the Shore is a 2002 novel by Japanese author Haruki Murakami", "authors": ["Haruki Murakami", "Philip Gabriel"] }, { "isbn": "9781471331435", "title": "1984", "description": "1984 was George Orwell’s chilling prophecy about the dystopian future", "authors": ["George Orwell"] } ] ``` -------------------------------- ### Running PGSync as a Daemon Source: https://github.com/toluaina/pgsync/blob/main/README.md This command starts the PGSync process in the background as a daemon, allowing it to run continuously without keeping the terminal session open, using the configuration specified in the schema file. ```Shell pgsync --config schema.json -d ``` -------------------------------- ### Executing pgsync bootstrap with test flag Source: https://github.com/toluaina/pgsync/blob/main/HISTORY.rst Instructs the user to run the 'bootstrap' command with the '-t' flag. This step is required after upgrading to version 1.1.0 to re-initialize the database for pgsync, specifically performing a test or setup phase of the bootstrap process. ```Command Line bootstrap -t ``` -------------------------------- ### Example Output with Attribute Renaming (JSON) Source: https://github.com/toluaina/pgsync/blob/main/README.md This JSON snippet illustrates how PGSync can be configured to rename attributes in the output document via the schema configuration. It shows an example where fields like 'title', 'description', and 'authors' are renamed to 'this_is_a_custom_title', 'desc', and 'contributors' respectively. ```json { "isbn": "9781471331435", "this_is_a_custom_title": "1984", "desc": "1984 was George Orwell’s chilling prophecy about the dystopian future", "contributors": ["George Orwell"] } ``` -------------------------------- ### Querying Elasticsearch/OpenSearch index via Curl Source: https://github.com/toluaina/pgsync/blob/main/README.md This command demonstrates how to send a GET request to an Elasticsearch or OpenSearch endpoint to retrieve data from the 'reservations' index using the Curl command-line tool, showing the results in a pretty-printed JSON format. ```Shell curl -X GET http://[Elasticsearch/OpenSearch host]:9201/reservations/_search?pretty=true ``` -------------------------------- ### Cloning Git Repository - Shell Source: https://github.com/toluaina/pgsync/blob/main/CONTRIBUTING.rst This command clones your forked pgsync repository from GitHub to your local machine. It's the first step in setting up your development environment after forking. ```shell $ git clone git@github.com:toluaina/pgsync.git ``` -------------------------------- ### Executing pgsync bootstrap command Source: https://github.com/toluaina/pgsync/blob/main/HISTORY.rst Instructs the user to run the main 'bootstrap' command. This step is required after upgrading to version 1.1.0, following the 'bootstrap -t' command, to complete the database re-initialization for pgsync. ```Command Line bootstrap ``` -------------------------------- ### Running Code Checks and Tests - Shell Source: https://github.com/toluaina/pgsync/blob/main/CONTRIBUTING.rst These commands are used to check code quality with flake8, run tests using setup.py or py.test, and test across multiple Python versions with tox. Ensure these pass before submitting a pull request. ```shell $ flake8 pgsync tests $ python setup.py test or py.test $ tox ``` -------------------------------- ### Committing and Pushing Changes - Shell Source: https://github.com/toluaina/pgsync/blob/main/CONTRIBUTING.rst These commands stage all local changes, create a new commit with a descriptive message, and push the changes to your forked repository on GitHub. This makes your work available for a pull request. ```shell $ git add . $ git commit -m "Your detailed description of your changes." $ git push origin name-of-your-bugfix-or-feature ``` -------------------------------- ### Deploying New Version (Maintainers) - Shell Source: https://github.com/toluaina/pgsync/blob/main/CONTRIBUTING.rst These commands are for maintainers to deploy a new version. They use bumpversion to update the version number (patch, minor, or major), push the commit to the main repository, and push the version tag. ```shell $ bumpversion patch # possible: major / minor / patch $ git push $ git push --tags ``` -------------------------------- ### Generated SQL Query for Document Creation (SQL) Source: https://github.com/toluaina/pgsync/blob/main/README.md This SQL query, automatically generated by PGSync based on the defined schema, fetches data from the 'book', 'book_author', and 'author' tables to construct the final JSON document. It utilizes PostgreSQL's JSON functions (JSON_BUILD_OBJECT, JSON_AGG) and joins to combine related data and aggregate authors for each book. ```sql SELECT JSON_BUILD_OBJECT( 'isbn', book_1.isbn, 'title', book_1.title, 'description', book_1.description, 'authors', anon_1.authors ) AS "JSON_BUILD_OBJECT_1", book_1.id FROM book AS book_1 LEFT OUTER JOIN (SELECT JSON_AGG(anon_2.anon) AS authors, book_author_1.book_isbn AS book_isbn FROM book_author AS book_author_1 LEFT OUTER JOIN (SELECT author_1.name AS anon, author_1.id AS id FROM author AS author_1) AS anon_2 ON anon_2.id = book_author_1.author_id GROUP BY book_author_1.book_isbn) AS anon_1 ON anon_1.book_isbn = book_1.isbn ``` -------------------------------- ### Running Specific Tests - Shell Source: https://github.com/toluaina/pgsync/blob/main/CONTRIBUTING.rst This command allows you to run tests only within a specific test file, which is useful for quickly checking changes related to that file without running the full test suite. ```shell $ py.test tests/test_base.py ``` -------------------------------- ### Configuring PostgreSQL for Logical Decoding Source: https://github.com/toluaina/pgsync/blob/main/README.md These settings in the postgresql.conf file are required to enable PostgreSQL's logical decoding feature, which PGSync uses to capture data changes. `wal_level` enables logging required data, `max_replication_slots` sets the maximum number of simultaneous slots, and `max_slot_wal_keep_size` limits disk usage by replication slots. ```SQL configuration wal_level = logical max_replication_slots = 1 max_slot_wal_keep_size = 100GB ``` -------------------------------- ### Defining PGSync Schema for Book Library (JSON) Source: https://github.com/toluaina/pgsync/blob/main/README.md This JSON snippet defines the schema used by PGSync to structure data from the relational Book library database into a document format. It specifies the pivot table ('book'), the columns to include from the pivot, and nested children tables ('author') joined via 'book_author' to be included in the resulting document. ```json { "table": "book", "columns": [ "isbn", "title", "description" ], "children": [ { "table": "author", "columns": [ "name" ] } ] } ``` -------------------------------- ### Creating Feature/Bugfix Branch - Shell Source: https://github.com/toluaina/pgsync/blob/main/CONTRIBUTING.rst This command creates a new Git branch for your specific bug fix or feature. It's recommended to make changes on a separate branch rather than directly on the main branch. ```shell $ git checkout -b name-of-your-bugfix-or-feature ``` === COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.