### Install and Show PyJedAI Package Source: https://github.com/ai-team-uoa/pyjedai/blob/main/docs/tutorials/TfIdfWorkflow.ipynb Installs the pyJedAI library from PyPI and then displays its installed version and details. This is a prerequisite for using the library's functionalities. ```python %pip install pyjedai -U ``` ```python %pip show pyjedai ``` -------------------------------- ### Install pyJedAI from source Source: https://github.com/ai-team-uoa/pyjedai/blob/main/README.md Installs pyJedAI locally by cloning the Git repository. Navigate to the cloned directory and use pip to install. This method is useful for developers or for using the latest unreleased features. ```bash git clone https://github.com/AI-team-UoA/pyJedAI.git cd pyJedAI pip install . ``` -------------------------------- ### Show pyJedAI installation details Source: https://github.com/ai-team-uoa/pyjedai/blob/main/docs/tutorials/CleanCleanERwithoutGT.ipynb Displays information about the installed pyJedAI package, including its version, summary, author, license, and installation location. This command is useful for verifying the installation and understanding package metadata. ```python ```python !pip show pyjedai ``` ``` -------------------------------- ### Install pyJedAI Source: https://github.com/ai-team-uoa/pyjedai/blob/main/docs/tutorials/SchemaMatching.ipynb Installs the pyJedAI library using pip. This is a prerequisite for using the schema matching functionalities. ```python !pip install pyjedai -U ``` -------------------------------- ### Optuna Trial Logging Source: https://github.com/ai-team-uoa/pyjedai/blob/main/docs/tutorials/Optuna.ipynb Logs from Optuna trials showing the finished value and parameters for each trial. It indicates the best trial found so far. ```log [I 2022-09-26 17:14:13,289] Trial 11 finished with value: 0.5274133516352982 and parameters: {'qgrams': 10, 'ratio': 0.9045625386867897, 'similarity_threshold': 0.6392297322807924}. Best is trial 10 with value: 0.6482633708392243. [I 2022-09-26 17:14:18,068] Trial 12 finished with value: 0.42621654591235925 and parameters: {'qgrams': 10, 'ratio': 0.9483018131894073, 'similarity_threshold': 0.669716499745008}. Best is trial 10 with value: 0.6482633708392243. [I 2022-09-26 17:14:23,897] Trial 13 finished with value: 0.22112929238626436 and parameters: {'qgrams': 7, 'ratio': 0.8961116303610029, 'similarity_threshold': 0.6036793964995847}. Best is trial 10 with value: 0.6482633708392243. [I 2022-09-26 17:14:28,293] Trial 14 finished with value: 0.4070833316320489 and parameters: {'qgrams': 9, 'ratio': 0.8983635553727757, 'similarity_threshold': 0.5808143891387648}. Best is trial 10 with value: 0.6482633708392243. [I 2022-09-26 17:14:33,109] Trial 15 finished with value: 0.42621654591235925 and parameters: {'qgrams': 10, 'ratio': 0.9488029595469605, 'similarity_threshold': 0.7454617001392468}. Best is trial 10 with value: 0.6482633708392243. [I 2022-09-26 17:14:40,257] Trial 16 finished with value: 0.2123846817548284 and parameters: {'qgrams': 6, 'ratio': 0.8777239752940389, 'similarity_threshold': 0.5114877692227677}. Best is trial 10 with value: 0.6482633708392243. [I 2022-09-26 17:14:45,396] Trial 17 finished with value: 0.3258435026582389 and parameters: {'qgrams': 8, 'ratio': 0.9204153410113451, 'similarity_threshold': 0.7435281257384602}. Best is trial 10 with value: 0.6482633708392243. [I 2022-09-26 17:14:50,167] Trial 18 finished with value: 0.3480318083295157 and parameters: {'qgrams': 8, 'ratio': 0.875183707875409, 'similarity_threshold': 0.5035164496571632}. Best is trial 10 with value: 0.6482633708392243. [I 2022-09-26 17:14:54,146] Trial 19 finished with value: 0.36391860524716446 and parameters: {'qgrams': 10, 'ratio': 0.8294161917860436, 'similarity_threshold': 0.6808222011656786}. Best is trial 10 with value: 0.6482633708392243. [I 2022-09-26 17:15:01,379] Trial 20 finished with value: 0.20993148042255388 and parameters: {'qgrams': 6, 'ratio': 0.9196762030928591, 'similarity_threshold': 0.5795471650862246}. Best is trial 10 with value: 0.6482633708392243. [I 2022-09-26 17:15:05,584] Trial 21 finished with value: 0.46330684097155167 and parameters: {'qgrams': 10, 'ratio': 0.8563838420348091, 'similarity_threshold': 0.23323189289645876}. Best is trial 10 with value: 0.6482633708392243. [I 2022-09-26 17:15:09,902] Trial 22 finished with value: 0.4070833316320489 and parameters: {'qgrams': 9, 'ratio': 0.8948541483847852, 'similarity_threshold': 0.2644175908762265}. Best is trial 10 with value: 0.6482633708392243. [I 2022-09-26 17:15:14,091] Trial 23 finished with value: 0.42745701777773953 and parameters: {'qgrams': 10, 'ratio': 0.8429055971815752, 'similarity_threshold': 0.4726717116974519}. Best is trial 10 with value: 0.6482633708392243. [I 2022-09-26 17:15:18,489] Trial 24 finished with value: 0.40750648259180683 and parameters: {'qgrams': 9, 'ratio': 0.876132015812083, 'similarity_threshold': 0.6726766450815258}. Best is trial 10 with value: 0.6482633708392243. [I 2022-09-26 17:15:23,589] Trial 25 finished with value: 0.3258414249026382 and parameters: {'qgrams': 8, 'ratio': 0.9275690010126776, 'similarity_threshold': 0.16506027753408292}. Best is trial 10 with value: 0.6482633708392243. [I 2022-09-26 17:15:27,909] Trial 26 finished with value: 0.46330684097155167 and parameters: {'qgrams': 10, 'ratio': 0.8601034741266624, 'similarity_threshold': 0.7695051840189288}. Best is trial 10 with value: 0.6482633708392243. [I 2022-09-26 17:15:31,804] Trial 27 finished with value: 0.4037641499510466 and parameters: {'qgrams': 9, 'ratio': 0.7662074140900922, 'similarity_threshold': 0.44798780615040656}. Best is trial 10 with value: 0.6482633708392243. [I 2022-09-26 17:15:36,144] Trial 28 finished with value: 0.3639219361818585 and parameters: {'qgrams': 10, 'ratio': 0.8174855005418883, 'similarity_threshold': 0.6110684353074003}. Best is trial 10 with value: 0.6482633708392243. [I 2022-09-26 17:15:43,271] Trial 29 finished with value: 0.24080227825457184 and parameters: {'qgrams': 7, 'ratio': 0.8389446905059951, 'similarity_threshold': 0.35743796412615136}. Best is trial 10 with value: 0.6482633708392243. Optuna trials finished ``` -------------------------------- ### Install pyJedAI using pip Source: https://github.com/ai-team-uoa/pyjedai/blob/main/README.md Installs the latest version of the pyJedAI library from PyPI. This is the recommended method for most users. Ensure you have Python 3.8 or higher installed. ```bash pip install pyjedai ``` -------------------------------- ### Start Ollama server in background using Python Source: https://github.com/ai-team-uoa/pyjedai/blob/main/docs/tutorials/LLMsMatching.ipynb Starts the Ollama server as a background process using Python's subprocess module. This is an alternative to manually starting the server in a new terminal. ```python # start ollama server as a background process import subprocess process = subprocess.Popen("ollama serve", shell=True) ``` -------------------------------- ### Show pyJedAI installation details Source: https://github.com/ai-team-uoa/pyjedai/blob/main/docs/tutorials/Demo.ipynb Displays information about the installed pyJedAI package, such as its version, location, and dependencies. Useful for verifying the installation. ```python %pip show pyjedai ``` -------------------------------- ### Install pyJedAI using pip Source: https://github.com/ai-team-uoa/pyjedai/blob/main/docs/tutorials/Demo.ipynb Installs the pyJedAI library and its latest version from PyPI. This is the first step to using the framework. ```python %pip install pyjedai -U ``` -------------------------------- ### Configure Optuna Study Parameters Source: https://github.com/ai-team-uoa/pyjedai/blob/main/docs/tutorials/Optuna.ipynb Sets up parameters for an Optuna hyper-parameter tuning study. Defines the database name, study title, and the storage location for the study results (using SQLite). ```python db_name = "pyjedai" title = "Test" storage_name = "sqlite:///{}.db".format(db_name) study_name = title # Unique identifier of the study. ``` -------------------------------- ### Install Ollama using Linux command Source: https://github.com/ai-team-uoa/pyjedai/blob/main/docs/tutorials/LLMsMatching.ipynb Installs Ollama, an open-source tool for running large language models locally, using a curl command on Linux. This is a prerequisite for using LLM-based entity matching. ```bash #Linux Command !curl -fsSL https://ollama.com/install.sh | sh ``` -------------------------------- ### Install pyJedAI Source: https://github.com/ai-team-uoa/pyjedai/blob/main/docs/tutorials/Spatial.ipynb Installs the pyJedAI library version 0.1.4 using pip. This is a prerequisite for using the pyJedAI functionalities. ```python !pip install pyjedai:0.1.4 ``` -------------------------------- ### Create SQLAlchemy Engine for PostgreSQL Source: https://github.com/ai-team-uoa/pyjedai/blob/main/docs/tutorials/Readers.ipynb Sets up a SQLAlchemy engine to connect to a PostgreSQL database. It requires database connection details such as address, port, username, password, and database name. ```python from sqlalchemy import create_engine POSTGRES_ADDRESS = 'db' ## INSERT YOUR DB ADDRESS POSTGRES_PORT = '5439' POSTGRES_USERNAME = 'username' ## CHANGE THIS TO YOUR POSTGRES USERNAME POSTGRES_PASSWORD = 'root' ## CHANGE THIS TO YOUR POSTGRES PASSWORD POSTGRES_DBNAME = 'database' ## CHANGE THIS TO YOUR DATABASE NAME postgres_str = ('postgresql://{username}:{password}@{ipaddress}:{port}/{dbname}'.format( username=POSTGRES_USERNAME, password=POSTGRES_PASSWORD, ipaddress=POSTGRES_ADDRESS, port=POSTGRES_PORT, dbname=POSTGRES_DBNAME )) # Create the connection cnx = create_engine(postgres_str) ``` -------------------------------- ### Load and Prepare Datasets with pyJedAI Source: https://github.com/ai-team-uoa/pyjedai/blob/main/docs/tutorials/Optuna.ipynb Loads two datasets (abt.csv and buy.csv) and a ground truth file (gt.csv) using pandas. It then initializes a pyJedAI Data object, specifying attribute columns and ID columns for each dataset. ```python from pyjedai.datamodel import Data data = Data( dataset_1=pd.read_csv("./../data/ccer/D2/abt.csv", sep='|', engine='python', na_filter=False).astype(str), attributes_1=['id','name','description'], id_column_name_1='id', dataset_2=pd.read_csv("./../data/ccer/D2/buy.csv", sep='|', engine='python', na_filter=False).astype(str), attributes_2=['id','name','description'], id_column_name_2='id', ground_truth=pd.read_csv("./../data/ccer/D2/gt.csv", sep='|', engine='python'), ) ``` -------------------------------- ### Import Libraries for Optuna and Data Handling Source: https://github.com/ai-team-uoa/pyjedai/blob/main/docs/tutorials/Optuna.ipynb Imports necessary libraries for data manipulation (pandas), visualization (plotly), logging, and hyper-parameter optimization with Optuna. Sets Plotly's default template. ```python import plotly.express as px import logging import sys import optuna import plotly import os import sys import pandas as pd from optuna.visualization import * import plotly.io as pio import plotly.express as px pio.templates.default = "plotly_white" ``` -------------------------------- ### Show pyJedAI package information using pip Source: https://github.com/ai-team-uoa/pyjedai/blob/main/docs/tutorials/LLMsMatching.ipynb Displays information about the installed pyJedAI package using pip. This command is useful for verifying the installation and checking the version. ```python !pip show pyjedai ``` -------------------------------- ### Optuna Trials Dataframe Output Source: https://github.com/ai-team-uoa/pyjedai/blob/main/docs/tutorials/Optuna.ipynb The resulting pandas DataFrame from the Optuna trials, displaying trial number, value, and parameters. This format is useful for analysis and comparison. ```text Result: number value params_qgrams params_ratio \ 0 0 0.303374 8 0.838095 1 1 0.203077 5 0.792963 2 2 0.191036 4 0.803869 ``` -------------------------------- ### Git Clone Repository Source: https://github.com/ai-team-uoa/pyjedai/blob/main/docs/pages/contribution.md Clones a Git repository from a specified URL to your local machine. This is the first step after forking a repository to get a local copy for development. ```git git clone ``` -------------------------------- ### Run Optuna Study for Hyperparameter Tuning Source: https://github.com/ai-team-uoa/pyjedai/blob/main/docs/tutorials/Optuna.ipynb This Python code initializes and runs an Optuna study to optimize the defined objective function. It sets up the study name, number of trials, and storage, then calls the `optimize` method with the objective function and desired number of trials. The output shows the progress and results of the optimization process. ```python study_name = title # Unique identifier of the study. num_of_trials = 30 study = optuna.create_study( directions=["maximize"], study_name=study_name, storage=storage_name, load_if_exists=True ) print("Optuna trials starting") study.optimize( objective, n_trials=num_of_trials, show_progress_bar=True ) print("Optuna trials finished") ``` -------------------------------- ### Install pyJedAI using pip Source: https://github.com/ai-team-uoa/pyjedai/blob/main/docs/tutorials/CleanCleanERwithoutGT.ipynb Installs the pyJedAI library and its dependencies from PyPI. The `-U` flag ensures that the latest version is installed or upgraded if already present. This is the primary method for obtaining the library. ```python ```python !pip install pyjedai -U ``` ``` -------------------------------- ### Initialize Entity Matching in Python Source: https://github.com/ai-team-uoa/pyjedai/blob/main/docs/tutorials/Reproducibility.ipynb Initializes the EntityMatching class with specified parameters for metric, tokenizer, vectorizer, q-gram size, and similarity threshold. This setup is crucial for defining how entity profiles are compared and scored. ```python from pyjedai.matching import EntityMatching em = EntityMatching( metric='cosine', tokenizer='char_tokenizer', vectorizer='tfidf', qgram=3, similarity_threshold=0.0 ) ``` -------------------------------- ### Load DataFrames into SQLite and Query Source: https://github.com/ai-team-uoa/pyjedai/blob/main/docs/tutorials/Readers.ipynb Demonstrates loading pandas DataFrames into an in-memory SQLite database using `to_sql` and then reading data back using `pd.read_sql`. This is useful for temporary data storage and querying. ```python from sqlite3 import connect conn = connect(':memory:') d1.to_sql('d1', conn) d2.to_sql('d2', conn) gt.to_sql('gt', conn) sql_d1 = pd.read_sql('SELECT * FROM d1', conn) sql_d2 = pd.read_sql('SELECT * FROM d2', conn) sql_gt = pd.read_sql('SELECT * FROM gt', conn) ``` -------------------------------- ### Import pyJedAI Workflow and Component Classes Source: https://github.com/ai-team-uoa/pyjedai/blob/main/docs/tutorials/Optuna.ipynb Imports various classes from pyJedAI for building and managing entity resolution workflows. This includes components for block building, block cleaning, comparison cleaning, matching, and clustering. ```python from pyjedai.workflow import WorkFlow, compare_workflows from pyjedai.block_building import StandardBlocking, QGramsBlocking, ExtendedQGramsBlocking, SuffixArraysBlocking, ExtendedSuffixArraysBlocking from pyjedai.block_cleaning import BlockFiltering, BlockPurging from pyjedai.comparison_cleaning import WeightedEdgePruning, WeightedNodePruning, CardinalityEdgePruning, CardinalityNodePruning, BLAST, ReciprocalCardinalityNodePruning, ReciprocalWeightedNodePruning, ComparisonPropagation from pyjedai.matching import EntityMatching from pyjedai.clustering import ConnectedComponentsClustering ``` -------------------------------- ### Git Create New Branch Source: https://github.com/ai-team-uoa/pyjedai/blob/main/docs/pages/contribution.md Creates and switches to a new branch for development. Branch names should be descriptive, e.g., 'feature/' or 'bugfix/'. ```git git checkout -b ``` -------------------------------- ### Generate Optuna Trials Dataframe Source: https://github.com/ai-team-uoa/pyjedai/blob/main/docs/tutorials/Optuna.ipynb This Python code snippet demonstrates how to generate a pandas DataFrame from Optuna study trials. It selects specific attributes for inclusion in the DataFrame. ```python study.trials_dataframe(attrs=("number", "value", "params", "state")) ``` -------------------------------- ### Display First Rows of RDF DataFrames Source: https://github.com/ai-team-uoa/pyjedai/blob/main/docs/tutorials/Readers.ipynb Displays the first two rows of the DataFrames created from RDF data using the `.head(2)` method. This allows for a quick check of the converted RDF triples. ```python d1.head(2) d2.head(2) ``` -------------------------------- ### Git Sync Fork with Upstream Source: https://github.com/ai-team-uoa/pyjedai/blob/main/docs/pages/contribution.md Synchronizes your local fork with the main repository by adding the upstream remote, fetching changes, merging them, and pushing to your fork. ```git git remote add upstream git fetch upstream git merge upstream/main git push origin main ``` -------------------------------- ### Import pandas Library Source: https://github.com/ai-team-uoa/pyjedai/blob/main/docs/tutorials/Readers.ipynb Imports the pandas library, which is essential for data manipulation and analysis in Python. This is a foundational step for most data reading operations. ```python import pandas as pd ``` -------------------------------- ### Display First Row of DataFrame Source: https://github.com/ai-team-uoa/pyjedai/blob/main/docs/tutorials/Readers.ipynb Shows the first row of a pandas DataFrame using the `.head(1)` method. This is useful for quickly inspecting the structure and content of the data after reading. ```python d1.head(1) ``` -------------------------------- ### Similarity Joins for Entity Resolution Source: https://context7.com/ai-team-uoa/pyjedai/llms.txt This section demonstrates similarity join techniques for entity resolution using pyJedAI. It includes examples of `EJoin` (threshold-based) and `TopKJoin` (top-K similar pairs), as well as integrating these into a `JoinWorkflow`. Dependencies include pyjedai.joins, pyjedai.workflow, and pyjedai.clustering. Various metrics and tokenizers can be specified. ```python from pyjedai.joins import EJoin, TopKJoin from pyjedai.workflow import JoinWorkflow from pyjedai.clustering import UniqueMappingClustering # EJoin - threshold-based similarity join ejoin = EJoin( similarity_threshold=0.5, metric='cosine', tokenizer='char_tokenizer', qgram=3 ) similarity_graph = ejoin.fit( data, attributes_1=['name', 'description'], attributes_2=['name', 'description'] ) ejoin.evaluate(similarity_graph, verbose=True) # TopKJoin - keeps top-K similar pairs topk = TopKJoin( K=10, metric='jaccard', tokenizer='white_space_tokenizer' ) graph = topk.fit(data) # Join-based Workflow join_workflow = JoinWorkflow( join=dict( method=EJoin, params=dict(similarity_threshold=0.5, metric='cosine'), attributes_1=['name'], attributes_2=['name'] ), clustering=dict(method=UniqueMappingClustering), name="JoinWorkflow" ) join_workflow.run(data, verbose=True) ``` -------------------------------- ### Plot Optimization History in Python Source: https://github.com/ai-team-uoa/pyjedai/blob/main/docs/tutorials/Optuna.ipynb Generates and displays a plot of the optimization history from a study object. This visualization is useful for tracking the progress of optimization trials over time. It typically shows metrics like trial number, objective values, and completion status. ```python fig = plot_optimization_history(study) fig.show() ``` -------------------------------- ### Plot Parameter Importances in Python Source: https://github.com/ai-team-uoa/pyjedai/blob/main/docs/tutorials/Optuna.ipynb Visualizes the importance of each parameter in an optimization study. This plot helps identify which parameters have the most significant impact on the objective function, guiding feature selection or parameter tuning efforts. ```python fig = plot_param_importances(study) fig.show() ``` -------------------------------- ### Initialize and Process with Connected Components Clustering Source: https://github.com/ai-team-uoa/pyjedai/blob/main/docs/tutorials/SimilarityJoins.ipynb Initializes the ConnectedComponentsClustering object and processes a similarity graph 'g' with associated data 'data' to generate equivalence clusters. A similarity threshold can be specified. ```python ec = ConnectedComponentsClustering() clusters = ec.process(g, data, similarity_threshold=0.3) ``` -------------------------------- ### Perform Similarity Join with EJoin Source: https://github.com/ai-team-uoa/pyjedai/blob/main/docs/tutorials/SimilarityJoins.ipynb Applies the EJoin algorithm for similarity joins. This involves specifying similarity parameters such as the threshold, metric, tokenization method, and q-gram size. The `fit` method then processes the data to generate candidate pairs. ```python from pyjedai.joins import EJoin join = EJoin(similarity_threshold = 0.5, metric = 'jaccard', tokenization = 'qgrams_multiset', qgrams = 2) g = join.fit(data) ``` -------------------------------- ### Configure and Run EmbeddingsNNWorkFlow with PyTorch Embeddings Source: https://github.com/ai-team-uoa/pyjedai/blob/main/docs/tutorials/WorkFlow.ipynb This snippet demonstrates how to initialize and execute an EmbeddingsNNWorkFlow. It configures block building with 'sminilm' vectorizer and 'faiss' similarity search, along with 'UniqueMappingClustering'. The workflow is then run on provided data. ```python w2 = EmbeddingsNNWorkFlow( block_building = dict( method=EmbeddingsNNBlockBuilding, params=dict(vectorizer='sminilm', similarity_search='faiss'), exec_params=dict(top_k=5, similarity_distance='euclidean', load_embeddings_if_exist=False, save_embeddings=False) ), clustering = dict(method=UniqueMappingClustering), name="EmbeddingsNNWorkFlow-Test" ) w2.run(data, verbose=True) ``` -------------------------------- ### Visualize Similarity Join Graph Source: https://github.com/ai-team-uoa/pyjedai/blob/main/docs/tutorials/SimilarityJoins.ipynb Visualizes the graph generated by the similarity join process using networkx. This helps in understanding the relationships and clusters identified by the join algorithm. ```python from networkx import draw draw(g) ``` -------------------------------- ### Uninstall pyJedAI using pip Source: https://github.com/ai-team-uoa/pyjedai/blob/main/docs/tutorials/LLMsMatching.ipynb Uninstalls the pyJedAI library from the current Python environment using pip. This is often done before installing a specific version or to ensure a clean installation. ```python !pip uninstall pyjedai -y ``` -------------------------------- ### Check Python Version Source: https://github.com/ai-team-uoa/pyjedai/blob/main/docs/tutorials/Spatial.ipynb Checks the installed Python version. This is useful for ensuring compatibility with the pyJedAI library. ```python !python --version ``` -------------------------------- ### Git Push Changes Source: https://github.com/ai-team-uoa/pyjedai/blob/main/docs/pages/contribution.md Pushes committed changes from a local branch to the remote repository. This makes your changes available for a pull request. ```git git push origin ``` -------------------------------- ### Initialize and Process with Valentine Schema Matching Source: https://github.com/ai-team-uoa/pyjedai/blob/main/docs/tutorials/SchemaMatching.ipynb This snippet demonstrates how to initialize the Valentine schema matching method using the Cupid matcher builder and then process the prepared PyJedAI Data object. It requires the pyjedai library and the Data object created previously. ```python from pyjedai.schema.matching import ValentineMethodBuilder, ValentineSchemaMatching sm = ValentineSchemaMatching(ValentineMethodBuilder.cupid_matcher()) sm.process(data) ``` -------------------------------- ### Git Add and Commit Changes Source: https://github.com/ai-team-uoa/pyjedai/blob/main/docs/pages/contribution.md Stages changes for commit and then commits them with a descriptive message. 'git add .' stages all changes in the current directory. ```git git add git commit -m "" git add . ``` -------------------------------- ### Import pyJedAI Utilities Source: https://github.com/ai-team-uoa/pyjedai/blob/main/docs/tutorials/SchemaMatching.ipynb Imports essential utilities from the pyJedAI library, including data handling, text cleaning, and printing functions for blocks, clusters, and candidate pairs. These are commonly used in schema matching tasks. ```python import os import sys import pandas as pd from pyjedai.utils import ( text_cleaning_method, print_clusters, print_blocks, print_candidate_pairs ) from pyjedai.evaluation import Evaluation from pyjedai.datamodel import Data ``` -------------------------------- ### Read JSON Data into DataFrame Source: https://github.com/ai-team-uoa/pyjedai/blob/main/docs/tutorials/Readers.ipynb Illustrates how to load data from a JSON file into a pandas DataFrame using `pd.read_json`. This method is suitable for JSON formatted data. ```python d1 = pd.read_json("../data/der/cora/cora.json") gt = pd.read_json("../data/der/cora/cora_gt.json") ``` -------------------------------- ### Run Blocking-Based Workflow - Python Source: https://github.com/ai-team-uoa/pyjedai/blob/main/docs/tutorials/WorkFlow.ipynb This code defines and runs a comprehensive entity resolution workflow using PyJedAI's BlockingBasedWorkFlow. It includes steps for block building (Q-Grams Blocking), block cleaning (Block Filtering, Block Purging), comparison cleaning (Cardinality Edge Pruning), entity matching, and clustering (Connected Components Clustering). ```python w1 = BlockingBasedWorkFlow( block_building = dict( method=QGramsBlocking, params=dict(qgrams=4), attributes_1=['name'], attributes_2=['name'] ), block_cleaning = [ dict( method=BlockFiltering, params=dict(ratio=0.6) ), dict( method=BlockPurging, params=dict(smoothing_factor=1.025) ) ], comparison_cleaning = dict(method=CardinalityEdgePruning), entity_matching = dict( method=EntityMatching, metric='sorensen_dice', similarity_threshold=0.5, attributes = ['description', 'name'] ), clustering = dict(method=ConnectedComponentsClustering) ) w1.run(data, verbose=False, workflow_tqdm_enable=True) ``` -------------------------------- ### Read Excel Data into DataFrame Source: https://github.com/ai-team-uoa/pyjedai/blob/main/docs/tutorials/Readers.ipynb Demonstrates reading data from an Excel file into a pandas DataFrame using `pd.read_excel`. This function can handle `.xls` and `.xlsx` file formats. ```python d1 = pd.read_excel("../data/der/cora/cora.xlsx") gt = pd.read_excel("../data/der/cora/cora_gt.xlsx") ``` -------------------------------- ### Initialize Data module Source: https://github.com/ai-team-uoa/pyjedai/blob/main/docs/tutorials/SimilarityJoins.ipynb Initializes the Data module from pyjedai, which serves as a central component for all subsequent workflow steps. It takes pandas DataFrames for the datasets, specifies the ID column, and provides attribute information. ```python from pyjedai.datamodel import Data d1 = pd.read_csv("./../data/der/cora/cora.csv", sep='|') gt = pd.read_csv("./../data/der/cora/cora_gt.csv", sep='|', header=None) attr = ['Entity Id','author', 'title'] data = Data( dataset_1=d1, id_column_name_1='Entity Id', ground_truth=gt, attributes_1=attr ) ``` -------------------------------- ### Display SPARQL Query Results Source: https://github.com/ai-team-uoa/pyjedai/blob/main/docs/tutorials/Readers.ipynb Displays the pandas DataFrame containing the results obtained from the DBpedia SPARQL query. This shows the retrieved information about Cristiano Ronaldo's birthplace. ```python d1 ``` -------------------------------- ### Read RDF Data and Convert to DataFrame Source: https://github.com/ai-team-uoa/pyjedai/blob/main/docs/tutorials/Readers.ipynb Loads RDF data from `.nt` files into `rdflib.Graph` objects and then converts them into pandas DataFrames. A helper function `rdf_to_df` is defined for this transformation. ```python import rdfpandas as rfd import rdflib rdfd1 = rdflib.Graph().parse('../data/rdf/restaurants/restaurant1.nt') rdfd2 = rdflib.Graph().parse('../data/rdf/restaurants/restaurant2.nt') def rdf_to_df(graph_parsed) -> pd.DataFrame: subject = [] predicate = [] rdfobject = [] df = pd.DataFrame(columns=['subject', 'predicate', 'object']) for s, p, o in graph_parsed: subject.append(s) predicate.append(p) rdfobject.append(o) df['predicate'] = predicate df['subject'] = subject df['object'] = rdfobject return df d1 = rdf_to_df(rdfd1) d2 = rdf_to_df(rdfd2) ``` -------------------------------- ### Configure and Run Blocking-Based Workflow Source: https://github.com/ai-team-uoa/pyjedai/blob/main/docs/tutorials/WorkFlow.ipynb Configures a BlockingBasedWorkFlow for entity resolution. This involves defining methods and parameters for block building, block cleaning, comparison cleaning, entity matching, and clustering. ```python w = BlockingBasedWorkFlow( block_building = dict( method=QGramsBlocking, params=dict(qgrams=3), attributes_1=['name'], attributes_2=['name'] ), block_cleaning = [ dict( method=BlockPurging, params=dict(smoothing_factor=1.025) ), dict( method=BlockFiltering, params=dict(ratio=0.8) ) ], comparison_cleaning = dict(method=CardinalityEdgePruning), entity_matching = dict( method=EntityMatching, metric='sorensen_dice', similarity_threshold=0.5, attributes = ['description', 'name'] ), clustering = dict(method=ConnectedComponentsClustering), name="Worflow-Test" ) ``` -------------------------------- ### Initialize Clustering Algorithms in Python Source: https://github.com/ai-team-uoa/pyjedai/blob/main/docs/tutorials/pyTorchWorkflow.ipynb Imports and initializes clustering algorithms from PyJEDAI, specifically ConnectedComponentsClustering and UniqueMappingClustering. These are used to group similar entities after block building. ```python from pyjedai.clustering import ConnectedComponentsClustering, UniqueMappingClustering ccc = UniqueMappingClustering() ``` -------------------------------- ### Run Default Blocking Workflow in Python Source: https://github.com/ai-team-uoa/pyjedai/blob/main/docs/tutorials/WorkFlow.ipynb Initializes and runs a blocking-based workflow using PyJedAI's best default configuration. This workflow is designed for efficient entity resolution by first blocking potential matches and then refining them. It requires the 'pyjedai' library and a 'data' object for processing. ```python w = BlockingBasedWorkFlow() w.best_blocking_workflow_ccer() w.run(data, verbose=True) ``` -------------------------------- ### Build Blocks with Pre-trained Embeddings Source: https://github.com/ai-team-uoa/pyjedai/blob/main/docs/tutorials/pyTorchWorkflow.ipynb Demonstrates how to build blocks for entity resolution using pre-trained embeddings from Gensim and pyTorch Sentence Transformers or Word Transformers. Custom models can also be specified. ```python # Available embeddings: # Gensim: { 'fasttext', 'glove', 'word2vec'} # pyTorch Sentence transformers : {'smpnet','st5','sdistilroberta','sminilm','sent_glove'} # pyTorch Word transformers :{'bert', 'distilbert', 'roberta', 'xlnet', 'albert'} # Custom Sentence transformers: vectorizer='model_name' and emb.build_blocks(..., custom_pretrained_model='sentence') # Custom Word transformers: vectorizer='model_name' and emb.build_blocks(..., custom_pretrained_model='word') ``` -------------------------------- ### Display First Row of SQLite Query Result Source: https://github.com/ai-team-uoa/pyjedai/blob/main/docs/tutorials/Readers.ipynb Shows the first row of a DataFrame queried from the SQLite database using `sql_d1.head(1)`. This verifies that data was loaded and can be retrieved correctly. ```python sql_d1.head(1) ``` -------------------------------- ### Build pyJedAI Docker image Source: https://github.com/ai-team-uoa/pyjedai/blob/main/README.md Builds a Docker image for pyJedAI. This allows for containerized deployment and ensures a consistent environment. The Dockerfile is available in the repository. ```bash docker build -f Dockerfile ``` -------------------------------- ### Import Connected Components Clustering from pyjedai Source: https://github.com/ai-team-uoa/pyjedai/blob/main/docs/tutorials/SimilarityJoins.ipynb Imports the necessary class for performing connected components clustering. This is the first step in utilizing the entity clustering functionality. ```python from pyjedai.clustering import ConnectedComponentsClustering ``` -------------------------------- ### Read Data from PostgreSQL into DataFrame Source: https://github.com/ai-team-uoa/pyjedai/blob/main/docs/tutorials/Readers.ipynb Reads data from a PostgreSQL table named 'd1' into a pandas DataFrame using `pd.read_sql` and the SQLAlchemy engine `cnx`. This demonstrates querying data from a remote database. ```python pd.read_sql('SELECT * FROM d1', cnx) ``` -------------------------------- ### Apply Blocking and Cleaning Techniques in PyJedAI Source: https://github.com/ai-team-uoa/pyjedai/blob/main/docs/tutorials/TfIdfWorkflow.ipynb Demonstrates the application of various blocking and cleaning methods within PyJedAI, including StandardBlocking, BlockPurging, BlockFiltering, and CardinalityEdgePruning. It also shows how to evaluate the results of these processes. ```python from pyjedai.block_building import StandardBlocking from pyjedai.block_cleaning import BlockFiltering from pyjedai.block_cleaning import BlockPurging from pyjedai.comparison_cleaning import WeightedEdgePruning, WeightedNodePruning, CardinalityEdgePruning, CardinalityNodePruning from pyjedai.matching import EntityMatching sb = StandardBlocking() blocks = sb.build_blocks(data) sb.evaluate(blocks, with_classification_report=True) cbbp = BlockPurging() blocks = cbbp.process(blocks, data, tqdm_disable=False) cbbp.evaluate(blocks, with_classification_report=True) bf = BlockFiltering(ratio=0.8) blocks = bf.process(blocks, data, tqdm_disable=False) wep = CardinalityEdgePruning(weighting_scheme='X2') candidate_pairs_blocks = wep.process(blocks, data) wep.evaluate(candidate_pairs_blocks, with_classification_report=True) ``` -------------------------------- ### Clean Dataset with PyJedAI (Optional) Source: https://github.com/ai-team-uoa/pyjedai/blob/main/docs/tutorials/CleanCleanER.ipynb Applies text cleaning operations to the dataset. This function allows for selective removal of stopwords, punctuation, numbers, and unicodes. The example shows disabling all cleaning options. ```python data.clean_dataset(remove_stopwords = False, remove_punctuation = False, remove_numbers = False, remove_unicodes = False) ``` -------------------------------- ### Initialize and Predict with EntityMatching in Python Source: https://github.com/ai-team-uoa/pyjedai/blob/main/docs/tutorials/CleanCleanER.ipynb Initializes the EntityMatching object with specified parameters like metric, tokenizer, vectorizer, qgram, and similarity threshold. It then predicts the similarity graph for candidate pairs. ```python em = EntityMatching( metric='cosine', tokenizer='char_tokenizer', vectorizer='tfidf', qgram=3, similarity_threshold=0.0 ) pairs_graph = em.predict(candidate_pairs_blocks, data, tqdm_disable=True) ``` -------------------------------- ### Initialize Entity Clustering in Python Source: https://github.com/ai-team-uoa/pyjedai/blob/main/docs/tutorials/Reproducibility.ipynb Initializes a clustering method, such as UniqueMappingClustering, for partitioning the similarity graph into equivalence clusters. The `process` method takes the similarity graph, data, and a similarity threshold as input. ```python from pyjedai.clustering import ConnectedComponentsClustering, UniqueMappingClustering ccc = UniqueMappingClustering() clusters = ccc.process(pairs_graph, data, similarity_threshold=0.17) ``` -------------------------------- ### Evaluate Entity Clustering Results Source: https://github.com/ai-team-uoa/pyjedai/blob/main/docs/tutorials/SimilarityJoins.ipynb Evaluates the generated clusters using the ConnectedComponentsClustering object. This step typically calculates performance metrics like precision, recall, and F1-score. ```python _ = ec.evaluate(clusters) ``` -------------------------------- ### Read CSV Data into DataFrame Source: https://github.com/ai-team-uoa/pyjedai/blob/main/docs/tutorials/Readers.ipynb Demonstrates reading data from a CSV file into a pandas DataFrame using `pd.read_csv`. It specifies the file path and the separator. The `gt` DataFrame is also read similarly, with no header specified. ```python d1 = pd.read_csv("../data/der/cora/cora.csv", sep='|') gt = pd.read_csv("../data/der/cora/cora_gt.csv", sep='|', header=None) ``` -------------------------------- ### Compare PyJedai Workflows with Visualization Source: https://github.com/ai-team-uoa/pyjedai/blob/main/docs/tutorials/WorkFlow.ipynb This code snippet illustrates how to compare multiple PyJedai workflows, including the previously configured EmbeddingsNNWorkFlow. It utilizes the `compare_workflows` function and enables visualization to present a comparative analysis of the workflows' performance metrics. ```python compare_workflows([w, w1, w2], with_visualization=True) ``` -------------------------------- ### Compare Multiple Workflows with Different Blocking Methods Source: https://context7.com/ai-team-uoa/pyjedai/llms.txt This snippet shows how to create and compare multiple workflow configurations using different blocking methods (StandardBlocking, QGramsBlocking, SuffixArraysBlocking). It iterates through blocking methods, configures BlockingBasedWorkFlow for each, runs them, and then compares their performance using `compare_workflows`. Dependencies include pyjedai.workflow, pyjedai.block_building, pyjedai.block_cleaning, pyjedai.comparison_cleaning, pyjedai.matching, and pyjedai.clustering. ```python from pyjedai.workflow import BlockingBasedWorkFlow, compare_workflows from pyjedai.block_building import StandardBlocking, QGramsBlocking, SuffixArraysBlocking from pyjedai.block_cleaning import BlockFiltering from pyjedai.comparison_cleaning import CardinalityEdgePruning from pyjedai.matching import EntityMatching from pyjedai.clustering import ConnectedComponentsClustering # Create multiple workflows with different block building methods block_methods = [StandardBlocking, QGramsBlocking, SuffixArraysBlocking] workflows = [] for method in block_methods: w = BlockingBasedWorkFlow( block_building=dict(method=method), block_cleaning=[dict(method=BlockFiltering, params=dict(ratio=0.8))], comparison_cleaning=dict(method=CardinalityEdgePruning), entity_matching=dict( method=EntityMatching, params=dict(metric='cosine', similarity_threshold=0.5) ), clustering=dict(method=ConnectedComponentsClustering), name=f"Workflow-{method.__name__}" ) w.run(data, workflow_tqdm_enable=True) workflows.append(w) # Compare all workflows comparison_df = compare_workflows(workflows, with_visualization=True) print(comparison_df) ``` -------------------------------- ### Plot Parameter Importances with Target Function in Python Source: https://github.com/ai-team-uoa/pyjedai/blob/main/docs/tutorials/Optuna.ipynb Generates a parameter importance plot for an optimization study, where the importance is calculated with respect to a custom target function. This allows for assessing parameter influence on specific aspects of the study, such as trial duration. ```python fig = optuna.visualization.plot_param_importances( study, target=lambda t: t.duration.total_seconds(), target_name="duration" ) fig.show() ``` -------------------------------- ### Query DBpedia SPARQL Endpoint and Normalize Results Source: https://github.com/ai-team-uoa/pyjedai/blob/main/docs/tutorials/Readers.ipynb Connects to the DBpedia SPARQL endpoint, executes a query to find Cristiano Ronaldo's birthplace, and converts the JSON results into a pandas DataFrame using `json_normalize`. Requires `SPARQLWrapper` and `pandas`. ```python from pandas import json_normalize from SPARQLWrapper import SPARQLWrapper, JSON sparql = SPARQLWrapper("http://dbpedia.org/sparql") sparql.setQuery(""" SELECT * WHERE { ?athlete rdfs:label "Cristiano Ronaldo"@en ; dbo:birthPlace ?place . ?place a dbo:City ; rdfs:label ?cityName . } """) sparql.setReturnFormat(JSON) results = sparql.query().convert() d1 = json_normalize(results["results"]["bindings"]) ``` -------------------------------- ### Read Data with Pandas and Format for PyJedAI Source: https://github.com/ai-team-uoa/pyjedai/blob/main/docs/tutorials/SchemaMatching.ipynb This snippet shows how to read two datasets and a ground truth file using pandas. It then formats this data into a PyJedAI Data object, specifying the datasets, relevant attributes, and ID column names for each dataset. ```python import pandas as pd from pyjedai.utils import Data d1 = pd.read_csv(r"C:\\Users\\nikol\\Desktop\\GitHub\\pyJedAI-Dev\\data\\ccer\\schema_matching\\authors\\authors1.csv") d2 = pd.read_csv(r"C:\\Users\\nikol\\Desktop\\GitHub\\pyJedAI-Dev\\data\\ccer\\schema_matching\\authors\\authors2.csv") gt = pd.read_csv(r"C:\\Users\\nikol\\Desktop\\GitHub\\pyJedAI-Dev\\data\\ccer\\schema_matching\\authors\\pairs.csv") data = Data( dataset_1=d1, attributes_1=['EID','Authors','Cited by','Title','Year','Source tittle','DOI'], id_column_name_1='EID', dataset_2=d2, attributes_2=['EID','Authors','Cited by','Country','Document Type','City','Access Type','aggregationType'], id_column_name_2='EID', ground_truth=gt, ) ``` -------------------------------- ### Plot Contour Visualization in Python Source: https://github.com/ai-team-uoa/pyjedai/blob/main/docs/tutorials/Optuna.ipynb Renders and displays a contour plot for an optimization study. Contour plots are effective for visualizing the objective function's landscape in a 2D parameter space, helping to identify optima and understand the function's shape. ```python fig = plot_contour(study) fig.show() ``` -------------------------------- ### Import Core pyJedAI Libraries Source: https://github.com/ai-team-uoa/pyjedai/blob/main/docs/tutorials/pyTorchWorkflow.ipynb Imports essential libraries for pyJedAI workflows, including pandas for data manipulation, networkx for graph operations, and specific modules from pyJedAI for evaluation and data modeling. ```python import os import sys import pandas as pd import networkx from networkx import draw, Graph from pyjedai.evaluation import Evaluation from pyjedai.datamodel import Data ``` -------------------------------- ### Load and Prepare Datasets for pyJedAI Source: https://github.com/ai-team-uoa/pyjedai/blob/main/docs/tutorials/pyTorchWorkflow.ipynb Loads datasets from CSV files using pandas and prepares them for pyJedAI. It specifies attribute columns, ID columns, and a ground truth dataset for evaluation. ```python d1 = pd.read_csv("../data/ccer/D2/abt.csv", sep='|', engine='python', na_filter=False).astype(str) d2 = pd.read_csv("../data/ccer/D2/buy.csv", sep='|', engine='python', na_filter=False).astype(str) gt = pd.read_csv("../data/ccer/D2/gt.csv", sep='|', engine='python') attr1 = d1.columns[1:].to_list() attr2 = d2.columns[1:].to_list() data = Data( dataset_1=d1, attributes_1=attr1, id_column_name_1='id', dataset_2=d2, attributes_2=attr2, id_column_name_2='id', ground_truth=gt ) ``` -------------------------------- ### Plot Parallel Coordinate Visualization in Python Source: https://github.com/ai-team-uoa/pyjedai/blob/main/docs/tutorials/Optuna.ipynb Creates and displays a parallel coordinate plot for an optimization study. This plot helps visualize the relationships between different parameters and their corresponding objective values across multiple trials. It can be used to identify promising parameter subspaces. ```python fig = plot_parallel_coordinate(study) fig.show() ``` -------------------------------- ### Get Workflow Results and Performance Metrics Source: https://context7.com/ai-team-uoa/pyjedai/llms.txt Demonstrates how to retrieve workflow results as a pandas DataFrame, extract final performance scores (F1, precision, recall), and visualize the workflow's performance. Assumes a 'workflow' object has been previously defined and run. ```python results_df = workflow.to_df() print(results_df) f1, precision, recall = workflow.get_final_scores() print(f"F1: {f1:.2f}%, Precision: {precision:.2f}%, Recall: {recall:.2f}%") workflow.visualize(separate=True) final_pairs_df = workflow.export_pairs() ``` -------------------------------- ### Initialize FAISS Embeddings Block Building in Python Source: https://github.com/ai-team-uoa/pyjedai/blob/main/docs/tutorials/pyTorchWorkflow.ipynb Initializes the EmbeddingsNNBlockBuilding class for similarity search using FAISS. It specifies the vectorizer ('sminilm') and similarity search method ('faiss'). This is a prerequisite for building blocks based on nearest neighbors. ```python from pyjedai.vector_based_blocking import EmbeddingsNNBlockBuilding emb = EmbeddingsNNBlockBuilding(vectorizer='sminilm', similarity_search='faiss') ``` -------------------------------- ### Plot Parallel Coordinate with Specific Parameters in Python Source: https://github.com/ai-team-uoa/pyjedai/blob/main/docs/tutorials/Optuna.ipynb Generates a parallel coordinate plot for an optimization study, focusing on a specified subset of parameters. This allows for a more targeted analysis of how certain parameters influence the optimization outcome. The `params` argument takes a list of parameter names. ```python fig = plot_parallel_coordinate(study, params=["qgrams"]) fig.show() ``` -------------------------------- ### Comparison Cleaning Methods in PyJedAI Source: https://github.com/ai-team-uoa/pyjedai/blob/main/docs/tutorials/CleanCleanERwithoutGT.ipynb Demonstrates the initialization of various comparison cleaning methods, such as CardinalityEdgePruning, for finer granularity cleaning of comparisons. These methods can be combined with different weighting schemes to refine the comparison set. ```python from pyjedai.comparison_cleaning import ( WeightedEdgePruning, WeightedNodePruning, CardinalityEdgePruning, CardinalityNodePruning, BLAST, ReciprocalCardinalityNodePruning, ReciprocalWeightedNodePruning, ComparisonPropagation ) ``` ```python wep = CardinalityEdgePruning(weighting_scheme='X2') candidate_pairs_blocks = wep.process(filtered_blocks, data, tqdm_disable=True) ``` -------------------------------- ### Plot Empirical Distribution Function (EDF) in Python Source: https://github.com/ai-team-uoa/pyjedai/blob/main/docs/tutorials/Optuna.ipynb Displays the empirical distribution function (EDF) of the objective values from an optimization study. The EDF plot provides insights into the distribution of performance across all trials, showing the proportion of trials that achieved a certain objective value or lower. ```python fig = plot_edf(study) fig.show() ``` -------------------------------- ### Print Schema Matching Results Source: https://github.com/ai-team-uoa/pyjedai/blob/main/docs/tutorials/SchemaMatching.ipynb This snippet prints the schema matching results to the console. This is useful for a quick inspection of the matches found by the Valentine method. ```python sm.print_matches() ``` -------------------------------- ### Initialize TopKJoin for Candidate Pair Filtering (Python) Source: https://github.com/ai-team-uoa/pyjedai/blob/main/docs/tutorials/LLMsMatching.ipynb Initializes the TopKJoin class from pyjedai for filtering candidate pairs. It configures the join with parameters like K (number of top-k results), similarity metric, and tokenization method. This step is crucial for reducing the number of pairs to be processed by more computationally intensive methods. ```python from pyjedai.joins import TopKJoin join = TopKJoin(K=5, metric='cosine', tokenization='qgrams_multiset', qgrams=5) graph = join.fit(data) ``` -------------------------------- ### Plot Slice Visualization in Python Source: https://github.com/ai-team-uoa/pyjedai/blob/main/docs/tutorials/Optuna.ipynb Generates and displays a slice plot for an optimization study, focusing on a specified set of parameters. Slice plots visualize the objective function's behavior by fixing all but two parameters and plotting the remaining two, which is useful for exploring high-dimensional spaces. ```python fig = plot_slice(study, params=["qgrams", "ratio"]) fig.show() ```