Wikidata Integrator

https://github.com/sulab/wikidataintegrator
Admin
Wikidata Integrator is a Python library for reading and writing to Wikidata/Wikibase that enables...
Tokens:19,248
Snippets:167
Trust Score:9.6
License:MIT
Update:6 months ago
Show doc for...
Tokens:
Raw
### Initialize WikidataIntegrator and RDF Graph

Source: https://github.com/sulab/wikidataintegrator/blob/main/notebooks/Wikidata_json2ttl.ipynb

Setup the environment by installing the library and importing necessary RDF and Wikidata modules.

```python
%%capture 
!pip install wikidataintegrator
from rdflib import Graph, URIRef, Literal, Namespace, BNode
from rdflib.namespace import RDF, RDFS, SKOS, XSD, OWL, PROV
from wikidataintegrator import wdi_core, wdi_config
import uuid
import urllib.parse
```

--------------------------------

### Install and Import WikidataIntegrator

Source: https://github.com/sulab/wikidataintegrator/blob/main/notebooks/CreateWikidataProperties.ipynb

Install the library and import core modules for interaction.

```python
%%capture
!pip install wikidataintegrator
from wikidataintegrator import wdi_core, wdi_login, wdi_config
```

--------------------------------

### Install WikidataIntegrator from source

Source: https://github.com/sulab/wikidataintegrator/blob/main/README.md

Clone the repository and install the WikidataIntegrator library from source. This method requires administrator rights or installation into a virtual environment.

```bash
git clone https://github.com/SuLab/WikidataIntegrator.git

cd WikidataIntegrator

python3 setup.py install
```

--------------------------------

### Install wikidataintegrator

Source: https://github.com/sulab/wikidataintegrator/blob/main/notebooks/LoginWikidata.ipynb

Install the library using pip within a Jupyter environment.

```python
%%capture
!pip install wikidataintegrator
```

--------------------------------

### Install WikidataIntegrator using pip

Source: https://github.com/sulab/wikidataintegrator/blob/main/README.md

Install the WikidataIntegrator library using pip. This is the recommended method for Python 3.8 and higher.

```bash
pip3 install wikidataintegrator
```

--------------------------------

### Parallel Processing Setup

Source: https://github.com/sulab/wikidataintegrator/blob/main/notebooks/CreateWikidataPropertiesParallel.ipynb

Imports necessary libraries for parallel processing using joblib and multiprocessing. This setup is typically used for optimizing performance on multi-core systems.

```python
from joblib import Parallel, delayed
import multiprocessing
     
# what are your inputs, and what operation do you want to 
```

--------------------------------

### Write Data Using Fast Run Mode

Source: https://github.com/sulab/wikidataintegrator/blob/main/README.md

This example demonstrates writing data to Wikidata using the fast run mode. It includes setting up the login instance, defining the fast run filter, and iterating over raw data to create and write WDItemEngine objects.

```Python
from wikidataintegrator import wdi_core, wdi_login

# login object
login_instance = wdi_login.WDLogin(user='<bot user name>', pwd='<bot password>')

fast_run_base_filter = {'P351': '', 'P703': 'Q15978631'}
fast_run = True

# We have raw data, which should be written to Wikidata, namely two human NCBI entrez gene IDs mapped to two Ensembl Gene IDs
# You can iterate over any data source as long as you can map the values to Wikidata properties.
raw_data = {
    '50943': 'ENST00000376197',
    '1029': 'ENST00000498124'
}

for entrez_id, ensembl in raw_data.items():
    # data type object
    entrez_gene_id = wdi_core.WDString(value=entrez_id, prop_nr='P351')
    ensembl_transcript_id = wdi_core.WDString(value=ensembl, prop_nr='P704')
    
    # data goes into a list, because many data objects can be provided to 
    data = [entrez_gene_id, ensembl_transcript_id]
    
    # Search for and then edit/create new item
    wd_item = wdi_core.WDItemEngine(data=data, fast_run=fast_run, fast_run_base_filter=fast_run_base_filter)
    wd_item.write(login_instance)
```

--------------------------------

### Test WikidataIntegrator Installation

Source: https://github.com/sulab/wikidataintegrator/blob/main/README.md

Verify the correct installation of WikidataIntegrator by importing `wdi_core` and retrieving a Wikidata item. This snippet retrieves the item for 'Human' (Q5) and prints its JSON representation.

```python
from wikidataintegrator import wdi_core

my_first_wikidata_item = wdi_core.WDItemEngine(wd_item_id='Q5')

# to check successful installation and retrieval of the data, you can print the json representation of the item
my_first_wikidata_item.get_wd_json_representation()
```

--------------------------------

### Create Publication Item from DOI

Source: https://github.com/sulab/wikidataintegrator/blob/main/README.md

This example demonstrates creating a Wikidata item for a publication using its DOI. It utilizes the PublicationHelper class and requires a login object.

```python
from wikidataintegrator import wdi_core, wdi_login, wdi_helpers
DOI = "<DOI>"
WDUSER = "<wikidata user>"
WDPASS = "<wikidata password>"
login = wdi_login.WDLogin(WDUSER, WDPASS)
print(wdi_helpers.PublicationHelper(DOI, id_type="doi", source="crossref").get_or_create(login))
```

--------------------------------

### Connect to Custom Wikibase Instances

Source: https://context7.com/sulab/wikidataintegrator/llms.txt

Configure Wikidata Integrator to work with custom Wikibase installations by providing the MediaWiki API, SPARQL endpoint, and Wikibase URLs during login and item engine creation. A factory method is available to create reusable engine classes for your specific instance.

```python
from wikidatator import wdi_core, wdi_login

# Login to custom Wikibase
login = wdi_login.WDLogin(
    user='admin',
    pwd='password',
    mediawiki_api_url='https://myinstance.wikibase.cloud/w/api.php'
)

# Create item on custom Wikibase
data = [wdi_core.WDString('test value', 'P1')]
item = wdi_core.WDItemEngine(
    data=data,
    mediawiki_api_url='https://myinstance.wikibase.cloud/w/api.php',
    sparql_endpoint_url='https://myinstance.wikibase.cloud/query/sparql',
    wikibase_url='https://myinstance.wikibase.cloud'
)

# Create a reusable engine class for your Wikibase
LocalItemEngine = wdi_core.WDItemEngine.wikibase_item_engine_factory(
    mediawiki_api_url='https://myinstance.wikibase.cloud/w/api.php',
    sparql_endpoint_url='https://myinstance.wikibase.cloud/query/sparql',
    name='MyWikibaseEngine'
)

# Use the custom engine
item = LocalItemEngine(data=data)
```

--------------------------------

### Execute SPARQL query and get results

Source: https://github.com/sulab/wikidataintegrator/blob/main/notebooks/RunSparqlQueryWDQS.ipynb

Execute the composed SPARQL query using WDFunctionsEngine. This returns the results as a Python dictionary.

```python
wdi_core.WDFunctionsEngine.execute_sparql_query(query)
```

--------------------------------

### Creating a local WDItemEngine factory

Source: https://github.com/sulab/wikidataintegrator/blob/main/doc/non_wikidata_wikibase.ipynb

Initialize a factory class configured for the local Wikibase instance.

```python
# create a WDItemEngine class with the urls preset
LocalItemEngine = WDItemEngine.wikibase_item_engine_factory(mediawiki_api_url, sparql_endpoint_url)
```

--------------------------------

### Login with Username and Password

Source: https://github.com/sulab/wikidataintegrator/blob/main/README.md

Instantiate WDLogin with bot username and password for authentication. This method is also usable with normal user accounts, but with lower writing limits.

```Python
login_instance = wdi_login.WDLogin(user='<bot user name>', pwd='<bot password>')
```

--------------------------------

### Create Database Release Items

Source: https://context7.com/sulab/wikidataintegrator/llms.txt

Use the Release helper to create Wikidata items for database releases, useful for references. Requires details like title, edition, and publication date.

```python
from wikidatator import wdi_login
from wikidatator.wdi_helpers import Release
import datetime

login = wdi_login.WDLogin(user='BotUsername', pwd='BotPassword')

# Create a database release item
release = Release(
    title='Ensembl Release 110',
    description='Release 110 of Ensembl',
    edition='110',
    edition_of_wdid='Q1344256',  # Ensembl QID
    archive_url='https://jul2023.archive.ensembl.org/',
    pub_date=datetime.date(2023, 7, 1),
    date_precision=10  # Month precision
)

# Get existing or create new release item
release_qid = release.get_or_create(login)
print(f'Release QID: {release_qid}')

# Get all releases for a database
all_releases = release.get_all_releases()
for edition, qid in all_releases.items():
    print(f'Edition {edition}: {qid}')
```

--------------------------------

### Define property creation function

Source: https://github.com/sulab/wikidataintegrator/blob/main/notebooks/CreateWikidataProperties.ipynb

Initializes a property creation function using WDItemEngine. Requires pre-configured login, api, and sparql objects.

```python
def createProperty(login=login, wdprop=None, label="", description="", property_datatype=""):
    if wdprop== None:
        s = []
    else:
        s = [wdi_core.WDUrl(wdprop, prop_nr="P1")]
    localEntityEngine = wdi_core.WDItemEngine.wikibase_item_engine_factory(api,sparql)
    item = localEntityEngine(data=s)
    item.set_label(label)
    item.set_description(description)
    print(item.write(login, entity_type="property", property_datatype=property_datatype))

#create mapping to wikidata property

#createProperty(login, label="property in wikidata", description="The same property in Wikidata", property_datatype="url")
```

--------------------------------

### Configure Wikidata Login

Source: https://github.com/sulab/wikidataintegrator/blob/main/notebooks/CreateWikidataPropertiesParallel.ipynb

Sets up connection details for a local wikibase instance and creates a login object. Ensure the API, SPARQL endpoint, and entity URI are correctly configured for your environment.

```python
from getpass import getpass
import pprint

wikibase = "https://demowb2.wiki.opencura.com/"
api = "http://localhost:8181/w/api.php"
sparql = "http://localhost:8282/proxy/wdqs/bigdata/namespace/wdq/sparql"
entityUri = wikibase.replace("https:", "http:")+"entity/"

WBUSER = "WikibaseAdmin"
WBPASS = "WikibaseDockerAdminPass"
login = wdi_login.WDLogin(WBUSER, WBPASS, mediawiki_api_url=api)
```

--------------------------------

### Authenticate with Wikidata

Source: https://github.com/sulab/wikidataintegrator/blob/main/notebooks/LoginWikidata.ipynb

Prompt the user for credentials and initialize a login session.

```python
WBUSER = getpass(prompt="username:")  
WBPASS = getpass(prompt='Enter your password: ')  
login = wdi_login.WDLogin(WBUSER, WBPASS)
```

--------------------------------

### Retrieve Wikidata Item JSON

Source: https://github.com/sulab/wikidataintegrator/blob/main/notebooks/gene-validity.ipynb

Initializes a WDItemEngine instance for a given QID and retrieves its full JSON representation.

```python
#wikidata_item = wdi_core.WDItemEngine(wd_item_id=HGNC_qid,data=statement, append_value=["P2293"])
wikidata_item = wdi_core.WDItemEngine(wd_item_id=HGNC_qid)
wikidata_item.get_wd_json_representation()
```

--------------------------------

### Retrieving items from local and public instances

Source: https://github.com/sulab/wikidataintegrator/blob/main/doc/non_wikidata_wikibase.ipynb

Demonstrates fetching labels from a local Wikibase item versus a standard Wikidata item.

```python
# we can now retrieve this item from our local wikibase
litem = LocalItemEngine(wd_item_id="Q2")
litem.get_label()
```

```python
# and can also still access regular wikidata
item = WDItemEngine(wd_item_id="Q2")
item.get_label()
```

--------------------------------

### Authenticate with WDLogin

Source: https://context7.com/sulab/wikidataintegrator/llms.txt

Handles authentication via username/password, OAuth, or custom Wikibase instances.

```python
from wikidataintegrator import wdi_login

# Basic username/password login
login = wdi_login.WDLogin(user='MyBotUsername', pwd='MyBotPassword')

# OAuth login (two-step process)
login = wdi_login.WDLogin(
    consumer_key='your_consumer_key',
    consumer_secret='your_consumer_secret'
)
login.continue_oauth()  # Will prompt for callback URL

# Login to a custom Wikibase instance
login = wdi_login.WDLogin(
    user='username',
    pwd='password',
    mediawiki_api_url='https://myinstance.wikibase.cloud/w/api.php'
)

# Get edit token for manual API calls
edit_token = login.get_edit_token()
session = login.get_session()
```

--------------------------------

### Import required libraries for Wikidata integration

Source: https://github.com/sulab/wikidataintegrator/blob/main/notebooks/gene-validity.ipynb

Initializes the necessary modules from wikidataintegrator and standard data science libraries.

```python
from wikidataintegrator import wdi_core, wdi_login
from wikidataintegrator.ref_handlers import update_retrieved_if_new_multiple_refs
import pandas as pd
import numpy as np
import requests
```

--------------------------------

### Authenticate with Wikidata API

Source: https://github.com/sulab/wikidataintegrator/blob/main/notebooks/gene-validity.ipynb

Use the WDLogin class to establish a session with the Wikidata API. Ensure credentials are provided as strings or retrieved from environment variables.

```python
print("Logging in...")
#if "WDUSER" in os.environ and "WDPASS" in os.environ:
#    WDUSER = os.environ['WDUSER']
#    WDPASS = os.environ['WDPASS']
#else:
#    raise ValueError("WDUSER and WDPASS must be specified in local.py or as environment variables")
login = wdi_login.WDLogin("<username>", "<password>")
```

--------------------------------

### Initialize Imports

Source: https://github.com/sulab/wikidataintegrator/blob/main/notebooks/setDescription.ipynb

Required imports for the wikidataintegrator library and data handling.

```python
from wikidataintegrator import wdi_core, wdi_login
import os
import pandas as pd
import pprint
```

--------------------------------

### Authenticating with a local Wikibase

Source: https://github.com/sulab/wikidataintegrator/blob/main/doc/non_wikidata_wikibase.ipynb

Perform login against the local MediaWiki API using WDLogin.

```python
# to login:
local_login = wdi_login.WDLogin("test", "password", mediawiki_api_url)
```

--------------------------------

### Import wikidataintegrator modules

Source: https://github.com/sulab/wikidataintegrator/blob/main/notebooks/LoginWikidata.ipynb

Import the core and login modules required for interacting with Wikidata.

```python
from wikidataintegrator import wdi_core, wdi_login
from getpass import getpass
```

--------------------------------

### Bind Namespaces to RDF Graph

Source: https://github.com/sulab/wikidataintegrator/blob/main/notebooks/Wikidata_json2ttl.ipynb

Initialize an RDF graph and bind namespaces defined in the WikidataIntegrator configuration.

```python
rdf_item = Graph()
ns = dict()
for prefix in wdi_config.prefix.keys():
    ns[prefix] = Namespace( wdi_config.prefix[prefix])
    rdf_item.namespace_manager.bind(prefix, ns[prefix])
```

--------------------------------

### Import wikidataintegrator core

Source: https://github.com/sulab/wikidataintegrator/blob/main/notebooks/RunSparqlQueryWDQS.ipynb

Import the necessary core module from the wikidataintegrator library to begin using its functionalities.

```python
from wikidataintegrator import wdi_core
```

--------------------------------

### Write and Log Simultaneously

Source: https://github.com/sulab/wikidataintegrator/blob/main/README.md

The try_write helper method simplifies writing data and logging actions concurrently. It captures exceptions and logs them, formatting messages using wdi_helpers.format_msg.

```python
wdi_helpers.try_write(item_to_write, login_obj, write_metadata)

```

--------------------------------

### Configuring local Wikibase URLs

Source: https://github.com/sulab/wikidataintegrator/blob/main/doc/non_wikidata_wikibase.ipynb

Define the MediaWiki API and SPARQL endpoint URLs for the local instance.

```python
# set urls
mediawiki_api_url = 'http://localhost:8181/w/api.php'
sparql_endpoint_url = 'http://localhost:8282/proxy/wdqs/bigdata/namespace/wdq/sparql'
```

--------------------------------

### Minimal Wikidata Bot

Source: https://github.com/sulab/wikidataintegrator/blob/main/README.md

Creates a minimal bot to interact with Wikidata. Requires a login instance, a data type object with a value, and a WDItemEngine object.

```Python
from wikidataintegrator import wdi_core, wdi_login
        
# login object
login_instance = wdi_login.WDLogin(user='<bot user name>', pwd='<bot password>')
        
# data type object, e.g. for a NCBI gene entrez ID
entrez_gene_id = wdi_core.WDString(value='<some_entrez_id>', prop_nr='P351')
    
# data goes into a list, because many data objects can be provided to 
data = [entrez_gene_id]
    
# Search for and then edit/create new item
wd_item = wdi_core.WDItemEngine(data=data)
wd_item.write(login_instance)
```

--------------------------------

### wikidataintegrator.wdi_helpers.release

Source: https://github.com/sulab/wikidataintegrator/blob/main/docs/source/wikidataintegrator.wdi_helpers.md

This module provides helper functions for managing releases.

```APIDOC
## wikidataintegrator.wdi_helpers.release

### Description
Helper functions for managing releases.

### Module Contents
This module contains functions related to releases.
```

--------------------------------

### Create New Wikidata Items

Source: https://context7.com/sulab/wikidataintegrator/llms.txt

Define data statements and create new items, optionally setting labels and descriptions.

```python
from wikidataintegrator import wdi_core, wdi_login

login = wdi_login.WDLogin(user='BotUsername', pwd='BotPassword')

# Create data statements for a new gene item
statements = [
    wdi_core.WDExternalID(value='1017', prop_nr='P351'),      # Entrez Gene ID
    wdi_core.WDExternalID(value='CDK2', prop_nr='P353'),      # Gene symbol
    wdi_core.WDItemID(value='Q15978631', prop_nr='P703'),     # Found in taxon: Homo sapiens
    wdi_core.WDItemID(value='Q7187', prop_nr='P31'),          # Instance of: gene
]

# Create item - WDI will search for existing item or create new one
item = wdi_core.WDItemEngine(data=statements, new_item=True)
item.set_label('CDK2', lang='en')
item.set_description('human gene', lang='en')
item.set_aliases(['cyclin-dependent kinase 2', 'p33 protein kinase'], lang='en')

# Write to Wikidata
qid = item.write(login, edit_summary='Creating CDK2 gene item')
print(f'Created item: {qid}')
```

--------------------------------

### Fetch Multiple Wikidata Items

Source: https://context7.com/sulab/wikidataintegrator/llms.txt

Use WDItemEngine.generate_item_instances to fetch multiple Wikidata items in a single API call. The 'login' parameter is optional and allows fetching more than 50 items.

```python
qids = ['Q5', 'Q7187', 'Q8054', 'Q11173']
items = wdi_core.WDItemEngine.generate_item_instances(
    items=qids,
    login=login  # Optional, allows fetching >50 items
)

for qid, item in items:
    label = item.get_label()
    print(f'{qid}: {label}')
```

--------------------------------

### Fastrun Mode for Bulk Updates

Source: https://context7.com/sulab/wikidataintegrator/llms.txt

Efficiently update Wikidata items by preloading data locally with fastrun mode. It checks if an update is needed before writing to Wikidata.

```python
from wikidatator import wdi_core, wdi_login

login = wdi_login.WDLogin(user='BotUsername', pwd='BotPassword')

# Define filter for human genes (has Entrez ID and taxon = human)
fast_run_base_filter = {
    'P351': '',           # Has Entrez Gene ID (any value)
    'P703': 'Q15978631'   # Found in taxon: Homo sapiens
}

# Data to sync from external source
gene_data = {
    '1017': {'symbol': 'CDK2', 'ensembl': 'ENSG00000123374'},
    '1029': {'symbol': 'CDKN2A', 'ensembl': 'ENSG00000147889'},
    '7157': {'symbol': 'TP53', 'ensembl': 'ENSG00000141510'},
}

for entrez_id, info in gene_data.items():
    statements = [
        wdi_core.WDExternalID(entrez_id, 'P351'),           # Entrez ID
        wdi_core.WDString(info['symbol'], 'P353'),         # Gene symbol
        wdi_core.WDExternalID(info['ensembl'], 'P594'),    # Ensembl gene ID
    ]

    # Fastrun checks if update is needed before writing
    item = wdi_core.WDItemEngine(
        data=statements,
        fast_run=True,
        fast_run_base_filter=fast_run_base_filter
    )

    if item.require_write:
        print(f'Updating {entrez_id}')
        item.write(login, edit_summary='Sync gene data')
    else:
        print(f'Skipping {entrez_id} - already up to date')

```

--------------------------------

### Use String and External ID Data Types

Source: https://context7.com/sulab/wikidataintegrator/llms.txt

Create claims using WDString and WDExternalID, including support for references.

```python
from wikidataintegrator import wdi_core

# Simple string value
string_claim = wdi_core.WDString(
    value='Some text value',
    prop_nr='P1448'  # Official name
)

# External identifier with references
reference = [
    wdi_core.WDItemID('Q905695', 'P248', is_reference=True),  # Stated in: UniProt
    wdi_core.WDTime('+2024-01-15T00:00:00Z', 'P813', is_reference=True)  # Retrieved date
]

external_id = wdi_core.WDExternalID(
    value='P12345',
    prop_nr='P352',  # UniProt ID
    references=[reference]
)
```

--------------------------------

### Authenticate Bot Account

Source: https://github.com/sulab/wikidataintegrator/blob/main/notebooks/setDescription.ipynb

Logs into Wikidata using environment variables WDUSER and WDPASS.

```python
print("Logging in...")
if "WDUSER" in os.environ and "WDPASS" in os.environ:
    WDUSER = os.environ['WDUSER']
    WDPASS = os.environ['WDPASS']
else:
    raise ValueError("WDUSER and WDPASS must be specified in local.py or as environment variables")
login = wdi_login.WDLogin(WDUSER, WDPASS)
```

--------------------------------

### Add references and qualifiers to statements

Source: https://context7.com/sulab/wikidataintegrator/llms.txt

Statements can be enriched with provenance via references and context via qualifiers.

```python
from wikidataintegrator import wdi_core

# Create reference block
reference = [
    wdi_core.WDItemID('Q905695', 'P248', is_reference=True),        # Stated in: UniProt
    wdi_core.WDExternalID('P12345', 'P352', is_reference=True),    # UniProt ID
    wdi_core.WDTime('+2024-01-15T00:00:00Z', 'P813', is_reference=True)  # Retrieved
]

# Create qualifiers
start_time = wdi_core.WDTime('+2020-01-01T00:00:00Z', 'P580', is_qualifier=True)
end_time = wdi_core.WDTime('+2024-12-31T00:00:00Z', 'P582', is_qualifier=True)

# Statement with references and qualifiers
statement = wdi_core.WDItemID(
    value='Q15978631',
    prop_nr='P703',
    references=[reference],
    qualifiers=[start_time, end_time]
)
```

--------------------------------

### Write Items with Logging and Error Handling

Source: https://context7.com/sulab/wikidataintegrator/llms.txt

The try_write function facilitates writing items to Wikidata with automatic logging and error handling. Ensure you have set up logging and have valid login credentials.

```python
from wikidatator import wdi_core, wdi_login
from wikidatator.wdi_helpers import try_write

login = wdi_login.WDLogin(user='BotUsername', pwd='BotPassword')

# Setup logging
wdi_core.WDItemEngine.setup_logging(
    log_dir='./logs',
    log_name='my_bot_run.log'
)

# Create item
data = [wdi_core.WDExternalID('1017', 'P351')]
item = wdi_core.WDItemEngine(data=data)

# Write with logging (returns True on success, Exception on failure)
success = try_write(
    wd_item=item,
    record_id='1017',
    record_prop='P351',
    login=login,
    edit_summary='Bot edit',
    write=True
)

if success is True:
    print('Write successful')
else:
    print(f'Write failed: {success}')
```

--------------------------------

### Importing wikidataintegrator modules

Source: https://github.com/sulab/wikidataintegrator/blob/main/doc/non_wikidata_wikibase.ipynb

Required imports for core functionality and authentication.

```python
from wikidataintegrator import wdi_core, wdi_login, wdi_helpers
from wikidataintegrator.wdi_core import WDItemEngine
```

--------------------------------

### Query Wikidata Properties via SPARQL

Source: https://github.com/sulab/wikidataintegrator/blob/main/notebooks/CreateWikidataProperties.ipynb

Execute a SPARQL query to retrieve property metadata and map datatypes using the configuration module.

```python
sparql = "https://data.lexbib.org/query/sparql"

query = """
SELECT DISTINCT ?property ?propertyLabel ?propertyDescription ?propType WHERE {
   ?property wikibase:directClaim ?p ;
             wikibase:propertyType ?propType ;
             schema:description ?propertyDescription ;
             rdfs:label ?propertyLabel .
   FILTER (lang(?propertyLabel) = 'en')
   FILTER (lang(?propertyDescription) = 'en')}
"""
propertiesSparql = wdi_core.WDItemEngine.execute_sparql_query(query, endpoint=sparql, as_dataframe=True)
propertiesSparql

datatype_map = {}
for datatype in wdi_config.property_value_types.keys():
    datatype_map[wdi_config.property_value_types[datatype]] = datatype
propertiesSparql['datatype']= ""
for index, row in propertiesSparql.iterrows():
    row["datatype"] = datatype_map[row["propType"]] 
propertiesSparql 
```

--------------------------------

### Authenticate with Wikibase

Source: https://github.com/sulab/wikidataintegrator/blob/main/notebooks/CreateWikidataProperties.ipynb

Configure API endpoints and perform login using credentials.

```python
from getpass import getpass
import pprint

# https://data.lexbib.org
wikibase = "https://lexbibclone.wiki.opencura.com/"
api = wikibase+"w/api.php"
sparql = wikibase+"query/sparql"
entityUri = wikibase.replace("https:", "http:")+"entity/"

WBUSER = getpass(prompt="username:")  
WBPASS = getpass(prompt='Enter your password: ')  
login = wdi_login.WDLogin(WBUSER, WBPASS, mediawiki_api_url=api)
```

--------------------------------

### Create Wikidata References

Source: https://github.com/sulab/wikidataintegrator/blob/main/notebooks/gene-validity.ipynb

Defines a function to generate a list of reference objects including stated in, retrieved, and URL properties.

```python
from datetime import datetime
import copy
def create_reference():
    refStatedIn = wdi_core.WDItemID(value="Q64403342", prop_nr="P248", is_reference=True)
    timeStringNow = datetime.now().strftime("+%Y-%m-%dT00:00:00Z")
    refRetrieved = wdi_core.WDTime(timeStringNow, prop_nr="P813", is_reference=True)
    refURL = wdi_core.WDUrl(value="https://search.clinicalgenome.org/kb/gene-validity/d910a9d8-516e-443d-acba-8d61f7574792--2018-06-07T14:23:53", prop_nr="P854", is_reference=True)

    return [refStatedIn, refRetrieved, refURL]
```

--------------------------------

### Execute SPARQL queries

Source: https://context7.com/sulab/wikidataintegrator/llms.txt

Run SPARQL queries against the Wikidata Query Service, with options for pandas DataFrame output and custom endpoints.

```python
from wikidataintegrator import wdi_core

# Simple query
query = '''
SELECT ?item ?itemLabel WHERE {
  ?item wdt:P31 wd:Q7187 .  # Instance of gene
  ?item wdt:P703 wd:Q15978631 .  # Found in taxon: Homo sapiens
  SERVICE wikibase:label { bd:serviceParam wikibase:language "en". }
}
LIMIT 10
'''

results = wdi_core.WDItemEngine.execute_sparql_query(query)
for result in results['results']['bindings']:
    print(f"{result['item']['value']}: {result['itemLabel']['value']}")

# Return as pandas DataFrame
import pandas as pd
df = wdi_core.WDItemEngine.execute_sparql_query(query, as_dataframe=True)
print(df.head())

# Query custom endpoint
results = wdi_core.WDItemEngine.execute_sparql_query(
    query,
    endpoint='https://query.wikidata.org/sparql',
    user_agent='MyBot/1.0 (contact@example.com)'
)
```

--------------------------------

### Search Wikidata Items

Source: https://context7.com/sulab/wikidataintegrator/llms.txt

Use get_wd_search_results to find items by label. Can return IDs and labels, or full information including descriptions.

```python
results = wdi_core.WDItemEngine.get_wd_search_results(
    search_string='insulin receptor',
    dict_id_label=True
)
for r in results:
    print(f"{r['id']}: {r['label']}")
```

```python
results = wdi_core.WDItemEngine.get_wd_search_results(
    search_string='CDK2',
    dict_id_all_info=True
)
for r in results:
    print(f"{r['id']}: {r['label']} - {r['description']}")
```

--------------------------------

### Retrieve Wikidata Items

Source: https://context7.com/sulab/wikidataintegrator/llms.txt

Load existing items by QID and access their properties, labels, and JSON representations.

```python
from wikidataintegrator import wdi_core

# Load an existing item by QID (Q5 = Human)
item = wdi_core.WDItemEngine(wd_item_id='Q5')

# Get the full JSON representation
json_data = item.get_wd_json_representation()

# Get label, description, and aliases
label = item.get_label(lang='en')
description = item.get_description(lang='en')
aliases = item.get_aliases(lang='en')

# Get all property numbers on the item
properties = item.get_property_list()
# Returns: ['P31', 'P279', 'P910', ...]
```

--------------------------------

### Fetch Description from Wikipathways

Source: https://github.com/sulab/wikidataintegrator/blob/main/notebooks/setDescription.ipynb

Retrieves the pathway description from the Wikipathways SPARQL endpoint.

```python
query = """
prefix dcterms:	<http://purl.org/dc/terms/>
SELECT ?item ?description WHERE { <"""
query += wdwpid + "> " 
query += """wdt:P2410 ?wpid ;
         wdt:P2888 ?wpiri .
  SERVICE <http://sparql.wikipathways.org/> {
      ?version dc:identifier ?wpiri ;
               dcterms:description ?description .
    }
}
"""
sparqlpandas(query, endpoint="https://query.wikidata.org/sparql")
```

```python
description = sparql(query, endpoint="https://query.wikidata.org/sparql")["results"]["bindings"][0]["description"]["value"]
print(description)
```

--------------------------------

### Iterate and create properties from Pandas

Source: https://github.com/sulab/wikidataintegrator/blob/main/notebooks/CreateWikidataProperties.ipynb

Iterates through a Pandas DataFrame to batch create properties based on mapped data types.

```python
for row in propertiesSparql.itertuples():
    print(datatype_map[row.propType])
    print(row)
    createProperty(login=login, label=row.propertyLabel, description=row.propertyDescription, property_datatype=datatype_map[row.propType])
```

--------------------------------

### Define WDUrl and WDCommonsMedia claims

Source: https://context7.com/sulab/wikidataintegrator/llms.txt

WDUrl is used for web links, while WDCommonsMedia handles Wikimedia Commons file references.

```python
from wikidataintegrator import wdi_core

# URL
website = wdi_core.WDUrl(
    value='https://example.com/resource',
    prop_nr='P856'  # Official website
)

# Commons media file
image = wdi_core.WDCommonsMedia(
    value='Example_image.jpg',
    prop_nr='P18'  # Image
)
```

--------------------------------

### Create a New Wikidata Property

Source: https://github.com/sulab/wikidataintegrator/blob/main/notebooks/CreateWikidataPropertiesParallel.ipynb

Defines a function to create a new property on a wikibase instance. It requires login credentials, a label, description, and the property's data type. If a Wikidata property ID is provided, it's linked.

```python
def createProperty(login=login, wdprop=None, label="", description="", property_datatype=""):
  if wdprop== None:
    s = []
  else:
    s = [wdi_core.WDUrl(wdprop, prop_nr="P1")]
  localEntityEngine = wdi_core.WDItemEngine.wikibase_item_engine_factory(api,sparql)
  item = localEntityEngine(data=s)
  item.set_label(label)
  item.set_description(description)
  print(item.write(login, entity_type="property", property_datatype=property_datatype))

#create mapping to wikidata property
if "property in wikidata" not in property_lookup:
  createProperty(login, label="property in wikidata", description="The same property in Wikidata", property_datatype="url")
```

--------------------------------

### Define Property Types and Definitions

Source: https://github.com/sulab/wikidataintegrator/blob/main/notebooks/Wikidata_json2ttl.ipynb

Functions to map Wikidata properties to OWL types and define their relationships within the RDF graph.

```python
properties = dict()
linked_items = []

def owlPropertyTypes(owlType):
    rdf_item.add((ns["wd"][pid], RDF.type, owlType))
    rdf_item.add((ns["p"][pid], RDF.type, owlType))
    rdf_item.add((ns["wdtn"][pid], RDF.type, owlType))
    rdf_item.add((ns["wdt"][pid], RDF.type, owlType))
    rdf_item.add((ns["pq"][pid], RDF.type, owlType))
    rdf_item.add((ns["pqn"][pid], RDF.type, owlType))
    rdf_item.add((ns["pqv"][pid], RDF.type, owlType))
    rdf_item.add((ns["pr"][pid], RDF.type, owlType))
    rdf_item.add((ns["prn"][pid], RDF.type, owlType))
    rdf_item.add((ns["prv"][pid], RDF.type,owlType))
    rdf_item.add((ns["ps"][pid], RDF.type, owlType))
    rdf_item.add((ns["psn"][pid], RDF.type, owlType))
    rdf_item.add((ns["psv"][pid], RDF.type, owlType))

def propdefs(pid, item, datatype):
    ## Properties and their derivatives
    object_properties = ["wikibase-item", 'external-id', 'string', 'commonsMedia',  'time', 'edtf', 'globe-coordinate', 'url', 'quantity', 'wikibase-property', 'monolingualtext', 'math', 'tabular-data', 'form', 'lexeme', 'geo-shape', 'musical-notation', 'sense']
    data_properties = ['external-id', 'string', 'time', 'edtf', 'globe-coordinate', 'quantity', 'monolingualtext', 'math', 'geo-shape', 'form', 'lexeme', 'musical-notation', 'sense' ]
    print(pid)  
    # ObjectProperty
    if datatype in object_properties:
        owlPropertyTypes(OWL.ObjectProperty)
    # Data Properties
    if datatype in data_properties:
        owlPropertyTypes(OWL.DatatypeProperty)  
    rdf_item.add((ns["wd"][pid], RDF.type, ns["wikibase"].Property))
    rdf_item.add((ns["wd"][pid], ns["wikibase"].directClaim, ns["wdt"][pid]))
    rdf_item.add((ns["wd"][pid], ns["wikibase"].claim, ns["p"][pid]))
    rdf_item.add((ns["wd"][pid], ns["wikibase"].statementProperty, ns["ps"][pid]))
    rdf_item.add((ns["wd"][pid], ns["wikibase"].statementValue, ns["psv"][pid]))
```

--------------------------------

### Define SPARQL Helper Functions

Source: https://github.com/sulab/wikidataintegrator/blob/main/notebooks/setDescription.ipynb

Utility functions to execute SPARQL queries against a specified endpoint, with optional pandas DataFrame output.

```python
def sparql(query, endpoint):
    query=query
    return wdi_core.WDItemEngine.execute_sparql_query(query, endpoint=endpoint)

def sparqlpandas(query, endpoint):
    return wdi_core.WDItemEngine.execute_sparql_query(query, endpoint=endpoint, as_dataframe=True)
```

--------------------------------

### wikidataintegrator.wdi_helpers.wikibase_helper

Source: https://github.com/sulab/wikidataintegrator/blob/main/docs/source/wikidataintegrator.wdi_helpers.md

This module offers helper functions for interacting with Wikibase.

```APIDOC
## wikidataintegrator.wdi_helpers.wikibase_helper

### Description
Helper functions for interacting with Wikibase.

### Module Contents
This module contains functions related to Wikibase interactions.
```

--------------------------------

### Parallel Property Creation Execution

Source: https://github.com/sulab/wikidataintegrator/blob/main/notebooks/CreateWikidataPropertiesParallel.ipynb

Utilizes multiprocessing to execute property creation stages in parallel across available CPU cores.

```python
inputs = range(10) 
def createPropertyStage(row, property_lookup):
    if row.propertyLabel in property_lookup:
        pass
    else:
        try:
            createProperty(login=login, wdprop=row.property, label=row.propertyLabel, description=row.propertyDescription, property_datatype=datatype_map[row.propType])
        except:
            print("Failed")
 
num_cores = multiprocessing.cpu_count()
     
results = Parallel(n_jobs=num_cores)(delayed(createPropertyStage)(i, property_lookup) for i in propertiesSparql.itertuples())
```

--------------------------------

### Login using OAuth1

Source: https://github.com/sulab/wikidataintegrator/blob/main/README.md

Authenticate using OAuth1 by providing consumer key and secret. The continue_oauth() method must be called afterwards to complete the process, which may involve user confirmation or direct callback handling for web applications.

```Python
login_instance = wdi_login.WDLogin(consumer_key='<your_consumer_key>', pwd='<your_consumer_secret>')
login_instance.continue_oauth()
```

--------------------------------

### List All Properties from a Wikidata Item

Source: https://github.com/sulab/wikidataintegrator/blob/main/notebooks/ObtainAWikidataItemAsPythonDict.ipynb

Access the 'claims' key of a loaded JSON item to retrieve a dictionary-like object containing all properties. Use the .keys() method to list them.

```python
json["claims"].keys()
```

--------------------------------

### Serialize RDF Item to Turtle (Second Instance)

Source: https://github.com/sulab/wikidataintegrator/blob/main/notebooks/Wikidata_json2ttl.ipynb

Serializes the current RDF item graph to a Turtle file named '3.ttl'. This might be used for comparison or as a separate output.

```python
rdf_item.serialize(format="turtle", destination="3.ttl")
```

--------------------------------

### Execute SPARQL Query

Source: https://github.com/sulab/wikidataintegrator/blob/main/README.md

Use this method to execute SPARQL queries against a Wikidata endpoint. You can specify custom prefixes and a user agent for the HTTP header.

```python
wdi_core.WDItemEngine.execute_sparql_query(query, prefix=None, endpoint='https://query.wikidata.org/sparql', user_agent='WDItemEngine')

```

--------------------------------

### Minimal Bot for Mass Import

Source: https://github.com/sulab/wikidataintegrator/blob/main/README.md

An enhanced minimal bot for mass creation or modification of Wikidata items by iterating over raw data.

```Python
from wikidataintegrator import wdi_core, wdi_login
        
# login object
login_instance = wdi_login.WDLogin(user='<bot user name>', pwd='<bot password>')
    
# We have raw data, which should be written to Wikidata, namely two human NCBI entrez gene IDs mapped to two Ensembl Gene IDs
raw_data = {
    '50943': 'ENST00000376197',
    '1029': 'ENST00000498124'
}
    
for entrez_id, ensembl in raw_data.items():
    # data type object
    entrez_gene_id = wdi_core.WDString(value=entrez_id, prop_nr='P351')
    ensembl_transcript_id = wdi_core.WDString(value=ensembl, prop_nr='P704')
        
    # data goes into a list, because many data objects can be provided to 
    data = [entrez_gene_id, ensembl_transcript_id]
        
    # Search for and then edit/create new item
    wd_item = wdi_core.WDItemEngine(data=data)
    wd_item.write(login_instance)
```

--------------------------------

### Write Wikidata Item

Source: https://github.com/sulab/wikidataintegrator/blob/main/notebooks/gene-validity.ipynb

Executes the write operation for a Wikidata item using an authenticated login object.

```python
# wikidata_item.write(login)
```

--------------------------------

### Bulk ID Mapping

Source: https://context7.com/sulab/wikidataintegrator/llms.txt

Efficiently map external identifiers to Wikidata QIDs using id_mapper. Supports filtering and options for handling duplicates or returning results as sets.

```python
from wikidatator.wdi_helpers import id_mapper, prop2qid

# Get all UniProt ID to QID mappings
uniprot_to_qid = id_mapper('P352')  # P352 = UniProt ID
print(f'P12345 -> {uniprot_to_qid.get("P12345")}')

# With filters (only human proteins)
human_proteins = id_mapper(
    'P352',
    filters=[('P703', 'Q15978631')]  # Found in taxon: Homo sapiens
)

# Raise error on duplicate mappings
try:
    mapping = id_mapper('P352', raise_on_duplicate=True)
except ValueError as e:
    print(f'Duplicate found: {e}')

# Return as set (multiple QIDs per ID possible)
mapping = id_mapper('P352', return_as_set=True)
# Returns: {'P12345': {'Q123', 'Q456'}, ...}

```

--------------------------------

### Dump Full JSON Output of a Wikidata Item

Source: https://github.com/sulab/wikidataintegrator/blob/main/notebooks/ObtainAWikidataItemAsPythonDict.ipynb

Simply reference the 'json' variable to print its entire structure for inspection. This is useful for understanding the complete data available for an item.

```python
json
```

--------------------------------

### Create Publication Items

Source: https://context7.com/sulab/wikidataintegrator/llms.txt

Use PublicationHelper to create Wikidata items for scientific publications from various external identifiers like DOI, PubMed ID, arXiv, or bioRxiv.

```python
from wikidatator import wdi_login
from wikidatator.wdi_helpers import PublicationHelper

login = wdi_login.WDLogin(user='BotUsername', pwd='BotPassword')

# Create publication from DOI via Crossref
pub = PublicationHelper(
    ext_id='10.1038/nature12373',
    id_type='doi',
    source='crossref'
)
qid, warnings, success = pub.get_or_create(login)
print(f'Created/found: {qid}')

# Create from PubMed ID via Europe PMC
pub = PublicationHelper(
    ext_id='23903654',
    id_type='pmid',
    source='europepmc'
)
qid, warnings, success = pub.get_or_create(login)

# Create from arXiv
pub = PublicationHelper(
    ext_id='2104.08653',
    id_type='arxiv',
    source='arxiv'
)
qid, warnings, success = pub.get_or_create(login)

# Create from bioRxiv
pub = PublicationHelper(
    ext_id='2021.01.15.426800',
    id_type='biorxiv',
    source='biorxiv'
)
qid, warnings, success = pub.get_or_create(login)
```

--------------------------------

### Transform Wikidata Labels, Descriptions, and Aliases

Source: https://github.com/sulab/wikidataintegrator/blob/main/notebooks/Wikidata_json2ttl.ipynb

A function to transform labels, descriptions, and aliases from a JSON item into RDF triples using RDFS.label, schema.name, skos.prefLabel, schema.description, and SKOS.altLabel. It also includes a call to process properties and their definitions.

```python
def transformLabels(qid, json_item):
    # Heading
    for language in json_item["labels"].keys():
        rdf_item.add((ns["wd"][qid], RDFS.label, Literal(json_item["labels"][language]["value"], language)))
        rdf_item.add((ns["wd"][qid], ns["schema"].name, Literal(json_item["labels"][language]["value"], language)))
        rdf_item.add((ns["wd"][qid], ns["skos"].prefLabel, Literal(json_item["labels"][language]["value"], language)))

    for language in json_item["descriptions"].keys():
        rdf_item.add((ns["wd"][qid], ns["schema"].description, Literal(json_item["descriptions"][language]["value"], language)))

    for language in json_item["aliases"].keys():
        for label in json_item["aliases"][language]:
            rdf_item.add((ns["wd"][qid], SKOS.altLabel, Literal(label, language))) 
                          
transformLabels(qid, json_item)     
    
for pid in properties.keys():
    pid_item = wdi_core.WDItemEngine(wd_item_id=pid).get_wd_json_representation()
    transformLabels(pid, pid_item)
    propdefs(pid, pid_item, properties[pid])
```

--------------------------------

### Update Wikidata Description

Source: https://github.com/sulab/wikidataintegrator/blob/main/notebooks/setDescription.ipynb

Sets the description for the Wikidata item and writes the changes.

```python
wdPage.set_description(description, lang="en")
wdPage.write(login)
```

--------------------------------

### Define WDItemID claims

Source: https://context7.com/sulab/wikidataintegrator/llms.txt

Use WDItemID to link to other Wikidata items, supporting both string and integer values, as well as optional qualifiers.

```python
from wikidataintegrator import wdi_core

# Link to another item (Q15978631 = Homo sapiens)
taxon_claim = wdi_core.WDItemID(
    value='Q15978631',
    prop_nr='P703'  # Found in taxon
)

# Can also use integer
taxon_claim = wdi_core.WDItemID(value=15978631, prop_nr='P703')

# With qualifiers
qualifier = wdi_core.WDItemID('Q21503252', 'P459', is_qualifier=True)  # Determination method
item_with_qualifier = wdi_core.WDItemID(
    value='Q15978631',
    prop_nr='P703',
    qualifiers=[qualifier]
)
```

--------------------------------

### Construct Wikidata Statement

Source: https://github.com/sulab/wikidataintegrator/blob/main/notebooks/gene-validity.ipynb

Creates a statement object with a deep-copied reference for a specific Wikidata property.

```python
reference = create_reference()
statement = [wdi_core.WDItemID(value=MONDO_qid, prop_nr="P2293", references=[copy.deepcopy(reference)])]
```

--------------------------------

### Batch Item Retrieval

Source: https://context7.com/sulab/wikidataintegrator/llms.txt

Efficiently retrieve multiple Wikidata items simultaneously using batch processing capabilities. This is useful for optimizing data fetching operations when dealing with a large number of items.

```python
from wikidatator import wdi_core, wdi_login

login = wdi_login.WDLogin(user='BotUsername', pwd='BotPassword')
```

--------------------------------

### Process Wikidata Sitelinks

Source: https://github.com/sulab/wikidataintegrator/blob/main/notebooks/Wikidata_json2ttl.ipynb

Iterates through sitelinks of a Wikidata item, extracting URLs, determining language, and adding RDF triples for schema.Article, schema.about, schema.inLanguage, and schema.inPartOf.

```python
# sitelinks
for sitelink in json_item['sitelinks'].keys():
    wiki = URIRef(json_item['sitelinks'][sitelink]["url"])
    print(json_item['sitelinks'][sitelink]["url"])
    partof = URIRef(json_item['sitelinks'][sitelink]["url"].split("wiki")[0])
    language = sitelink.replace("wiki", "")
    rdf_item.add((wiki, RDF.type, ns["schema"].Article))
    rdf_item.add((wiki, schema.about, ns["wd"][qid]))
    rdf_item.add((wiki, schema.inLanguage, ns["wd"][qid]))
    rdf_item.add((wiki, schema.inPartOf, partof))
```

--------------------------------

### Check Wikidata Item Conformance to Shape Expression

Source: https://github.com/sulab/wikidataintegrator/blob/main/README.md

Tests a Wikidata item for conformance to a given Shape Expression (ShEx). Requires the item ID (qid) and the Shape Expression ID (eid).

```Python
from wikidataintegrator import wdi_core

print("Q560 conforms to E46"+wdi_core.WDItemEngine.check_shex_conformance(qid="Q560", eid="E46"))
```

--------------------------------

### Process Linked Wikidata Items

Source: https://github.com/sulab/wikidataintegrator/blob/main/notebooks/Wikidata_json2ttl.ipynb

Iterates through a list of linked QIDs, adds them as Wikidata items to the RDF graph, and transforms their labels.

```python
for linked_qid in linked_items:
    print(linked_qid)
    rdf_item.add((ns["wd"][linked_qid], RDF.type, ns["wikibase"].Item))
    linked_qid_item = wdi_core.WDItemEngine(wd_item_id=linked_qid).get_wd_json_representation()
    transformLabels(linked_qid, linked_qid_item)
```

--------------------------------

### Execute SPARQL Queries

Source: https://github.com/sulab/wikidataintegrator/blob/main/README.md

Executes a SPARQL query against a specified endpoint.

```APIDOC
## POST wdi_core.WDItemEngine.execute_sparql_query()

### Description
Executes a SPARQL query string against a Wikidata-compatible endpoint.

### Parameters
#### Request Body
- **query** (string) - Required - The SPARQL query string.
- **prefix** (dict) - Optional - Custom prefixes if not using standard Wikidata prefixes.
- **endpoint** (string) - Optional - The SPARQL endpoint URL.
- **user_agent** (string) - Optional - User agent string for the HTTP header.
```

--------------------------------

### Define Fast Run Filter Dictionary

Source: https://github.com/sulab/wikidataintegrator/blob/main/README.md

This dictionary specifies the properties and their corresponding item QIDs to filter data for fast run mode. Use an empty string for the value if it's a literal.

```Python
fast_run_base_filter = {'P351': '', 'P703': 'Q15978631'}
```

--------------------------------

### Serialize RDF Item to Turtle

Source: https://github.com/sulab/wikidataintegrator/blob/main/notebooks/Wikidata_json2ttl.ipynb

Serializes the current RDF item graph to a Turtle file named '2.ttl'.

```python
rdf_item.serialize(format="turtle", destination="2.ttl")
```
Tokens:
Raw