ComProScanner

https://slimeslab.github.io/ComProScanner
Admin
ComProScanner is a comprehensive Python package designed to extract composition-property...
Tokens:178,778
Snippets:671
Trust Score:8.3
Update:10 months ago
Show doc for...
Tokens:
Raw
### Basic ComProScanner Installation

Source: https://slimeslab.github.io/ComProScanner/getting-started/installation

Installs the latest stable version of ComProScanner and its dependencies from the Python Package Index (PyPI) using pip. This is the recommended method for most users.

```shell
pip install comproscanner
```

--------------------------------

### Build Documentation Locally

Source: https://slimeslab.github.io/ComProScanner/about/contribution

These commands install the necessary packages (`mkdocs` and `mkdocs-material`) to build and serve the project's documentation locally. After installation, `mkdocs serve` starts a local development server.

```shell
pip install mkdocs mkdocs-material
mkdocs serve
```

--------------------------------

### Verify ComProScanner Installation

Source: https://slimeslab.github.io/ComProScanner/getting-started/installation

Checks if ComProScanner has been installed correctly by importing the library and printing its version number. Successful execution without errors indicates a proper installation.

```python
import comproscanner
print(comproscanner.__version__)
```

--------------------------------

### ComProScanner Source Code Installation

Source: https://slimeslab.github.io/ComProScanner/getting-started/installation

Installs ComProScanner directly from its source code repository. This method is useful for developers who want to contribute to the project or work with the latest unreleased features. It involves cloning the repository and then installing the package in editable mode.

```shell
git clone https://github.com/slimeslab/ComProScanner.git
cd comproscanner
pip install -e .
```

--------------------------------

### Install ComProScanner Package

Source: https://slimeslab.github.io/ComProScanner/index_q=

Installs the ComProScanner package using pip. This is the first step to start using the library for extracting composition-property relationships from scientific articles.

```bash
pip install comproscanner

```

--------------------------------

### Initialize ComProScanner Instance

Source: https://slimeslab.github.io/ComProScanner/getting-started/quick-start

Initializes the ComProScanner with a `main_property_keyword`. This keyword is used for organizing associated files and directories, ensuring a structured approach to data management during the scanning process.

```python
from comproscanner import ComProScanner

scanner = ComProScanner(main_property_keyword="piezoelectric")

```

--------------------------------

### ComProScanner Full Workflow for d33 Extraction

Source: https://slimeslab.github.io/ComProScanner/getting-started/quick-start_q=

Demonstrates the complete minimal workflow for extracting piezoelectric coefficient (d33) data using ComProScanner. It covers initializing the scanner, collecting metadata, processing articles from specified sources, and extracting composition-property relationships.

```python
from comproscanner import ComProScanner

# Step 1: Initialize with property of interest
scanner = ComProScanner(main_property_keyword="piezoelectric")

# Step 2: Collect metadata
scanner.collect_metadata()

# Step 3: Define property keywords for filtering and process articles from specific sources
property_keywords = {
    "exact_keywords": ["d33"],
    "substring_keywords": [" d 33 "]
}
scanner.process_articles(
    property_keywords=property_keywords,
    source_list=["elsevier", "springer"]
)

# Step 4: Extract composition-property relationships
scanner.extract_composition_property_data(
    main_extraction_keyword="d33"
)

```

--------------------------------

### Complete ComProScanner Workflow for Piezoelectric Data Extraction

Source: https://slimeslab.github.io/ComProScanner/getting-started/quick-start

Demonstrates the full ComProScanner workflow to extract piezoelectric coefficient (d33) data. This involves initializing the scanner, collecting metadata, processing articles from specified sources, and finally extracting the composition-property relationships.

```python
from comproscanner import ComProScanner

# Step 1: Initialize with property of interest
scanner = ComProScanner(main_property_keyword="piezoelectric")

# Step 2: Collect metadata
scanner.collect_metadata()

# Step 3: Define property keywords for filtering and process articles from specific sources
property_keywords = {
    "exact_keywords": ["d33"],
    "substring_keywords": [" d 33 "]
}
s scanner.process_articles(
    property_keywords=property_keywords,
    source_list=["elsevier", "springer"]
)

# Step 4: Extract composition-property relationships
scanner.extract_composition_property_data(
    main_extraction_keyword="d33"
)

```

--------------------------------

### Comproscanner JSON Data Structure Example

Source: https://slimeslab.github.io/ComProScanner/getting-started/quick-start_q=

This JSON object demonstrates the structured data extracted by Comproscanner. It includes composition details, synthesis steps and techniques, and article metadata such as DOI, title, authors, and keywords. The format is designed for organized storage and retrieval of scientific research data.

```json
"10.1016/j.apradiso.2024.111655": {
    "composition_data": {
      "compositions_property_values": {
        "Eu1.90Dy0.10Ge2O7": 0.66,
        "Eu1.90La0.10Ge2O7": 0.36,
        "Eu1.90Ho0.10Ge2O7": 0.62
      },
      "property_unit": "pC/N",
      "family": "RE2B2O7"
    },
    "synthesis_data": {
      "method": "solid-state reaction",
      "precursors": [
        "Eu2O3",
        "GeO2",
        "Dy2O3",
        "La2O3",
        "Ho2O3"
      ],
      "steps": [
        "Starting materials Eu2O3, GeO2, Dy2O3, La2O3 and Ho2O3 were combined in stoichiometric ratios with each dopant at 5 mol%.,
        "Samples were first heated at 800°C for 2 hours in pure alumina crucibles under open atmosphere.",
        "Materials were then heated to 1150°C for 10 hours followed by slow cooling.",
        "Resulting materials were ground into powder for further characterization.",
        "Ceramic discs were formed from obtained powder materials with 1 mm thickness and 10 mm diameter.",
        "Ceramic discs were compacted using uniaxial pressing under 250 MPa pressure with 2 wt% of 5 wt% PVA aqueous solution as binder.",
        "Samples were heated at 600°C for 30 minutes to eliminate organic additives.",
        "Sintering was conducted at 1400°C for 4 hours.",
        "Silver paste was applied to disc surfaces and fired at 650°C for 1 hour to form surface electrodes.",
        "Electric field of 9-18 kV/mm was applied in silicon oil bath at 120°C for 30 minutes followed by 24-hour aging."
      ],
      "characterization_techniques": [
        "TG/DTA",
        "XRD",
        "SEM",
        "EDX",
        "photoluminescence spectroscopy",
        "LCR meter",
        "d33 meter"
      ]
    },
    "article_metadata": {
      "doi": "10.1016/j.apradiso.2024.111655",
      "title": "Novel smart materials with high curie temperatures: Eu1.90Dy0.10Ge2O7, Eu1.90La0.10Ge2O7 and Eu1.90Ho0.10Ge2O7",
      "journal": "Applied Radiation and Isotopes",
      "year": "2025",
      "isOpenAccess": false,
      "authors": [
        {
          "name": "Esra Öztürk",
          "affiliation_id": "60020484",
          "affiliation_name": "Hacettepe Üniversitesi",
          "affiliation_country": "Turkey"
        },
        {
          "name": "Nilgun Kalaycioglu Ozpozan",
          "affiliation_id": "122321412",
          "affiliation_name": "Erciyes Ün.",
          "affiliation_country": "Türkiye"
        },
        {
          "name": "Volkan Kalem",
          "affiliation_id": "60193845",
          "affiliation_name": "Konya Technical University",
          "affiliation_country": "Turkey"
        }
      ],
      "keywords": [
        "Curie"
      ]
    }
  }
```

--------------------------------

### ComProScanner Troubleshooting: Virtual Environment

Source: https://slimeslab.github.io/ComProScanner/getting-started/installation

Addresses potential dependency conflicts by setting up and activating a new virtual environment before installing ComProScanner. This isolates the project's dependencies from other Python projects on the system.

```shell
python -m venv compro_env
source compro_env/bin/activate  # On Windows: compro_env\Scripts\activate
pip install comproscanner
```

--------------------------------

### Optional LLM Provider Dependencies for ComProScanner

Source: https://slimeslab.github.io/ComProScanner/getting-started/installation

Installs additional Python packages required to integrate ComProScanner with specific Large Language Model (LLM) providers. Choose the packages corresponding to the LLM services you intend to use.

```shell
# For Anthropic Claude
pip install langchain-anthropic

# For Google Gemini
pip install langchain-google-genai

# For Ollama (local models)
pip install langchain-ollama

# For TogetherAI Model Integration
pip install langchain-together

# For OpenRouter Model Integration
pip install langchain-openrouter

# For Cohere Model Integration
pip install langchain-cohere
```

--------------------------------

### Install Project Dependencies

Source: https://slimeslab.github.io/ComProScanner/about/contribution

This command installs the project dependencies in editable mode, including development dependencies. The `.[dev]` part ensures that optional dependencies for development are also installed.

```shell
pip install -e ".[dev]"
```

--------------------------------

### Collect Metadata with ComProScanner

Source: https://slimeslab.github.io/ComProScanner/getting-started/quick-start

Collects metadata for scientific articles related to piezoelectric materials from the Scopus database, focusing on entries from the last two years. This step helps in identifying relevant literature for further processing.

```python
scanner.collect_metadata(
    base_queries=["piezoelectric", "piezoelectricity"]
)

```

--------------------------------

### ComProScanner Environment Variables Configuration

Source: https://slimeslab.github.io/ComProScanner/getting-started/installation

Sets up essential API keys and configuration paths required for ComProScanner to function, particularly for accessing publisher APIs and LLM models. These should be stored in a `.env` file in the project directory for security.

```dotenv
# Publisher TDM API Keys (for direct article access)
SCOPUS_API_KEY=your_scopus_api_key # for Elsevier as well as metadata retrieval
WILEY_API_KEY=your_wiley_api_key
SPRINGER_OPENACCESS_API_KEY=your_springer_openaccess_api_key # Springer provides two separate keys for open access and TDM API
SPRINGER_TDM_API_KEY=your_springer_tdm_api_key
IOP_papers_path=local_path_to_iop_papers # IOP Publishing provides XML articles in bulk through SFTP access

# API Keys for LLM Models (at least one is required which will be used for data extraction)
OPENAI_API_KEY=your_openai_api_key
DEEPSEEK_API_KEY=your_deepseek_api_key
ANTHROPIC_API_KEY=your_anthropic_api_key
GEMINI_API_KEY=your_google_api_key
OPENROUTER_API_KEY=your_openrouter_api_key

# Hugging Face API Key (for accessing thellert/physbert_cased model for embeddings)
HF_TOKEN=your_huggingface_api_key

# Neo4j Configuration (for knowledge graph visualization)
NEO4J_URI=bolt://localhost:7687
NEO4J_USER=neo4j
NEO4J_PASSWORD=your_password
NEO4J_DATABASE=neo4j
```

--------------------------------

### Install Langchain Ollama Dependency and Ollama

Source: https://slimeslab.github.io/ComProScanner/rag-config_q=

Installs the Langchain Ollama integration package and provides instructions for installing Ollama locally, enabling the use of local LLMs with Langchain.

```bash
pip install langchain-ollama

# Install Ollama locally
# Visit: https://ollama.ai/download

```

--------------------------------

### JSON Data Structure Example

Source: https://slimeslab.github.io/ComProScanner/getting-started/quick-start

This JSON object represents the structured data extracted for a scientific article. It includes details about material composition, synthesis procedures, and bibliographic information. The structure is designed for easy parsing and analysis of research data.

```json
{
    "10.1016/j.apradiso.2024.111655": {
        "composition_data": {
          "compositions_property_values": {
            "Eu1.90Dy0.10Ge2O7": 0.66,
            "Eu1.90La0.10Ge2O7": 0.36,
            "Eu1.90Ho0.10Ge2O7": 0.62
          },
          "property_unit": "pC/N",
          "family": "RE2B2O7"
        },
        "synthesis_data": {
          "method": "solid-state reaction",
          "precursors": [
            "Eu2O3",
            "GeO2",
            "Dy2O3",
            "La2O3",
            "Ho2O3"
          ],
          "steps": [
            "Starting materials Eu2O3, GeO2, Dy2O3, La2O3 and Ho2O3 were combined in stoichiometric ratios with each dopant at 5 mol%.",
            "Samples were first heated at 800°C for 2 hours in pure alumina crucibles under open atmosphere.",
            "Materials were then heated to 1150°C for 10 hours followed by slow cooling.",
            "Resulting materials were ground into powder for further characterization.",
            "Ceramic discs were formed from obtained powder materials with 1 mm thickness and 10 mm diameter.",
            "Ceramic discs were compacted using uniaxial pressing under 250 MPa pressure with 2 wt% of 5 wt% PVA aqueous solution as binder.",
            "Samples were heated at 600°C for 30 minutes to eliminate organic additives.",
            "Sintering was conducted at 1400°C for 4 hours.",
            "Silver paste was applied to disc surfaces and fired at 650°C for 1 hour to form surface electrodes.",
            "Electric field of 9-18 kV/mm was applied in silicon oil bath at 120°C for 30 minutes followed by 24-hour aging."
          ],
          "characterization_techniques": [
            "TG/DTA",
            "XRD",
            "SEM",
            "EDX",
            "photoluminescence spectroscopy",
            "LCR meter",
            "d33 meter"
          ]
        },
        "article_metadata": {
          "doi": "10.1016/j.apradiso.2024.111655",
          "title": "Novel smart materials with high curie temperatures: Eu1.90Dy0.10Ge2O7, Eu1.90La0.10Ge2O7 and Eu1.90Ho0.10Ge2O7",
          "journal": "Applied Radiation and Isotopes",
          "year": "2025",
          "isOpenAccess": false,
          "authors": [
            {
              "name": "Esra Öztürk",
              "affiliation_id": "60020484",
              "affiliation_name": "Hacettepe Üniversitesi",
              "affiliation_country": "Turkey"
            },
            {
              "name": "Nilgun Kalaycioglu Ozpozan",
              "affiliation_id": "122321412",
              "affiliation_name": "Erciyes Ün.",
              "affiliation_country": "Türkiye"
            },
            {
              "name": "Volkan Kalem",
              "affiliation_id": "60193845",
              "affiliation_name": "Konya Technical University",
              "affiliation_country": "Turkey"
            }
          ],
          "keywords": [
            "Curie"
          ]
        }
      }
    }

```

--------------------------------

### Create and Populate .env File

Source: https://slimeslab.github.io/ComProScanner/about/contribution

This command copies the example environment file (`.env.example`) to a new file named `.env`. You should then add your specific API keys and configurations to this `.env` file.

```shell
cp .env.example .env
```

--------------------------------

### ComproScanner Initialization and Configuration

Source: https://slimeslab.github.io/ComProScanner/api

This snippet illustrates the initialization of the ComproScanner, defining various optional arguments that control data extraction, saving behavior, LLM and RAG configurations, and logging.

```python
is_extract_synthesis_data (bool, optional): A flag to indicate if the synthesis data should be extracted. Defaults to True.
        is_save_csv (bool, optional): A flag to indicate if the results should be saved in the CSV file. Defaults to False.
        is_save_relevant (bool, optional): A flag to indicate if only papers with composition-property data should be saved. If True, only saves papers with composition data. If False, saves all processed papers. Defaults to True.
        llm (LLM, optional): An instance of the LLM class. Defaults to None.
        materials_data_identifier_query (str, optional): Query to identify the materials data. Must be an 'yes/no' answer. Defaults to "Is there any material chemical composition and corresponding {main_property_keyword} value mentioned in the paper? GIVE ONE WORD ANSWER. Either yes or no."
        model (str: optional): The model to use (defaults to "gpt-4o-mini")
        api_base (str, optional): Base URL for standard API endpoints
        base_url (str, optional): Base URL for the model service
        api_key (str, optional): API key for the model service
        output_log_folder (str, optional): Base folder path to save logs. Logs will be saved in {output_log_folder}/{doi}/ subdirectory. Logs will be in JSON format if is_log_json is True, otherwise plain text. Defaults to None.
        task_output_folder (str, optional): Base folder path to save task outputs. Task outputs will be saved as .txt files in {task_output_folder}/{doi}/ subdirectory. Defaults to None.
        is_log_json (bool, optional): Flag to save logs in JSON format. Defaults to False.
        verbose (bool, optional): Flag to enable verbose output inside the terminal (defaults to True)
        temperature (float, optional): Temperature for text generation - controls randomness (defaults to 0.1)
        top_p (float, optional): Nucleus sampling parameter for text generation - controls diversity (defaults to 0.9)
        timeout (int, optional): Request timeout in seconds (defaults to 60)
        frequency_penalty (float, optional): Frequency penalty for text generation
        max_tokens (int, optional): Maximum tokens for completion
        rag_db_path (str, optional): Path to the vector database. Defaults to 'db'.
        embedding_model (str, optional): Name of the embedding model for RAG. Defaults to 'huggingface:thellert/physbert_cased'.
        rag_chat_model (str, optional): Name of the chat model for RAG. Defaults to 'gpt-4o-mini'.
        rag_max_tokens (int, optional): Maximum tokens for completion for RAG. Defaults to 512.
        rag_top_k (int, optional): Top k value for sampling for RAG. Defaults to 3.
        rag_base_url (str, optional): Base URL for the RAG model service.
        **flow_optional_args: Optional arguments for the MaterialsFlow class.
```

--------------------------------

### Upgrade ComProScanner

Source: https://slimeslab.github.io/ComProScanner/getting-started/installation

Updates an existing ComProScanner installation to the latest available version on PyPI. This command ensures you are using the most recent stable release with all the latest features and bug fixes.

```shell
pip install --upgrade comproscanner
```

--------------------------------

### Basic Usage of ComProScanner

Source: https://slimeslab.github.io/ComProScanner/index_q=

Demonstrates the basic workflow of ComProScanner, including initializing the scanner, collecting metadata, processing articles from specified sources, and extracting composition-property data.

```python
from comproscanner import ComProScanner

# Initialize with property of interest
scanner = ComProScanner(main_property_keyword="piezoelectric")

# Collect metadata
scanner.collect_metadata(
    base_queries=["piezoelectric", "piezoelectricity"],
)

# Define property keywords for filtering
property_keywords = {
    "exact_keywords": ["d33"],
    "substring_keywords": [" d 33 "]
}

# Process articles
scanner.process_articles(
    property_keywords=property_keywords,
    source_list=["elsevier", "springer"]
)

# Extract data
scanner.extract_composition_property_data(
    main_extraction_keyword="d33"
)

```

--------------------------------

### ComProScanner Data Visualization Examples (Python)

Source: https://slimeslab.github.io/ComProScanner/usage/visualization/overview

Demonstrates how to use the ComProScanner data_visualizer module to create visualizations such as pie charts for material family distributions and knowledge graphs from extracted data sources. Requires 'results.json' as input.

```python
from comproscanner import data_visualizer

# Pie chart
fig = data_visualizer.plot_family_pie_chart(
    data_sources=["results.json"],
    output_file="families.png"
)

# Knowledge graph
data_visualizer.create_knowledge_graph(
    result_file="results.json"
)
```

--------------------------------

### Evaluate Extraction Quality with ComProScanner

Source: https://slimeslab.github.io/ComProScanner/getting-started/quick-start

Evaluates the quality of the extracted data against ground truth using both semantic and agentic evaluation methods. The results are saved to specified JSON output files for further analysis.

```python
from comproscanner import evaluate_semantic, evaluate_agentic

# Semantic evaluation
semantic_results = evaluate_semantic(
    ground_truth_file="ground_truth.json",
    test_data_file="extracted_results.json",
    output_file="semantic_evaluation.json"
)

# Agentic evaluation (more advanced)
agentic_results = evaluate_agentic(
    ground_truth_file="ground_truth.json",
    test_data_file="extracted_results.json",
    output_file="agentic_evaluation.json"
)

```

--------------------------------

### Visualize Extracted Data using ComProScanner

Source: https://slimeslab.github.io/ComProScanner/getting-started/quick-start

Generates visualizations from the extracted results, including a pie chart for material family distribution and a knowledge graph. This helps in understanding the patterns and relationships within the data.

```python
from comproscanner import data_visualizer

# Plot material families distribution
fig = data_visualizer.plot_family_pie_chart(
    data_sources=["extracted_results.json"],
    output_file="family_distribution.png"
)

# Create knowledge graph
data_visualizer.create_knowledge_graph(
    result_file="extracted_results.json"
)

```

--------------------------------

### Configure Material Data Identification Query

Source: https://slimeslab.github.io/ComProScanner/api

Sets up the query to identify material data, including optional parameters for language model, API, logging, output folders, RAG settings, and generation parameters like temperature and top_p. It validates required inputs like `main_extraction_keyword` and `test_doi_list_file` when applicable.

```python
def materials_data_identifier_query(
    main_extraction_keyword: str,
    materials_data_identifier_query: str = None,
    model: str = "gpt-4o-mini",
    api_base: str = None,
    base_url: str = None,
    api_key: str = None,
    output_log_folder: str = None,
    task_output_folder: str = None,
    is_log_json: bool = False,
    verbose: bool = True,
    temperature: float = 0.1,
    top_p: float = 0.9,
    timeout: int = 60,
    frequency_penalty: float = None,
    max_tokens: int = None,
    rag_db_path: str = 'db',
    embedding_model: str = 'huggingface:thellert/physbert_cased',
    rag_chat_model: str = 'gpt-4o-mini',
    rag_max_tokens: int = 512,
    rag_top_k: int = 3,
    rag_base_url: str = None,
    flow_optional_args: dict = None,
    is_test_data_preparation: bool = False,
    test_doi_list_file: str = None,
    total_test_data: int = None,
    test_random_seed: int = None,
    checked_doi_list_file: str = None,
    main_property_keyword: str,
    json_results_file: str = None,
    start_row: int = 0,
    num_rows: int = None,
    is_only_consider_test_doi_list: bool = False,
) -> None:
    """
    Identifies if a paper contains mentions of material chemical composition and corresponding property values.

    Args:
        materials_data_identifier_query (str, optional): Query to identify the materials data. Must be an 'yes/no' answer. Defaults to "Is there any material chemical composition and corresponding {main_property_keyword} value mentioned in the paper? GIVE ONE WORD ANSWER. Either yes or no."
        model (str: optional): The model to use (defaults to "gpt-4o-mini")
        api_base (str, optional): Base URL for standard API endpoints
        base_url (str, optional): Base URL for the model service
        api_key (str, optional): API key for the model service
        output_log_folder (str, optional): Base folder path to save logs. Logs will be saved in {output_log_folder}/{doi}/ subdirectory. Logs will be in JSON format if is_log_json is True, otherwise plain text. Defaults to None.
        task_output_folder (str, optional): Base folder path to save task outputs. Task outputs will be saved as .txt files in {task_output_folder}/{doi}/ subdirectory. Defaults to None.
        is_log_json (bool, optional): Flag to save logs in JSON format. Defaults to False.
        verbose (bool, optional): Flag to enable verbose output inside the terminal (defaults to True)
        temperature (float, optional): Temperature for text generation - controls randomness (defaults to 0.1)
        top_p (float, optional): Nucleus sampling parameter for text generation - controls diversity (defaults to 0.9)
        timeout (int, optional): Request timeout in seconds (defaults to 60)
        frequency_penalty (float, optional): Frequency penalty for text generation
        max_tokens (int, optional): Maximum tokens for completion
        rag_db_path (str, optional): Path to the vector database. Defaults to 'db'.
        embedding_model (str, optional): Name of the embedding model for RAG. Defaults to 'huggingface:thellert/physbert_cased'.
        rag_chat_model (str, optional): Name of the chat model for RAG. Defaults to 'gpt-4o-mini'.
        rag_max_tokens (int, optional): Maximum tokens for completion for RAG. Defaults to 512.
        rag_top_k (int, optional): Top k value for sampling for RAG. Defaults to 3.
        rag_base_url (str, optional): Base URL for the RAG model service.
        **flow_optional_args: Optional arguments for the MaterialsFlow class.

    Raises:
        ValueErrorHandler: If main_extraction_keyword is not provided.
    """
    if main_extraction_keyword is None:
        logger.error(
            "main_extraction_keyword cannot be None. Please provide a valid keyword. Exiting..."
        )
        raise ValueErrorHandler(
            message="Please provide main_extraction_keyword to proceed for identifying sentences based on property."
        )
    if is_test_data_preparation and test_doi_list_file is None:
        logger.error("Test data file name is required for test data preparation.")
        raise ValueErrorHandler(
            message="Test data file name is required for test data preparation."
        )
    self.is_test_data_preparation = is_test_data_preparation
    if is_test_data_preparation:
        self.test_doi_list_file = test_doi_list_file
        if total_test_data is None:
            self.total_test_data = 50
        else:
            self.total_test_data = total_test_data
    llm_config = LLMConfig(
        model=model,
        api_base=api_base,
        base_url=base_url,
        api_key=api_key,
        temperature=temperature,
        top_p=top_p,
        timeout=timeout,
        frequency_penalty=frequency_penalty,
        max_tokens=max_tokens,
    )
    llm = llm_config.get_llm()
    rag_config = RAGConfig(
        rag_db_path=rag_db_path,
        embedding_model=embedding_model,
        rag_chat_model=rag_chat_model,
        rag_max_tokens=rag_max_tokens,
        rag_top_k=rag_top_k,
        rag_base_url=rag_base_url,
    )
    if materials_data_identifier_query is None:
        materials_data_identifier_query = f"Is there any material chemical composition and corresponding {self.main_property_keyword} value mentioned in the paper? Give one word answer. Either yes or no."
    preparator = MatPropDataPreparator(
        main_property_keyword=self.main_property_keyword,
        main_extraction_keyword=main_extraction_keyword,
        json_results_file=json_results_file,
        start_row=start_row,
        num_rows=num_rows,
        is_test_data_preparation=is_test_data_preparation,
        test_doi_list_file=test_doi_list_file,
        is_only_consider_test_doi_list=is_only_consider_test_doi_list,
        total_test_data=total_test_data,
        test_random_seed=test_random_seed,
        checked_doi_list_file=checked_doi_list_file,
    )

```

--------------------------------

### Process Wiley, IOP, and PDF Articles - Python

Source: https://slimeslab.github.io/ComProScanner/api

This code snippet demonstrates the processing of articles from different sources: Wiley, IOP (Institute of Physics), and local PDF files. It initializes specific processors for each source type, passing relevant configuration parameters such as main property keywords, batch sizes, date ranges, and RAG (Retrieval-Augmented Generation) configurations. The processors are then invoked to handle the respective article types.

```Python
        # Process Wiley articles
        if "wiley" in source_list:
            from .article_processors.wiley_processor import WileyArticleProcessor

            wiley_processor = WileyArticleProcessor(
                main_property_keyword=self.main_property_keyword,
                property_keywords=property_keywords,
                sql_batch_size=sql_batch_size,
                csv_batch_size=csv_batch_size,
                start_row=start_row,
                end_row=end_row,
                doi_list=doi_list,
                is_sql_db=is_sql_db,
                rag_config=rag_config,
            )
            wiley_processor.process_wiley_articles()

        # Process IOP articles
        if "iop" in source_list:
            from .article_processors.iop_processor import IOPArticleProcessor

            iop_processor = IOPArticleProcessor(
                main_property_keyword=self.main_property_keyword,
                property_keywords=property_keywords,
                sql_batch_size=sql_batch_size,
                csv_batch_size=csv_batch_size,
                start_row=start_row,
                end_row=end_row,
                doi_list=doi_list,
                is_sql_db=is_sql_db,
                rag_config=rag_config,
            )
            iop_processor.process_iop_articles()

        # Process PDFs
        if "pdfs" in source_list:
            from .article_processors.pdfs_processor import PDFsProcessor

            pdf_processor = PDFsProcessor(
                folder_path=folder_path,
                main_property_keyword=self.main_property_keyword,
                property_keywords=property_keywords,
                sql_batch_size=sql_batch_size,
                csv_batch_size=csv_batch_size,
                is_sql_db=is_sql_db,
                rag_config=rag_config,
            )
            pdf_processor.process_pdfs()

```

--------------------------------

### Clone ComProScanner Repository

Source: https://slimeslab.github.io/ComProScanner/about/contribution

This command clones the ComProScanner repository from GitHub to your local machine. After cloning, navigate into the project directory.

```shell
git clone https://github.com/slimeslab/ComProScanner.git
cd comproscanner
```

--------------------------------

### Visualize Single Model Evaluation Results

Source: https://slimeslab.github.io/ComProScanner/getting-started/quick-start_q=

Visualizes the evaluation metrics for a single model using a bar chart. This function requires the path to the evaluation results JSON file and an output file path for the generated plot.

```python
from comproscanner import eval_visualizer

# Plot single model evaluation
fig = eval_visualizer.plot_single_bar_chart(
    result_file="semantic_evaluation.json",
    output_file="evaluation_metrics.png"
)

```

--------------------------------

### Clean Extracted Data with ComProScanner

Source: https://slimeslab.github.io/ComProScanner/getting-started/quick-start

Cleans the extracted data by removing entries based on abbreviations, periodic elements, and resolving arithmetic expressions or fractional compositions. It also standardizes bracket usage in the results stored in a JSON file.

```python
scanner.clean_data(
    json_results_file="extracted_results.json"
)

```

--------------------------------

### Write Unit Tests with Pytest

Source: https://slimeslab.github.io/ComProScanner/about/contribution_q=

Example Python code demonstrating how to write unit tests using pytest for the ComProScanner project. These tests cover valid and invalid DOI inputs for composition extraction, ensuring the function behaves as expected.

```python
import pytest
from your_module import extract_composition_data

def test_extract_composition_valid_doi():
    """Test composition extraction with valid DOI."""
    result = extract_composition_data('10.1016/j.example.2024.1', 'd33')
    assert isinstance(result, dict)
    assert 'compositions' in result

def test_extract_composition_invalid_doi():
    """Test composition extraction with invalid DOI."""
    with pytest.raises(ValueError):
        extract_composition_data('invalid-doi', 'd33')

```

--------------------------------

### Visualize Multiple Model Comparisons

Source: https://slimeslab.github.io/ComProScanner/getting-started/quick-start_q=

Compares the evaluation metrics of multiple models using radar charts. This function requires a list of result file paths, corresponding model names, and an output file path for the comparison plot.

```python
from comproscanner import eval_visualizer

# Compare multiple models
fig = eval_visualizer.plot_multiple_radar_charts(
    result_sources=["model1_eval.json", "model2_eval.json"],
    model_names=["GPT-4", "Claude"],
    output_file="model_comparison.png"
)

```
Tokens:
Raw