### Basic ComProScanner Installation Source: https://slimeslab.github.io/ComProScanner/getting-started/installation Installs the latest stable version of ComProScanner and its dependencies from the Python Package Index (PyPI) using pip. This is the recommended method for most users. ```shell pip install comproscanner ``` -------------------------------- ### Build Documentation Locally Source: https://slimeslab.github.io/ComProScanner/about/contribution These commands install the necessary packages (`mkdocs` and `mkdocs-material`) to build and serve the project's documentation locally. After installation, `mkdocs serve` starts a local development server. ```shell pip install mkdocs mkdocs-material mkdocs serve ``` -------------------------------- ### Verify ComProScanner Installation Source: https://slimeslab.github.io/ComProScanner/getting-started/installation Checks if ComProScanner has been installed correctly by importing the library and printing its version number. Successful execution without errors indicates a proper installation. ```python import comproscanner print(comproscanner.__version__) ``` -------------------------------- ### ComProScanner Source Code Installation Source: https://slimeslab.github.io/ComProScanner/getting-started/installation Installs ComProScanner directly from its source code repository. This method is useful for developers who want to contribute to the project or work with the latest unreleased features. It involves cloning the repository and then installing the package in editable mode. ```shell git clone https://github.com/slimeslab/ComProScanner.git cd comproscanner pip install -e . ``` -------------------------------- ### Install ComProScanner Package Source: https://slimeslab.github.io/ComProScanner/index_q= Installs the ComProScanner package using pip. This is the first step to start using the library for extracting composition-property relationships from scientific articles. ```bash pip install comproscanner ``` -------------------------------- ### Initialize ComProScanner Instance Source: https://slimeslab.github.io/ComProScanner/getting-started/quick-start Initializes the ComProScanner with a `main_property_keyword`. This keyword is used for organizing associated files and directories, ensuring a structured approach to data management during the scanning process. ```python from comproscanner import ComProScanner scanner = ComProScanner(main_property_keyword="piezoelectric") ``` -------------------------------- ### ComProScanner Full Workflow for d33 Extraction Source: https://slimeslab.github.io/ComProScanner/getting-started/quick-start_q= Demonstrates the complete minimal workflow for extracting piezoelectric coefficient (d33) data using ComProScanner. It covers initializing the scanner, collecting metadata, processing articles from specified sources, and extracting composition-property relationships. ```python from comproscanner import ComProScanner # Step 1: Initialize with property of interest scanner = ComProScanner(main_property_keyword="piezoelectric") # Step 2: Collect metadata scanner.collect_metadata() # Step 3: Define property keywords for filtering and process articles from specific sources property_keywords = { "exact_keywords": ["d33"], "substring_keywords": [" d 33 "] } scanner.process_articles( property_keywords=property_keywords, source_list=["elsevier", "springer"] ) # Step 4: Extract composition-property relationships scanner.extract_composition_property_data( main_extraction_keyword="d33" ) ``` -------------------------------- ### Complete ComProScanner Workflow for Piezoelectric Data Extraction Source: https://slimeslab.github.io/ComProScanner/getting-started/quick-start Demonstrates the full ComProScanner workflow to extract piezoelectric coefficient (d33) data. This involves initializing the scanner, collecting metadata, processing articles from specified sources, and finally extracting the composition-property relationships. ```python from comproscanner import ComProScanner # Step 1: Initialize with property of interest scanner = ComProScanner(main_property_keyword="piezoelectric") # Step 2: Collect metadata scanner.collect_metadata() # Step 3: Define property keywords for filtering and process articles from specific sources property_keywords = { "exact_keywords": ["d33"], "substring_keywords": [" d 33 "] } s scanner.process_articles( property_keywords=property_keywords, source_list=["elsevier", "springer"] ) # Step 4: Extract composition-property relationships scanner.extract_composition_property_data( main_extraction_keyword="d33" ) ``` -------------------------------- ### Comproscanner JSON Data Structure Example Source: https://slimeslab.github.io/ComProScanner/getting-started/quick-start_q= This JSON object demonstrates the structured data extracted by Comproscanner. It includes composition details, synthesis steps and techniques, and article metadata such as DOI, title, authors, and keywords. The format is designed for organized storage and retrieval of scientific research data. ```json "10.1016/j.apradiso.2024.111655": { "composition_data": { "compositions_property_values": { "Eu1.90Dy0.10Ge2O7": 0.66, "Eu1.90La0.10Ge2O7": 0.36, "Eu1.90Ho0.10Ge2O7": 0.62 }, "property_unit": "pC/N", "family": "RE2B2O7" }, "synthesis_data": { "method": "solid-state reaction", "precursors": [ "Eu2O3", "GeO2", "Dy2O3", "La2O3", "Ho2O3" ], "steps": [ "Starting materials Eu2O3, GeO2, Dy2O3, La2O3 and Ho2O3 were combined in stoichiometric ratios with each dopant at 5 mol%., "Samples were first heated at 800°C for 2 hours in pure alumina crucibles under open atmosphere.", "Materials were then heated to 1150°C for 10 hours followed by slow cooling.", "Resulting materials were ground into powder for further characterization.", "Ceramic discs were formed from obtained powder materials with 1 mm thickness and 10 mm diameter.", "Ceramic discs were compacted using uniaxial pressing under 250 MPa pressure with 2 wt% of 5 wt% PVA aqueous solution as binder.", "Samples were heated at 600°C for 30 minutes to eliminate organic additives.", "Sintering was conducted at 1400°C for 4 hours.", "Silver paste was applied to disc surfaces and fired at 650°C for 1 hour to form surface electrodes.", "Electric field of 9-18 kV/mm was applied in silicon oil bath at 120°C for 30 minutes followed by 24-hour aging." ], "characterization_techniques": [ "TG/DTA", "XRD", "SEM", "EDX", "photoluminescence spectroscopy", "LCR meter", "d33 meter" ] }, "article_metadata": { "doi": "10.1016/j.apradiso.2024.111655", "title": "Novel smart materials with high curie temperatures: Eu1.90Dy0.10Ge2O7, Eu1.90La0.10Ge2O7 and Eu1.90Ho0.10Ge2O7", "journal": "Applied Radiation and Isotopes", "year": "2025", "isOpenAccess": false, "authors": [ { "name": "Esra Öztürk", "affiliation_id": "60020484", "affiliation_name": "Hacettepe Üniversitesi", "affiliation_country": "Turkey" }, { "name": "Nilgun Kalaycioglu Ozpozan", "affiliation_id": "122321412", "affiliation_name": "Erciyes Ün.", "affiliation_country": "Türkiye" }, { "name": "Volkan Kalem", "affiliation_id": "60193845", "affiliation_name": "Konya Technical University", "affiliation_country": "Turkey" } ], "keywords": [ "Curie" ] } } ``` -------------------------------- ### ComProScanner Troubleshooting: Virtual Environment Source: https://slimeslab.github.io/ComProScanner/getting-started/installation Addresses potential dependency conflicts by setting up and activating a new virtual environment before installing ComProScanner. This isolates the project's dependencies from other Python projects on the system. ```shell python -m venv compro_env source compro_env/bin/activate # On Windows: compro_env\Scripts\activate pip install comproscanner ``` -------------------------------- ### Optional LLM Provider Dependencies for ComProScanner Source: https://slimeslab.github.io/ComProScanner/getting-started/installation Installs additional Python packages required to integrate ComProScanner with specific Large Language Model (LLM) providers. Choose the packages corresponding to the LLM services you intend to use. ```shell # For Anthropic Claude pip install langchain-anthropic # For Google Gemini pip install langchain-google-genai # For Ollama (local models) pip install langchain-ollama # For TogetherAI Model Integration pip install langchain-together # For OpenRouter Model Integration pip install langchain-openrouter # For Cohere Model Integration pip install langchain-cohere ``` -------------------------------- ### Install Project Dependencies Source: https://slimeslab.github.io/ComProScanner/about/contribution This command installs the project dependencies in editable mode, including development dependencies. The `.[dev]` part ensures that optional dependencies for development are also installed. ```shell pip install -e ".[dev]" ``` -------------------------------- ### Collect Metadata with ComProScanner Source: https://slimeslab.github.io/ComProScanner/getting-started/quick-start Collects metadata for scientific articles related to piezoelectric materials from the Scopus database, focusing on entries from the last two years. This step helps in identifying relevant literature for further processing. ```python scanner.collect_metadata( base_queries=["piezoelectric", "piezoelectricity"] ) ``` -------------------------------- ### ComProScanner Environment Variables Configuration Source: https://slimeslab.github.io/ComProScanner/getting-started/installation Sets up essential API keys and configuration paths required for ComProScanner to function, particularly for accessing publisher APIs and LLM models. These should be stored in a `.env` file in the project directory for security. ```dotenv # Publisher TDM API Keys (for direct article access) SCOPUS_API_KEY=your_scopus_api_key # for Elsevier as well as metadata retrieval WILEY_API_KEY=your_wiley_api_key SPRINGER_OPENACCESS_API_KEY=your_springer_openaccess_api_key # Springer provides two separate keys for open access and TDM API SPRINGER_TDM_API_KEY=your_springer_tdm_api_key IOP_papers_path=local_path_to_iop_papers # IOP Publishing provides XML articles in bulk through SFTP access # API Keys for LLM Models (at least one is required which will be used for data extraction) OPENAI_API_KEY=your_openai_api_key DEEPSEEK_API_KEY=your_deepseek_api_key ANTHROPIC_API_KEY=your_anthropic_api_key GEMINI_API_KEY=your_google_api_key OPENROUTER_API_KEY=your_openrouter_api_key # Hugging Face API Key (for accessing thellert/physbert_cased model for embeddings) HF_TOKEN=your_huggingface_api_key # Neo4j Configuration (for knowledge graph visualization) NEO4J_URI=bolt://localhost:7687 NEO4J_USER=neo4j NEO4J_PASSWORD=your_password NEO4J_DATABASE=neo4j ``` -------------------------------- ### Install Langchain Ollama Dependency and Ollama Source: https://slimeslab.github.io/ComProScanner/rag-config_q= Installs the Langchain Ollama integration package and provides instructions for installing Ollama locally, enabling the use of local LLMs with Langchain. ```bash pip install langchain-ollama # Install Ollama locally # Visit: https://ollama.ai/download ``` -------------------------------- ### JSON Data Structure Example Source: https://slimeslab.github.io/ComProScanner/getting-started/quick-start This JSON object represents the structured data extracted for a scientific article. It includes details about material composition, synthesis procedures, and bibliographic information. The structure is designed for easy parsing and analysis of research data. ```json { "10.1016/j.apradiso.2024.111655": { "composition_data": { "compositions_property_values": { "Eu1.90Dy0.10Ge2O7": 0.66, "Eu1.90La0.10Ge2O7": 0.36, "Eu1.90Ho0.10Ge2O7": 0.62 }, "property_unit": "pC/N", "family": "RE2B2O7" }, "synthesis_data": { "method": "solid-state reaction", "precursors": [ "Eu2O3", "GeO2", "Dy2O3", "La2O3", "Ho2O3" ], "steps": [ "Starting materials Eu2O3, GeO2, Dy2O3, La2O3 and Ho2O3 were combined in stoichiometric ratios with each dopant at 5 mol%.", "Samples were first heated at 800°C for 2 hours in pure alumina crucibles under open atmosphere.", "Materials were then heated to 1150°C for 10 hours followed by slow cooling.", "Resulting materials were ground into powder for further characterization.", "Ceramic discs were formed from obtained powder materials with 1 mm thickness and 10 mm diameter.", "Ceramic discs were compacted using uniaxial pressing under 250 MPa pressure with 2 wt% of 5 wt% PVA aqueous solution as binder.", "Samples were heated at 600°C for 30 minutes to eliminate organic additives.", "Sintering was conducted at 1400°C for 4 hours.", "Silver paste was applied to disc surfaces and fired at 650°C for 1 hour to form surface electrodes.", "Electric field of 9-18 kV/mm was applied in silicon oil bath at 120°C for 30 minutes followed by 24-hour aging." ], "characterization_techniques": [ "TG/DTA", "XRD", "SEM", "EDX", "photoluminescence spectroscopy", "LCR meter", "d33 meter" ] }, "article_metadata": { "doi": "10.1016/j.apradiso.2024.111655", "title": "Novel smart materials with high curie temperatures: Eu1.90Dy0.10Ge2O7, Eu1.90La0.10Ge2O7 and Eu1.90Ho0.10Ge2O7", "journal": "Applied Radiation and Isotopes", "year": "2025", "isOpenAccess": false, "authors": [ { "name": "Esra Öztürk", "affiliation_id": "60020484", "affiliation_name": "Hacettepe Üniversitesi", "affiliation_country": "Turkey" }, { "name": "Nilgun Kalaycioglu Ozpozan", "affiliation_id": "122321412", "affiliation_name": "Erciyes Ün.", "affiliation_country": "Türkiye" }, { "name": "Volkan Kalem", "affiliation_id": "60193845", "affiliation_name": "Konya Technical University", "affiliation_country": "Turkey" } ], "keywords": [ "Curie" ] } } } ``` -------------------------------- ### Create and Populate .env File Source: https://slimeslab.github.io/ComProScanner/about/contribution This command copies the example environment file (`.env.example`) to a new file named `.env`. You should then add your specific API keys and configurations to this `.env` file. ```shell cp .env.example .env ``` -------------------------------- ### ComproScanner Initialization and Configuration Source: https://slimeslab.github.io/ComProScanner/api This snippet illustrates the initialization of the ComproScanner, defining various optional arguments that control data extraction, saving behavior, LLM and RAG configurations, and logging. ```python is_extract_synthesis_data (bool, optional): A flag to indicate if the synthesis data should be extracted. Defaults to True. is_save_csv (bool, optional): A flag to indicate if the results should be saved in the CSV file. Defaults to False. is_save_relevant (bool, optional): A flag to indicate if only papers with composition-property data should be saved. If True, only saves papers with composition data. If False, saves all processed papers. Defaults to True. llm (LLM, optional): An instance of the LLM class. Defaults to None. materials_data_identifier_query (str, optional): Query to identify the materials data. Must be an 'yes/no' answer. Defaults to "Is there any material chemical composition and corresponding {main_property_keyword} value mentioned in the paper? GIVE ONE WORD ANSWER. Either yes or no." model (str: optional): The model to use (defaults to "gpt-4o-mini") api_base (str, optional): Base URL for standard API endpoints base_url (str, optional): Base URL for the model service api_key (str, optional): API key for the model service output_log_folder (str, optional): Base folder path to save logs. Logs will be saved in {output_log_folder}/{doi}/ subdirectory. Logs will be in JSON format if is_log_json is True, otherwise plain text. Defaults to None. task_output_folder (str, optional): Base folder path to save task outputs. Task outputs will be saved as .txt files in {task_output_folder}/{doi}/ subdirectory. Defaults to None. is_log_json (bool, optional): Flag to save logs in JSON format. Defaults to False. verbose (bool, optional): Flag to enable verbose output inside the terminal (defaults to True) temperature (float, optional): Temperature for text generation - controls randomness (defaults to 0.1) top_p (float, optional): Nucleus sampling parameter for text generation - controls diversity (defaults to 0.9) timeout (int, optional): Request timeout in seconds (defaults to 60) frequency_penalty (float, optional): Frequency penalty for text generation max_tokens (int, optional): Maximum tokens for completion rag_db_path (str, optional): Path to the vector database. Defaults to 'db'. embedding_model (str, optional): Name of the embedding model for RAG. Defaults to 'huggingface:thellert/physbert_cased'. rag_chat_model (str, optional): Name of the chat model for RAG. Defaults to 'gpt-4o-mini'. rag_max_tokens (int, optional): Maximum tokens for completion for RAG. Defaults to 512. rag_top_k (int, optional): Top k value for sampling for RAG. Defaults to 3. rag_base_url (str, optional): Base URL for the RAG model service. **flow_optional_args: Optional arguments for the MaterialsFlow class. ``` -------------------------------- ### Upgrade ComProScanner Source: https://slimeslab.github.io/ComProScanner/getting-started/installation Updates an existing ComProScanner installation to the latest available version on PyPI. This command ensures you are using the most recent stable release with all the latest features and bug fixes. ```shell pip install --upgrade comproscanner ``` -------------------------------- ### Basic Usage of ComProScanner Source: https://slimeslab.github.io/ComProScanner/index_q= Demonstrates the basic workflow of ComProScanner, including initializing the scanner, collecting metadata, processing articles from specified sources, and extracting composition-property data. ```python from comproscanner import ComProScanner # Initialize with property of interest scanner = ComProScanner(main_property_keyword="piezoelectric") # Collect metadata scanner.collect_metadata( base_queries=["piezoelectric", "piezoelectricity"], ) # Define property keywords for filtering property_keywords = { "exact_keywords": ["d33"], "substring_keywords": [" d 33 "] } # Process articles scanner.process_articles( property_keywords=property_keywords, source_list=["elsevier", "springer"] ) # Extract data scanner.extract_composition_property_data( main_extraction_keyword="d33" ) ``` -------------------------------- ### ComProScanner Data Visualization Examples (Python) Source: https://slimeslab.github.io/ComProScanner/usage/visualization/overview Demonstrates how to use the ComProScanner data_visualizer module to create visualizations such as pie charts for material family distributions and knowledge graphs from extracted data sources. Requires 'results.json' as input. ```python from comproscanner import data_visualizer # Pie chart fig = data_visualizer.plot_family_pie_chart( data_sources=["results.json"], output_file="families.png" ) # Knowledge graph data_visualizer.create_knowledge_graph( result_file="results.json" ) ``` -------------------------------- ### Evaluate Extraction Quality with ComProScanner Source: https://slimeslab.github.io/ComProScanner/getting-started/quick-start Evaluates the quality of the extracted data against ground truth using both semantic and agentic evaluation methods. The results are saved to specified JSON output files for further analysis. ```python from comproscanner import evaluate_semantic, evaluate_agentic # Semantic evaluation semantic_results = evaluate_semantic( ground_truth_file="ground_truth.json", test_data_file="extracted_results.json", output_file="semantic_evaluation.json" ) # Agentic evaluation (more advanced) agentic_results = evaluate_agentic( ground_truth_file="ground_truth.json", test_data_file="extracted_results.json", output_file="agentic_evaluation.json" ) ``` -------------------------------- ### Visualize Extracted Data using ComProScanner Source: https://slimeslab.github.io/ComProScanner/getting-started/quick-start Generates visualizations from the extracted results, including a pie chart for material family distribution and a knowledge graph. This helps in understanding the patterns and relationships within the data. ```python from comproscanner import data_visualizer # Plot material families distribution fig = data_visualizer.plot_family_pie_chart( data_sources=["extracted_results.json"], output_file="family_distribution.png" ) # Create knowledge graph data_visualizer.create_knowledge_graph( result_file="extracted_results.json" ) ``` -------------------------------- ### Configure Material Data Identification Query Source: https://slimeslab.github.io/ComProScanner/api Sets up the query to identify material data, including optional parameters for language model, API, logging, output folders, RAG settings, and generation parameters like temperature and top_p. It validates required inputs like `main_extraction_keyword` and `test_doi_list_file` when applicable. ```python def materials_data_identifier_query( main_extraction_keyword: str, materials_data_identifier_query: str = None, model: str = "gpt-4o-mini", api_base: str = None, base_url: str = None, api_key: str = None, output_log_folder: str = None, task_output_folder: str = None, is_log_json: bool = False, verbose: bool = True, temperature: float = 0.1, top_p: float = 0.9, timeout: int = 60, frequency_penalty: float = None, max_tokens: int = None, rag_db_path: str = 'db', embedding_model: str = 'huggingface:thellert/physbert_cased', rag_chat_model: str = 'gpt-4o-mini', rag_max_tokens: int = 512, rag_top_k: int = 3, rag_base_url: str = None, flow_optional_args: dict = None, is_test_data_preparation: bool = False, test_doi_list_file: str = None, total_test_data: int = None, test_random_seed: int = None, checked_doi_list_file: str = None, main_property_keyword: str, json_results_file: str = None, start_row: int = 0, num_rows: int = None, is_only_consider_test_doi_list: bool = False, ) -> None: """ Identifies if a paper contains mentions of material chemical composition and corresponding property values. Args: materials_data_identifier_query (str, optional): Query to identify the materials data. Must be an 'yes/no' answer. Defaults to "Is there any material chemical composition and corresponding {main_property_keyword} value mentioned in the paper? GIVE ONE WORD ANSWER. Either yes or no." model (str: optional): The model to use (defaults to "gpt-4o-mini") api_base (str, optional): Base URL for standard API endpoints base_url (str, optional): Base URL for the model service api_key (str, optional): API key for the model service output_log_folder (str, optional): Base folder path to save logs. Logs will be saved in {output_log_folder}/{doi}/ subdirectory. Logs will be in JSON format if is_log_json is True, otherwise plain text. Defaults to None. task_output_folder (str, optional): Base folder path to save task outputs. Task outputs will be saved as .txt files in {task_output_folder}/{doi}/ subdirectory. Defaults to None. is_log_json (bool, optional): Flag to save logs in JSON format. Defaults to False. verbose (bool, optional): Flag to enable verbose output inside the terminal (defaults to True) temperature (float, optional): Temperature for text generation - controls randomness (defaults to 0.1) top_p (float, optional): Nucleus sampling parameter for text generation - controls diversity (defaults to 0.9) timeout (int, optional): Request timeout in seconds (defaults to 60) frequency_penalty (float, optional): Frequency penalty for text generation max_tokens (int, optional): Maximum tokens for completion rag_db_path (str, optional): Path to the vector database. Defaults to 'db'. embedding_model (str, optional): Name of the embedding model for RAG. Defaults to 'huggingface:thellert/physbert_cased'. rag_chat_model (str, optional): Name of the chat model for RAG. Defaults to 'gpt-4o-mini'. rag_max_tokens (int, optional): Maximum tokens for completion for RAG. Defaults to 512. rag_top_k (int, optional): Top k value for sampling for RAG. Defaults to 3. rag_base_url (str, optional): Base URL for the RAG model service. **flow_optional_args: Optional arguments for the MaterialsFlow class. Raises: ValueErrorHandler: If main_extraction_keyword is not provided. """ if main_extraction_keyword is None: logger.error( "main_extraction_keyword cannot be None. Please provide a valid keyword. Exiting..." ) raise ValueErrorHandler( message="Please provide main_extraction_keyword to proceed for identifying sentences based on property." ) if is_test_data_preparation and test_doi_list_file is None: logger.error("Test data file name is required for test data preparation.") raise ValueErrorHandler( message="Test data file name is required for test data preparation." ) self.is_test_data_preparation = is_test_data_preparation if is_test_data_preparation: self.test_doi_list_file = test_doi_list_file if total_test_data is None: self.total_test_data = 50 else: self.total_test_data = total_test_data llm_config = LLMConfig( model=model, api_base=api_base, base_url=base_url, api_key=api_key, temperature=temperature, top_p=top_p, timeout=timeout, frequency_penalty=frequency_penalty, max_tokens=max_tokens, ) llm = llm_config.get_llm() rag_config = RAGConfig( rag_db_path=rag_db_path, embedding_model=embedding_model, rag_chat_model=rag_chat_model, rag_max_tokens=rag_max_tokens, rag_top_k=rag_top_k, rag_base_url=rag_base_url, ) if materials_data_identifier_query is None: materials_data_identifier_query = f"Is there any material chemical composition and corresponding {self.main_property_keyword} value mentioned in the paper? Give one word answer. Either yes or no." preparator = MatPropDataPreparator( main_property_keyword=self.main_property_keyword, main_extraction_keyword=main_extraction_keyword, json_results_file=json_results_file, start_row=start_row, num_rows=num_rows, is_test_data_preparation=is_test_data_preparation, test_doi_list_file=test_doi_list_file, is_only_consider_test_doi_list=is_only_consider_test_doi_list, total_test_data=total_test_data, test_random_seed=test_random_seed, checked_doi_list_file=checked_doi_list_file, ) ``` -------------------------------- ### Process Wiley, IOP, and PDF Articles - Python Source: https://slimeslab.github.io/ComProScanner/api This code snippet demonstrates the processing of articles from different sources: Wiley, IOP (Institute of Physics), and local PDF files. It initializes specific processors for each source type, passing relevant configuration parameters such as main property keywords, batch sizes, date ranges, and RAG (Retrieval-Augmented Generation) configurations. The processors are then invoked to handle the respective article types. ```Python # Process Wiley articles if "wiley" in source_list: from .article_processors.wiley_processor import WileyArticleProcessor wiley_processor = WileyArticleProcessor( main_property_keyword=self.main_property_keyword, property_keywords=property_keywords, sql_batch_size=sql_batch_size, csv_batch_size=csv_batch_size, start_row=start_row, end_row=end_row, doi_list=doi_list, is_sql_db=is_sql_db, rag_config=rag_config, ) wiley_processor.process_wiley_articles() # Process IOP articles if "iop" in source_list: from .article_processors.iop_processor import IOPArticleProcessor iop_processor = IOPArticleProcessor( main_property_keyword=self.main_property_keyword, property_keywords=property_keywords, sql_batch_size=sql_batch_size, csv_batch_size=csv_batch_size, start_row=start_row, end_row=end_row, doi_list=doi_list, is_sql_db=is_sql_db, rag_config=rag_config, ) iop_processor.process_iop_articles() # Process PDFs if "pdfs" in source_list: from .article_processors.pdfs_processor import PDFsProcessor pdf_processor = PDFsProcessor( folder_path=folder_path, main_property_keyword=self.main_property_keyword, property_keywords=property_keywords, sql_batch_size=sql_batch_size, csv_batch_size=csv_batch_size, is_sql_db=is_sql_db, rag_config=rag_config, ) pdf_processor.process_pdfs() ``` -------------------------------- ### Clone ComProScanner Repository Source: https://slimeslab.github.io/ComProScanner/about/contribution This command clones the ComProScanner repository from GitHub to your local machine. After cloning, navigate into the project directory. ```shell git clone https://github.com/slimeslab/ComProScanner.git cd comproscanner ``` -------------------------------- ### Visualize Single Model Evaluation Results Source: https://slimeslab.github.io/ComProScanner/getting-started/quick-start_q= Visualizes the evaluation metrics for a single model using a bar chart. This function requires the path to the evaluation results JSON file and an output file path for the generated plot. ```python from comproscanner import eval_visualizer # Plot single model evaluation fig = eval_visualizer.plot_single_bar_chart( result_file="semantic_evaluation.json", output_file="evaluation_metrics.png" ) ``` -------------------------------- ### Clean Extracted Data with ComProScanner Source: https://slimeslab.github.io/ComProScanner/getting-started/quick-start Cleans the extracted data by removing entries based on abbreviations, periodic elements, and resolving arithmetic expressions or fractional compositions. It also standardizes bracket usage in the results stored in a JSON file. ```python scanner.clean_data( json_results_file="extracted_results.json" ) ``` -------------------------------- ### Write Unit Tests with Pytest Source: https://slimeslab.github.io/ComProScanner/about/contribution_q= Example Python code demonstrating how to write unit tests using pytest for the ComProScanner project. These tests cover valid and invalid DOI inputs for composition extraction, ensuring the function behaves as expected. ```python import pytest from your_module import extract_composition_data def test_extract_composition_valid_doi(): """Test composition extraction with valid DOI.""" result = extract_composition_data('10.1016/j.example.2024.1', 'd33') assert isinstance(result, dict) assert 'compositions' in result def test_extract_composition_invalid_doi(): """Test composition extraction with invalid DOI.""" with pytest.raises(ValueError): extract_composition_data('invalid-doi', 'd33') ``` -------------------------------- ### Visualize Multiple Model Comparisons Source: https://slimeslab.github.io/ComProScanner/getting-started/quick-start_q= Compares the evaluation metrics of multiple models using radar charts. This function requires a list of result file paths, corresponding model names, and an output file path for the comparison plot. ```python from comproscanner import eval_visualizer # Compare multiple models fig = eval_visualizer.plot_multiple_radar_charts( result_sources=["model1_eval.json", "model2_eval.json"], model_names=["GPT-4", "Claude"], output_file="model_comparison.png" ) ```