### Install and Run Ollama Locally Source: https://github.com/google/langextract/blob/main/examples/ollama/README.md Install Ollama, pull a model, and start the server. Keep the server running in a separate terminal before executing the demo script. ```bash # Install and start Ollama ollama pull gemma2:2b ollama serve # Keep this running in a separate terminal # Run the demo python demo_ollama.py ``` -------------------------------- ### Copy and Rename Example Directory Source: https://github.com/google/langextract/blob/main/examples/custom_provider_plugin/README.md Copy the existing custom provider plugin example directory and rename the internal package directory to start creating your own provider. ```bash # Copy this example directory cp -r examples/custom_provider_plugin/ ~/langextract-myprovider/ # Rename the package directory cd ~/langextract-myprovider/ mv langextract_provider_example langextract_myprovider ``` -------------------------------- ### Install LangExtract from Source Source: https://github.com/google/langextract/blob/main/README.md Installs LangExtract from its source code repository. Use '-e .' for basic installation or '.[dev]'/'[test]' for development or testing respectively. ```bash git clone https://github.com/google/langextract.git cd langextract # For basic installation: pip install -e . # For development (includes linting tools): pip install -e ".[dev]" # For testing (includes pytest): pip install -e ".[test]" ``` -------------------------------- ### Install Custom Provider Plugin Source: https://github.com/google/langextract/blob/main/examples/custom_provider_plugin/README.md Install the custom provider plugin in development mode from its directory. ```bash # Navigate to this example directory first cd examples/custom_provider_plugin # Install in development mode pip install -e . ``` -------------------------------- ### Provider Package Structure Example Source: https://github.com/google/langextract/blob/main/langextract/providers/README.md Example directory structure for a custom LangExtract provider plugin package. Includes pyproject.toml for entry point configuration. ```text langextract-yourprovider/ ├── pyproject.toml # Package config with entry point ├── README.md # Documentation ├── LICENSE # License file └── langextract_yourprovider/ # Package directory ├── __init__.py # Exports provider class ├── provider.py # Provider implementation └── schema.py # (Optional) Custom schema ``` -------------------------------- ### Using an Installed Third-Party Provider Source: https://github.com/google/langextract/blob/main/langextract/providers/README.md Shows how to install and immediately use a third-party provider without needing explicit imports after installation. The provider is automatically discovered and available for use with the `langextract.extract` function. ```bash pip install langextract-yourprovider ``` ```python # Use it immediately - no imports needed! import langextract as lx result = lx.extract( text_or_documents="...", prompt_description="Extract key information", examples=[...], model_id="yourmodel-latest", # Automatically finds the provider ) ``` -------------------------------- ### Install LangExtract from PyPI Source: https://github.com/google/langextract/blob/main/README.md Installs the LangExtract library using pip. Recommended for most users. Consider using a virtual environment for isolated installations. ```bash pip install langextract ``` ```bash python -m venv langextract_env source langextract_env/bin/activate # On Windows: langextract_env\Scripts\activate pip install langextract ``` -------------------------------- ### Install and Test Custom Provider Source: https://github.com/google/langextract/blob/main/examples/custom_provider_plugin/README.md Installs the custom provider in development mode and runs a test to verify its registration with the LangExtract router. ```bash # Install in development mode pip install -e . # Test your provider python -c " from langextract.providers import load_plugins_once, router load_plugins_once() print('Provider registered:', any('myprovider' in str(e) for e in router.list_entries())) " ``` -------------------------------- ### Clone Repository and Install Dependencies Source: https://github.com/google/langextract/blob/main/README.md Steps to clone the LangExtract repository and install necessary testing dependencies using pip. ```bash # Clone the repository git clone https://github.com/google/langextract.git cd langextract # Install with test dependencies pip install -e ".[test]" ``` -------------------------------- ### Custom Schema Implementation - from_examples Source: https://github.com/google/langextract/blob/main/examples/custom_provider_plugin/README.md Implement the from_examples class method to analyze example data and build a schema. ```python from langextract.core import schema as core_schema class CustomProviderSchema(core_schema.BaseSchema): @classmethod def from_examples(cls, examples_data, attribute_suffix="_attributes"): # Analyze examples to find patterns # Build schema based on extraction classes and attributes seen schema_dict = {} return cls(schema_dict) ``` -------------------------------- ### Basic Extraction with Examples Source: https://github.com/google/langextract/blob/main/skills/langextract-usage/SKILL.md Perform a basic extraction using `lx.extract()`, providing example data and a prompt description. The result is iterated to print extracted information. ```python import langextract as lx examples = [ lx.data.ExampleData( text="Patient takes lisinopril 10mg daily for hypertension.", extractions=[ lx.data.Extraction( extraction_class="medication", extraction_text="lisinopril", attributes={"dose": "10mg", "frequency": "daily"}, ), lx.data.Extraction( extraction_class="condition", extraction_text="hypertension", attributes={"status": "active"}, ), ], ) ] result = lx.extract( text_or_documents="Patient is prescribed metformin 500mg twice daily.", prompt_description="Extract medications and conditions with attributes.", examples=examples, model_id="gemini-2.5-flash", ) for e in result.extractions: print(e.extraction_class, e.extraction_text) print(f" char_interval: {e.char_interval}") print(f" attributes: {e.attributes}") ``` -------------------------------- ### Install External Plugin Source: https://github.com/google/langextract/blob/main/langextract/providers/README.md This command installs an external provider plugin for LangExtract. Once installed, the provider becomes available for use within LangExtract without further configuration. ```bash pip install langextract-yourprovider # That's it! The provider is now available in langextract ``` -------------------------------- ### Install Optional Dependencies Source: https://github.com/google/langextract/blob/main/langextract/providers/README.md Install optional dependencies, such as the 'openai' package, to resolve 'Missing Dependencies' errors. ```python InferenceConfigError: OpenAI provider requires openai package ``` ```bash pip install langextract[openai] ``` -------------------------------- ### Test Schema Creation Source: https://github.com/google/langextract/blob/main/langextract/providers/README.md Test schema creation with `Schema.from_examples(examples)` to ensure schema constraints are applied correctly. ```python # Schema constraints not being applied ``` ```python Schema.from_examples(examples) ``` -------------------------------- ### Define Extraction Prompt and Examples Source: https://github.com/google/langextract/blob/main/examples/notebooks/romeo_juliet_extraction.ipynb Defines the prompt for the extraction task and provides a high-quality example of expected output. This helps guide the model's extraction process. ```python import langextract as lx import textwrap # Define the extraction task prompt = textwrap.dedent(""" Extract characters, emotions, and relationships in order of appearance. Use exact text for extractions. Do not paraphrase or overlap entities. Provide meaningful attributes for each entity to add context.""") # Provide a high-quality example examples = [ lx.data.ExampleData( text="ROMEO. But soft! What light through yonder window breaks? It is the east, and Juliet is the sun.", extractions=[ lx.data.Extraction( extraction_class="character", extraction_text="ROMEO", attributes={"emotional_state": "wonder"} ), lx.data.Extraction( extraction_class="emotion", extraction_text="But soft!", attributes={"feeling": "gentle awe"} ), lx.data.Extraction( extraction_class="relationship", extraction_text="Juliet is the sun", attributes={"type": "metaphor"} ), ] ) ] ``` -------------------------------- ### Install LangExtract Source: https://github.com/google/langextract/blob/main/examples/notebooks/romeo_juliet_extraction.ipynb Installs the langextract library using pip. This is a prerequisite for using the library. ```python # Install LangExtract %pip install -q langextract ``` -------------------------------- ### Define Example Data for LangExtract Source: https://github.com/google/langextract/blob/main/README.md Prepare example data including text and expected extractions with attributes for LangExtract. Ensure extractions are verbatim from the text and in order. ```python examples = [ lx.data.ExampleData( text="ROMEO. But soft! What light through yonder window breaks? It is the east, and Juliet is the sun.", extractions=[ lx.data.Extraction( extraction_class="character", extraction_text="ROMEO", attributes={"emotional_state": "wonder"} ), lx.data.Extraction( extraction_class="emotion", extraction_text="But soft!", attributes={"feeling": "gentle awe"} ), lx.data.Extraction( extraction_class="relationship", extraction_text="Juliet is the sun", attributes={"type": "metaphor"} ), ] ) ] ``` -------------------------------- ### Run LangExtract with Input Text and Examples Source: https://github.com/google/langextract/blob/main/README.md Execute the extraction process using LangExtract by providing input text, prompt description, examples, and a model ID. The 'gemini-3.5-flash' model is recommended. ```python # The input text to be processed input_text = "Lady Juliet gazed longingly at the stars, her heart aching for Romeo" # Run the extraction result = lx.extract( text_or_documents=input_text, prompt_description=prompt, examples=examples, model_id="gemini-3.5-flash", ) ``` -------------------------------- ### Install LangExtract Source: https://github.com/google/langextract/blob/main/skills/langextract-usage/SKILL.md Install the LangExtract library and optional support for OpenAI models using pip. ```bash pip install langextract # For OpenAI model support: pip install langextract[openai] ``` -------------------------------- ### Define Extraction Task with Prompt and Example Source: https://github.com/google/langextract/blob/main/README.md This snippet demonstrates how to define an extraction task using a Python prompt and an example. It imports the necessary libraries and sets up the prompt string with specific instructions for extraction, including using exact text and providing attributes. ```python import langextract as lx import textwrap # 1. Define the prompt and extraction rules prompt = textwrap.dedent(""" Extract characters, emotions, and relationships in order of appearance. Use exact text for extractions. Do not paraphrase or overlap entities. Provide meaningful attributes for each entity to add context.""" ) ``` -------------------------------- ### Run Ollama and Demo with Docker Compose Source: https://github.com/google/langextract/blob/main/examples/ollama/README.md Use Docker Compose to run both Ollama and the LangExtract demo in separate containers for a production-ready setup. ```bash # Runs both Ollama and the demo in containers docker-compose up ``` -------------------------------- ### Test Custom Provider Plugin Source: https://github.com/google/langextract/blob/main/examples/custom_provider_plugin/README.md Run the test script to verify the functionality of the custom provider plugin. This must be executed from the example directory. ```bash # Test the provider (must be run from this directory) python test_example_provider.py ``` -------------------------------- ### Install and Use Pre-commit Hooks Source: https://github.com/google/langextract/blob/main/CONTRIBUTING.md Install the pre-commit package and its git hooks for automatic code formatting checks before each commit. You can also run all checks manually on all files. ```bash # Install pre-commit pip install pre-commit # Install the git hooks pre-commit install # Run manually on all files pre-commit run --all-files ``` -------------------------------- ### Define Example Data for Medication Extraction Source: https://github.com/google/langextract/blob/main/docs/examples/medication_examples.md Define example data with medication groups to guide the extraction process. This includes text and corresponding extractions with attributes like 'medication_group'. ```python import langextract as lx # Define example data with medication groups examples = [ lx.data.ExampleData( text="Patient takes Aspirin 100mg daily for heart health and Simvastatin 20mg at bedtime.", extractions=[ # First medication group lx.data.Extraction( extraction_class="medication", extraction_text="Aspirin", attributes={"medication_group": "Aspirin"} # Group identifier ), lx.data.Extraction( extraction_class="dosage", extraction_text="100mg", attributes={"medication_group": "Aspirin"} ), lx.data.Extraction( extraction_class="frequency", extraction_text="daily", attributes={"medication_group": "Aspirin"} ), lx.data.Extraction( extraction_class="condition", extraction_text="heart health", attributes={"medication_group": "Aspirin"} ), # Second medication group lx.data.Extraction( extraction_class="medication", extraction_text="Simvastatin", attributes={"medication_group": "Simvastatin"} ), lx.data.Extraction( extraction_class="dosage", extraction_text="20mg", attributes={"medication_group": "Simvastatin"} ), lx.data.Extraction( extraction_class="frequency", extraction_text="at bedtime", attributes={"medication_group": "Simvastatin"} ) ] ) ] ``` -------------------------------- ### Python Example: Batch Processing Shakespeare with LangExtract Source: https://github.com/google/langextract/blob/main/docs/examples/batch_api_example.md Demonstrates downloading text, configuring batch settings, and running extraction using Vertex AI Batch API via the langextract library. Ensure to replace 'your-gcp-project' with your actual GCP Project ID. ```python import requests import textwrap import langextract as lx import logging # Configure logging to see progress (both in console and file) logging.basicConfig( level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s', handlers=[ logging.FileHandler("batch_process.log"), logging.StreamHandler() ] ) # 1. Download Text (Shakespeare's Romeo and Juliet) url = "https://www.gutenberg.org/files/1513/1513-0.txt" print(f"Downloading {url}...") text = requests.get(url).text # Process first ~20 pages (approx. 60k characters). text_subset = text[:60000] print(f"Processing first {len(text_subset)} characters...") # 2. Define Prompt & Examples prompt = textwrap.dedent(""" Extract characters and emotions from the text. Use exact text from the input for extraction_text.""") examples = [ lx.data.ExampleData( text="ROMEO. But soft! What light through yonder window breaks?", extractions=[ lx.data.Extraction(extraction_class="character", extraction_text="ROMEO"), lx.data.Extraction(extraction_class="emotion", extraction_text="But soft!"), ] ) ] # 3. Configure Batch Settings batch_config = { "enabled": True, "threshold": 10, "poll_interval": 30, "timeout": 3600, # Set to True to cache results in GCS. Add timestamp to prompt to force re-run. "enable_caching": True, # Retention policy for GCS bucket (days). None for permanent. "retention_days": 30, } # 4. Run Extraction # langextract will automatically chunk the text and submit a batch job. results = lx.extract( text_or_documents=text_subset, prompt_description=prompt, examples=examples, model_id="gemini-3.5-flash", max_char_buffer=500, batch_length=1000, language_model_params={ "vertexai": True, "project": "your-gcp-project", # TODO: Replace with your Project ID. "location": "us-central1", "batch": batch_config } ) ## GCS File Structure The library automatically creates and manages a GCS bucket for you, named: `langextract-{project}-{location}-batch` Inside this bucket, data is organized as follows: - **Input**: `batch-input/{job_name}.jsonl` - **Output**: `batch-input/{job_name}/dest/prediction-model-{timestamp}/predictions.jsonl` - **Cache**: `cache/{hash}.json` (Individual cached results) ## Cost Optimization & Caching LangExtract's batch processing is designed to minimize costs: 1. **Cost Efficiency**: Vertex AI Batch predictions are typically ~50% cheaper than online predictions. 2. **Smart Caching**: - Results are cached in your GCS bucket (`cache/` directory). - **Instant Retrieval**: Re-running identical prompts fetches results directly from storage, bypassing model inference. - **Reduced Inference**: You avoid paying for redundant model calls on previously processed data. - **Lifecycle Management**: Use `retention_days` (e.g., 30) to automatically clean up old data and manage storage usage. ## Analyze Results print(f"Extracted {len(results.extractions)} entities.") print("First 5 extractions:") for extraction in results.extractions[:5]: print(f"- {extraction.extraction_class}: {extraction.extraction_text}") ``` -------------------------------- ### Build and Publish External Plugin Source: https://github.com/google/langextract/blob/main/langextract/providers/README.md These commands are used to build a distribution package for an external LangExtract provider and upload it to PyPI. This makes the provider easily installable by users. ```bash pip install build twine python -m build twine upload dist/* ``` -------------------------------- ### Full Example of lx.extract with resolver_params Source: https://github.com/google/langextract/blob/main/skills/langextract-usage/references/resolver-params.md Demonstrates how to use resolver_params to configure extraction behavior, including suppressing parse errors, customizing index suffixes, and enabling/tuning fuzzy alignment. ```python result = lx.extract( text_or_documents=text, examples=examples, prompt_description=prompt, model_id="gemini-2.5-flash", resolver_params={ "suppress_parse_errors": True, "extraction_index_suffix": "_index", "enable_fuzzy_alignment": True, "fuzzy_alignment_threshold": 0.75, "fuzzy_alignment_algorithm": "lcs", "fuzzy_alignment_min_density": 1 / 3, "accept_match_lesser": True, }, ) ``` -------------------------------- ### Copy LangExtract Skill Source: https://github.com/google/langextract/blob/main/skills/langextract-usage/README.md Use this command to statically install the LangExtract skill by copying its directory to the tool's skill path. ```bash cp -R skills/langextract-usage ``` -------------------------------- ### Key Parameters for lx.extract() Source: https://github.com/google/langextract/blob/main/skills/langextract-usage/SKILL.md Illustrates the key parameters available for the `lx.extract()` function, including text input, prompt description, examples, model ID, and performance tuning options. ```python result = lx.extract( text_or_documents=text, # str, URL, or list of Documents prompt_description=prompt, # what to extract (optional) examples=examples, # few-shot examples (required) model_id="gemini-2.5-flash", # model to use extraction_passes=1, # >1 for higher recall (costs more) max_char_buffer=1000, # chunk size batch_length=10, # chunks per batch max_workers=10, # parallel workers (provider-dependent) context_window_chars=None, # cross-chunk context for coreference ) ``` -------------------------------- ### Sample Output: LangExtract Batch Processing Source: https://github.com/google/langextract/blob/main/docs/examples/batch_api_example.md Displays the expected output format for extracted entities after running the batch processing example. Shows the total count and the first five extracted items. ```text Extracted 767 entities. First 5 extractions: - character: ESCALUS - character: MERCUTIO - character: PARIS - character: Page to Paris - character: MONTAGUE ``` -------------------------------- ### Symlink LangExtract Skill Source: https://github.com/google/langextract/blob/main/skills/langextract-usage/README.md Use this command to install the LangExtract skill by creating a symbolic link. This allows the skill to automatically track updates from the repository. ```bash ln -s $(pwd)/skills/langextract-usage ``` -------------------------------- ### Full Japanese Information Extraction Pipeline Source: https://github.com/google/langextract/blob/main/docs/examples/japanese_extraction.md This example shows the complete process of extracting named entities (Person, Location, Organization) from Japanese text using LangExtract. It highlights the initialization and use of `UnicodeTokenizer` for proper character segmentation, along with defining extraction prompts and few-shot examples. ```python import langextract as lx from langextract.core import tokenizer # Japanese text with entities (Person, Location, Organization) # "Mr. Tanaka from Tokyo works at Google." input_text = "東京出身の田中さんはGoogleで働いています。" # Define extraction prompt prompt_description = "Extract named entities including Person, Location, and Organization." # Define example data (few-shot examples help the model understand the task) examples = [ lx.data.ExampleData( text="大阪の山田さんはソニーに入社しました。", # Mr. Yamada from Osaka joined Sony. extractions=[ lx.data.Extraction(extraction_class="Location", extraction_text="大阪"), lx.data.Extraction(extraction_class="Person", extraction_text="山田"), lx.data.Extraction(extraction_class="Organization", extraction_text="ソニー"), ] ) ] # 1. Initialize the UnicodeTokenizer # Essential for Japanese to ensure correct grapheme segmentation. unicode_tokenizer = tokenizer.UnicodeTokenizer() # 2. Run Extraction with the Custom Tokenizer result = lx.extract( text_or_documents=input_text, prompt_description=prompt_description, examples=examples, model_id="gemini-3.5-flash", tokenizer=unicode_tokenizer, # <--- Pass the tokenizer here api_key="your-api-key-here" # Optional if env var is set ) # 3. Display Results print(f"Input: {input_text}\n") print("Extracted Entities:") for entity in result.extractions: position_info = "" if entity.char_interval: start, end = entity.char_interval.start_pos, entity.char_interval.end_pos position_info = f" (pos: {start}-{end})" print(f"• {entity.extraction_class}: {entity.extraction_text}{position_info}") # Expected Output: # Input: 東京出身の田中さんはGoogleで働いています。 # # Extracted Entities: # • Location: 東京 (pos: 0-2) # • Person: 田中 (pos: 5-7) # • Organization: Google (pos: 10-16) ``` -------------------------------- ### Provider Registration Example Source: https://github.com/google/langextract/blob/main/langextract/providers/README.md Register a custom provider class with specific model ID patterns. This class must inherit from base_model.BaseLanguageModel and implement the infer method. ```python from langextract.core import base_model from langextract.providers import router # Gemini provider registration: @router.register( r'^GeminiLanguageModel$', # Explicit: model_id="GeminiLanguageModel" r'^gemini', # Prefix: model_id="gemini-3.5-flash" r'^palm' # Legacy: model_id="palm-2" ) class GeminiLanguageModel(base_model.BaseLanguageModel): def __init__(self, model_id: str, api_key: str = None, **kwargs): # Initialize Gemini client ... def infer(self, batch_prompts, **kwargs): # Call Gemini API ... ``` -------------------------------- ### Extract Entities from Sample Text Source: https://github.com/google/langextract/blob/main/examples/notebooks/romeo_juliet_extraction.ipynb Performs entity extraction on a short input text using the defined prompt and examples. Displays the extracted entities and their attributes. ```python # Simple extraction from a short text input_text = "Lady Juliet gazed longingly at the stars, her heart aching for Romeo" result = lx.extract( text_or_documents=input_text, prompt_description=prompt, examples=examples, model_id="gemini-3.5-flash", ) # Display results print(f"Extracted {len(result.extractions)} entities:\n") for extraction in result.extractions: print(f"• {extraction.extraction_class}: '{extraction.extraction_text}'") if extraction.attributes: for key, value in extraction.attributes.items(): print(f" - {key}: {value}") ``` -------------------------------- ### Run Ollama Integration Tests Source: https://github.com/google/langextract/blob/main/README.md Execute Ollama integration tests locally. This requires Ollama to be installed and running with the gemma2:2b model. The test automatically detects Ollama availability. ```bash # Test Ollama integration (requires Ollama running with gemma2:2b model) tox -e ollama-integration ``` -------------------------------- ### Perform Medication Extraction and Process Results Source: https://github.com/google/langextract/blob/main/docs/examples/medication_examples.md Perform extraction using the defined examples and then process the results to group and display extracted medications. This includes printing the input text and iterating through the results to group by 'medication_group'. ```python input_text = "Patient takes Aspirin 100mg daily for heart health and Simvastatin 20mg at bedtime." prompt_description = "Extract medication information." result = lx.extract( text_or_documents=input_text, prompt_description=prompt_description, examples=examples, model_id="gemini-2.5-pro", api_key="your-api-key-here" # Optional if LANGEXTRACT_API_KEY environment variable is set ) # Display grouped medications print(f"Input text: {input_text.strip()}\n") print("Extracted Medications:") # Group by medication medication_groups = {} for extraction in result.extractions: if not extraction.attributes or "medication_group" not in extraction.attributes: print(f"Warning: Missing medication_group for {extraction.extraction_text}") continue group_name = extraction.attributes["medication_group"] medication_groups.setdefault(group_name, []).append(extraction) # Print each medication group for med_name, extractions in medication_groups.items(): print(f"\n* {med_name}") for extraction in extractions: position_info = "" if extraction.char_interval: start, end = extraction.char_interval.start_pos, extraction.char_interval.end_pos position_info = f" (pos: {start}-{end})" print(f" • {extraction.extraction_class.capitalize()}: {extraction.extraction_text}{position_info}") # Save and visualize the results lx.io.save_annotated_documents( [result], output_name="medical_relationship_extraction.jsonl", output_dir="." ) ``` -------------------------------- ### Extract Text with Local Ollama Gemma2 Model Source: https://github.com/google/langextract/blob/main/README.md Run local inference using Ollama for models like Gemma2. Ensure Ollama is installed, the model is pulled, and the server is running. ```python import langextract as lx result = lx.extract( text_or_documents=input_text, prompt_description=prompt, examples=examples, model_id="gemma2:2b", # Automatically selects Ollama provider model_url="http://localhost:11434", ) ``` -------------------------------- ### Configure Entry Point in pyproject.toml Source: https://github.com/google/langextract/blob/main/langextract/providers/README.md This TOML configuration sets up the build system and defines the project metadata, including the entry point for LangExtract providers. It specifies the package name, version, dependencies, and how the provider should be registered. ```toml [build-system] requires = ["setuptools>=61.0", "wheel"] build-backend = "setuptools.build_meta" [project] name = "langextract-yourprovider" version = "0.1.0" dependencies = ["langextract>=1.0.0"] [project.entry-points."langextract.providers"] yourprovider = "langextract_yourprovider:YourProviderLanguageModel" ``` -------------------------------- ### Explicit Provider Selection with Factory Source: https://github.com/google/langextract/blob/main/langextract/providers/README.md Demonstrates three methods for explicitly selecting a provider when creating a model instance using the factory. This is useful when multiple providers support the same model ID or when a specific provider is desired. ```python import langextract as lx # Method 1: Using factory directly with provider parameter config = lx.factory.ModelConfig( model_id="gpt-4", provider="OpenAILanguageModel", # Explicit provider provider_kwargs={"api_key": "..."} ) model = lx.factory.create_model(config) # Method 2: Using provider without model_id (uses provider's default) config = lx.factory.ModelConfig( provider="GeminiLanguageModel", # Will use default gemini-3.5-flash provider_kwargs={"api_key": "..."} ) model = lx.factory.create_model(config) # Method 3: Auto-detection (when no conflicts exist) config = lx.factory.ModelConfig( model_id="gemini-3.5-flash" # Provider auto-detected ) model = lx.factory.create_model(config) ``` -------------------------------- ### Passing Common Parameters to lx.extract Source: https://github.com/google/langextract/blob/main/langextract/providers/README.md Pass common parameters like prompt_description, examples, max_workers, and max_char_buffer to the lx.extract function. These are handled by LangExtract itself. ```python # 1. Common parameters handled by lx.extract itself: result = lx.extract( text_or_documents="Your document", model_id="gemini-3.5-flash", prompt_description="Extract key facts", examples=[...], # Used for few-shot prompting (required) max_workers=4, # Parallel processing (provider-dependent) max_char_buffer=3000, # Document chunking ) ``` -------------------------------- ### Build and Upload Package to PyPI Source: https://github.com/google/langextract/blob/main/examples/custom_provider_plugin/README.md Builds the distribution package for the custom provider and uploads it to the Python Package Index (PyPI). ```bash # Build package python -m build # Upload to PyPI twine upload dist/* ``` -------------------------------- ### Build and Run LangExtract with Docker Source: https://github.com/google/langextract/blob/main/README.md Builds a Docker image for LangExtract and runs a container. The container can be configured with an API key and execute a Python script. ```bash docker build -t langextract . docker run --rm -e LANGEXTRACT_API_KEY="your-api-key" langextract python your_script.py ``` -------------------------------- ### Configure Entry Point for External Plugin Source: https://github.com/google/langextract/blob/main/langextract/providers/README.md This TOML configuration defines the build system and project details for an external LangExtract provider. It includes two patterns for registering entry points: direct class registration and module self-registration. ```toml [build-system] requires = ["setuptools>=61.0", "wheel"] build-backend = "setuptools.build_meta" [project] name = "langextract-myprovider" version = "0.1.0" dependencies = ["langextract>=1.0.0", "your-sdk"] [project.entry-points."langextract.providers"] # Pattern 1: Register the class directly myprovider = "langextract_myprovider:MyProviderLanguageModel" # Pattern 2: Register a module that self-registers # myprovider = "langextract_myprovider" ``` -------------------------------- ### Format Code with isort and pyink Source: https://github.com/google/langextract/blob/main/README.md Manually run code formatters isort and pyink with specific configurations for sorting imports and line length. ```bash # Or run formatters separately isort langextract tests --profile google --line-length 80 pyink langextract tests --config pyproject.toml ``` -------------------------------- ### Set up Gemini API Key Source: https://github.com/google/langextract/blob/main/examples/notebooks/romeo_juliet_extraction.ipynb Sets up the Gemini API key required for LangExtract to function. It checks if the key is already set in the environment and prompts the user to enter it if not. ```python # Set up your Gemini API key # Get your key from: https://aistudio.google.com/app/apikey import os from getpass import getpass if 'GEMINI_API_KEY' not in os.environ: os.environ['GEMINI_API_KEY'] = getpass('Enter your Gemini API key: ') ``` -------------------------------- ### Accessing Grounded Extractions and Source Positions Source: https://github.com/google/langextract/blob/main/skills/langextract-usage/SKILL.md Filter extractions to include only those with defined character intervals (grounded) and demonstrate how to access the matched text using start and end positions. ```python # Filter to grounded extractions only (have source positions) grounded = [e for e in result.extractions if e.char_interval] # Access source position for e in grounded: start = e.char_interval.start_pos end = e.char_interval.end_pos matched_text = result.text[start:end] ``` -------------------------------- ### Package Configuration for Provider Entry Point Source: https://github.com/google/langextract/blob/main/examples/custom_provider_plugin/README.md Define the provider entry point in pyproject.toml to allow LangExtract to discover your custom provider. ```toml [project.entry-points."langextract.providers"] custom_gemini = "langextract_provider_example:CustomGeminiProvider" ``` -------------------------------- ### Run Auto-Formatter Source: https://github.com/google/langextract/blob/main/CONTRIBUTING.md Execute the auto-formatter script to ensure code style consistency. This script organizes imports and formats code according to Google's Python Style Guide. ```bash # Run the auto-formatter ./autoformat.sh ``` -------------------------------- ### Declare custom provider entry point in pyproject.toml Source: https://github.com/google/langextract/blob/main/skills/langextract-usage/references/providers.md For external packages, declare an entry point in your pyproject.toml file under [project.entry-points."langextract.providers"] to make your custom provider discoverable by LangExtract. ```toml [project.entry-points."langextract.providers"] my-model = "my_package.provider:MyProvider" ``` -------------------------------- ### Define Custom Provider Schema Source: https://github.com/google/langextract/blob/main/langextract/providers/README.md Implement a custom schema by inheriting from `schema.BaseSchema`. This class defines how to process and structure extraction data, including methods for building the schema from examples and converting it to provider configuration. ```python from langextract.core import schema class MyProviderSchema(schema.BaseSchema): def __init__(self, schema_dict: dict): self._schema_dict = schema_dict @property def schema_dict(self) -> dict: return self._schema_dict @classmethod def from_examples(cls, examples_data, attribute_suffix="_attributes"): """Build schema from example extractions.""" # Analyze examples to determine structure extraction_types = {} for example in examples_data: for extraction in example.extractions: class_name = extraction.extraction_class if class_name not in extraction_types: extraction_types[class_name] = set() if extraction.attributes: extraction_types[class_name].update(extraction.attributes.keys()) # Build JSON schema schema_dict = { "type": "object", "properties": { "extractions": { "type": "array", "items": {"type": "object"} # Simplified } } } return cls(schema_dict) def to_provider_config(self) -> dict: """Convert to provider-specific configuration.""" return { "response_schema": self._schema_dict, "structured_output": True } @property def requires_raw_output(self) -> bool: """Return True if provider emits raw JSON without fence markers.""" return True ``` -------------------------------- ### Manually Trigger Plugin Loading Source: https://github.com/google/langextract/blob/main/langextract/providers/README.md Manually trigger plugin loading with `lx.providers.load_plugins_once()` to resolve 'Plugin Not Loading' issues. ```python # Your plugin isn't being discovered ``` ```python lx.providers.load_plugins_once() ``` -------------------------------- ### Basic Medication NER with LangExtract Source: https://github.com/google/langextract/blob/main/docs/examples/medication_examples.md Extracts medication details like name, dosage, route, frequency, and duration from clinical text. Requires specifying extraction classes and providing example data for guidance. The output can be saved and visualized interactively. ```python import langextract as lx # Text with a medication mention input_text = "Patient took 400 mg PO Ibuprofen q4h for two days." # Define extraction prompt prompt_description = "Extract medication information including medication name, dosage, route, frequency, and duration in the order they appear in the text." # Define example data with entities in order of appearance examples = [ lx.data.ExampleData( text="Patient was given 250 mg IV Cefazolin TID for one week.", extractions=[ lx.data.Extraction(extraction_class="dosage", extraction_text="250 mg"), lx.data.Extraction(extraction_class="route", extraction_text="IV"), lx.data.Extraction(extraction_class="medication", extraction_text="Cefazolin"), lx.data.Extraction(extraction_class="frequency", extraction_text="TID"), # TID = three times a day lx.data.Extraction(extraction_class="duration", extraction_text="for one week") ] ) ] result = lx.extract( text_or_documents=input_text, prompt_description=prompt_description, examples=examples, model_id="gemini-2.5-pro", api_key="your-api-key-here" # Optional if LANGEXTRACT_API_KEY environment variable is set ) # Display entities with positions print(f"Input: {input_text}\n") print("Extracted entities:") for entity in result.extractions: position_info = "" if entity.char_interval: start, end = entity.char_interval.start_pos, entity.char_interval.end_pos position_info = f" (pos: {start}-{end})" print(f"• {entity.extraction_class.capitalize()}: {entity.extraction_text}{position_info}") # Save and visualize the results lx.io.save_annotated_documents([result], output_name="medical_ner_extraction.jsonl", output_dir=".") # Generate the interactive visualization html_content = lx.visualize("medical_ner_extraction.jsonl") with open("medical_ner_visualization.html", "w") as f: if hasattr(html_content, 'data'): f.write(html_content.data) # For Jupyter/Colab else: f.write(html_content) print("Interactive visualization saved to medical_ner_visualization.html") ``` -------------------------------- ### Implement Provider with Entry Point Registration Source: https://github.com/google/langextract/blob/main/langextract/providers/README.md This Python code implements a custom language model provider for LangExtract. It uses the `@router.register` decorator to define supported model ID patterns and includes an `infer` method for processing prompts. ```python # langextract_myprovider/__init__.py import os from langextract.core import base_model, types from langextract.providers import router @router.register(r'^mymodel', r'^custom', priority=10) class MyProviderLanguageModel(base_model.BaseLanguageModel): def __init__(self, model_id: str, api_key: str = None, **kwargs): super().__init__() self.model_id = model_id self.api_key = api_key or os.environ.get('MYPROVIDER_API_KEY') # Initialize your client self.client = MyProviderClient(api_key=self.api_key) def infer(self, batch_prompts, **kwargs): # Implement inference for prompt in batch_prompts: result = self.client.generate(prompt, **kwargs) yield [types.ScoredOutput(score=1.0, output=result)] ``` -------------------------------- ### Extract Entities from Full Text Document Source: https://github.com/google/langextract/blob/main/docs/examples/longer_text_example.md This snippet demonstrates how to process a large text document directly from a URL using LangExtract. It utilizes a detailed prompt and examples to improve extraction accuracy for complex literary works. The code also includes parameters for parallel processing and saving/visualizing the results. ```python import langextract as lx import textwrap from collections import Counter, defaultdict # Define comprehensive prompt and examples for complex literary text prompt = textwrap.dedent(""" Extract characters, emotions, and relationships from the given text. Provide meaningful attributes for every entity to add context and depth. Important: Use exact text from the input for extraction_text. Do not paraphrase. Extract entities in order of appearance with no overlapping text spans. Note: In play scripts, speaker names appear in ALL-CAPS followed by a period.""") examples = [ lx.data.ExampleData( text=textwrap.dedent(""" ROMEO. But soft! What light through yonder window breaks? It is the east, and Juliet is the sun. JULIET. O Romeo, Romeo! Wherefore art thou Romeo?"""), extractions=[ lx.data.Extraction( extraction_class="character", extraction_text="ROMEO", attributes={"emotional_state": "wonder"} ), lx.data.Extraction( extraction_class="emotion", extraction_text="But soft!", attributes={"feeling": "gentle awe", "character": "Romeo"} ), lx.data.Extraction( extraction_class="relationship", extraction_text="Juliet is the sun", attributes={"type": "metaphor", "character_1": "Romeo", "character_2": "Juliet"} ), lx.data.Extraction( extraction_class="character", extraction_text="JULIET", attributes={"emotional_state": "yearning"} ), lx.data.Extraction( extraction_class="emotion", extraction_text="Wherefore art thou Romeo?", attributes={"feeling": "longing question", "character": "Juliet"} ), ] ) ] # Process Romeo & Juliet directly from Project Gutenberg print("Downloading and processing Romeo and Juliet from Project Gutenberg...") result = lx.extract( text_or_documents="https://www.gutenberg.org/files/1513/1513-0.txt", prompt_description=prompt, examples=examples, model_id="gemini-3.5-flash", extraction_passes=3, # Multiple passes for improved recall max_workers=20, # Parallel processing for speed max_char_buffer=1000 # Smaller contexts for better accuracy ) print(f"Extracted {len(result.extractions)} entities from {len(result.text):,} characters") # Save and visualize the results lx.io.save_annotated_documents([result], output_name="romeo_juliet_extractions.jsonl", output_dir=".") # Generate the interactive visualization html_content = lx.visualize("romeo_juliet_extractions.jsonl") with open("romeo_juliet_visualization.html", "w") as f: if hasattr(html_content, 'data'): f.write(html_content.data) # For Jupyter/Colab else: f.write(html_content) print("Interactive visualization saved to romeo_juliet_visualization.html") ``` -------------------------------- ### Manual Formatting with isort and pyink Source: https://github.com/google/langextract/blob/main/CONTRIBUTING.md Manually run the import sorter and code formatter. These commands target specific directories to avoid formatting non-source files. ```bash isort langextract tests ``` ```bash pyink langextract tests --config pyproject.toml ``` -------------------------------- ### Run Linting with Pylint Source: https://github.com/google/langextract/blob/main/README.md Execute the Pylint linter on the LangExtract project code and tests before submitting a Pull Request. ```bash pylint --rcfile=.pylintrc langextract tests ``` -------------------------------- ### Configure API Keys Source: https://github.com/google/langextract/blob/main/skills/langextract-usage/SKILL.md Set environment variables for API keys for Gemini and OpenAI models. Ollama does not require an API key. ```bash # Gemini (checks GEMINI_API_KEY then LANGEXTRACT_API_KEY) export GEMINI_API_KEY="your_key" # OpenAI (checks OPENAI_API_KEY then LANGEXTRACT_API_KEY) export OPENAI_API_KEY="your_key" # Ollama: no API key needed, just a running Ollama server ``` -------------------------------- ### Using Factory for Advanced Control Source: https://github.com/google/langextract/blob/main/langextract/providers/README.md Use the factory.create_model function to explicitly specify both the model and the provider, which is useful when multiple providers support the same model ID. ```python # When you need explicit provider selection or advanced configuration from langextract import factory # Specify both model and provider (useful when multiple providers support same model) config = factory.ModelConfig( model_id="gemma2:2b", provider="OllamaLanguageModel", # Explicitly use Ollama provider_kwargs={ "model_url": "http://localhost:11434" } ) model = factory.create_model(config) ``` -------------------------------- ### Run Full CI Matrix with Tox Source: https://github.com/google/langextract/blob/main/README.md Reproduce the full Continuous Integration matrix locally using tox, which includes pylint and pytest for Python 3.10 and 3.11. ```bash tox # runs pylint + pytest on Python 3.10 and 3.11 ``` -------------------------------- ### List Available Providers Source: https://github.com/google/langextract/blob/main/langextract/providers/README.md Check available patterns with `langextract.providers.router.list_entries()` to resolve 'Provider Not Found' errors. ```python InferenceConfigError: No provider registered for model_id='unknown-model' ``` -------------------------------- ### Create External Plugin Package Structure Source: https://github.com/google/langextract/blob/main/langextract/providers/README.md This illustrates the basic directory structure for an external LangExtract provider package. It includes the main package directory, pyproject.toml for configuration, and README.md for documentation. ```bash langextract-myprovider/ ├── pyproject.toml ├── README.md └── langextract_myprovider/ └── __init__.py ``` -------------------------------- ### Migrate Registry Import Source: https://github.com/google/langextract/blob/main/langextract/_compat/README.md Update imports from langextract.registry to langextract.plugins. ```python from langextract.registry import * → from langextract.plugins import * ``` -------------------------------- ### Direct Provider Usage Source: https://github.com/google/langextract/blob/main/langextract/providers/README.md Import and instantiate a provider class directly for more granular control. This bypasses the automatic provider selection mechanism. ```python import langextract as lx # Direct import if you prefer (optional) from langextract.providers.gemini import GeminiLanguageModel model = GeminiLanguageModel( model_id="gemini-3.5-flash", api_key="your-key" ) outputs = model.infer(["prompt1", "prompt2"]) ``` -------------------------------- ### Migrate Provider Registry Import Source: https://github.com/google/langextract/blob/main/langextract/_compat/README.md Update imports for provider registry from langextract.providers.registry to langextract.providers.router. ```python from langextract.providers.registry import * → from langextract.providers.router import * ``` -------------------------------- ### Set LangExtract API Key via .env File Source: https://github.com/google/langextract/blob/main/README.md Adds the LANGEXTRACT_API_KEY to a .env file and configures gitignore to keep it secure. This is the recommended method for managing API keys. ```bash # Add API key to .env file cat >> .env << 'EOF' LANGEXTRACT_API_KEY=your-api-key-here EOF # Keep your API key secure echo '.env' >> .gitignore ``` -------------------------------- ### Custom Schema Implementation - to_provider_config Source: https://github.com/google/langextract/blob/main/examples/custom_provider_plugin/README.md Implement the to_provider_config method to convert the schema into provider configuration arguments. ```python def to_provider_config(self): # Convert schema to provider kwargs return { "response_schema": self._schema_dict, "enable_structured_output": True } ``` -------------------------------- ### Run Full Test Suite with Tox Source: https://github.com/google/langextract/blob/main/CONTRIBUTING.md Execute the complete testing suite across multiple Python versions using tox. This ensures compatibility and adherence to standards. ```bash tox # runs pylint + pytest on Python 3.10 and 3.11 ``` -------------------------------- ### Create Custom Provider Plugin Source: https://github.com/google/langextract/blob/main/examples/custom_provider_plugin/README.md Use the provider plugin generator script to create a new plugin structure with schema support. ```bash python scripts/create_provider_plugin.py MyProvider --with-schema ``` -------------------------------- ### Migrate GeminiLanguageModel Import Source: https://github.com/google/langextract/blob/main/langextract/_compat/README.md Update imports for GeminiLanguageModel from langextract.inference to langextract.providers.gemini. ```python from langextract.inference import GeminiLanguageModel → from langextract.providers.gemini import GeminiLanguageModel ``` -------------------------------- ### Auto-format Code Source: https://github.com/google/langextract/blob/main/README.md Automatically format all code within the LangExtract project using the provided script. ```bash # Auto-format all code ./autoformat.sh ``` -------------------------------- ### Custom Provider Implementation Source: https://github.com/google/langextract/blob/main/examples/custom_provider_plugin/README.md Implement a custom provider by inheriting from base_model.BaseLanguageModel and registering it with the router. ```python from langextract.core import base_model from langextract.providers import router @router.register( r'^gemini', # Pattern for model IDs this provider handles ) class CustomGeminiProvider(base_model.BaseLanguageModel): def __init__(self, model_id: str, **kwargs): # Initialize your backend client pass def infer(self, batch_prompts, **kwargs): # Call your backend API and return results pass ``` -------------------------------- ### Enable OpenAI Batch API for Large Workloads Source: https://github.com/google/langextract/blob/main/README.md Configure LangExtract to use the OpenAI Batch API for large, non-latency-sensitive workloads. Batch mode is opt-in and falls back to realtime calls below a specified threshold. ```python result = lx.extract( text_or_documents=documents, prompt_description=prompt, examples=examples, model_id="gpt-4o-mini", language_model_params={ "batch": { "enabled": True, "threshold": 50, "poll_interval": 10, } }, ) ``` -------------------------------- ### Migrate OllamaLanguageModel Import Source: https://github.com/google/langextract/blob/main/langextract/_compat/README.md Update imports for OllamaLanguageModel from langextract.inference to langextract.providers.ollama. ```python from langextract.inference import OllamaLanguageModel → from langextract.providers.ollama import OllamaLanguageModel ```