### Setup Development Environment and Install Dependencies Source: https://github.com/coderamp-labs/gitingest/commit/f8d397e66e3382d12f8a0ed05d291a39db830bda Instructions for setting up the development environment and installing project dependencies, likely using a package manager like pip. ```bash cd gitingest ``` ```bash git checkout -S -b your-branch ``` ```bash git config --global commit.gpgSign true ``` ```bash git commit -S -m "Your commit message" ``` -------------------------------- ### Add Minio Setup to compose.yml Source: https://github.com/coderamp-labs/gitingest/commit/414e85189fb9055491530ba8c0665c798474451e This snippet details the addition of Minio service configuration to the compose.yml file. It includes volume mounting for setup scripts and defines the entrypoint and command for the Minio container. Dependencies include Docker. ```yaml + - S3_BUCKET_NAME=${S3_BUCKET_NAME:-gitingest-bucket} + volumes: + - ./.docker/minio/setup.sh:/setup.sh:ro + entrypoint: sh + command: -c /setup.sh + +volumes: + minio-data: + driver: local ``` -------------------------------- ### Start Metrics Server (Python) Source: https://github.com/coderamp-labs/gitingest/commit/1016f6ecb3b1b066d541d1eba1ddffec49b15f16 Starts a metrics server on a specified host and port using uvicorn. It logs the server's startup details and accepts 'host' and 'port' as parameters. The default host is '127.0.0.1' and the default port is 9090. ```python logger.info("Starting metrics server on %s:%s", host, port) uvicorn.run(metrics_app, host=host, port=port) ``` -------------------------------- ### Python Package Installation with Dev Extras Source: https://github.com/coderamp-labs/gitingest/issues/352 This command shows how to install the project's Python dependencies, including development extras, using `pip`. This is used in the CI pipeline for setting up the development environment. ```bash pip install -e .[dev] ``` -------------------------------- ### Docker Compose Service Configuration Source: https://github.com/coderamp-labs/gitingest/commit/d061b4877a253ba3f0480d329f025427c7f70177 Example of a service definition in a Docker Compose file, specifying image, ports, dependencies, and environment variables for a Minio setup. ```yaml services: minio-setup: image: minio/minio:latest ports: - "9000:9000" # API port - "9001:9001" # Console port environment: *minio-environment volumes: - minio_data:/export ``` -------------------------------- ### Update pip install command for Docker Source: https://github.com/coderamp-labs/gitingest/issues/449 This snippet shows the corrected pip install command. It ensures that the '[server]' extra dependencies are included, which is crucial for the proper launch of the Docker container. This fix addresses a common issue where the container fails to start due to missing server-related components. ```shell pip install "gitingest[server]" ``` -------------------------------- ### Setting up Development Environment Source: https://github.com/coderamp-labs/gitingest/commit/f8d397e66e3382d12f8a0ed05d291a39db830bda This code block provides the bash command to set up the development environment and install dependencies for Gitingest. It's a standard step for new contributors. ```bash cd gitingest ``` bash ``` ``` -------------------------------- ### Setup MinIO Bucket and User with Shell Script Source: https://github.com/coderamp-labs/gitingest/commit/414e85189fb9055491530ba8c0665c798474451e This shell script automates the setup of a MinIO bucket and user. It configures the MinIO client, creates a bucket, sets anonymous download policy, adds a user with specified keys, creates a bucket policy, and attaches it to the user. It requires environment variables like S3_BUCKET_NAME, MINIO_ROOT_USER, MINIO_ROOT_PASSWORD, S3_ACCESS_KEY, and S3_SECRET_KEY. ```shell #!/bin/sh # Simple script to set up MinIO bucket and user # Based on example from MinIO issues # Format bucket name to ensure compatibility BUCKET_NAME=$(echo "${S3_BUCKET_NAME}" | tr '[:upper:]' '[:lower:]' | tr '_' '-') # Configure MinIO client mc alias set myminio http://minio:9000 ${MINIO_ROOT_USER} ${MINIO_ROOT_PASSWORD} # Remove bucket if it exists (for clean setup) mc rm -r --force myminio/${BUCKET_NAME} || true # Create bucket mc mb myminio/${BUCKET_NAME} # Set bucket policy to allow downloads mc anonymous set download myminio/${BUCKET_NAME} # Create user with access and secret keys mc admin user add myminio ${S3_ACCESS_KEY} ${S3_SECRET_KEY} || echo "User already exists" # Create policy for the bucket echo '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Action":["s3:*"],"Resource":["arn:aws:s3:::'${BUCKET_NAME}'/*","arn:aws:s3:::'${BUCKET_NAME}'"]}]}' > /tmp/policy.json # Apply policy mc admin policy create myminio gitingest-policy /tmp/policy.json || echo "Policy already exists" mc admin policy attach myminio gitingest-policy --user ${S3_ACCESS_KEY} echo "MinIO setup completed successfully" echo "Bucket: ${BUCKET_NAME}" echo "Access via console: http://localhost:9001" ``` -------------------------------- ### Python Development Installation Source: https://github.com/coderamp-labs/gitingest/issues/352 This command shows how to install the project in editable mode with development dependencies. It uses `pip` and specifies the `[dev]` extra. ```bash pip -e .[dev] ``` -------------------------------- ### Setup Python Environment with Caching Source: https://github.com/coderamp-labs/gitingest/commit/b683e59b5b1a31d27cc5c6ce8fb62da9b660613b This snippet configures the GitHub Actions environment to use Python version 3.13. It also sets up pip caching for faster dependency installations, using 'pyproject.toml' to define cache dependencies. ```yaml uses: actions/setup-python@v5 with: python-version: "3.13" cache: pip cache-dependency-path: pyproject.toml ``` -------------------------------- ### Install Python Build Tools and Twine (YAML) Source: https://github.com/coderamp-labs/gitingest/commit/016817d5590c1412498b7532f6e854d20239c6be This step ensures the necessary tools for building and publishing Python packages are installed. It upgrades pip and then installs the build and twine packages. ```yaml + python -m pip install --upgrade pip + python -m pip install build twine ``` -------------------------------- ### Install project dependencies using Poetry Source: https://github.com/coderamp-labs/gitingest/commit/d061b4877a253ba3f0480d329f025427c7f70177 This command installs the project's dependencies as defined in the pyproject.toml file using Poetry. It ensures that all required packages are available for the project. ```bash # requirements.txt (generated from pyproject.toml) black==23.3.0 flake8==6.0.0 loguru==0.6.0 pydantic==1.10.6 requests==2.28.1 ``` -------------------------------- ### Docker Configuration for Prometheus Exporter Source: https://github.com/coderamp-labs/gitingest/commit/1016f6ecb3b1b066d541d1eba1ddffec49b15f16 This Dockerfile snippet sets up an environment to run a Python application, likely the Prometheus exporter. It installs dependencies from requirements.txt and sets the working directory. The configuration suggests a production-ready setup for deploying the application. ```dockerfile FROM python:3.9-slim WORKDIR /app COPY requirements.txt requirements.txt RUN pip install --no-cache-dir -r requirements.txt COPY . . CMD ["uvicorn", "src.server.main:app", "--host", "0.0.0.0", "--port", "8000"] ``` -------------------------------- ### Commit: Add Example .env File for Environment Configuration Source: https://github.com/coderamp-labs/gitingest/issues/408 This commit introduces an example .env file, which serves as a template for environment-specific configuration. This is crucial for managing settings like Sentry DSNs and other sensitive information. ```git commit feat: add example .env file for environment configuration ``` -------------------------------- ### Docker: Implement 3-stage build for CSS compilation and Python installation Source: https://github.com/coderamp-labs/gitingest/issues/352 This documentation outlines the Docker configuration, replacing a single-stage image with a more efficient 3-stage build. It details the roles of `css-builder` for Tailwind compilation, `python-builder` for Python project installation, and the runtime image for executing the application. ```plaintext Docker * replace single-stage image with 3-stage build • css-builder (Node 20 alpine) → compiles Tailwind • python-builder installs project with PEP 621 metadata • runtime image copies site-packages + compiled CSS, runs as uid 1000 ``` -------------------------------- ### Python Prometheus Exporter Setup Source: https://github.com/coderamp-labs/gitingest/commit/1016f6ecb3b1b066d541d1eba1ddffec49b15f16 This snippet demonstrates the setup of a Prometheus exporter in Python, likely for a web server or API. It imports necessary libraries and defines a metrics server. The code appears to be part of a larger application that exposes metrics for monitoring. ```python from prometheus_client import Gauge, Counter, Histogram, CollectorRegistry, generate_latest from starlette.responses import PlainTextResponse from starlette.routing import Route from starlette.applications import Starlette registry = CollectorRegistry() github_rate_limit_gauge = Gauge( "github_rate_limit_gauge", "github rate limit gauge", labelnames=["resource", "type"], registry=registry, ) github_rate_limit_counter = Counter( "github_rate_limit_counter", "github rate limit counter", labelnames=["resource", "type"], registry=registry, ) github_rate_limit_histogram = Histogram( "github_rate_limit_histogram", "github rate limit histogram", labelnames=["resource", "type"], registry=registry, ) def metrics(request): if request.method == "GET": return PlainTextResponse(generate_latest(registry)) return PlainTextResponse("Invalid request method") routes = [ Route("/metrics", endpoint=metrics), ] app = Starlette(routes=routes) ``` -------------------------------- ### Python: Log partial clone setup Source: https://github.com/coderamp-labs/gitingest/commit/d061b4877a253ba3f0480d329f025427c7f70177 Logs information about setting up a partial clone for a subpath. It uses the 'logging' module and requires 'config' and 'token' as inputs. ```python logger.info("Setting up partial clone for subpath", extra={"subpath": config.subpath}) ``` -------------------------------- ### Set up APIRouter in Python Source: https://github.com/coderamp-labs/gitingest/commit/1016f6ecb3b1b066d541d1eba1ddffec49b15f16 This code snippet shows the basic setup of an APIRouter instance, which is commonly used in Python web frameworks like FastAPI for defining API endpoints and organizing routes. ```python router = APIRouter() ``` -------------------------------- ### Install Dependencies and Run Tests (YAML) Source: https://github.com/coderamp-labs/gitingest/commit/016817d5590c1412498b7532f6e854d20239c6be Installs project dependencies, including development extras, and runs tests using pytest. It ensures pip is up-to-date before installing packages. ```yaml - name: Install dependencies run: | python -m pip install --upgrade pip python -m pip install ".[dev]" - name: Run tests run: pytest ``` -------------------------------- ### Ensure Git Installation and Long Path Support (Python) Source: https://github.com/coderamp-labs/gitingest/commit/b8e375f71cae7d980cf431396c4414a6dbd0588c Ensures Git is installed and checks for Windows long path support. If not enabled, it prints a warning and instructions on how to enable it, noting the need for administrator privileges. ```python import asyncio import base64 import ctypes import os import re import sys from typing import Final # Assuming run_command and Colors are defined elsewhere # For demonstration purposes, defining placeholder classes/functions: class Colors: BROWN = "\033[33m" END = "\033[0m" RED = "\033[91m" async def run_command(cmd: str, *args) -> tuple[str, str]: # Placeholder for actual command execution if cmd == "git" and args == ("config", "--system", "core.longpaths"): # Simulate success when admin rights are present, failure otherwise if ctypes.windll.shell32.IsUserAnAdmin(): return "true\n", "" else: raise RuntimeError("Permission denied") return "", "" async def ensure_git_installed() -> None: """ Ensures Git is installed and accessible. Raises: RuntimeError: If Git is not installed or not accessible. RuntimeError: If checking the long path setting fails on Windows. """ try: await run_command("git", "--version") except RuntimeError as exc: msg = "Git is not installed or not accessible. Please install Git." raise RuntimeError(msg) from exc if sys.platform == "win32": try: if ctypes.windll.shell32.IsUserAnAdmin(): stdout, _ = await run_command("git", "config", "--system", "core.longpaths") if stdout.decode().strip().lower() == "true": return else: # Attempt to check without admin rights, might fail try: await run_command("git", "config", "--system", "core.longpaths") except RuntimeError: # Ignore if checking 'core.longpaths' fails due to lack of administrator rights. pass print( f"{Colors.BROWN}WARN{Colors.END}: {Colors.RED}Git clone may fail on Windows " f"due to long file paths:{Colors.END}", ) print(f"{Colors.RED}To avoid this issue, consider enabling long path support with:{Colors.END}") print(f"{Colors.RED} git config --system core.longpaths true{Colors.END}") print(f"{Colors.RED}Note: This command may require administrator privileges.{Colors.END}") except RuntimeError as exc: msg = ( "Unable to verify or access Git long path configuration. " "Run this application as Administrator or configure it manually." ) raise RuntimeError(msg) from exc ``` -------------------------------- ### List of Example Repositories in Python Source: https://github.com/coderamp-labs/gitingest/commit/932bfef85db66704985c83f3f7c427756bd14023 This snippet defines a Python list named EXAMPLE_REPOS. Each element in the list is a dictionary containing 'name' and 'url' keys, representing example GitHub repositories. This is likely used for testing or demonstration purposes. ```python EXAMPLE_REPOS: list[dict[str, str]] = [ {"name": "Gitingest", "url": "https://github.com/coderamp-labs/gitingest"}, ] ``` -------------------------------- ### Apply Top Margin to Example Repositories Section Source: https://github.com/coderamp-labs/gitingest/issues/331 This commit fixes an issue where the top margin for the example repositories section was not applied when a token is present. This ensures consistent spacing and layout. ```commit fix(ui): apply top margin to example repositories section if token is present ``` -------------------------------- ### Install Development Dependencies Source: https://github.com/coderamp-labs/gitingest/commit/016817d5590c1412498b7532f6e854d20239c6be Installs the necessary Python packages for development, including upgrading pip and installing from a requirements file. This ensures a consistent development environment. ```bash pip install --upgrade pip pip install -r requirements-dev.txt ``` -------------------------------- ### MinIO Server Command with Console Address Source: https://github.com/coderamp-labs/gitingest/commit/d061b4877a253ba3f0480d329f025427c7f70177 This snippet shows the command to start the MinIO server, specifying the data directory and the console address. It's a common way to launch MinIO in a containerized environment. ```shell command: server /data --console-address ":9001" ``` -------------------------------- ### Install Dependencies for Docker Container (Shell) Source: https://github.com/coderamp-labs/gitingest/commit/998cea15b4f79c5d6f840b5d3d916f83c8be3a07 This command installs project dependencies for the Docker container. The `--no-cache-dir` flag prevents caching, and `--timeout 1000` sets a connection timeout. The `.[server]` argument specifies the package to install. ```shell pip install --no-cache-dir --timeout 1000 .[server] ``` -------------------------------- ### Ensure Git Installation in Python Source: https://github.com/coderamp-labs/gitingest/commit/d061b4877a253ba3f0480d329f025427c7f70177 This snippet demonstrates checking and ensuring that Git is installed on the system using Python. It relies on an `ensure_git_installed` function and logs the status. ```python logger.debug("Ensuring git is installed") await ensure_git_installed() ``` -------------------------------- ### Dockerfile EXPOSE and CMD Source: https://github.com/coderamp-labs/gitingest/commit/d061b4877a253ba3f0480d329f025427c7f70177 This snippet details the Dockerfile configuration for the project. It shows which ports are exposed for external access and the command used to start the application within the container. ```dockerfile EXPOSE 8000 EXPOSE 9090 CMD ["python", "-m", "uvicorn", "server.main:app", "--host", "0.0.0.0", "--port", "8000"] ``` -------------------------------- ### Docker: MinIO Setup Script Source: https://github.com/coderamp-labs/gitingest/commit/414e85189fb9055491530ba8c0665c798474451e A shell script to set up MinIO, a self-hosted object storage service compatible with Amazon S3. This is likely used for local development or testing of S3 integrations. ```bash #!/bin/bash # Exit immediately if a command exits with a non-zero status. set -e # Default values MINIO_ROOT_USER=${MINIO_ROOT_USER:-minioadmin} MINIO_ROOT_PASSWORD=${MINIO_ROOT_PASSWORD:-minioadmin} MINIO_CONTAINER_NAME=${MINIO_CONTAINER_NAME:-minio} # Check if MinIO is already running if [ $(docker ps -q -f name=$MINIO_CONTAINER_NAME) ]; then echo "MinIO container '$MINIO_CONTAINER_NAME' is already running." exit 0 fi # Check if MinIO container exists but is stopped if [ $(docker ps -aq -f status=exited -f name=$MINIO_CONTAINER_NAME) ]; then echo "Starting existing MinIO container '$MINIO_CONTAINER_NAME'வுகளை." docker start $MINIO_CONTAINER_NAME exit 0 fi # Pull the MinIO Docker image echo "Pulling MinIO Docker image..." docker pull minio/minio:latest # Run the MinIO container echo "Starting MinIO container '$MINIO_CONTAINER_NAME'வுகளை." docker run -d \ --name $MINIO_CONTAINER_NAME \ -p 9000:9000 \ -p 9001:9001 \ -e "MINIO_ROOT_USER=$MINIO_ROOT_USER" \ -e "MINIO_ROOT_PASSWORD=$MINIO_ROOT_PASSWORD" \ minio/minio:latest server /data --console-address ":9001" # Wait for MinIO to start echo "Waiting for MinIO to start..." sleep 5 # Create a bucket if it doesn't exist (example: 'my-bucket') # You might want to make this dynamic or configurable BUCKET_NAME="gitingest-data" echo "Creating bucket '$BUCKET_NAME' if it does not exist..." docker exec -it $MINIO_CONTAINER_NAME /usr/bin/mc alias set myminio http://localhost:9000 $MINIO_ROOT_USER $MINIO_ROOT_PASSWORD docker exec -it $MINIO_CONTAINER_NAME /usr/bin/mc mb myminio/$BUCKET_NAME || echo "Bucket '$BUCKET_NAME' already exists." echo "MinIO setup complete. Access the console at http://localhost:9001" ``` -------------------------------- ### Ignore Installer and Log Files Source: https://github.com/coderamp-labs/gitingest/commit/b683e59b5b1a31d27cc5c6ce8fb62da9b660613b This section defines patterns for files related to installation and logging that should be ignored. It includes common log files generated by package managers like pip and temporary directories used for installation. ```gitignore pip-log.txt pip-delete-this-directory.txt ``` -------------------------------- ### Python Package Installation (PEP 621) Source: https://github.com/coderamp-labs/gitingest/issues/352 This refers to the use of PEP 621 for defining project metadata, which is used during the Python package installation process. ```python installs project with PEP 621 metadata ``` -------------------------------- ### Pre-commit Fixer Configuration (example) Source: https://github.com/coderamp-labs/gitingest/issues/352 This example shows a hypothetical configuration for a pre-commit hook, likely removed as part of the tooling changes. The `requirements-dev.txt` was dropped in favor of `[dev]` extras. ```yaml # .pre-commit-config.yaml (example of a removed config) - repo: local hooks: - id: requirements-fixer name: Fix Requirements entry: scripts/fix_requirements.sh language: script files: ^requirements-dev\.txt$ ``` -------------------------------- ### Docker Build Command Source: https://github.com/coderamp-labs/gitingest/commit/998cea15b4f79c5d6f840b5d3d916f83c8be3a07 This snippet shows a Dockerfile command for installing Python packages. It uses pip to install dependencies from the current directory and sets a timeout for the installation process. The `set -eux` command ensures that the script exits immediately if a command exits with a non-zero status, and that all commands are printed as they are executed. ```dockerfile RUN set -eux; \ pip install --no-cache-dir --timeout 1000 . ``` -------------------------------- ### Update README with `include_submodules` example Source: https://github.com/coderamp-labs/gitingest/commit/38c23171a14556a2cdd05c0af8219f4dc789defd This snippet represents a change in the README.md file, showing an example of how to use the new `--include-submodules` flag when running the gitingest CLI. This documents the functionality for users. ```markdown ```bash gitingest ingest --repo --include-submodules ``` ``` -------------------------------- ### FastAPI App Setup and Routing in Python Source: https://github.com/coderamp-labs/gitingest/commit/2b1f228ae1f6d1f7ee471794d258b13fcac25a96 Configures a FastAPI application, including middleware, exception handlers, and route inclusion. It serves static files and handles various HTTP requests for different endpoints. ```python from pathlib import Path from dotenv import load_dotenv from fastapi import FastAPI from fastapi.responses import FileResponse, HTMLResponse from fastapi.staticfiles import StaticFiles from slowapi.errors import RateLimitExceeded from starlette.middleware.trustedhost import TrustedHostMiddleware from server.routers import dynamic, index, ingest from server.server_utils import lifespan, limiter, rate_limit_exception_handler # Load environment variables from .env file load_dotenv() # Initialize FastAPI app app = FastAPI( title="LLMtxt API", version="0.1.0", description="API for interacting with Large Language Models and text processing.", lifespan=lifespan, limiter=limiter, exception_handlers={"RateLimitExceeded": rate_limit_exception_handler}, ) # Add TrustedHostMiddleware to mitigate Host header attacks app.add_middleware(TrustedHostMiddleware, allowed_hosts=["*"]) # Mount static files for serving frontend assets app.mount("/static", StaticFiles(directory="src/static"), name="static") # Define health check endpoint @app.get("/health") async def health_check() -> dict[str, str]: return {"status": "healthy"} # Respond to HTTP HEAD requests for the root URL @app.head("/", include_in_schema=False) async def head_root() -> HTMLResponse: return HTMLResponse(content=None, headers={"content-type": "text/html; charset=utf-8"}) # Serve robots.txt file @app.get("/robots.txt", include_in_schema=False) async def robots() -> FileResponse: return FileResponse("src/static/robots.txt") # Include routers for modular endpoints app.include_router(index) app.include_router(ingest) app.include_router(dynamic) # Endpoint to serve the main application file @app.get("/") async def llm_txt() -> FileResponse: return FileResponse("src/static/index.html") ``` -------------------------------- ### Update Dockerfile for Python application Source: https://github.com/coderamp-labs/gitingest/commit/d061b4877a253ba3f0480d329f025427c7f70177 This Dockerfile defines the build process for a Python application. It sets up the environment, installs dependencies, and prepares the application for execution. ```dockerfile # Dockerfile FROM python:3.10-slim WORKDIR /app COPY requirements.txt . RUN pip install --no-cache-dir -r requirements.txt COPY . . RUN adduser --system --group appuser RUN chown -R appuser:appuser /app USER appuser CMD ["python", "src/gitingest/__main__.py"] ``` -------------------------------- ### Docker Compose: Set Minio Entrypoint to Shell Source: https://github.com/coderamp-labs/gitingest/commit/d061b4877a253ba3f0480d329f025427c7f70177 This snippet demonstrates setting the entrypoint for a Minio service in a Docker Compose file to 'sh'. This configuration ensures that the Minio container starts with a shell environment. ```yaml entrypoint: sh ``` -------------------------------- ### Configure Docker for S3 and Local Volumes Source: https://github.com/coderamp-labs/gitingest/commit/414e85189fb9055491530ba8c0665c798474451e This snippet configures Docker environment variables for S3 access (secret key and bucket name) and defines volume mounts for a setup script and local data storage. It specifies an entrypoint and command to execute the setup script. ```dockerfile - S3_SECRET_KEY=${S3_SECRET_KEY:-gitingest123} - S3_BUCKET_NAME=${S3_BUCKET_NAME:-gitingest-bucket} volumes: - ./.docker/minio/setup.sh:/setup.sh:ro entrypoint: sh command: -c /setup.sh volumes: minio-data: driver: local ``` -------------------------------- ### Start Prometheus Metrics Server Source: https://github.com/coderamp-labs/gitingest/commit/1016f6ecb3b1b066d541d1eba1ddffec49b15f16 This Python function starts an HTTP server to expose Prometheus metrics. It takes optional host and port arguments, defaulting to '127.0.0.1' and 9090 respectively. The server is designed to run on a separate port and is intended for local network access. ```python def start_metrics_server(host: str = "127.0.0.1", port: int = 9090) -> None: """Start the metrics server on a separate port.""" # Code to start the server would go here. ``` -------------------------------- ### Project Development Setup (pyproject.toml) Source: https://github.com/coderamp-labs/gitingest/issues/352 This snippet describes the addition of a '[dev]' extra for development dependencies and the removal of the `requirements-dev.txt` file. It also mentions the integration of a pre-commit fixer for improved code quality. ```toml add `[dev]` extra, drop requirements-dev.txt & its pre-commit fixer ``` -------------------------------- ### CI/CD Pipeline Configuration (ci.yml) Source: https://github.com/coderamp-labs/gitingest/issues/352 This snippet shows the CI/CD configuration for caching dependencies using pyproject.toml and installing the project with development extras. It is designed for a typical Python project setup. ```yaml ci.yml: cache by pyproject.toml, install with `pip -e .[dev]` ``` -------------------------------- ### Python Imports for Web Server Setup Source: https://github.com/coderamp-labs/gitingest/commit/1016f6ecb3b1b066d541d1eba1ddffec49b15f16 This snippet showcases essential Python imports for building a web server, including Path for file system operations, dotenv for environment variable loading, and specific components from slowapi and starlette for request handling, middleware, and error management. ```python from pathlib import Path from dotenv import load_dotenv from slowapi.errors import RateLimitExceeded from starlette.middleware.trustedhost import TrustedHostMiddleware ``` -------------------------------- ### JavaScript: Toggle Access Settings Source: https://github.com/coderamp-labs/gitingest/commit/b39ef5416c1f8a7993a8249161d2a898b7387595 This JavaScript function, `toggleAccessSettings`, manipulates the visibility of access settings and related UI elements. It gets references to container, checkbox, and examples elements by their IDs and toggles their display styles. ```javascript function toggleAccessSettings() { const container = document.getElementById('accessSettingsContainer'); const checkbox = document.getElementById('showAccessSettings'); const examples = document.getElementById('exampleRepositories') if (checkbox.checked) { container.style.display = 'block'; examples.style.display = 'block'; } else { container.style.display = 'none'; examples.style.display = 'none'; } } ``` -------------------------------- ### Open Local Server URL Source: https://github.com/coderamp-labs/gitingest/commit/d1f8a80826ca38ec105a1878742fe351d4939d6e This snippet provides the URL to access the locally running server. After starting the server, you can open this URL in your web browser to confirm it's working correctly. ```text http://localhost:8000 ``` -------------------------------- ### List of Application Environment Variables Source: https://github.com/coderamp-labs/gitingest/commit/590e55a4d28a4f5c0beafbd12c525828fa79e221 This section outlines the environment variables available for configuring the Gitingest application. It specifies the purpose of each variable and its default value if applicable, aiding in setup and customization. ```markdown ### Environment Variables The application can be configured using the following environment variables: - **ALLOWED_HOSTS**: Comma-separated list of allowed hostnames (default: "gitingest.com, *.gitingest.com, localhost, 127.0.0.1") ``` -------------------------------- ### Clone Git Repository (Python) Source: https://github.com/coderamp-labs/gitingest/commit/d061b4877a253ba3f0480d329f025427c7f70177 Clones a Git repository based on the provided configuration. It handles repository existence checks, commit resolution, command execution, partial clone setup, and submodule updates. Dependencies include `gitingest.utils.logging_config`, `gitingest.utils.os_utils`, `gitingest.utils.timeout_wrapper`, and Git installation. ```python import logging from pathlib import Path from typing import TYPE_CHECKING from gitingest.constants import DEFAULT_TIMEOUT from gitingest.git.commands import checkout_partial_clone, create_git_command, ensure_git_installed, is_github_host, resolve_commit from gitingest.git.repo_exists import check_repo_exists from gitingest.utils.logging_config import get_logger from gitingest.utils.os_utils import ensure_directory_exists_or_create from gitingest.utils.timeout_wrapper import async_timeout if TYPE_CHECKING: from gitingest.schemas import CloneConfig logger = get_logger(__name__) @async_timeout(DEFAULT_TIMEOUT) async def clone_repo(config: CloneConfig, *, token: str | None = None) -> None: """Clones a Git repository based on the provided configuration. Args: config: The CloneConfig object containing repository details. token: Optional authentication token for private repositories. """ url: str = config.url local_path: str = config.local_path partial_clone: bool = config.subpath != "/" logger.info( "Starting git clone operation", extra={ "url": url, "local_path": local_path, "partial_clone": partial_clone, "subpath": config.subpath, "branch": config.branch, "tag": config.tag, "commit": config.commit, "include_submodules": config.include_submodules, }, ) logger.debug("Ensuring git is installed") await ensure_git_installed() logger.debug("Creating local directory", extra={"parent_path": str(Path(local_path).parent)}) await ensure_directory_exists_or_create(Path(local_path).parent) logger.debug("Checking if repository exists", extra={"url": url}) if not await check_repo_exists(url, token=token): logger.error("Repository not found", extra={"url": url}) msg = "Repository not found. Make sure it is public or that you have provided a valid token." raise ValueError(msg) logger.debug("Resolving commit reference") commit = await resolve_commit(config, token=token) logger.debug("Resolved commit", extra={"commit": commit}) clone_cmd = ["git"] if token and is_github_host(url): # Use SSH URL if token is provided and it's a GitHub host for better security # Note: This assumes SSH keys are configured correctly on the machine. # For HTTPS with token, the URL format would be different (e.g., https://@github.com/user/repo.git) # However, the current implementation directly uses the provided URL which might be HTTPS. # A more robust solution might involve parsing the URL and constructing the token-authenticated HTTPS URL. pass # Placeholder for potential SSH URL logic if needed clone_cmd += [url, local_path] # Clone the repository logger.info("Executing git clone command", extra={"command": " ".join([*clone_cmd[:-1], "", local_path])}) await run_command(*clone_cmd) logger.info("Git clone completed successfully") # Checkout the subpath if it is a partial clone if partial_clone: logger.info("Setting up partial clone for subpath", extra={"subpath": config.subpath}) await checkout_partial_clone(config, token=token) logger.debug("Partial clone setup completed") git = create_git_command(["git"], local_path, url, token) # Ensure the commit is locally available logger.debug("Fetching specific commit", extra={"commit": commit}) await run_command(*git, "fetch", "--depth=1", "origin", commit) # Write the work-tree at that commit logger.info("Checking out commit", extra={"commit": commit}) await run_command(*git, "checkout", commit) # Update submodules if config.include_submodules: logger.info("Updating submodules") await run_command(*git, "submodule", "update", "--init", "--recursive", "--depth=1") logger.debug("Submodules updated successfully") logger.info("Git clone operation completed successfully", extra={"local_path": local_path}) ``` -------------------------------- ### Main Server Application Entrypoint Source: https://github.com/coderamp-labs/gitingest/commit/1016f6ecb3b1b066d541d1eba1ddffec49b15f16 This Python code defines the main application entrypoint for the server using FastAPI and Uvicorn. It includes imports and the main app object, likely for running the web service. ```python from fastapi import FastAPI from uvicorn import Config, Server import server.routes.auth import server.routes.auth.endpoints import server.routes.chat import server.routes.chat.endpoints import server.routes.chat.response_models import server.routes.chat.schemas import server.routes.endpoints import server.routes.endpoints.endpoints import server.routes.endpoints.schemas import server.routes.health import server.routes.health.endpoints import server.routes.health.schemas import server.routes.main import server.routes.main.endpoints import server.routes.main.schemas import server.routes.users import server.routes.users.endpoints import server.routes.users.schemas app = FastAPI( title="LLM API", description="LLM API for serving OpenAI compatible data", version="0.1.0", contact={ "name": "Husein Abdelgawad", "email": "husein.abdelgawad@gmail.com", }, license_info={ "name": "Apache 2.0", "url": "https://www.apache.org/licenses/LICENSE-2.0.html", }, ) app.include_router(server.routes.auth.router) app.include_router(server.routes.chat.router) app.include_router(server.routes.endpoints.router) app.include_router(server.routes.health.router) app.include_router(server.routes.main.router) app.include_router(server.routes.users.router) def main(): config = Config(app, host="0.0.0.0", port=8000) server = Server(config) server.run() if __name__ == "__main__": main() ``` -------------------------------- ### Python Logging Setup in GitIngest Source: https://github.com/coderamp-labs/gitingest/commit/d061b4877a253ba3f0480d329f025427c7f70177 This snippet demonstrates how to configure and initialize a logger within the GitIngest project. It imports a logging utility and assigns a logger instance to the current module, preparing it to capture log messages. ```python from gitingest.utils.logging_config import get_logger # Initialize logger for this module logger = get_logger(__name__) ``` -------------------------------- ### Run Local Server with Python Source: https://github.com/coderamp-labs/gitingest/commit/d1f8a80826ca38ec105a1878742fe351d4939d6e This snippet demonstrates how to run a local development server using Python. It's a common command for testing web applications locally. Ensure Python is installed and the server module is available. ```shell python -m server ``` -------------------------------- ### Python Main Server Logic Source: https://github.com/coderamp-labs/gitingest/commit/1016f6ecb3b1b066d541d1eba1ddffec49b15f16 This Python snippet represents the main entry point for the server application. It initializes a Starlette application, includes routers for different functionalities (like ingest), and sets up event listeners for application startup and shutdown. It also integrates the metrics server. ```python from starlette.applications import Starlette from starlette.routing import Mount, Route from starlette.middleware import Middleware from src.server.metrics_server import app as metrics_app from src.server.routers.ingest import ingest_router async def startup_event(): print("Server starting...") async def shutdown_event(): print("Server shutting down...") routes = [ Mount("/ingest", app=ingest_router, name="ingest"), Mount("/metrics", app=metrics_app, name="metrics"), ] middleware = [ # Add middleware here if needed ] app = Starlette( routes=routes, middleware=middleware, on_startup=[startup_event], on_shutdown=[shutdown_event], ) ``` -------------------------------- ### Docker Multi-Stage Build Configuration Source: https://github.com/coderamp-labs/gitingest/issues/352 This outlines the multi-stage Docker build process, separating concerns into distinct build stages for CSS compilation, Python environment setup, and the final runtime image. ```dockerfile FROM node:20-alpine AS css-builder RUN npm install -g postcss-cli COPY package.json tailwind.config.js ./ RUN npm install COPY ./src/input.css . RUN npx tailwindcss -i ./input.css -o ./site.css FROM python:3.10-alpine AS python-builder COPY pyproject.toml ./ RUN pip install .[dev] FROM python:3.10-alpine COPY --from=python-builder /usr/local/lib/python3.10/site-packages /usr/local/lib/python3.10/site-packages COPY --from=css-builder /site.css /app/static/css/site.css USER 1000 ``` -------------------------------- ### Jinja Conditional Logic for Examples Source: https://github.com/coderamp-labs/gitingest/commit/b39ef5416c1f8a7993a8249161d2a898b7387595 Uses Jinja templating to conditionally display an 'exampleRepositories' div. The div's visibility is determined by the presence of a 'token' variable, affecting its top margin. ```html {% if show_examples %}

Try these example repositories:

=3.1.27", "requests>=2.28.1", "typer[all]>=0.4.2", ] [tool.setuptools.packages.find] where = ["src"] [project.urls] "Homepage" = "https://github.com/coderamp-labs/gitingest" "Bug Tracker" = "https://github.com/coderamp-labs/gitingest/issues" ``` -------------------------------- ### Enhance docstrings with usage examples Source: https://github.com/coderamp-labs/gitingest/commit/fdcbc53cadde6a5dc3c3626120df1935b63693b2 This Python code snippet demonstrates improved docstrings for better clarity and includes comprehensive usage examples. These examples help users understand how to effectively utilize the gitingest CLI for various tasks. ```python """ Run the gitingest CLI. Examples: # Run with default settings: gitingest # Run and save to a file: gitingest --output my_digest.txt # Run with a specific GitHub token: gitingest --github-token "YOUR_GITHUB_TOKEN" """ ``` -------------------------------- ### Build and Run Application with Docker Compose Source: https://github.com/coderamp-labs/gitingest/commit/414e85189fb9055491530ba8c0665c798474451e Instructions for building and running the application in a production environment using Docker Compose. This involves two commands: one for building the Docker images and another for starting the services in detached mode. ```bash docker compose --profile prod build ``` ```bash docker compose --profile prod up -d ``` -------------------------------- ### Start Metrics Server in a Separate Thread in Python Source: https://github.com/coderamp-labs/gitingest/commit/1016f6ecb3b1b066d541d1eba1ddffec49b15f16 This code snippet demonstrates how to conditionally start a metrics server in a separate thread. It checks an environment variable 'GITINGEST_METRICS_ENABLED' to determine if the server should be started. The server host and port are also configurable via environment variables. ```python if os.getenv("GITINGEST_METRICS_ENABLED", "false").lower() == "true": metrics_host = os.getenv("GITINGEST_METRICS_HOST", "127.0.0.1") metrics_port = int(os.getenv("GITINGEST_METRICS_PORT", "9090")) metrics_thread = threading.Thread( target=start_metrics_server, args=(metrics_host, metrics_port), daemon=True, ) metrics_thread.start() ``` -------------------------------- ### Build Frontend Assets and Python Package for PyPI Source: https://github.com/coderamp-labs/gitingest/commit/016817d5590c1412498b7532f6e854d20239c6be This snippet defines a GitHub Actions workflow that builds frontend assets (Tailwind CSS) and then proceeds to build a Python package (wheel and sdist). It handles dependency installation for both frontend and backend, asset uploading, and artifact downloading to ensure a cohesive build process before publishing. ```yaml name: Publish to PyPI on: release: types: [created] # Run when you click “Publish release” workflow_dispatch: # ... or run it manually from the Actions tab permissions: contents: read jobs: frontend-build: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - name: Setup Node uses: actions/setup-node@v4 with: node-version: 20 cache: npm cache-dependency-path: package-lock.json - name: Install deps + build Tailwind run: | npm ci npm run build:css - name: Upload built CSS uses: actions/upload-artifact@v4 with: name: frontend-assets path: src/static/css/site.css if-no-files-found: error # ── Build wheel/sdist (needs CSS) and upload “dist/” ──────────── release-build: needs: frontend-build runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 # Grab site.css produced above - uses: actions/download-artifact@v4 with: name: frontend-assets path: src/static/css/ - name: Set up Python 3.13 uses: actions/setup-python@v5 with: python-version: "3.13" cache: pip cache-dependency-path: pyproject.toml - name: Install build backend run: | python -m pip install --upgrade pip python -m pip install build twine - name: Build package run: python -m build - name: Upload dist artefact uses: actions/upload-artifact@v4 with: name: dist path: dist/ # ── Publish to PyPI (only if “dist/” succeeded) ───────────────── pypi-publish: needs: release-build runs-on: ubuntu-latest environment: pypi # Creates the “pypi” environment in repo-settings permissions: id-token: write # OIDC token for trusted publishing steps: - uses: actions/download-artifact@v4 with: name: dist path: dist/ - uses: pypa/gh-action-pypi-publish@release/v1 with: verbose: true ``` -------------------------------- ### Initialize FastAPI App with Rate Limiter Source: https://github.com/coderamp-labs/gitingest/commit/2df0eb43989731ae40a9dd82d310ff76a794a46d Initializes a FastAPI application instance and assigns a pre-configured rate limiter to its state. This setup is crucial for managing request rates to the API. It disables the default OpenAPI and Redoc documentation URLs. ```python # Initialize the FastAPI application app = FastAPI(docs_url=None, redoc_url=None) app.state.limiter = limiter ``` -------------------------------- ### Initialize Sentry SDK with Configuration in Python Source: https://github.com/coderamp-labs/gitingest/commit/590e55a4d28a4f5c0beafbd12c525828fa79e221 This code initializes the Sentry SDK with various configuration parameters retrieved from environment variables. It includes the DSN, PII sending preference, and sampling rate for transactions. ```python sentry_sdk.init( dsn=sentry_dsn, # Add data like request headers and IP for users send_default_pii=send_default_pii, # Set traces_sample_rate to capture transactions for tracing traces_sample_rate=traces_sample_rate, ) ```