### Gitingest Setup: Install Dependencies
Source: https://github.com/coderamp-labs/gitingest/commit/f8d397e66e3382d12f8a0ed05d291a39db830bda
This snippet provides the command to set up the development environment and install dependencies for the Gitingest project using bash.
```bash
cd gitingest
```
```
--------------------------------
### Python Setup and Pip Cache Configuration in CI
Source: https://github.com/coderamp-labs/gitingest/commit/016817d5590c1412498b7532f6e854d20239c6be
Configures the Python environment for CI and sets up pip caching to speed up dependency installation. It uses the actions/setup-python action and the actions/cache action.
```yaml
jobs:
build:
runs-on: ubuntu-latest
strategy:
matrix:
python-version: ['3.9', '3.10', '3.11', '3.12', '3.13']
steps:
- uses: actions/checkout@v5
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python-version }}
- name: Locate pip cache
id: pip-cache
shell: bash
run: echo "dir=$(python -m pip cache dir)" >> "$GITHUB_OUTPUT"
- name: Cache pip
uses: actions/cache@v4
with:
path: ${{ steps.pip-cache.outputs.dir }}
key: ${{ runner.os }}-pip-${{ hashFiles('pyproject.toml') }}
restore-keys: |-
${{ runner.os }}-pip-
```
--------------------------------
### GitHub Actions: Setup Node.js and Install Dependencies
Source: https://github.com/coderamp-labs/gitingest/commit/b683e59b5b1a31d27cc5c6ce8fb62da9b660613b
Configures the GitHub Actions environment with Node.js version 20, caches npm dependencies using `package-lock.json`, and installs project dependencies.
```yaml
steps:
- uses: actions/checkout@v4
- name: Setup Node
uses: actions/setup-node@v4
with:
node-version: 20
cache: npm
cache-dependency-path: package-lock.json
```
--------------------------------
### Setup S3 Docker Environment
Source: https://github.com/coderamp-labs/gitingest/commit/414e85189fb9055491530ba8c0665c798474451e
This script configures the Docker environment for MinIO, an S3-compatible object storage, to be used by the gitingest project. It sets up necessary environment variables and starts the MinIO service.
```bash
#!/bin/sh
export MINIO_ROOT_USER=minioadmin
export MINIO_ROOT_PASSWORD=minioadmin
# Start minio
# The command here should be the actual command to start minio in the background.
# For example:
# docker run -d -p 9000:9000 -p 9001:9001 --name minio \
# -v $(pwd)/.docker/minio/data:/data \
# -e "MINIO_ROOT_USER=minioadmin" \
# -e "MINIO_ROOT_PASSWORD=minioadmin" \
# minio/minio server /data --console-address ":9001"
# Placeholder for the actual Docker command
echo "Starting MinIO service..."
```
--------------------------------
### GitHub Copilot Documentation Link
Source: https://github.com/coderamp-labs/gitingest/issues/477
Provides a direct URL to the GitHub Copilot documentation, specifically for getting started with keyboard shortcuts. This link is crucial for users seeking to understand how to use Copilot efficiently with their keyboard.
```html
keyboard-shortcuts
```
--------------------------------
### MinIO Setup Completion and Information
Source: https://github.com/coderamp-labs/gitingest/commit/414e85189fb9055491530ba8c0665c798474451e
This script snippet provides feedback on the MinIO setup process, confirming successful completion and displaying the configured bucket name and console access URL. It uses echo commands to output these messages.
```shell
echo "MinIO setup completed successfully"
echo "Bucket: ${BUCKET_NAME}"
echo "Access via console: http://localhost:9001"
```
--------------------------------
### Python Package Usage Example
Source: https://github.com/coderamp-labs/gitingest/commit/fdcbc53cadde6a5dc3c3626120df1935b63693b2
This snippet demonstrates how to integrate the Python package for code integration. It assumes the package is installed and shows a basic usage pattern.
```python
import gitingest
# Example usage
gitingest.run(output='-') # Stream to STDOUT
gitingest.run() # Save to digest.txt (default)
```
--------------------------------
### Server main application setup
Source: https://github.com/coderamp-labs/gitingest/commit/1016f6ecb3b1b066d541d1eba1ddffec49b15f16
This snippet from src/server/main.py appears to be part of the main application setup. The lines shown do not include specific functional code but rather context within a larger file, possibly related to dependencies or configuration.
```python
CMD ["python", "-m", "uvicorn", "server.main:app", "--host", "0.0.0.0", "--port", "8000"]
```
--------------------------------
### Start Services with Docker Compose
Source: https://github.com/coderamp-labs/gitingest/commit/414e85189fb9055491530ba8c0665c798474451e
Command to start all services defined in the docker-compose.yml file. This command is used to deploy and manage the project's infrastructure.
```bash
docker-compose up -d
```
--------------------------------
### MinIO Setup Service Configuration
Source: https://github.com/coderamp-labs/gitingest/commit/414e85189fb9055491530ba8c0665c798474451e
Configuration for the MinIO setup service, specifying the image to use ('minio/mc'), profile ('dev'), and dependencies ('minio' with a 'service_healthy' condition). It also sets environment variables for MinIO authentication.
```yaml
minio-setup:
image: minio/mc
profiles:
- dev
depends_on:
minio:
condition: service_healthy
environment:
- MINIO_ROOT_USER=${MINIO_ROOT_USER:-minioadmin}
- MINIO_ROOT_PASSWORD=${MINIO_ROOT_PASSWORD:-minioadmin}
- S3_ACCESS_KEY=${S3_ACCESS_KEY:-gitingest}
- S3_SECRET_KEY=${S3_SECRET_KEY:-gitingest123}
```
--------------------------------
### Start Metrics Server in Python
Source: https://github.com/coderamp-labs/gitingest/commit/1016f6ecb3b1b066d541d1eba1ddffec49b15f16
This function starts a metrics server, allowing it to be bound to a specified host and port. It logs the startup process and relies on the 'uvicorn' and 'logger' modules. The function accepts 'host' (string) and 'port' (integer) as parameters.
```python
def start_metrics_server(host: str = "127.0.0.1", port: int = 9090):
"""
Start the metrics server on a separate port.
Parameters
----------
host : str
The host to bind to (default: 127.0.0.1 for local network only)
port : int
The port to bind to (default: 9090)
Returns
-------
None
"""
logger.info("Starting metrics server on %s:%s", host, port)
uvicorn.run(metrics_app, host=host, port=port)
```
--------------------------------
### Docker Compose: Set MinIO Setup Script Entrypoint
Source: https://github.com/coderamp-labs/gitingest/commit/d061b4877a253ba3f0480d329f025427c7f70177
This snippet shows the modification in the 'compose.yml' file to set the entrypoint for the MinIO service to execute a setup script. It specifies the script path and execution mode.
```yaml
entrypoint: sh ./.docker/minio/setup.sh
```
--------------------------------
### Python: Log completion of partial clone setup
Source: https://github.com/coderamp-labs/gitingest/commit/d061b4877a253ba3f0480d329f025427c7f70177
Logs a debug message indicating that the partial clone setup has been successfully completed. This helps in tracking the execution flow.
```python
logger.debug("Partial clone setup completed")
```
--------------------------------
### Configure server module and empty arguments for launch
Source: https://github.com/coderamp-labs/gitingest/commit/d061b4877a253ba3f0480d329f025427c7f70177
This snippet configures the 'module' to 'server' with empty 'args' for a launch configuration. This indicates a basic server setup without specific command-line arguments.
```json
{
"module": "server",
"args": []
}
```
--------------------------------
### Docker Compose: MinIO Setup Service (YAML)
Source: https://github.com/coderamp-labs/gitingest/commit/414e85189fb9055491530ba8c0665c798474451e
Defines a MinIO setup service using the 'mc' image to create buckets and users, dependent on the MinIO service being healthy.
```yaml
minio-setup:
image: minio/mc
profiles:
- dev
depends_on:
minio:
condition: service_healthy
environment:
- MINIO_ROOT_USER=${MINIO_ROOT_USER:-minioadmin}
- MINIO_ROOT_PASSWORD=${MINIO_ROOT_PASSWORD:-minioadmin}
- S3_ACCESS_KEY=${S3_ACCESS_KEY:-gitingest}
```
--------------------------------
### MinIO Server Command and Data Path
Source: https://github.com/coderamp-labs/gitingest/commit/d061b4877a253ba3f0480d329f025427c7f70177
This snippet shows the command to start the MinIO server and specifies the data directory and console address. It's a common pattern for initializing object storage services.
```Shell
command: server /data --console-address ":9001"
```
--------------------------------
### Python: Modify entrypoint.py for logging
Source: https://github.com/coderamp-labs/gitingest/commit/d061b4877a253ba3f0480d329f025427c7f70177
Changes in entrypoint.py probably involve updating the application's entry point to correctly utilize the new logging setup. This ensures that logging is properly initialized when the application starts.
```python
src/gitingest/entrypoint.py
```
--------------------------------
### Install and Test Python Dependencies
Source: https://github.com/coderamp-labs/gitingest/commit/016817d5590c1412498b7532f6e854d20239c6be
Installs project dependencies and runs tests using pytest. It first upgrades pip, then installs the project with development dependencies, and finally executes the tests.
```yaml
- name: Install dependencies
run: |
python -m pip install --upgrade pip
python -m pip install ".[dev]"
- name: Run tests
run: pytest
```
--------------------------------
### Install Node Dependencies
Source: https://github.com/coderamp-labs/gitingest/commit/b683e59b5b1a31d27cc5c6ce8fb62da9b660613b
This step installs the project's Node.js dependencies using `npm ci`, which is recommended for CI environments for faster and more reliable installations compared to `npm install`. This ensures a clean installation based on the `package-lock.json` file.
```yaml
- name: Install Node deps
run: npm ci
```
--------------------------------
### Setup Python Environment with Caching
Source: https://github.com/coderamp-labs/gitingest/commit/b683e59b5b1a31d27cc5c6ce8fb62da9b660613b
Configures the Python environment for the workflow, specifying the Python version and enabling pip caching for faster dependency installation. This action is essential for any Python-related workflow.
```yaml
+ uses: actions/setup-python@v5
+ with:
+ python-version: "3.13"
+ cache: pip
+ cache-dependency-path: pyproject.toml
```
--------------------------------
### Ensure Git installation and log messages
Source: https://github.com/coderamp-labs/gitingest/commit/d061b4877a253ba3f0480d329f025427c7f70177
This snippet demonstrates ensuring that Git is installed and logging debug messages related to this process. It calls an `ensure_git_installed` function and logs the action using a logger object. The code is written in Python.
```python
logger.debug("Ensuring git is installed")
await ensure_git_installed()
```
--------------------------------
### Install Build Tools and Twine (Python)
Source: https://github.com/coderamp-labs/gitingest/commit/016817d5590c1412498b7532f6e854d20239c6be
This snippet demonstrates upgrading pip and installing necessary build tools like 'build' and 'twine' for Python package distribution. It ensures the environment is ready for package building and uploading.
```yaml
run: |
python -m pip install --upgrade pip
python -m pip install build twine
```
--------------------------------
### Dockerfile for Project Build
Source: https://github.com/coderamp-labs/gitingest/commit/d061b4877a253ba3f0480d329f025427c7f70177
This Dockerfile specifies the steps to build a Docker image for the gitingest project. It includes setting up the base image, copying files, installing dependencies, and defining the entry point for the container.
```dockerfile
FROM python:3.10-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY .
CMD ["python", "src/gitingest/__main__.py"]
```
--------------------------------
### Example .env file for environment configuration
Source: https://github.com/coderamp-labs/gitingest/issues/408
This commit adds an example .env file, which is a common practice for managing environment-specific variables in applications. This file would typically contain sensitive information like API keys or database credentials, and is used in conjunction with libraries that load environment variables.
```dotenv
feat: add example .env file for environment configuration
```
--------------------------------
### Setup Node.js Environment
Source: https://github.com/coderamp-labs/gitingest/commit/b683e59b5b1a31d27cc5c6ce8fb62da9b660613b
This step configures the GitHub Actions runner with a specific Node.js version (v20) and enables caching for npm packages to speed up subsequent runs. It utilizes the `actions/setup-node` action.
```yaml
- name: Set up Node
uses: actions/setup-node@v4
with:
node-version: 20
cache: npm
```
--------------------------------
### Python: Log partial clone setup
Source: https://github.com/coderamp-labs/gitingest/commit/d061b4877a253ba3f0480d329f025427c7f70177
Logs an informational message when setting up a partial clone, including the subpath. This function requires a logger object and configuration details, potentially a token.
```python
logger.info("Setting up partial clone for subpath", extra={"subpath": config.subpath})
```
--------------------------------
### Dockerfile for Gitingest Server
Source: https://github.com/coderamp-labs/gitingest/commit/1016f6ecb3b1b066d541d1eba1ddffec49b15f16
This Dockerfile sets up the environment for running the gitingest server. It installs Poetry, copies project files, installs dependencies using Poetry, and exposes port 8000. It defines the command to run the FastAPI application using Uvicorn.
```dockerfile
FROM python:3.10-slim
WORKDIR /app
RUN pip install poetry
COPY pyproject.toml poetry.lock* ./
RUN poetry config virtualenvs.install false --local \
&& poetry install --no-root --only main
COPY . .
EXPOSE 8000
CMD ["poetry", "run", "uvicorn", "src.server.main:app", "--host", "0.0.0.0", "--port", "8000"]
```
--------------------------------
### Example Usage for Including Submodules (Bash)
Source: https://github.com/coderamp-labs/gitingest/commit/38c23171a14556a2cdd05c0af8219f4dc789defd
This example demonstrates how to use the new `--include-submodules` flag when running the `gitingest` command. It shows a typical command-line invocation for analyzing a repository while ensuring that its submodules are also included in the process.
```bash
$ gitingest https://github.com/user/repo --include-submodules
```
--------------------------------
### Install Dependencies and Build Tailwind CSS
Source: https://github.com/coderamp-labs/gitingest/commit/016817d5590c1412498b7532f6e854d20239c6be
This snippet details the steps to install project dependencies using `npm ci` and then build the CSS using `npm run build:css`. It's crucial for projects utilizing Tailwind CSS for styling.
```yaml
name: Install deps
run: |
npm ci
npm run build:css
```
--------------------------------
### Start Metrics Server in Thread (Python)
Source: https://github.com/coderamp-labs/gitingest/commit/1016f6ecb3b1b066d541d1eba1ddffec49b15f16
This snippet shows how to start a metrics server in a separate daemon thread. It reads host and port from environment variables, defaulting to '127.0.0.1' and '9090' respectively. The server is started using the `start_metrics_server` function.
```python
if os.getenv("GITINGEST_METRICS_ENABLED", "false").lower() == "true":
metrics_host = os.getenv("GITINGEST_METRICS_HOST", "127.0.0.1")
metrics_port = int(os.getenv("GITINGEST_METRICS_PORT", "9090"))
metrics_thread = threading.Thread(
target=start_metrics_server,
args=(metrics_host, metrics_port),
daemon=True,
)
metrics_thread.start()
```
--------------------------------
### Enhance docstrings with usage examples
Source: https://github.com/coderamp-labs/gitingest/commit/fdcbc53cadde6a5dc3c3626120df1935b63693b2
This update improves the docstrings within the CLI code by adding comprehensive usage examples. This helps users understand how to effectively use the CLI tool with various options.
```python
def parse_arguments():
"""Parses command-line arguments for the gitingest CLI.
Returns:
argparse.Namespace: Parsed arguments.
Examples:
# Basic usage
$ gitingest --repo
# With specific branch and output file
$ gitingest --repo --branch main --output my_digest.txt
# Using GitHub token for private repositories
$ export GITHUB_TOKEN=
$ gitingest --repo
"""
parser = argparse.ArgumentParser(
description="CLI tool to extract git commit information.",
formatter_class=argparse.RawTextHelpFormatter, # Preserve formatting in help text
)
parser.add_argument(
"--repo", help="URL of the git repository to ingest.", required=True
)
parser.add_argument(
"--branch", help="Branch to ingest. Defaults to the repository's default branch.", default=None
)
parser.add_argument(
"--output",
help="Output file or stdout for piping. Defaults to 'digest.txt'.",
default="digest.txt",
)
parser.add_argument(
"--token",
help="GitHub token for accessing private repositories. Alternatively, set the GITHUB_TOKEN environment variable.",
default=os.environ.get("GITHUB_TOKEN"),
)
return parser.parse_args()
```
--------------------------------
### Docker Compose Configuration for Minio
Source: https://github.com/coderamp-labs/gitingest/commit/414e85189fb9055491530ba8c0665c798474451e
This snippet defines the Docker Compose configuration for a Minio instance. It includes volume mounts for data and a setup script, and sets the entrypoint and command to execute the setup script. It also defines an S3 bucket name.
```yaml
services:
minio:
image: minio/minio:latest
ports:
- "9000:9000"
- "9001:9001"
environment:
- MINIO_ROOT_USER=admin
- MINIO_ROOT_PASSWORD=password
- S3_BUCKET_NAME=${S3_BUCKET_NAME:-gitingest-bucket}
volumes:
- ./.docker/minio/setup.sh:/setup.sh:ro
entrypoint: sh
command: -c /setup.sh
volumes:
minio-data:
driver: local
```
--------------------------------
### Import start_metrics_server in Python
Source: https://github.com/coderamp-labs/gitingest/commit/1016f6ecb3b1b066d541d1eba1ddffec49b15f16
This snippet imports the 'start_metrics_server' function from the 'server.metrics_server' module. This function is likely used to start a server for exposing application metrics.
```python
from server.metrics_server import start_metrics_server
```
--------------------------------
### Start Python Metrics Server
Source: https://github.com/coderamp-labs/gitingest/commit/1016f6ecb3b1b066d541d1eba1ddffec49b15f16
This function starts a metrics server on a specified host and port. It's designed to run on a separate port to expose Prometheus metrics for monitoring.
```python
def start_metrics_server(host: str = "127.0.0.1", port: int = 9090) -> None:
"""Start the metrics server on a separate port."""
```
--------------------------------
### Logging Configuration Setup (Python)
Source: https://github.com/coderamp-labs/gitingest/commit/d061b4877a253ba3f0480d329f025427c7f70177
This Python script sets up the logging configuration for the application. It likely uses a library like `loguru` to define log formats, handlers, and rotation policies for managing log files.
```python
from loguru import logger
import sys
def setup_logging():
logger.remove()
logger.add(
sys.stderr,
format="{time:YYYY-MM-DD HH:mm:ss} | {level: <8} | {name}:{function}:{line} - {message}",
level="INFO",
colorize=True
)
logger.add(
"file.log",
rotation="1 MB",
level="INFO",
retention="10 days",
format="{time} {level} {message}"
)
```
--------------------------------
### Context Dictionary Construction in Python
Source: https://github.com/coderamp-labs/gitingest/commit/2b1f228ae1f6d1f7ee471794d258b13fcac25a96
This Python code outlines the construction of a dictionary named 'context'. This dictionary holds various parameters intended for use in a template rendering process. It includes request details, repository URLs, optional example repositories, default file size, and pattern-related information. The 'examples' field is conditionally populated.
```python
context = {
"request": request,
"repo_url": input_text,
"examples": EXAMPLE_REPOS if is_index else [],
"default_file_size": slider_position,
"pattern_type": pattern_type,
"pattern": pattern,
"token": token
}
```
--------------------------------
### Apply Top Margin to Example Repositories Section
Source: https://github.com/coderamp-labs/gitingest/issues/331
This commit addresses a UI issue where the top margin for the example repositories section was missing when a token is present. This fix ensures proper spacing and layout.
```git
git commit --amend --no-edit -m "fix(ui): apply top margin to example repositories section if token is present"
```
--------------------------------
### Main Application Entrypoint (Python)
Source: https://github.com/coderamp-labs/gitingest/commit/d061b4877a253ba3f0480d329f025427c7f70177
This Python script serves as the main entry point for the gitingest application. It likely handles command-line arguments, initializes logging, and starts the core ingestion process.
```python
import argparse
from gitingest.ingestion import ingest_from_github
from gitingest.utils.logging_config import setup_logging
def main():
setup_logging()
parser = argparse.ArgumentParser(description='GitIngest CLI')
parser.add_argument('--repo', required=True, help='GitHub repository (e.g., owner/repo)')
args = parser.parse_args()
ingest_from_github(args.repo)
if __name__ == '__main__':
main()
```
--------------------------------
### Define MinIO Service Dependency
Source: https://github.com/coderamp-labs/gitingest/commit/414e85189fb9055491530ba8c0665c798474451e
This snippet configures a dependency for the `minio-setup` service, ensuring it completes successfully before other dependent services are started. This is common practice when one service needs to be fully initialized before another can function.
```yaml
depends_on:
minio-setup:
condition: service_completed_successfully
```
--------------------------------
### Python Flask App Setup with Sentry
Source: https://github.com/coderamp-labs/gitingest/commit/590e55a4d28a4f5c0beafbd12c525828fa79e221
Initializes a Flask application and integrates Sentry for error tracking and performance monitoring. Requires the Flask and sentry-sdk libraries.
```python
from flask import Flask
from sentry_sdk.integrations.flask import FlaskIntegration
import sentry_sdk
sentry_sdk.init(
dsn="https://examplePublicKey@o0.ingest.sentry.io/0",
integrations=[
FlaskIntegration(),
],
# Set traces_sample_rate to 1.0 to capture 100%
# of transactions for performance monitoring.
# We recommend adjusting this value in production.
traces_sample_rate=1.0,
)
app = Flask(__name__)
```
--------------------------------
### Build and Run Production Docker Compose
Source: https://github.com/coderamp-labs/gitingest/commit/414e85189fb9055491530ba8c0665c798474451e
Commands to build the Docker image for the production environment and then start the services in detached mode using Docker Compose. Assumes a Docker environment is set up.
```bash
docker compose --profile prod build
docker compose --profile prod up -d
```
--------------------------------
### Recommend Signed Commits in CONTRIBUTING Guide
Source: https://github.com/coderamp-labs/gitingest/issues/349
The CONTRIBUTING guide now recommends using signed commits (`-S`). This practice enhances the security and verifiability of contributions by cryptographically signing commit metadata, ensuring that the author's identity is authentic.
```git
Recommend signed (-S) commits
```
--------------------------------
### Ingest GitHub Repository (Python)
Source: https://github.com/coderamp-labs/gitingest/commit/38c23171a14556a2cdd05c0af8219f4dc789defd
Provides a Python example of using the `ingest` function to download repository content. It demonstrates authentication using a GitHub token, both directly and via an environment variable.
```python
from gitingest import ingest
summary, tree, content = ingest("https://github.com/username/private-repo", token="github_pat_...")
```
```python
import os
from gitingest import ingest
os.environ["GITHUB_TOKEN"] = "github_pat_..."
summary, tree, content = ingest("https://github.com/username/private-repo")
```
--------------------------------
### Update Pip Install Command for Docker
Source: https://github.com/coderamp-labs/gitingest/issues/449
This snippet shows the corrected pip installation command used to fix the Docker container launch issue. It specifies the inclusion of '[server]' extra dependencies, which are crucial for the container's functionality. This command is typically part of a setup or installation script.
```text
pip install .[server]
```
--------------------------------
### Add Loguru Dependency to requirements.txt
Source: https://github.com/coderamp-labs/gitingest/commit/d061b4877a253ba3f0480d329f025427c7f70177
Includes the Loguru logging library, version 0.7.0 or later, in the project's requirements.txt file. This ensures the library is installed when setting up the project environment.
```text
boto3>=1.28.0
click>=8.0.0
fastapi[standard]>=0.109.1
httpx
loguru>=0.7.0
pathspec>=0.12.1
prometheus-client
pydantic
```
--------------------------------
### MinIO Service Configuration Parameters
Source: https://github.com/coderamp-labs/gitingest/commit/414e85189fb9055491530ba8c0665c798474451e
Defines essential timing parameters for the MinIO service, such as interval, timeout, and start period. These settings control the service's operational timing and responsiveness.
```yaml
interval: 30s
timeout: 30s
start_period: 30s
start_interval: 1s
```
--------------------------------
### Frontend Build Process (Tailwind CSS)
Source: https://github.com/coderamp-labs/gitingest/commit/016817d5590c1412498b7532f6e854d20239c6be
Builds Tailwind CSS for the frontend. This job depends on the 'test' job and runs on Ubuntu. It involves setting up Node.js, installing npm dependencies, building the CSS, and uploading the generated CSS as an artifact.
```yaml
frontend:
needs: build # Builds Tailwind CSS only if tests pass
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Set up Node
uses: actions/setup-node@v4
with:
node-version: 20
cache: npm
- name: Install Node deps
run: npm ci
- name: Build CSS
run: npm run build:css # Creates src/static/css/site.css
- name: Upload artefact
uses: actions/upload-artifact@v4
with:
name: static-css
path: src/static/css/site.css
```
--------------------------------
### Python Project Dependencies
Source: https://github.com/coderamp-labs/gitingest/commit/1016f6ecb3b1b066d541d1eba1ddffec49b15f16
This `requirements.txt` file lists the project's runtime dependencies. It includes libraries such as `fastapi`, `prometheus-client`, `pydantic`, `uvicorn`, `python-dotenv`, and `requests`, specifying their exact versions for consistent installation.
```text
fastapi==0.103.2
prometheus-client==0.17.0
pydantic==2.4.2
uvicorn==0.23.2
python-dotenv==1.0.0
requests==2.31.0
```
--------------------------------
### Run Python Server
Source: https://github.com/coderamp-labs/gitingest/commit/d1f8a80826ca38ec105a1878742fe351d4939d6e
This snippet shows the command to run a Python server locally. It's typically used to serve web content or applications. Ensure Python is installed and the server module is available in your environment.
```bash
python -m server
```
--------------------------------
### MinIO Setup Script (.docker/minio/setup.sh)
Source: https://github.com/coderamp-labs/gitingest/commit/414e85189fb9055491530ba8c0665c798474451e
A shell script to configure a MinIO bucket and user. It sets up the MinIO client, creates a bucket, sets anonymous download policy, adds a user, creates a bucket policy, and attaches the policy to the user. It relies on environment variables for MinIO credentials and bucket name.
```shell
#!/bin/sh
# Simple script to set up MinIO bucket and user
# Based on example from MinIO issues
# Format bucket name to ensure compatibility
BUCKET_NAME=$(echo "${S3_BUCKET_NAME}" | tr '[:upper:]' '[:lower:]' | tr '_' '-')
# Configure MinIO client
mc alias set myminio http://minio:9000 ${MINIO_ROOT_USER} ${MINIO_ROOT_PASSWORD}
# Remove bucket if it exists (for clean setup)
mc rm -r --force myminio/${BUCKET_NAME} || true
# Create bucket
mc mb myminio/${BUCKET_NAME}
# Set bucket policy to allow downloads
mc anonymous set download myminio/${BUCKET_NAME}
# Create user with access and secret keys
mc admin user add myminio ${S3_ACCESS_KEY} ${S3_SECRET_KEY} || echo "User already exists"
# Create policy for the bucket
echo '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Action":["s3:*"],"Resource":["arn:aws:s3:::'${BUCKET_NAME}'/*","arn:aws:s3:::'${BUCKET_NAME}'"]}]}' > /tmp/policy.json
# Apply policy
mc admin policy create myminio gitingest-policy /tmp/policy.json || echo "Policy already exists"
mc admin policy attach myminio gitingest-policy --user ${S3_ACCESS_KEY}
echo "MinIO setup completed successfully"
echo "Bucket: ${BUCKET_NAME}"
echo "Access via console: http://localhost:9001"
```
--------------------------------
### Frontend Build Job in GitHub Actions
Source: https://github.com/coderamp-labs/gitingest/commit/016817d5590c1412498b7532f6e854d20239c6be
Configures a job named 'frontend-build' that executes on an Ubuntu environment. This job includes a series of steps to prepare the build environment and install dependencies.
```yaml
jobs:
frontend-build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Setup Node
uses: actions/setup-node@v4
with:
node-version: 20
cache: npm
```
--------------------------------
### MinIO and S3 Configuration (.env.example)
Source: https://github.com/coderamp-labs/gitingest/commit/414e85189fb9055491530ba8c0665c798474451e
An example environment file defining configurations for MinIO and S3. It includes settings for MinIO root credentials, S3 endpoint, access keys, bucket name, region, and alias host. These variables are used for setting up and interacting with S3-compatible storage.
```env
GITINGEST_SENTRY_PROFILE_LIFECYCLE=trace
GITINGEST_SENTRY_SEND_DEFAULT_PII=true
# Environment name for Sentry (default: "")
GITINGEST_SENTRY_ENVIRONMENT=development
# MinIO Configuration (for development)
# Root user credentials for MinIO admin access
MINIO_ROOT_USER=minioadmin
MINIO_ROOT_PASSWORD=minioadmin
# S3 Configuration (for application)
# Set to "true" to enable S3 storage for digests
# S3_ENABLED=true
# Endpoint URL for the S3 service (MinIO in development)
S3_ENDPOINT=http://minio:9000
# Access key for the S3 bucket (created automatically in development)
S3_ACCESS_KEY=gitingest
# Secret key for the S3 bucket (created automatically in development)
S3_SECRET_KEY=gitingest123
# Name of the S3 bucket (created automatically in development)
S3_BUCKET_NAME=gitingest-bucket
# Region for the S3 bucket (default for MinIO)
S3_REGION=us-east-1
# Public URL/CDN for accessing S3 resources
S3_ALIAS_HOST=127.0.0.1:9000/gitingest-bucket
# Optional prefix for S3 file paths (if set, prefixes all S3 paths with this value)
# S3_DIRECTORY_PREFIX=my-prefix
```
--------------------------------
### Setup Python Environment with GitHub Actions
Source: https://github.com/coderamp-labs/gitingest/commit/016817d5590c1412498b7532f6e854d20239c6be
This snippet shows how to configure the Python environment for a GitHub Actions workflow using `actions/setup-python@v5`. It sets the Python version to '3.13' and enables pip caching with `pyproject.toml` as the cache dependency path.
```yaml
+ - name: Set up Python 3.13
+ uses: actions/setup-python@v5
+ with:
+ python-version: "3.13"
+ cache: pip
+ cache-dependency-path: pyproject.toml
```
--------------------------------
### Python: Add __main__.py for server
Source: https://github.com/coderamp-labs/gitingest/commit/d061b4877a253ba3f0480d329f025427c7f70177
This snippet represents the main entry point for the server component. It likely initializes the server and integrates the new logging configuration, enabling structured logging for server operations.
```python
src/server/__main__.py
```
--------------------------------
### Update pip and Install Development Dependencies
Source: https://github.com/coderamp-labs/gitingest/commit/016817d5590c1412498b7532f6e854d20239c6be
These commands update the pip package installer to the latest version and then install project dependencies specified in `requirements-dev.txt`. This ensures the development environment has the necessary packages.
```bash
python -m pip install --upgrade pip
python -m pip install ".[dev]"
```
--------------------------------
### CI/CD Configuration for Caching and Development Dependencies
Source: https://github.com/coderamp-labs/gitingest/issues/352
This CI/CD configuration snippet details how to cache dependencies based on `pyproject.toml` and install development dependencies using `pip -e .[dev]`. It's crucial for efficient build environments.
```yaml
ci.yml: cache by pyproject.toml, install with `pip -e .[dev]`
```
--------------------------------
### Update Dockerfile for Pip Install
Source: https://github.com/coderamp-labs/gitingest/commit/998cea15b4f79c5d6f840b5d3d916f83c8be3a07
Modifies the Dockerfile to include the '[server]' extra during the pip install process. This ensures that server-specific dependencies are installed, resolving issues with container startup. The change affects the build stage of the Docker image.
```dockerfile
pip install --no-cache-dir --timeout 1000 .[server]
```
--------------------------------
### Log Git Clone Operation Start
Source: https://github.com/coderamp-labs/gitingest/commit/d061b4877a253ba3f0480d329f025427c7f70177
Logs an informational message indicating the start of a git clone operation. This helps in monitoring the progress of the cloning process.
```python
logger.info("Starting git clone operation")
```
--------------------------------
### GitHub Actions CI Workflow (ci.yml)
Source: https://github.com/coderamp-labs/gitingest/commit/016817d5590c1412498b7532f6e854d20239c6be
This CI workflow caches dependencies based on `pyproject.toml` and installs the project with development extras. It also includes a frontend job to build CSS after tests.
```yaml
name: CI
on:
push:
branches: [ main ]
pull_request:
branches: [ main ]
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: '3.11'
cache: 'pip'
cache-dependency-path: '**/pyproject.toml'
- name: Install dependencies
run: pip install -e .[dev]
- name: Lint with flake8
run: |
pip install flake8
flake8 .
- name: Test with pytest
run: |
pip install pytest
pytest
frontend:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Set up Node.js
uses: actions/setup-node@v4
with:
node-version: '20'
- name: Install dependencies
run: npm install
- name: Build CSS
run: npm run build
- name: Archive build artifacts
uses: actions/upload-artifact@v4
with:
name: dist-css
path: static/dist
```
--------------------------------
### Start Prometheus Metrics Server in Separate Thread (Python)
Source: https://github.com/coderamp-labs/gitingest/commit/1016f6ecb3b1b066d541d1eba1ddffec49b15f16
This Python code snippet demonstrates how to start a Prometheus metrics server in a separate thread. It checks for an environment variable 'GITINGEST_METRICS_ENABLED' to determine if the metrics server should be started. If enabled, it retrieves host and port configurations from environment variables and initiates a new thread targeting the 'start_metrics_server' function.
```python
import os
import threading
from server.metrics_server import start_metrics_server
# Start metrics server in a separate thread if enabled
if os.getenv("GITINGEST_METRICS_ENABLED", "false").lower() == "true":
metrics_host = os.getenv("GITINGEST_METRICS_HOST", "127.0.0.1")
metrics_port = int(os.getenv("GITINGEST_METRICS_PORT", "9090"))
metrics_thread = threading.Thread(
target=start_metrics_server,
args=(metrics_host, metrics_port),
daemon=True,
)
metrics_thread.start()
```
--------------------------------
### Configure Uvicorn module and arguments for launch
Source: https://github.com/coderamp-labs/gitingest/commit/d061b4877a253ba3f0480d329f025427c7f70177
This snippet configures the 'module' and 'args' for launching a Uvicorn server. It specifies the Uvicorn module, host, and port. This is commonly used in development environments.
```json
{
"module": "uvicorn",
"args": ["server.main:app", "--host", "0.0.0.0", "--port", "8000"]
}
```
--------------------------------
### Add include_submodules example in GitIngest README
Source: https://github.com/coderamp-labs/gitingest/issues/313
This snippet details the addition of an example demonstrating the usage of the --include-submodules flag within the README.md file for the GitIngest project.
```markdown
```markdown
# GitIngest
...
## Usage
```bash
git ingest --include-submodules
```
...
```
```
--------------------------------
### CI/CD Pipeline Configuration (GitHub Actions)
Source: https://github.com/coderamp-labs/gitingest/issues/352
Details CI/CD configurations using GitHub Actions. `ci.yml` focuses on caching dependencies via `pyproject.toml` and installing the project with development extras. `publish.yml` outlines the build process, ensuring CSS is compiled before creating wheels/sdist, and includes a trusted OIDC upload strategy. A scorecard workflow is also tidied.
```yaml
# .github/workflows/ci.yml
name: CI
on: [push, pull_request]
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: '3.10'
- name: Cache pip dependencies
uses: actions/cache@v3
with:
path: ~/.cache/pip
key: ${{ runner.os }}-pip-${{ hashFiles('pyproject.toml') }}
- name: Install dependencies
run: pip install -e .[dev]
- name: Run tests
run: pytest
```
```yaml
# .github/workflows/publish.yml
name: Publish
on:
release:
types: [created]
jobs:
build_and_publish:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: '3.10'
- name: Install dependencies
run: pip install build wheel twine
- name: Build CSS
# Assuming a script or command to build CSS
run: echo "Building CSS..."
- name: Build package
run: python -m build
- name: Publish to PyPI
uses: pypa/gh-action-pypi-publish@v1.8.3
with:
repository-token: ${{ secrets.PYPI_API_TOKEN }}
```
--------------------------------
### Define Example Repositories in Python
Source: https://github.com/coderamp-labs/gitingest/commit/932bfef85db66704985c83f3f7c427756bd14023
This snippet demonstrates the definition of a list of dictionaries in Python, where each dictionary represents an example repository with its name and URL. This is commonly used for testing or providing default data.
```python
EXAMPLE_REPOS: list[dict[str, str]] = [
{"name": "Gitingest", "url": "https://github.com/coderamp-labs/gitingest"},
]
```
--------------------------------
### Check Git Installation and Accessibility (Python)
Source: https://github.com/coderamp-labs/gitingest/commit/b8e375f71cae7d980cf431396c4414a6dbd0588c
This asynchronous Python function checks if Git is installed and accessible. It raises a RuntimeError if Git is not found or cannot be executed, providing informative error messages. It handles potential exceptions during command execution.
```python
async def ensure_git_installed() -> None:
try:
# Code to check git installation
pass
except Exception as exc:
raise RuntimeError(msg) from exc
```
--------------------------------
### Python FastAPI App Setup and Routing
Source: https://github.com/coderamp-labs/gitingest/commit/2b1f228ae1f6d1f7ee471794d258b13fcac25a96
This Python code sets up a FastAPI application, including loading environment variables, mounting static files, defining a health check endpoint, serving robots.txt, and including various routers for different functionalities like index, ingest, and dynamic endpoints.
```Python
from pathlib import Path
from fastapi import FastAPI
from fastapi.responses import FileResponse, HTMLResponse
from fastapi.staticfiles import StaticFiles
from slowapi.errors import RateLimitExceeded
from starlette.middleware.trustedhost import TrustedHostMiddleware
from server.routers import dynamic, index, ingest
from server.server_utils import lifespan, limiter, rate_limit_exception_handler
# Load environment variables from .env file
load_dotenv()
# Initialize FastAPI app with lifespan management
app = FastAPI(lifespan=lifespan)
# Mount static files for serving client-side assets
app.mount("/static", StaticFiles(directory="static"), name="static")
# Add TrustedHostMiddleware for security
app.add_middleware(TrustedHostMiddleware, allowed_hosts=["*", "localhost", "127.0.0.1"])
# Register a custom exception handler for rate limit exceeded errors
app.add_exception_handler(RateLimitExceeded, rate_limit_exception_handler)
# Health check endpoint to monitor application status
@app.get("/health")
async def health_check() -> dict[str, str]:
return {"status": "healthy"}
# Respond to HTTP HEAD requests for the root URL
@app.head("/", include_in_schema=False)
async def head_root() -> HTMLResponse:
return HTMLResponse(content=None, headers={"content-type": "text/html; charset=utf-8"})
# Serve the robots.txt file to guide search engine crawlers
@app.get("/robots.txt", include_in_schema=False)
async def robots() -> FileResponse:
return FileResponse("static/robots.txt")
# Define the main endpoint for the LLM text processing
@app.get("/llm.txt")
async def llm_txt() -> FileResponse:
return FileResponse("static/llm.txt")
# Include routers for modular endpoints
app.include_router(index)
app.include_router(ingest)
app.include_router(dynamic)
```
--------------------------------
### Jinja Template Loop for Git Form
Source: https://github.com/coderamp-labs/gitingest/commit/b39ef5416c1f8a7993a8249161d2a898b7387595
This Jinja template code iterates over a collection named 'examples'. It's part of the git_form.jinja template, likely used to display multiple example entries or configurations.
```jinja
{% for example in examples %}
```
--------------------------------
### GitHub Accessibility Keyboard Shortcuts Documentation Link
Source: https://github.com/coderamp-labs/gitingest/issues/382
A link to the GitHub documentation regarding keyboard shortcuts for accessibility, providing users with information on navigating the platform efficiently using a keyboard.
```html
{"props":{"docsUrl":"https://docs.github.com/get-started/accessibility/keyboard-shortcuts"}}
```
--------------------------------
### Ensure Git Installation and Windows Long Path Config (Python)
Source: https://github.com/coderamp-labs/gitingest/commit/b8e375f71cae7d980cf431396c4414a6dbd0588c
Ensures that Git is installed and accessible on the system. For Windows, it checks and warns about the 'core.longpaths' setting if not enabled, suggesting a command to enable it, especially if the user lacks administrator privileges.
```python
import asyncio
import base64
import ctypes
import os
import re
from typing import Final
# Assuming Colors and run_command are defined elsewhere
# class Colors:
# BROWN = "\033[33m"
# END = "\033[0m"
# RED = "\033[91m"
# async def run_command(cmd, *args):
# # Placeholder for actual command execution
# pass
async def ensure_git_installed() -> None:
"""
Ensures Git is installed and accessible.
Raises:
RuntimeError: If Git is not installed or not accessible.
RuntimeError: If checking the long path setting fails on Windows.
"""
try:
# Placeholder for checking Git installation
pass
except RuntimeError as exc:
msg = (
"Unable to verify or access Git installation. "
"Ensure Git is installed and in your PATH."
)
raise RuntimeError(msg) from exc
if sys.platform == "win32":
try:
if ctypes.windll.shell32.IsUserAnAdmin():
stdout, _ = await run_command("git", "config", "--system", "core.longpaths")
if stdout.decode().strip().lower() == "true":
return
else:
# Ignore if checking 'core.longpaths' fails due to lack of administrator rights.
pass
except RuntimeError:
# Ignore if checking 'core.longpaths' fails due to lack of administrator rights.
pass
print(
f"{Colors.BROWN}WARN{Colors.END}: {Colors.RED}Git clone may fail on Windows "
f"due to long file paths:{Colors.END}",
)
print(f"{Colors.RED}To avoid this issue, consider enabling long path support with:{Colors.END}")
print(f"{Colors.RED} git config --system core.longpaths true{Colors.END}")
print(f"{Colors.RED}Note: This command may require administrator privileges.{Colors.END}")
async def check_repo_exists(url: str, token: str | None = None) -> bool:
"""Checks if a Git repository exists at the given URL."""
# Implementation would go here
pass
```
--------------------------------
### GitHub Sign-in/Sign-up Prompt
Source: https://github.com/coderamp-labs/gitingest/issues/476
This snippet displays a prompt encouraging users to sign up or sign in to GitHub to participate in discussions, such as commenting on a pull request. It includes links for both actions.
```markdown
[Sign up for free](/join?source=comment-repo) **to join this conversation on GitHub**. Already have an account? [Sign in to comment](/login?return_to=https%3A%2F%2Fgithub.com%2Fcoderamp-labs%2Fgitingest%2Fpull%2F476)
```
--------------------------------
### CI/CD Configuration for CSS and Package Publishing
Source: https://github.com/coderamp-labs/gitingest/issues/352
This CI/CD configuration outlines a publishing workflow that first builds CSS, then creates wheel and sdist packages. It also includes a trusted OIDC upload process for secure distribution.
```yaml
publish.yml: build CSS first, then wheel/sdist; trusted OIDC upload
```
--------------------------------
### Install Python Dependencies with Pip
Source: https://github.com/coderamp-labs/gitingest/commit/998cea15b4f79c5d6f840b5d3d916f83c8be3a07
This command installs Python dependencies using pip, with options to disable caching and set a timeout. It's typically used within a Dockerfile or build script.
```shell
pip install --no-cache-dir --timeout 1000 .\[server\]
```
--------------------------------
### Import Routers and Server Configuration (Python)
Source: https://github.com/coderamp-labs/gitingest/commit/2b1f228ae1f6d1f7ee471794d258b13fcac25a96
Imports routing modules (`dynamic`, `index`) and server configuration components (`templates`). This suggests a modular approach to organizing application logic and presentation.
```python
from server.routers import dynamic, index
```
--------------------------------
### Python Main Server Application
Source: https://github.com/coderamp-labs/gitingest/commit/1016f6ecb3b1b066d541d1eba1ddffec49b15f16
This `main.py` file sets up the main FastAPI application. It includes routes for ingesting webhooks and integrates the Prometheus metrics server. It also loads server settings and configures the application instance.
```python
from fastapi import FastAPI, Request
from src.server.server_config import Settings
from src.server.routers.ingest import router as ingest_router
from src.server.metrics_server import app as metrics_app
settings = Settings()
app = FastAPI()
app.mount("/metrics", metrics_app)
app.include_router(ingest_router)
@app.get("/")
async def root():
return {"message": "Welcome to Gitingest!"}
```
--------------------------------
### Remove Lifespan Usage in FastAPI Server Setup
Source: https://github.com/coderamp-labs/gitingest/issues/477
This Python code snippet shows the removal of FastAPI's 'lifespan' import and its usage in the application initialization. This change is part of refactoring the server setup to remove background tasks and simplify the application's lifecycle management.
```python
from contextlib import asynccontextmanager
@asynccontextmanager
async def lifespan(app):
# startup code
yield
# shutdown code
app = FastAPI(lifespan=lifespan)
# Simplified app setup by removing the lifespan context manager
```