### Start Auto-News Services using Make

Source: https://github.com/finaldie/auto-news/wiki/Docker-Installation

This command starts the deployed services for the Auto-News project. Once started, the services will automatically pull sources every hour. This command should be executed from the root of the project's source code folder.

```bash
make start
```

--------------------------------

### Deploy Auto-News Services using Make

Source: https://github.com/finaldie/auto-news/wiki/Docker-Installation

This command installs project dependencies and deploys the necessary services for the Auto-News project. It assumes you are in the root of the project's source code folder. Ensure all prerequisites are met before execution.

```bash
make deps && make deploy
```

--------------------------------

### Rebuild Docker Images

Source: https://github.com/finaldie/auto-news/wiki/Docker-Installation

Commands to rebuild the Docker images for the project. This process involves stopping the current services, building new images, and then starting the services with the updated images.

```bash
make stop && make build && make start
```

--------------------------------

### Redeploy Services After .env Changes

Source: https://github.com/finaldie/auto-news/wiki/Docker-Installation

Instructions to redeploy services after modifying the .env file. This process involves stopping services, deploying the changes, and then starting the services again.

```bash
make stop && make deploy && make start
```

--------------------------------

### Install ArgoCD Manifests for Auto-news

Source: https://github.com/finaldie/auto-news/wiki/Installation-using-ArgoCD

This command installs the necessary ArgoCD manifests for the Auto-news application. It deploys the application into the ArgoCD 'default' project and the Kubernetes 'auto-news' namespace. Ensure you have the 'make' utility installed and the project cloned locally.

```bash
make k8s-argocd-install
```

--------------------------------

### Install Auto-News Helm Chart

Source: https://github.com/finaldie/auto-news/wiki/Installation-using-Helm

Installs the Auto-News Helm chart using a make command. This is a prerequisite for deploying the application on Kubernetes. No specific inputs or outputs are detailed beyond the execution of the command.

```bash
make k8s-helm-install
```

--------------------------------

### Upgrade Project to Latest Code

Source: https://github.com/finaldie/auto-news/wiki/Docker-Installation

Steps to upgrade the project to the latest code version from the repository. This includes pulling changes, stopping services, deploying updates, and restarting services.

```bash
git pull && make stop && make deploy && make start
```

--------------------------------

### Prepare Environment File for Kubernetes

Source: https://github.com/finaldie/auto-news/wiki/Installation-using-Helm

Creates a default environment file for Kubernetes deployment. This file, typically located at `build/.env.k8s`, should be populated with necessary environment variables, similar to the docker-compose setup.

```bash
make prepare-env
```

--------------------------------

### Stop and Restart Services with Make

Source: https://github.com/finaldie/auto-news/wiki/Docker-Installation

Commands to stop and restart all services managed by the make utility. These are fundamental operations for service management within the project.

```bash
# Stop all services
make stop

# Restart all services
make stop && make start
```

--------------------------------

### Docker Compose Deployment Commands (Bash)

Source: https://context7.com/finaldie/auto-news/llms.txt

This set of Bash commands outlines the process for deploying the auto-news application using Docker Compose. It includes steps for installing dependencies, building and tagging the Docker image, deploying the services, initializing the environment, starting the application, and managing logs and status.

```bash
# Build and deploy
make deps
make build repo=finaldie/auto-news tag=0.9.15
make deploy
make init
make start

# Enable Airflow DAGs
make enable_dags

# Monitor logs
make logs

# Check status
make ps

# Stop services
make stop

# Upgrade to new version
make upgrade
```

--------------------------------

### Restart Service (Makefile)

Source: https://github.com/finaldie/auto-news/wiki/Content-Filtering

Commands to restart the service after modifying environment variables. This includes initialization, stopping, deploying, and starting the service.

```bash
make init && make stop && make deploy && make start
```

--------------------------------

### Update Helm Repositories

Source: https://github.com/finaldie/auto-news/wiki/Installation-using-Helm

Updates the local Helm repositories to ensure the latest chart information is available. This is a prerequisite for installing or upgrading Helm-based applications.

```bash
make helm-repo-update
```

--------------------------------

### Complete Article Processing Pipeline Setup in Python

Source: https://context7.com/finaldie/auto-news/llms.txt

Sets up the environment variables required for the complete article processing pipeline, including Notion token, OpenAI API key, and the OpenAI model to be used. It also specifies the LLM provider, configuring the system for article processing operations.

```python
import os
from ops_article import OperatorArticle

# Set environment variables
os.environ["NOTION_TOKEN"] = "secret_xxx"
os.environ["OPENAI_API_KEY"] = "sk-xxx"
os.environ["OPENAI_MODEL"] = "gpt-4o-mini"
os.environ["LLM_PROVIDER"] = "openai"

```

--------------------------------

### Modify and Re-apply Environment Variables

Source: https://github.com/finaldie/auto-news/wiki/Installation-using-Helm

Updates environment variables by modifying the `build/.env` file, then re-creates Kubernetes secrets and restarts the services. This involves deleting and creating secrets, followed by uninstalling and installing the Helm chart.

```bash
make k8s-secret-delete && make k8s-secret-create
make k8s-helm-uninstall && k8s-helm-install
```

--------------------------------

### ArgoCD Deployment Commands (Bash)

Source: https://context7.com/finaldie/auto-news/llms.txt

This section provides Bash commands for deploying the auto-news application using ArgoCD. It covers installing the ArgoCD application into a specified namespace and enabling Airflow DAGs post-deployment. It also mentions monitoring the deployment through the ArgoCD dashboard.

```bash
# Install ArgoCD application
make k8s-argocd-install namespace=auto-news

# Enable DAGs after deployment
make k8s-airflow-dags-enable namespace=auto-news

# Monitor deployment in ArgoCD dashboard
# Access: https://argocd.example.com/applications/auto-news
```

--------------------------------

### Kubernetes Port-Forwarding for Services

Source: https://github.com/finaldie/auto-news/wiki/Installation-using-Helm

Establishes port-forwarding to access various services of the Auto-News deployment locally. This is useful for debugging and accessing UIs like Airflow, Milvus, and Adminer. It requires kubectl to be installed and configured.

```bash
kubectl port-forward service/auto-news-webserver 8080:8080 --namespace auto-news --address=0.0.0.0
```

```bash
kubectl port-forward service/auto-news-milvus-attu -n auto-news 9100:3001 --address=0.0.0.0
```

```bash
kubectl port-forward service/auto-news-adminer -n auto-news 8070:8080 --address=0.0.0.0
```

--------------------------------

### Upgrade Auto-News to Latest Release

Source: https://github.com/finaldie/auto-news/wiki/Installation-using-Helm

Upgrades the Auto-News project to the latest version. This process involves pulling the latest code, deleting a specific Airflow job, and then re-installing the Helm chart. Ensure you are in the root directory of the auto-news project before starting.

```bash
> git pull

# Remove the latest completed airflow-init-user job first
> kubectl delete job -n auto-news airflow-init-user

# Install latest release
> make k8s-helm-install
```

--------------------------------

### Kubernetes Helm Deployment Commands (Bash)

Source: https://context7.com/finaldie/auto-news/llms.txt

These Bash commands detail the deployment of the auto-news project on Kubernetes using Helm. The process involves adding Helm repositories, creating a namespace, configuring the environment, generating secrets, building and pushing Docker images, and finally installing the application via Helm, along with enabling Airflow DAGs and port forwarding for access.

```bash
# Add Helm repositories
make helm-repo-update

# Create namespace
make k8s-namespace-create namespace=auto-news

# Configure environment
make k8s-env-create
# Edit build/.env.k8s with your configuration

# Create secrets
make k8s-secret-create namespace=auto-news

# Build and push Docker image
make k8s-docker-build repo=myrepo/auto-news tag=1.0.0
make k8s-docker-push repo=myrepo/auto-news tag=1.0.0

# Install with Helm
make k8s-helm-install namespace=auto-news

# Enable DAGs
make k8s-airflow-dags-enable namespace=auto-news

# Port forwarding for access
kubectl port-forward service/auto-news-webserver 8080:8080 -n auto-news
kubectl port-forward service/auto-news-milvus-attu 9100:3001 -n auto-news
```

--------------------------------

### Initialize and Pull Articles from Notion using Python

Source: https://context7.com/finaldie/auto-news/llms.txt

Initializes the `OperatorArticle` to pull articles from Notion inbox databases. This function is part of the content pulling operation and returns a dictionary of page IDs and their associated data. It serves as the starting point for processing content from Notion.

```python
from ops_article import OperatorArticle

# Initialize article operator
operator = OperatorArticle()

# Pull articles from Notion inbox databases
# Returns: dict {page_id: page_data}
pages = operator.pull()

# Example output structure:
# {
#   "abc123": {
#     "name": "Article Title",
#     "source_url": "https://example.com/article",
#     "created_time": "2025-01-15T10:30:00.000Z",
#     "last_edited_time": "2025-01-15T10:30:00.000Z",
#     "source": "Article"
#   }
# }
```

--------------------------------

### Configure Reddit Filtering Thresholds (.env)

Source: https://github.com/finaldie/auto-news/wiki/Content-Filtering

Sets the minimum score threshold for Reddit categories to filter content. Modify the `REDDIT_FILTER_MIN_SCORES` variable in `build/.env`. Values are formatted as `Category:threshold[,Category2:threshold2,...]`. Restart the service after changes using `make init && make stop && make deploy && make start`.

```bash
# In build/.env
...
# Value Format: Category:threshold[,Category2:threshold2,
          ...]
REDDIT_FILTER_MIN_SCORES=Game:4.5,AI:5
```

--------------------------------

### Configure Twitter Filtering Thresholds (.env)

Source: https://github.com/finaldie/auto-news/wiki/Content-Filtering

Sets the minimum score threshold for Twitter lists to filter content. Modify the `TWITTER_FILTER_MIN_SCORES` variable in `build/.env`. Values are formatted as `ListName:threshold[,ListName2:threshold2,...]`. This feature requires a paid Twitter account. Restart the service after changes using `make init && make stop && make deploy && make start`.

```bash
# In build/.env
...
# Value Format: ListName:threshold[,ListName2:threshold2,
          ...]
TWITTER_FILTER_MIN_SCORES=Famous:4.5,AI:5
```

--------------------------------

### Cache Notion Inbox Processing Times (Python)

Source: https://context7.com/finaldie/auto-news/llms.txt

Utilizes DBClient to set and get the creation timestamp for Notion inbox items. This is useful for tracking when items were last processed to avoid redundant operations. It caches the timestamp associated with a source and category.

```python
client.set_notion_inbox_created_time(
    source="Article",
    category="default",
    t="2025-01-15T10:30:00.000Z"
)

last_time = client.get_notion_inbox_created_time("Article", "default")
print(f"Last processed: {last_time.decode('utf-8')}")
```

--------------------------------

### Refine Text, Split Bilingual Summary, Get Top Items (Python)

Source: https://context7.com/finaldie/auto-news/llms.txt

Offers utilities for text manipulation, including cleaning extra whitespace from content, splitting a combined bilingual summary and its translation, and retrieving the top N items from a scored list. These functions are part of the 'utils' module.

```Python
from utils import refine_content, splitSummaryTranslation, get_top_items

# Refine content (remove extra whitespace)
text = "Line 1\n\n\n\nLine 2\n\n\nLine 3"
cleaned = refine_content(text)
print(cleaned)  # "Line 1\nLine 2\nLine 3"

# Split bilingual summary
bilingual = "English summary\n===
Chinese translation"
summary, translation = splitSummaryTranslation(bilingual)
print(f"Summary: {summary}")
print(f"Translation: {translation}")

# Get top items from scored list
items = [("item1", 0.9), ("item2", 0.7), ("item3", 0.95), ("item4", 0.6)]
top_k = get_top_items(items, k=2)
print(top_k)  # [("item3", 0.95), ("item1", 0.9)]
```

--------------------------------

### Create Kubernetes Environment File

Source: https://github.com/finaldie/auto-news/wiki/Installation-using-Helm

Generates a specific environment file tailored for Kubernetes. This step is crucial as it separates sensitive information like tokens and passwords into Kubernetes Secrets, rather than embedding them directly into the Docker image.

```bash
make k8s-env-create
```

--------------------------------

### Create Kubernetes Namespace

Source: https://github.com/finaldie/auto-news/wiki/Installation-using-Helm

Creates a dedicated Kubernetes namespace named 'auto-news' for deploying the application. This helps in organizing and isolating resources.

```bash
make k8s-namespace-create

```

```bash
# Verification command:
kubectl get namespace | grep auto-news
```

--------------------------------

### Google Gemini Agent Configuration and Usage (Python)

Source: https://context7.com/finaldie/auto-news/llms.txt

Initializes and uses a Google Gemini LLM agent for content generation. This snippet shows setting the Google API key, initializing the agent with a specific model and temperature, setting up a default prompt, and then running the agent to generate content based on input text. The generated text is accessed via the `.text` attribute.

```python
from llm_agent import LLMAgentGemini
import os

os.environ["GOOGLE_API_KEY"] = "AIzaXXX"

# Initialize Gemini agent
agent = LLMAgentGemini(
    model_name="gemini-1.5-flash-latest",
    temperature=0
)
agent.init_prompt()  # Uses default summary prompt

# Generate content
text = "Long article content..."
response = agent.run(text)

print(response.text)  # Access generated text
```

--------------------------------

### Build and Push Custom Docker Image for Kubernetes

Source: https://github.com/finaldie/auto-news/wiki/Installation-using-Helm

This section covers building a customized Docker image for auto-news and pushing it to a registry, such as Dockerhub. It is optional if using the default image. The `repository` and `tag` in `helm/values.yaml` must be updated after pushing the image.

```bash
# Build docker image, replace <repo_and_imagename> and <x.y.z> accordingly
make k8s-docker-build repo=<repo_and_imagename> tag=<x.y.z>

# Push docker image to the registry (e.g. dockerhub)
docker login

# Replace <repo_and_imagename> and <x.y.z> accordingly
make k8s-docker-push repo=<repo_and_imagename> tag=<x.y.z>
```

--------------------------------

### Summarization Agent Configuration and Usage (Python)

Source: https://context7.com/finaldie/auto-news/llms.txt

Initializes and runs a summarization LLM agent. This snippet demonstrates setting up the environment variables for the LLM provider, configuring the agent's prompt and LLM parameters, and then executing the summarization process on a given text. It supports chunking for long documents.

```python
from llm_agent import LLMAgentSummary
import os

# Configure LLM provider
os.environ["LLM_PROVIDER"] = "openai"  # or "google", "ollama"
os.environ["OPENAI_MODEL"] = "gpt-4o-mini"
os.environ["OPENAI_API_KEY"] = "sk-xxx"
os.environ["TRANSLATION_LANG"] = ""  # Leave empty for no translation

# Initialize agent
agent = LLMAgentSummary()
agent.init_prompt(translation_enabled=False)
agent.init_llm(
    provider="openai",
    model_name="gpt-4o-mini",
    temperature=0,
    chain_type="map_reduce",
    verbose=True
)

# Generate summary
long_text = """
[Long article content here, can be 50,000+ characters]
"""

summary = agent.run(
    text=long_text,
    chunk_size=10240,    # Chars per chunk
    chunk_overlap=256    # Overlap between chunks
)

print(f"Summary: {summary}")
# Output: "1. Key point one
# 2. Key point two
# 3. ..."
```

--------------------------------

### Complete YouTube Pipeline with OperatorYoutube

Source: https://context7.com/finaldie/auto-news/llms.txt

Orchestrates a complete pipeline for processing YouTube videos, from pulling transcripts to publishing summarized and ranked content to Notion. Requires API keys for Notion and OpenAI to be set as environment variables.

```python
from ops_youtube import OperatorYoutube
import os

os.environ["NOTION_TOKEN"] = "secret_xxx"
os.environ["OPENAI_API_KEY"] = "sk-xxx"
os.environ["YOUTUBE_TRANSCRIPT_LANGS"] = "en,es"  # Try English, then Spanish

operator = OperatorYoutube()

# Process YouTube videos
videos = operator.pull(data_folder="data/youtube", run_id="run_456")
print(f"Extracted {len(videos)} video transcripts")

deduped = operator.dedup(videos, target="inbox")
summarized = operator.summarize(deduped)
ranked = operator.rank(summarized)

stats = operator.push(ranked, targets=["notion"], topk=3)
print(f"Published {stats['total']} videos")
```

--------------------------------

### Enable Airflow DAGs

Source: https://github.com/finaldie/auto-news/wiki/Installation-using-Helm

Enables Airflow Directed Acyclic Graphs (DAGs) for the Auto-News project. This command is essential for the scheduled tasks within Airflow to function correctly. Verification involves checking pods and accessing the Airflow UI via port-forwarding.

```bash
make k8s-airflow-dags-enable
```

--------------------------------

### Environment Configuration Variables (.env)

Source: https://context7.com/finaldie/auto-news/llms.txt

This .env file defines various environment variables crucial for configuring the auto-news application. It covers settings for LLM providers, embedding models, API keys for different services (OpenAI, Google, Notion, Twitter, Reddit, YouTube), database connections (Milvus, MySQL), and Redis.

```bash
# .env configuration file

# LLM Provider Selection
LLM_PROVIDER=openai  # openai, google, ollama
OPENAI_MODEL=gpt-4o-mini
OPENAI_API_KEY=sk-xxx
GOOGLE_MODEL=gemini-1.5-flash-latest
GOOGLE_API_KEY=AIza-xxx
OLLAMA_MODEL=llama3
OLLAMA_URL=http://localhost:11434

# Embedding Configuration
EMBEDDING_PROVIDER=openai  # openai, hf, hf_inst, ollama
EMBEDDING_MODEL=text-embedding-ada-002
EMBEDDING_MAX_LENGTH=5000
TEXT_CHUNK_SIZE=10240
TEXT_CHUNK_OVERLAP=256

# Notion Integration
NOTION_TOKEN=secret_xxx
NOTION_ENTRY_PAGE_ID=page_xxx

# Content Sources
CONTENT_SOURCES=Twitter,Reddit,Article,Youtube,RSS

# Twitter API
TWITTER_API_KEY=xxx
TWITTER_API_KEY_SECRET=xxx
TWITTER_ACCESS_TOKEN=xxx
TWITTER_ACCESS_TOKEN_SECRET=xxx
TWITTER_FILTER_MIN_SCORES=AI:4.5,Life:4,Game:5

# Reddit API
REDDIT_CLIENT_ID=xxx
REDDIT_CLIENT_SECRET=xxx
REDDIT_PULLING_COUNT=25
REDDIT_FILTER_MIN_SCORES=Game:4.5,AI:5

# YouTube Configuration
YOUTUBE_TRANSCRIPT_LANGS=en,es,zh

# Database Configuration
MILVUS_HOST=milvus-standalone
MILVUS_PORT=19530
MILVUS_SIMILARITY_METRICS=L2  # L2, IP, COSINE

MYSql_HOST=mysql-db
MYSql_PORT=3306
MYSql_USER=bot
MYSql_PASSWORD=bot
MYSql_DATABASE=bot

BOT_REDIS_URL=redis://:@redis:6379/1
BOT_REDIS_KEY_EXPIRE_TIME=604800  # 1 week

# Processing Options
TRANSLATION_LANG=  # Leave empty or set to target language
SUMMARY_MAX_LENGTH=20000
RSS_ENABLE_CLASSIFICATION=false
REDDIT_ENABLE_CLASSIFICATION=false
```

--------------------------------

### Configure Helm Values for Custom Docker Image

Source: https://github.com/finaldie/auto-news/wiki/Installation-using-Helm

After building and pushing a custom Docker image, update the `repository` and `tag` fields within the `helm/values.yaml` file to point to your newly created image. This ensures Kubernetes deploys your custom image.

```yaml
airflow:
  images:
    airflow:
      repository: finaldie/auto-news
      tag: 0.9.2
```

--------------------------------

### Python Complete RSS Processing Pipeline

Source: https://context7.com/finaldie/auto-news/llms.txt

This Python script demonstrates a complete RSS processing pipeline. It configures environment variables for Notion, OpenAI, and Milvus, then executes the pull, dedup, score, filter, summarize, rank, and push steps for processing RSS articles and publishing them to Notion.

```python
import os
from ops_rss import OperatorRSS

# Configure environment
os.environ["NOTION_TOKEN"] = "secret_xxx"
os.environ["OPENAI_API_KEY"] = "sk-xxx"
os.environ["RSS_ENABLE_CLASSIFICATION"] = "false"  # Disable LLM ranking for speed
os.environ["MILVUS_HOST"] = "milvus-standalone"
os.environ["MILVUS_PORT"] = "19530"

operator = OperatorRSS()

# Full pipeline
articles = operator.pull()
print(f"Pulled {len(articles)} RSS articles")

deduped = operator.dedup(articles, target="inbox")
print(f"After dedup: {len(deduped)} articles")

scored = operator.score(deduped, max_distance=0.45)
print(f"Scored {len(scored)} articles")

filtered = operator.filter(scored, k=5, min_score=4.0)
print(f"Filtered to top {len(filtered)} articles")

summarized = operator.summarize(filtered)
ranked = operator.rank(summarized)  # Uses default -0.02 if classification disabled

stats = operator.push(ranked, targets=["notion"], topk=3)
print(f"Published {stats['total']} articles")
```

--------------------------------

### Redis Cache Management with DBClient

Source: https://context7.com/finaldie/auto-news/llms.txt

Initializes a database client, defaulting to Redis, for managing cache operations. This client can be used for various database interactions, including caching.

```python
from db_cli import DBClient

# Initialize database client (defaults to Redis)
client = DBClient()
```

--------------------------------

### Create Kubernetes Secret

Source: https://github.com/finaldie/auto-news/wiki/Installation-using-Helm

Creates a Kubernetes Secret resource, typically named 'airflow-secrets', within the 'auto-news' namespace. This secret will store sensitive configuration data for the application.

```bash
make k8s-secret-create

```

```bash
# Verification command:
kubectl get secret -n auto-news | grep airflow-secrets
```

--------------------------------

### Video Content Summarization with LLMAgent

Source: https://context7.com/finaldie/auto-news/llms.txt

Summarizes video transcripts using an LLMAgent with a map_reduce strategy. It prioritizes the '__transcript' field, truncates content, and generates a numbered list of key points, caching results in Redis. Expects a list of video objects as input.

```python
# Summarize video transcripts
summarized_videos = operator.summarize(list(videos.values()))

# Summary process:
# 1. Uses "__transcript" field if available
# 2. Truncates to SUMMARY_MAX_LENGTH (default: 20000 chars)
# 3. Applies LLMAgentSummary with map_reduce strategy
# 4. Generates numbered list of key points
# 5. Caches summary in Redis

# Example:
# {
#   "title": "ML Tutorial",
#   "__transcript": "...",
#   "__summary": "Key takeaways:\n1. Machine learning basics\n2. Neural network architecture\n3. Training techniques"
# }
```

--------------------------------

### Translation Agent Configuration and Usage (Python)

Source: https://context7.com/finaldie/auto-news/llms.txt

Demonstrates how to initialize and use an LLM agent for text translation. This involves specifying the target language during prompt initialization and configuring the LLM. The agent then translates the provided English text into the target language.

```python
from llm_agent import LLMAgentTranslation

agent = LLMAgentTranslation()
agent.init_prompt(trans_lang="Chinese")  # Target language
agent.init_llm(provider="openai", model_name="gpt-4o-mini")

# Translate text
english_text = """
This article discusses the latest advancements in artificial intelligence
and their impact on software development.
"""

translation = agent.run(english_text)
print(translation)
# Output: "这篇文章讨论了人工智能的最新进展及其对软件开发的影响。"
```

--------------------------------

### Category and Ranking Agent Configuration and Usage (Python)

Source: https://context7.com/finaldie/auto-news/llms.txt

Sets up and utilizes an LLM agent for analyzing content to extract topics and an overall score. This involves initializing the agent, setting up its LLM parameters, and then running the analysis on provided text. The output is a JSON string containing topics and the score.

```python
from llm_agent import LLMAgentCategoryAndRanking
import json

agent = LLMAgentCategoryAndRanking()
agent.init_prompt()  # Uses default ranking prompt
agent.init_llm(provider="openai", model_name="gpt-4o-mini", temperature=0)

# Analyze content
content = """
Article about new developments in transformer models for natural language processing.
Discusses attention mechanisms and their applications in ChatGPT and similar systems.
"""

response = agent.run(content)

# Parse JSON response
ranking_data = json.loads(response)

print(f"Topics: {ranking_data['topics']}")
# [{"topic": "Transformers", "category": "AI", "score": 0.9}, ...]

print(f"Overall score: {ranking_data['overall_score']}")
# 4.5
```

--------------------------------

### Python Article Processing Pipeline

Source: https://context7.com/finaldie/auto-news/llms.txt

This Python code outlines a general pipeline for processing articles. It includes steps for pulling articles, deduplication, summarization using an LLM, ranking, and finally pushing the processed content to Notion. Error handling is included.

```python
from ops_news import OperatorArticle

operator = OperatorArticle()

try:
    # Step 1: Pull articles
    pages = operator.pull()
    print(f"Pulled {len(pages)} articles")

    # Step 2: Deduplicate
    deduped = operator.dedup(pages, target="inbox")
    print(f"After dedup: {len(deduped)} new articles")

    # Step 3: Summarize with LLM
    summarized = operator.summarize(deduped)
    print(f"Summarized {len(summarized)} articles")

    # Step 4: Rank and categorize
    ranked = operator.rank(summarized)
    print(f"Ranked {len(ranked)} articles")

    # Step 5: Publish to Notion
    stats = operator.push(ranked, targets=["notion"], topk=3)
    print(f"Published: {stats['total']} items, Errors: {stats['error']}")

except Exception as e:
    print(f"Pipeline error: {e}")
```

--------------------------------

### Generate AI Summaries for Articles in Python

Source: https://context7.com/finaldie/auto-news/llms.txt

Generates AI-powered summaries for articles, checking a Redis cache first for existing summaries. If not cached, it extracts content, falls back to web scraping if needed, truncates content, uses `LLMAgentSummary` with a map-reduce strategy, and caches the result in Redis with a TTL. For Arxiv papers, it includes metadata extraction.

```python
# Generate AI summaries for articles
# Returns: list of pages with added "__summary" field
summarized_pages = operator.summarize(deduplicated_pages)

# Summary process:
# 1. Checks Redis cache for existing summary
# 2. If not cached, extracts page content from Notion
# 3. Falls back to web scraping if content is empty
# 4. Truncates to SUMMARY_MAX_LENGTH (default: 20000 chars)
# 5. Calls LLMAgentSummary with map_reduce strategy
# 6. Caches result in Redis with TTL (default: 1 week)
# 7. For Arxiv papers, includes metadata extraction

# Example output:
# {
#   "name": "Article Title",
#   "source_url": "https://example.com/article",
#   "__summary": "Concise numbered list summary:\n1. Key point one...\n2. Key point two..."
# }
```

--------------------------------

### Run Shell Commands, Retry Function, Protected Run (Python)

Source: https://context7.com/finaldie/auto-news/llms.txt

Provides utilities for executing shell commands, with error checking for success and capturing output. Includes a robust retry mechanism with exponential backoff for unstable functions and a protected run function that catches and logs exceptions without crashing the program.

```Python
from utils import run_shell_command, retry, prun

# Run shell command
success, output = run_shell_command("ls -la")
if success:
    print(output)
else:
    print(f"Error: {output}")

# Retry function with exponential backoff
def unstable_function(**kwargs):
    # Function that might fail intermittently
    pass

result = retry(unstable_function, retries=3, param1="value")

# Protected run (catches exceptions)
def risky_operation(**kwargs):
    # Might raise exception
    pass

prun(risky_operation, param1="value")  # Won't crash, prints traceback
```

--------------------------------

### Publish Processed Content to Notion in Python

Source: https://context7.com/finaldie/auto-news/llms.txt

Publishes processed content to the Notion ToRead database, returning statistics on published items and errors. It retrieves target database IDs, selects top topics/categories, creates Notion pages with metadata and summaries, tags content, assigns AI ratings, marks items as visited in Redis, and updates creation times for incremental sync.

```python
# Publish processed content to Notion ToRead database
# Returns: dict {"total": int, "error": int}
stats = operator.push(ranked_pages, targets=["notion"], topk=3)

# Publishing process:
# 1. Gets target database ID from Notion index
# 2. Selects top-3 topics and categories by score
# 3. Creates Notion page with:
#    - Title, URL, source metadata
#    - Summary in page body
#    - Topics and categories as multi-select tags
#    - AI rating as number property
# 4. Marks item as visited in Redis
# 5. Updates last created time for incremental sync

print(f"Published {stats['total']} items, {stats['error']} errors")
```

--------------------------------

### Airflow DAG for Daily Journal Processing (Python)

Source: https://context7.com/finaldie/auto-news/llms.txt

This Airflow DAG, journal_daily.py, sets up a daily workflow for processing journal data. It consists of two main tasks: pulling data from specified sources and then saving the journal entries. Both tasks are executed using BashOperators.

```python
# dags/journal_daily.py
with DAG(
    'journal_daily',
    schedule_interval="30 3 */1 * *",  # 03:30 daily
    max_active_runs=1,
) as dag:

    pull_journal = BashOperator(
        task_id='pull',
        bash_command='python3 af_journal_pull.py --sources=Journal'
    )

    save_journal = BashOperator(
        task_id='save',
        bash_command='python3 af_journal_save.py --sources=Journal'
    )

    pull_journal >> save_journal
```

--------------------------------

### Load Web Content, Unshorten URL, Load Video Transcript (Python)

Source: https://context7.com/finaldie/auto-news/llms.txt

Loads web article content, resolves shortened URLs, and extracts video transcripts. Supports fallback to audio transcription and caching for video transcripts. Requires helper functions from a 'utils' module.

```Python
from utils import load_web, urlUnshorten, load_video_transcript

# Load web article content
url = "https://example.com/article"
content = load_web(url)
print(f"Extracted {len(content)} characters")

# Unshorten URL
short_url = "https://bit.ly/xxx"
full_url = urlUnshorten(short_url)
print(f"Full URL: {full_url}")

# Load video transcript with fallback to audio transcription
transcript, metadata = load_video_transcript(
    url="https://youtube.com/watch?v=xxx",
    audio_url="https://youtube.com/watch?v=xxx",
    page_id="unique_id",
    data_folder="data/videos",
    run_id="run_123",
    audio2text=True,        # Enable Whisper transcription fallback
    enable_cache=True       # Use Redis cache
)

print(f"Transcript: {transcript[:200]}...")
print(f"Metadata: {metadata}")
# Metadata includes: title, author, length, publish_date
```

--------------------------------

### Airflow DAG for News Pulling (Python)

Source: https://context7.com/finaldie/auto-news/llms.txt

Configuration for an Airflow DAG named 'news_pulling' that automates the process of fetching news articles. It defines tasks for initialization, pulling data from sources, saving to Notion, and finalization, with specified schedules and retry logic.

```Python
# dags/news_pulling.py
from airflow import DAG
from airflow.operators.bash import BashOperator
from datetime import timedelta

default_args = {
    'owner': 'airflow',
    'depends_on_past': False,
    'retries': 1,
    'retry_delay': timedelta(minutes=5),
}

with DAG(
    'news_pulling',
    default_args=default_args,
    schedule_interval="15 * * * *",  # Every hour at minute 15
    max_active_runs=1,
    tags=['NewsBot'],
) as dag:

    # Initialize run
    start = BashOperator(
        task_id='start',
        bash_command='cd ~/airflow/run/auto-news/src && python3 af_start.py --start {{ ds }}'
    )

    # Pull from all sources
    pull = BashOperator(
        task_id='pull',
        bash_command='cd ~/airflow/run/auto-news/src && python3 af_pull.py ' 
                     '--start {{ ds }} --run-id={{ run_id }}'
    )

    # Save to Notion
    save = BashOperator(
        task_id='save',
        bash_command='cd ~/airflow/run/auto-news/src && python3 af_save.py ' 
                     '--start {{ ds }} --run-id={{ run_id }} ' 
                     '--targets=notion --dedup=True'
    )

    # Finish
    finish = BashOperator(
        task_id='finish',
        bash_command='cd ~/airflow/run/auto-news/src && python3 af_end.py'
    )

    start >> pull >> save >> finish
```

--------------------------------

### Rank Articles and Categorize using LLM in Python

Source: https://context7.com/finaldie/auto-news/llms.txt

Ranks articles using LLM-based category and topic extraction, checking Redis cache first. It sends the summary to `LLMAgentCategoryAndRanking`, parses the JSON response for topics, categories, and scores, assigns an `overall_score` as `__rate`, and caches the results in Redis using a specific key format.

```python
# Rank articles using LLM-based category and topic extraction
# Returns: list with "__topics", "__categories", "__rate", "__feedback"
ranked_pages = operator.rank(summarized_pages)

# Ranking process:
# 1. Checks Redis cache for existing ranking
# 2. Sends summary to LLMAgentCategoryAndRanking
# 3. Parses JSON response with topics, categories, and scores
# 4. Assigns overall_score as __rate
# 5. Caches in Redis with key: notion_ranking_item_id_{source}_{category}_{id}

# Example output:
# {
#   "name": "Article Title",
#   "__summary": "...",
#   "__topics": [
#     {"topic": "Machine Learning", "category": "Technology", "score": 0.9},
#     {"topic": "Neural Networks", "category": "AI", "score": 0.85}
#   ],
#   "__rate": 4.5,
#   "__feedback": "Highly relevant technical article"
# }
```

--------------------------------

### Python RSS Feed Discovery and Pulling

Source: https://context7.com/finaldie/auto-news/llms.txt

This Python code uses the OperatorRSS class to pull articles from configured RSS feeds. It queries Notion for feed configurations, fetches recent articles using feedparser, generates unique IDs, and normalizes metadata. The output is a dictionary of articles.

```python
from ops_rss import OperatorRSS

# Initialize RSS operator
operator = OperatorRSS()

# Pull RSS articles from configured feeds
# Returns: dict {page_id: page_data}
articles = operator.pull()
```

--------------------------------

### Reddit Post Processing Pipeline with OperatorReddit

Source: https://context7.com/finaldie/auto-news/llms.txt

Executes a full pipeline for processing Reddit posts, including pulling, deduplicating, summarizing, ranking, and filtering by minimum scores before publishing to Notion. Requires Reddit API credentials and classification to be enabled via environment variables.

```python
import os
from ops_reddit import OperatorReddit

os.environ["REDDIT_CLIENT_ID"] = "client_id"
os.environ["REDDIT_CLIENT_SECRET"] = "secret"
os.environ["REDDIT_FILTER_MIN_SCORES"] = "MachineLearning:4.5,Technology:4.0"
os.environ["REDDIT_ENABLE_CLASSIFICATION"] = "true"

operator = OperatorReddit()

# Full pipeline
posts = operator.pull(
    pulling_count=25,
    pulling_interval=3,
    data_folder="data/reddit",
    run_id="run_123"
)

deduped = operator.dedup(posts, target="toread")
summarized = operator.summarize(deduped)
ranked = operator.rank(summarized)

# Filter by per-list minimum scores
filtered = operator.score(ranked, min_scores={"MachineLearning": 4.5})

stats = operator.push(filtered, targets=["notion"], topk=3)
print(f"Published {stats['total']} Reddit posts")
```

--------------------------------

### Parse ISO Datetime, Convert UTC to Pacific Time (Python)

Source: https://context7.com/finaldie/auto-news/llms.txt

Includes functions for parsing datetime strings in ISO format and converting UTC timestamps to Pacific Daylight Time (PDT) strings. Requires importing datetime from the standard library and specific utilities.

```Python
from utils import parseDataFromIsoFormat, convertUTC2PDT_str
from datetime import datetime

# Parse ISO format datetime
iso_string = "2025-01-15T10:30:00.000Z"
dt = parseDataFromIsoFormat(iso_string)
print(dt)  # datetime object

# Convert UTC to Pacific time
utc_time = "2025-01-15T18:30:00.000Z"
pdt_time = convertUTC2PDT_str(utc_time)
print(pdt_time)  # datetime in Pacific timezone
```

--------------------------------

### Python YouTube Video Transcript Extraction

Source: https://context7.com/finaldie/auto-news/llms.txt

This Python code snippet initializes the OperatorYoutube class to pull YouTube videos from a Notion inbox. It extracts video transcripts using the youtube-transcript-api, falling back to audio transcription with Whisper if necessary. Transcripts are cached in Redis, and the output includes the '__transcript' field within the video data.

```python
from ops_youtube import OperatorYoutube

# Initialize YouTube operator
operator = OperatorYoutube()

# Pull YouTube videos from Notion inbox
# Returns: dict {page_id: page_data with "__transcript"}
videos = operator.pull(data_folder="data/youtube", run_id="run_123")
```

--------------------------------

### Subreddit Post Pulling with OperatorReddit

Source: https://context7.com/finaldie/auto-news/llms.txt

Pulls posts from configured subreddits using the OperatorReddit. It queries Notion for subreddit configurations, uses the RedditAgent (PRAW wrapper) to fetch hot/top posts, and returns extracted data grouped by subreddit name. Parameters control the number of posts and interval between requests.

```python
from ops_reddit import OperatorReddit

operator = OperatorReddit()

# Pull posts from configured subreddits
posts = operator.pull(
    pulling_count=25,      # Posts per subreddit
    pulling_interval=3,    # Seconds between requests
    data_folder="data/reddit",
    run_id="run_789"
)

# Pull process:
# 1. Queries Notion for subreddit list configurations
# 2. Uses RedditAgent (PRAW wrapper) with credentials
# 3. Pulls hot/top posts from each subreddit
# 4. Extracts title, selftext, url, score, comments
# 5. Returns grouped by list_name

# Example output:
# {
#   "MachineLearning": [
#     {
#       "post_id": "abc123",
#       "title": "New paper on transformers",
#       "selftext": "Discussion about...",
#       "url": "https://reddit.com/r/MachineLearning/...",
#       "score": 342,
#       "num_comments": 67,
#       "list_name": "MachineLearning"
#     }
#   ]
# }
```

--------------------------------

### Twitter Content Processing and Publishing

Source: https://context7.com/finaldie/auto-news/llms.txt

Processes pulled Twitter data by deduplicating, summarizing, ranking, and filtering tweets based on minimum score thresholds before publishing to Notion. Requires the OperatorTwitter instance.

```python
# Deduplicate tweets
deduped_tweets = operator.dedup(tweets, target="toread")

# Summarize tweets (extracts thread context if available)
summarized_tweets = operator.summarize(deduped_tweets)

# Rank tweets by relevance
ranked_tweets = operator.rank(summarized_tweets)

# Filter by minimum score threshold
filtered_tweets = operator.score(
    ranked_tweets,
    min_scores={"AI_Experts": 4.5, "Tech_News": 4.0}  # Per-list thresholds
)

# Publish to Notion
stats = operator.push(filtered_tweets, targets=["notion"], topk=5)
```

--------------------------------

### Cache ToRead Item Publication Status (Python)

Source: https://context7.com/finaldie/auto-news/llms.txt

Uses DBClient to mark an item as published to 'ToRead' and check its publication status. This prevents duplicate publishing. It caches the status associated with a source, category, and item ID.

```python
# Mark item as published to ToRead
client.set_notion_toread_item_id("Article", "default", "page_123")

# Check if already published
exists = client.get_notion_toread_item_id("Article", "default", "page_123")
if exists:
    print("Already published, skipping...")
```

--------------------------------

### Cache Embedding Vectors with Milvus (Python)

Source: https://context7.com/finaldie/auto-news/llms.txt

Leverages DBClient to cache and retrieve embedding vectors in Milvus. This is crucial for efficient semantic search, storing vectors associated with a provider, model, source, and item ID. The embedding is stored as a JSON serialized byte string.

```python
from db_cli import DBClient

client = DBClient()

# Cache embedding vector
embedding_vector = [0.123, 0.456, 0.789, ...]  # 1536-dim for OpenAI

client.set_milvus_embedding_item_id(
    provider="openai",
    model_name="text-embedding-ada-002",
    source="Article",
    item_id="page_123",
    embed=embedding_vector
)

# Retrieve cached embedding
cached_embed = client.get_milvus_embedding_item_id(
    provider="openai",
    model_name="text-embedding-ada-002",
    source="Article",
    item_id="page_123"
)

# Returns bytes (JSON serialized) or None
if cached_embed:
    import json
    vector = json.loads(cached_embed.decode('utf-8'))
    print(f"Retrieved {len(vector)}-dimensional vector")
```

--------------------------------

### Airflow DAG for Weekly Collection (Python)

Source: https://context7.com/finaldie/auto-news/llms.txt

This Airflow DAG, collect_weekly.py, defines a workflow for weekly data collection. It includes a conditional task to only run on Saturdays or if forced, followed by tasks for collecting and publishing data. It relies on BashOperators to execute Python scripts.

```python
from airflow import DAG
from airflow.operators.python import BranchPythonOperator

def should_run(**kwargs):
    """Only run on Saturday"""
    next_exec_date = kwargs['next_execution_date']
    force = kwargs["dag_run"].conf.setdefault("force_to_run", False)

    if force or next_exec_date.weekday() == 5:  # Saturday
        return "start"
    return "finish"

with DAG(
    'collection_weekly',
    schedule_interval="30 2 */1 * *",  # 02:30 daily
    max_active_runs=1,
) as dag:

    condition = BranchPythonOperator(
        task_id='condition',
        python_callable=should_run
    )

    collect = BashOperator(
        task_id='collect',
        bash_command='python3 af_collect.py --collection-type=weekly --min-rating=4'
    )

    publish = BashOperator(
        task_id='publish',
        bash_command='python3 af_publish.py --min-rating=4.5'
    )

    condition >> [collect, finish]
    collect >> publish
```

--------------------------------

### Python RSS Semantic Similarity Scoring

Source: https://context7.com/finaldie/auto-news/llms.txt

This Python code snippet scores RSS articles based on user reading history using semantic similarity. It utilizes OperatorMilvus to generate embeddings from article titles, list names, and summaries, comparing them against user history in Milvus. The output includes a '__relevant_score' for each article.

```python
# Score RSS articles based on user's reading history
# Returns: list with "__relevant_score" field
scored_articles = operator.score(
    list(articles.values()),
    max_distance=0.45  # Similarity threshold (lower = more similar)
)
```

--------------------------------

### Cache Page Metadata (Python)

Source: https://context7.com/finaldie/auto-news/llms.txt

Uses DBClient to cache and retrieve arbitrary page metadata as JSON. This allows for flexible storage of page-specific information, such as user ratings or categories, identified by an item ID.

```python
import json
from db_cli import DBClient

client = DBClient()

# Cache page metadata
page_data = {
    "user_rating": 4.5,
    "last_edited_time": "2025-01-15T10:30:00.000Z",
    "categories": ["AI", "Technology"]
}

client.set_page_item_id(
    item_id="page_123",
    json_data=json.dumps(page_data)
)

# Retrieve page data
cached_data = client.get_page_item_id("page_123")
if cached_data:
    page_info = json.loads(cached_data.decode('utf-8'))
    print(f"User rating: {page_info['user_rating']}")
```

--------------------------------

### Deduplicate Articles using Redis Cache in Python

Source: https://context7.com/finaldie/auto-news/llms.txt

Deduplicates articles by checking against a Redis cache to identify and filter out already processed items. This function ensures that only new content is passed on for further processing, preventing redundant operations. It utilizes Redis keys in the format `notion_toread_item_id_{source}_{category}_{id}`.

```python
# Deduplicate articles against Redis cache
# Returns: list of new, unseen pages
deduplicated_pages = operator.dedup(pages, target="inbox")

# The dedup process:
# 1. Checks each page_id against Redis key: notion_toread_item_id_{source}_{category}_{id}
# 2. Filters out already-processed items
# 3. Returns only new content to avoid redundant processing
```

--------------------------------

### Cache LLM Summary Response (Python)

Source: https://context7.com/finaldie/auto-news/llms.txt

Uses DBClient to cache and retrieve the summarized content of an item. This prevents re-computation of summaries for the same item. The summary is stored as a JSON string. It caches the summary associated with a source, category, and item ID.

```python
client.set_notion_summary_item_id(
    source="Article",
    category="default",
    item_id="page_123",
    s='{"summary": "AI-generated summary..."}'
)

cached_summary = client.get_notion_summary_item_id("Article", "default", "page_123")
# Returns bytes or None if not found/expired
```