### Dreamer Initialization Output Source: https://context7.com/eesiz/clawdreamer/llms.txt Example output from the setup script, showing the creation of directories and the LanceDB 'memories' table. ```text # Dreamer home: /home/user/.dreamer # Created: /home/user/.dreamer/episodes # Created: /home/user/.dreamer/episodes/archive # Created: /home/user/.dreamer/lancedb # Created: /home/user/.dreamer/dream-log # Created: /home/user/.dreamer/memory-archive # Created: /home/user/.dreamer/workspace # Created: /home/user/.dreamer/workspace/docs # Created LanceDB 'memories' table (1536-dim vectors) ``` -------------------------------- ### Initialize Dreamer and Create Example Episode Source: https://context7.com/eesiz/clawdreamer/llms.txt Initialize the Dreamer data directory and create an example episode file for testing purposes. ```bash python setup.py --example ``` -------------------------------- ### Setup and Initialization: setup.py Source: https://context7.com/eesiz/clawdreamer/llms.txt Initializes the Dreamer data directory structure and creates the LanceDB memories table. Optionally generates an example episode file. ```APIDOC ## Setup and Initialization: setup.py ### Description Initializes the Dreamer data directory structure and creates the LanceDB memories table with the required schema for vector storage. Optionally generates an example episode file for testing. ### Usage ```bash # Initialize with default path (~/.dreamer) python setup.py # Custom data directory python setup.py --home /path/to/data # Initialize and create example episode python setup.py --example ``` ### Output ``` # Dreamer home: /home/user/.dreamer # Created: /home/user/.dreamer/episodes # Created: /home/user/.dreamer/episodes/archive # Created: /home/user/.dreamer/lancedb # Created: /home/user/.dreamer/dream-log # Created: /home/user/.dreamer/memory-archive # Created: /home/user/.dreamer/workspace # Created: /home/user/.dreamer/workspace/docs # Created LanceDB 'memories' table (1536-dim vectors) ``` ``` -------------------------------- ### Systemd Timer Setup for Dreamer Source: https://context7.com/eesiz/clawdreamer/llms.txt Example systemd configuration for running Dreamer as a nightly cron job at 3 AM. This includes service and timer unit files, along with commands to enable and start the timer. ```bash # /etc/systemd/system/dreamer.service [Unit] Description=Dreamer memory consolidation After=network.target [Service] Type=oneshot User=youruser Environment="DREAMER_HOME=/home/youruser/.dreamer" Environment="OPENAI_API_KEY=sk-..." WorkingDirectory=/path/to/dreamer ExecStart=/usr/bin/python3 dreamer.py --verbose StandardOutput=append:/var/log/dreamer.log StandardError=append:/var/log/dreamer.log # /etc/systemd/system/dreamer.timer [Unit] Description=Run Dreamer nightly at 3 AM [Timer] OnCalendar=*-*-* 03:00:00 Persistent=true [Install] WantedBy=timers.target # Enable and start sudo systemctl enable --now dreamer.timer sudo systemctl status dreamer.timer ``` -------------------------------- ### Install Dependencies and Run Dreamer Source: https://github.com/eesiz/clawdreamer/blob/master/README.md Install project dependencies, set up environment variables, initialize the data directory and LanceDB table, and then run the main Dreamer script. Ensure your OpenAI API key is configured in the .env file. ```bash # 1. Install dependencies pip install -r requirements.txt # 2. Set up environment cp .env.example .env # Edit .env with your OpenAI API key # 3. Initialize data directory + LanceDB table python setup.py --example # 4. Run python dreamer.py --verbose ``` -------------------------------- ### Enable and Start session-flush Systemd Timer Source: https://github.com/eesiz/clawdreamer/blob/master/README.md Enables and starts the systemd timer for the `session-flush` script, which automatically generates episode files at 2 AM daily to prevent data loss. ```bash sudo systemctl enable --now session-flush.timer ``` -------------------------------- ### Standalone Dreamer Usage Example Source: https://github.com/eesiz/clawdreamer/blob/master/README.md Demonstrates how to use Dreamer standalone by specifying episode files and pointing to a LanceDB store via the DREAMER_HOME environment variable. Run `python setup.py` to initialize the LanceDB table. ```bash $DREAMER_HOME/episodes/2024-03-15.md $DREAMER_HOME/episodes/2024-03-16.md ``` -------------------------------- ### Dreamer Configuration Parameters Source: https://context7.com/eesiz/clawdreamer/llms.txt Configure Dreamer using environment variables. Defaults are defined in config.py. This example shows settings for data directory, embedding providers, LLM providers, and tunable parameters. ```bash # .env file example # Data directory DREAMER_HOME=~/.dreamer # Embedding provider: openai (default), ollama, sentence-transformers DREAMER_EMBEDDING_PROVIDER=openai OPENAI_API_KEY=sk-... DREAMER_EMBEDDING_DIM=1536 # Ollama settings (if using ollama) OLLAMA_BASE_URL=http://localhost:11434 OLLAMA_EMBEDDING_MODEL=nomic-embed-text # LLM provider: openai (default), ollama, minimax DREAMER_LLM_PROVIDER=openai OLLAMA_LLM_MODEL=qwen2.5:3b # Tunable parameters (in config.py) # CLUSTER_SIMILARITY=0.75 # Threshold for grouping chunks # DEDUP_SIMILARITY=0.90 # Skip if existing memory is this similar # CONTRADICTION_SIMILARITY=0.70 # Conflict detection threshold # IMPORTANCE_DECAY_RATE=0.05 # Daily decay rate # SOFT_DELETE_THRESHOLD=0.15 # Below this = memory deleted # MAX_EPISODES_PER_RUN=7 # Max days processed per cycle # MAX_NEW_MEMORIES=20 # Cap on new memories per cycle ``` -------------------------------- ### Example Episode File Content Source: https://github.com/eesiz/clawdreamer/blob/master/README.md Markdown content for an episode file, including session notes, user interactions, and API details. This format is used for daily AI agent experiences. ```markdown # Session Notes - 2024-03-15 ## User asked about deployment Discussed Docker setup. User prefers docker-compose over raw Docker commands. Decided to use nginx as reverse proxy. ## API Integration Connected to the payment API. Key endpoint: POST /v1/charges Rate limit: 100 req/min. Auth via Bearer token. ``` -------------------------------- ### Episode File Format Example Source: https://context7.com/eesiz/clawdreamer/llms.txt Episodes are markdown files representing daily AI agent experiences. Dreamer processes files named YYYY-MM-DD.md or YYYY-MM-DD-slug.md from the episodes directory. ```markdown # Session Notes - 2024-03-15 ## Project Setup Discussed deployment strategy. Decided on Docker Compose with nginx reverse proxy. Database: PostgreSQL 16 with pgvector extension for embeddings. ## API Integration Connected to the payment gateway API. - Endpoint: POST /v1/charges - Rate limit: 100 req/min - Auth: Bearer token in header - Webhook URL configured for payment confirmations. ## User Preferences User prefers dark mode UI. Font size: 14px. Keyboard shortcut for save: Ctrl+S (not Cmd+S). ``` -------------------------------- ### Schedule Daily Cron Job Source: https://github.com/eesiz/clawdreamer/blob/master/README.md Example cron job entry to run the dreamer script daily at 3 AM. Ensure the path and log redirection are correctly configured for your environment. ```bash # Example: run daily at 3 AM 0 3 * * * cd /path/to/dreamer && python3 dreamer.py --verbose >> dream-log/cron.log 2>&1 ``` -------------------------------- ### Dreamer Cycle JSON Output Source: https://context7.com/eesiz/clawdreamer/llms.txt Example JSON output structure detailing the results of the NREM, REM, and Dream Log phases, including memory statistics and file paths. ```json { "nrem": { "episodes": 3, "chunks": 45, "clusters": 12, "created": 8, "skipped_dup": 3, "docs_created": 1, "processed_dates": ["2024-03-15", "2024-03-16", "2024-03-17"], "created_ids": ["uuid-1", "uuid-2", "..."] }, "rem": { "total_memories": 156, "conflicts_found": 4, "merged": 2, "consolidated": 1, "deleted": 3, "decayed": 45, "soft_deleted": 2, "archived": 3 }, "dream_log": "/home/user/.dreamer/dream-log/2024-03-18_0300.md", "elapsed_seconds": 45.2 } ``` -------------------------------- ### OpenClaw Configuration for Dreamer Integration Source: https://github.com/eesiz/clawdreamer/blob/master/README.md Configure OpenClaw's plugins to enable the memory-lancedb integration and the session-memory hook. This setup ensures that semantic memories are stored in LanceDB and conversation data is saved to episode files. ```json { "plugins": { "slots": { "memory": "memory-lancedb" }, "entries": { "memory-lancedb": { "enabled": true, "config": { "embedding": { "apiKey": "${OPENAI_API_KEY}", "model": "text-embedding-3-small" }, "autoCapture": true, "autoRecall": true } } } }, "hooks": { "internal": { "enabled": true, "entries": { "session-memory": { "enabled": true } } } } } ``` -------------------------------- ### Initialize Dreamer with Default Path Source: https://context7.com/eesiz/clawdreamer/llms.txt Initialize the Dreamer data directory structure and create the LanceDB memories table using the default home path (~/.dreamer). ```bash python setup.py ``` -------------------------------- ### Main Entry Point: dreamer.py Source: https://context7.com/eesiz/clawdreamer/llms.txt Orchestrates the complete dream cycle (NREM, REM, Dream Log). Supports running individual phases and provides verbose logging. ```APIDOC ## Main Entry Point: dreamer.py ### Description Runs the complete dream cycle: NREM (episode processing), REM (memory integration), and Dream Log (report generation). Supports running individual phases and provides verbose logging for debugging. ### Usage ```bash # Full dream cycle (NREM -> REM -> Dream Log) python dreamer.py --verbose # NREM phase only (episode -> semantic conversion) python dreamer.py --nrem-only # REM phase only (decay + pruning existing memories) python dreamer.py --rem-only ``` ### Expected JSON Output ```json { "nrem": { "episodes": 3, "chunks": 45, "clusters": 12, "created": 8, "skipped_dup": 3, "docs_created": 1, "processed_dates": ["2024-03-15", "2024-03-16", "2024-03-17"], "created_ids": ["uuid-1", "uuid-2", "..."] }, "rem": { "total_memories": 156, "conflicts_found": 4, "merged": 2, "consolidated": 1, "deleted": 3, "decayed": 45, "soft_deleted": 2, "archived": 3 }, "dream_log": "/home/user/.dreamer/dream-log/2024-03-18_0300.md", "elapsed_seconds": 45.2 } ``` ``` -------------------------------- ### Initialize Dreamer with Custom Path Source: https://context7.com/eesiz/clawdreamer/llms.txt Initialize the Dreamer data directory structure and create the LanceDB memories table using a specified custom home path. ```bash python setup.py --home /path/to/data ``` -------------------------------- ### Run Full Dreamer Cycle Source: https://context7.com/eesiz/clawdreamer/llms.txt Execute the complete dream cycle including NREM, REM, and Dream Log phases. Use the --verbose flag for detailed logging. ```bash python dreamer.py --verbose ``` -------------------------------- ### Dreamer Project File Structure Source: https://github.com/eesiz/clawdreamer/blob/master/DESIGN.md This outlines the directory and file structure for the Dreamer project. It includes configuration, main entry point, modules for NREM and REM phases, embedding generation, LanceDB operations, LLM interactions, and dream log generation. ```plaintext dreamer/ ├── DESIGN.md # this document ├── config.py # configuration ├── dreamer.py # main entry point ├── nrem.py # Phase 1: episode -> semantic conversion ├── rem.py # Phase 2: integration + pruning ├── embedder.py # OpenAI embedding generation ├── lancedb_store.py # LanceDB read/write operations ├── llm.py # LLM calls (summarization/classification) ├── dream_log.py # dream log generation └── requirements.txt ``` -------------------------------- ### NREM Phase: run_nrem() Source: https://context7.com/eesiz/clawdreamer/llms.txt Processes episode markdown files by chunking content, clustering similar chunks, analyzing clusters with LLM, and storing new semantic memories in LanceDB. ```APIDOC ## NREM Phase: run_nrem() ### Description Processes episode markdown files by chunking content, clustering similar chunks via embedding similarity, analyzing clusters with LLM to extract key facts, and storing new semantic memories in LanceDB after deduplication. ### Usage ```python from nrem import run_nrem # Run NREM phase to process episode files result = run_nrem() ``` ### Result Structure ```json { "episodes": 3, # Number of episode files processed "chunks": 45, # Text chunks extracted "clusters": 12, # Semantic clusters formed "created": 8, # New memories stored "skipped_dup": 3, # Duplicates skipped "docs_created": 1, # Reference docs generated "processed_dates": ["2024-03-15", "2024-03-16"], "created_ids": ["abc-123", "def-456"] # IDs for REM phase } ``` ``` -------------------------------- ### Configure Telegram Alerts for Dreamer Source: https://github.com/eesiz/clawdreamer/blob/master/README.md Sets environment variables to enable and configure Telegram alerts for Dreamer. This bypasses the AI agent and sends error notifications directly to the operator. ```bash export DREAMER_ALERT_PROVIDER=telegram export DREAMER_ALERT_TELEGRAM_BOT_TOKEN=123456:ABC-DEF export DREAMER_ALERT_TELEGRAM_CHAT_ID=your_chat_id ``` -------------------------------- ### Dream Log Generation Steps Source: https://github.com/eesiz/clawdreamer/blob/master/DESIGN.md This describes the process for generating a summary report of the Dreamer's consolidation activities. The log includes details on new memories created, merged, consolidated, deleted, and any conflicts resolved. ```plaintext 15. Generate summary report -> dream-log/YYYY-MM-DD_HHMM.md - New semantic memories created - Memories merged/consolidated - Memories soft-deleted - Conflicts found and resolved ``` -------------------------------- ### Add, Update, and Delete Memories Source: https://context7.com/eesiz/clawdreamer/llms.txt Manage memories by adding new ones with text, vectors, and metadata, updating their importance or content, or deleting them entirely. ```python mem_id = add_memory( text="User prefers Docker Compose over raw Docker commands.", vector=[0.1, 0.2, ...], # 1536-dim embedding importance=0.75, category="preference" ) ``` ```python update_importance(mem_id, new_importance=0.85) ``` ```python update_memory_text( mem_id, new_text="User prefers Docker Compose. Uses nginx as reverse proxy.", new_vector=[0.15, 0.22, ...] ) ``` ```python delete_memory(mem_id) ``` -------------------------------- ### Load All Memories from LanceDB Source: https://context7.com/eesiz/clawdreamer/llms.txt Load all existing memories from the LanceDB vector store using the `load_all_memories` function. ```python from lancedb_store import ( load_all_memories, add_memory, update_importance, update_memory_text, delete_memory, search_similar ) # Load all memories from database memories = load_all_memories() ``` -------------------------------- ### NREM Phase: Episode to Semantic Memory Conversion Steps Source: https://github.com/eesiz/clawdreamer/blob/master/DESIGN.md This outlines the process for converting raw episodic data into structured semantic memories. It involves chunking, embedding generation, clustering, LLM summarization, deduplication against existing memories, and storing new semantic memories with dynamic importance scoring. ```plaintext 1. Load episodes (episodes/*.md) 2. Chunk into semantic units 3. Generate embeddings 4. Cluster similar chunks (cosine similarity > 0.75) 5. LLM summarization per cluster -> extract patterns/principles 6. Dedup check against existing LanceDB memories (similarity > 0.9) 7. Store new semantic memories (dynamic importance scoring) ``` -------------------------------- ### Send Error Alerts with send_alert() Source: https://context7.com/eesiz/clawdreamer/llms.txt Use this function to send failure notifications to operators via Telegram, Slack, or a generic webhook when errors occur. Configuration is done via environment variables. ```python from alerts import send_alert # Automatic alerting on errors (called by dreamer.py) try: # ... dreamer operations except Exception as e: send_alert(e) # Returns True if delivered ``` ```bash # Configuration via environment variables: # DREAMER_ALERT_PROVIDER=telegram # DREAMER_ALERT_TELEGRAM_BOT_TOKEN=123456:ABC-DEF # DREAMER_ALERT_TELEGRAM_CHAT_ID=your_chat_id # Or for Slack: # DREAMER_ALERT_PROVIDER=slack # DREAMER_ALERT_SLACK_WEBHOOK_URL=https://hooks.slack.com/... # Or generic webhook (POST JSON): # DREAMER_ALERT_PROVIDER=webhook # DREAMER_ALERT_WEBHOOK_URL=https://your-endpoint.com/alert # Payload: {"source": "dreamer", "text": "...", "error": "..."} ``` -------------------------------- ### Run NREM Phase in Python Source: https://context7.com/eesiz/clawdreamer/llms.txt Execute the NREM phase using the `run_nrem` function. This processes episode files, chunks content, clusters similar chunks, analyzes clusters with an LLM, and stores new semantic memories in LanceDB after deduplication. ```python from nrem import run_nrem # Run NREM phase to process episode files result = run_nrem() # Result structure: # { # "episodes": 3, # Number of episode files processed # "chunks": 45, # Text chunks extracted # "clusters": 12, # Semantic clusters formed # "created": 8, # New memories stored # "skipped_dup": 3, # Duplicates skipped # "docs_created": 1, # Reference docs generated # "processed_dates": ["2024-03-15", "2024-03-16"], # "created_ids": ["abc-123", "def-456"] # IDs for REM phase # } ``` -------------------------------- ### REM Phase: run_rem() Source: https://context7.com/eesiz/clawdreamer/llms.txt Integrates new memories with existing ones by detecting conflicts, classifying relationships, merging or consolidating, applying importance decay, and archiving processed episodes. ```APIDOC ## REM Phase: run_rem() ### Description Integrates new memories with existing ones by detecting conflicts, classifying relationships (state_change/different_aspects/unrelated), merging or consolidating as appropriate, applying importance decay to old memories, and archiving processed episodes. ### Usage ```python from rem import run_rem nrem_result = { "created_ids": ["abc-123", "def-456"], "processed_dates": ["2024-03-15"] } rem_result = run_rem(nrem_result) ``` ### Result Structure ```json { "total_memories": 156, "conflicts_found": 4, "merged": 2, # State changes merged "consolidated": 1, # Different aspects combined "deleted": 3, # Memories removed "split_created": 1, # New memories from splits "decayed": 45, # Importance decremented "soft_deleted": 2, # Below threshold, removed "archived": 1, # Episode files archived "merge_details": [...], # Detailed merge info "consolidation_details": [...] } ``` ``` -------------------------------- ### Search for Similar Memories Source: https://context7.com/eesiz/clawdreamer/llms.txt Find memories similar to a given vector embedding, specifying the number of results and a similarity threshold. ```python matches = search_similar( vector=[0.1, 0.2, ...], top_k=5, threshold=0.3 ) ``` -------------------------------- ### Generate Text Embeddings Source: https://context7.com/eesiz/clawdreamer/llms.txt Create vector embeddings for text using batch processing for efficiency or single embedding for individual texts. Supports configurable providers like OpenAI, Ollama, and sentence-transformers. ```python from embedder import embed_texts, embed_single, cosine_similarity texts = [ "User prefers dark mode UI", "API rate limit is 100 req/min", "Database uses PostgreSQL 16" ] vectors = embed_texts(texts) ``` ```python vector = embed_single("Deploy with Docker Compose") ``` ```python sim = cosine_similarity(vectors[0], vectors[1]) ``` -------------------------------- ### LanceDB Store Operations Source: https://context7.com/eesiz/clawdreamer/llms.txt Handles all vector database operations including loading, adding, updating, searching, and deleting memories with their embeddings. ```APIDOC ## LanceDB Store Operations ### Description The LanceDB store module handles all vector database operations including loading, adding, updating, searching, and deleting memories with their embeddings. ### Usage ```python from lancedb_store import ( load_all_memories, add_memory, update_importance, update_memory_text, delete_memory, search_similar ) # Load all memories from database memories = load_all_memories() ``` ``` -------------------------------- ### LLM Functions for Memory Analysis Source: https://context7.com/eesiz/clawdreamer/llms.txt Utilize LLM-powered functions to analyze clusters of text, classify relationships between memories, merge state changes, and consolidate related information into comprehensive memories. ```python from llm import ( analyze_cluster, classify_relationship, merge_state_change, consolidate_aspects, llm_call ) chunks = [ "Discussed deployment strategy. Decided on Docker Compose.", "Will use nginx as reverse proxy for the deployment." ] existing_files = ["docs/api-guide.md", "skills/deploy/SKILL.md"] result = analyze_cluster(chunks, existing_files) ``` ```python classification = classify_relationship( "Model set to gemini-pro for chat completions", "Changed model to claude-3 for better reasoning" ) ``` ```python merged = merge_state_change( newer_text="Model is claude-3 for chat completions", older_text="Model set to gemini-pro for chat completions" ) ``` ```python consolidated = consolidate_aspects( "API endpoint: POST /v1/charges", "API rate limit: 100 req/min, auth via Bearer token" ) ``` ```python response = llm_call( prompt="Summarize this in one sentence: ...", system="You are a helpful assistant.", max_tokens=100 ) ``` -------------------------------- ### Generate Dream Log Report Source: https://context7.com/eesiz/clawdreamer/llms.txt Create a detailed markdown report of each dream cycle, documenting memory creation, merging, consolidation, and deletion activities for auditing purposes. ```python from dream_log import write_dream_log nrem_result = { "episodes": 3, "chunks": 45, "clusters": 12, "created": 8, "skipped_dup": 3, "docs_created": 1, "processed_dates": ["2024-03-15", "2024-03-16"] } rem_result = { "total_memories": 156, "conflicts_found": 4, "merged": 2, "consolidated": 1, "deleted": 3, "decayed": 45, "soft_deleted": 2, "archived": 2, "merge_details": [ {"before": ["old1", "old2"], "after": "merged", "kept_id": "abc"} ], "consolidation_details": [] } log_path = write_dream_log(nrem_result, rem_result) ``` -------------------------------- ### REM Phase: Semantic Memory Integration and Pruning Steps Source: https://github.com/eesiz/clawdreamer/blob/master/DESIGN.md This details the integration and pruning of semantic memories. It includes detecting conflicts between new and existing memories, classifying conflicts, resolving them through merging or consolidation, applying importance decay, soft-deleting low-importance memories, and archiving processed episode files. ```plaintext 8. Load all semantic memories 9. Detect conflicts between NEW and EXISTING (O(N*M), not O(N^2)) 10. Classify: state_change / different_aspects / unrelated 11. Resolve: merge (state changes) or consolidate (different aspects) 12. Apply importance decay (unrecalled memories fade) 13. Soft-delete memories below threshold (0.15) 14. Archive processed episode files ``` -------------------------------- ### Run NREM Phase Only Source: https://context7.com/eesiz/clawdreamer/llms.txt Execute only the NREM phase of the dream cycle, which focuses on episode processing and semantic conversion. ```bash python dreamer.py --nrem-only ``` -------------------------------- ### Run REM Phase in Python Source: https://context7.com/eesiz/clawdreamer/llms.txt Execute the REM phase using the `run_rem` function, passing the results from the NREM phase. This integrates new memories with existing ones, detects conflicts, classifies relationships, merges or consolidates memories, applies importance decay, and archives processed episodes. ```python from rem import run_rem # Run REM phase with NREM results nrem_result = { "created_ids": ["abc-123", "def-456"], "processed_dates": ["2024-03-15"] } rem_result = run_rem(nrem_result) # Result structure: # { # "total_memories": 156, # "conflicts_found": 4, # "merged": 2, # State changes merged # "consolidated": 1, # Different aspects combined # "deleted": 3, # Memories removed # "split_created": 1, # New memories from splits # "decayed": 45, # Importance decremented # "soft_deleted": 2, # Below threshold, removed # "archived": 1, # Episode files archived # "merge_details": [...], # Detailed merge info # "consolidation_details": [...] # } ``` -------------------------------- ### Run REM Phase Only Source: https://context7.com/eesiz/clawdreamer/llms.txt Execute only the REM phase of the dream cycle, which handles memory decay and pruning of existing memories. ```bash python dreamer.py --rem-only ``` === COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.