### Docker Compose for PostgreSQL and pgAdmin4 Setup

Source: https://context7.com/opplieam/facebook_page_crawler/llms.txt

Starts PostgreSQL and pgAdmin4 containers using Docker Compose. Provides access points for database management.

```bash
# Start PostgreSQL and pgAdmin4 containers
docker-compose up -d

# Access points:
# - PostgreSQL: localhost:5432
# - pgAdmin4 web interface: localhost:5555

# Create a database named 'fb_page' using pgAdmin4 or psql
# Then create the required tables:
python create_sql_table.py \
  -a user="postgres" \
  -a password="password" \
  -a host="localhost" \
  -a port="5432" \
  -a database="fb_page"

# Expected output:
# {'user': 'postgres', 'dbname': 'fb_page', ...}
# You are connected to - ('PostgreSQL 12.x ...',)
# Added posts, image_urls and comments table
# PostgreSQL connection is closed
```

--------------------------------

### Start Docker Services

Source: https://context7.com/opplieam/facebook_page_crawler/llms.txt

Commands to start the Docker services defined in the docker-compose.yml file. Use default credentials or specify custom ones via environment variables.

```bash
# Start services with default credentials
docker-compose up -d

# Or with custom credentials via environment variables
POSTGRES_USER=myuser POSTGRES_PASSWORD=mysecret docker-compose up -d
```

--------------------------------

### Create and Activate Conda Environment

Source: https://context7.com/opplieam/facebook_page_crawler/llms.txt

Instructions for creating a Conda environment from the 'environment.yml' file and activating it. This installs Scrapy, psycopg2, and demjson.

```bash
# Create the conda environment from the provided file
conda env create -f environment.yml

# Activate the environment (default name: fb-crawler)
source activate fb-crawler

# Or if using newer conda versions
conda activate fb-crawler
```

--------------------------------

### Run FBP Crawler to CSV

Source: https://github.com/opplieam/facebook_page_crawler/blob/master/README.md

Example command to run the FBP crawler spider and store the extracted data in a CSV file. Requires username, password, and page ID.

```bash
scrapy crawl fb_page \
-a username="your_email@email.com" \
-a password="your_password" \
-a page_id="page_id_you_want_scrape" \
-o output.csv
```

--------------------------------

### Run FBP Crawler to JSON

Source: https://github.com/opplieam/facebook_page_crawler/blob/master/README.md

Example command to run the FBP crawler spider and store the extracted data in a JSON file. Requires username, password, and page ID.

```bash
scrapy crawl fb_page \
-a username="your_email@email.com" \
-a password="your_password" \
-a page_id="page_id_you_want_scrape" \
-o output.json
```

--------------------------------

### Verify Scrapy Installation

Source: https://context7.com/opplieam/facebook_page_crawler/llms.txt

Verify that Scrapy is installed correctly within the activated Conda environment by checking its version and listing available spiders.

```bash
python -c "import scrapy; print(f'Scrapy version: {scrapy.__version__}')"
# Output: Scrapy version: 1.7.3

# Navigate to spider directory and verify
cd facebook_crawler
scrapy list
# Output: fb_page
```

--------------------------------

### Run FBP Crawler with Database Output

Source: https://github.com/opplieam/facebook_page_crawler/blob/master/README.md

Command to run the FBP crawler spider and store the extracted data directly into the configured database. Requires username, password, page ID, and the database argument.

```bash
scrapy crawl fb_page \
-a username="your_email@email.com" \
-a password="yourpassword" \
-a page_id="page_id_you_want_to_scrape" \
-a database=true
```

--------------------------------

### Create SQL Table for FBP Crawler

Source: https://github.com/opplieam/facebook_page_crawler/blob/master/README.md

Python script to create necessary SQL tables for storing FBP crawler data. Requires database connection parameters.

```python
python create_sql_table.py \
-a user="postgres" \
-a password="password" \
-a host="localhost" \
-a port="5432" \
-a database="fb_page"
```

--------------------------------

### Run FBP Crawler with JSON Output

Source: https://context7.com/opplieam/facebook_page_crawler/llms.txt

Execute the FBP crawler spider with authentication credentials and a target page ID, outputting results to a JSON file. Ensure you are in the project directory.

```bash
cd facebook_crawler

scrapy crawl fb_page \
  -a username="your_email@email.com" \
  -a password="your_password" \
  -a page_id="ejeab" \
  -o output.json
```

--------------------------------

### Configure PostgreSQL Database Settings in settings.py

Source: https://context7.com/opplieam/facebook_page_crawler/llms.txt

Specifies the database connection parameters within the Scrapy project's settings.py file. Enables the FacebookPostgresPipeline for data storage.

```python
# In facebook_crawler/settings.py, uncomment and configure:
DB_SETTINGS = {
    'db': 'fb_page',
    'user': 'postgres',
    'password': 'password',
    'host': 'localhost',
    'port': '5432'
}

# The pipeline is already enabled by default:
ITEM_PIPELINES = {
   'facebook_crawler.pipelines.FacebookPostgresPipeline': 300,
}
```

--------------------------------

### Crawl Facebook Pages with Database and File Output

Source: https://context7.com/opplieam/facebook_page_crawler/llms.txt

This command crawls Facebook pages, stores data in the database, and also outputs the data to a JSON file. Useful for both persistent storage and immediate review.

```bash
scrapy crawl fb_page \
  -a username="your_email@email.com" \
  -a password="your_password" \
  -a page_id="ejeab" \
  -a database=true \
  -o backup_output.json
```

--------------------------------

### Configure Database Settings in settings.py

Source: https://github.com/opplieam/facebook_page_crawler/blob/master/README.md

Configuration snippet for database settings within the Scrapy project's settings.py file. Specifies connection details for the database.

```python
DB_SETTINGS = {
    'db': 'fb_page',
    'user': 'postgres',
    'password': 'password',
    'host': 'localhost',
    'port': '5432'
}
```

--------------------------------

### Docker Compose for PostgreSQL and pgAdmin4

Source: https://context7.com/opplieam/facebook_page_crawler/llms.txt

This Docker Compose configuration sets up a PostgreSQL database and pgAdmin4 for managing it. Credentials can be configured via environment variables.

```yaml
version: '3.5'
services:
  postgres-db:
    container_name: postgres
    image: postgres
    volumes:
      - ~/.postgres-data:/var/lib/postgresql/data
    environment:
      POSTGRES_USER: ${POSTGRES_USER:-postgres}
      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:-password}
    ports:
      - 5432:5432
    networks:
      - pgnetwork
  pgadmin4:
    container_name: pgadmin4
    image: dpage/pgadmin4
    ports:
      - 5555:80
    environment:
      PGADMIN_DEFAULT_EMAIL: ${PGADMIN_DEFAULT_EMAIL:-pgadmin4}
      PGADMIN_DEFAULT_PASSWORD: ${PGADMIN_DEFAULT_PASSWORD:-admin}
    networks:
      - pgnetwork
networks:
  pgnetwork:
    driver: bridge
```

--------------------------------

### Run FBP Crawler for Multiple Pages

Source: https://context7.com/opplieam/facebook_page_crawler/llms.txt

Scrape data from multiple Facebook Pages in a single crawl session by providing a comma-separated list of page IDs. The output will be saved to a JSON file.

```bash
scrapy crawl fb_page \
  -a username="your_email@email.com" \
  -a password="your_password" \
  -a page_id="ejeab,Viciousant,AnotherPage" \
  -o multi_page_output.json
```

--------------------------------

### Run FBP Crawler with CSV Output

Source: https://context7.com/opplieam/facebook_page_crawler/llms.txt

Crawl a Facebook Page using the FBP crawler and export the scraped data in CSV format. This command requires the same authentication and page ID parameters.

```bash
scrapy crawl fb_page \
  -a username="your_email@email.com" \
  -a password="your_password" \
  -a page_id="Viciousant" \
  -o output.csv
```

--------------------------------

### Crawl Facebook Pages with Database Storage

Source: https://context7.com/opplieam/facebook_page_crawler/llms.txt

Use this command to crawl Facebook pages and store the data directly into a PostgreSQL database. Ensure the database is configured and accessible.

```bash
scrapy crawl fb_page \
  -a username="your_email@email.com" \
  -a password="your_password" \
  -a page_id="ejeab,Viciousant" \
  -a database=true
```

--------------------------------

### Scrapy Settings Configuration

Source: https://context7.com/opplieam/facebook_page_crawler/llms.txt

Key settings in 'settings.py' for the Facebook crawler. This includes setting a browser-like user agent, disabling robots.txt, configuring download delays, and enabling the PostgreSQL pipeline.

```python
# facebook_crawler/settings.py

BOT_NAME = 'facebook_crawler'
SPIDER_MODULES = ['facebook_crawler.spiders']
NEWSPIDER_MODULE = 'facebook_crawler.spiders'

# Browser-like user agent to avoid blocking
USER_AGENT = 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36'

# Disable robots.txt compliance (required for Facebook)
ROBOTSTXT_OBEY = False

# Rate limiting: 2.5 second delay between requests
DOWNLOAD_DELAY = 2.5

# Enable the PostgreSQL pipeline
ITEM_PIPELINES = {
   'facebook_crawler.pipelines.FacebookPostgresPipeline': 300,
}

# Optional: Enable HTTP caching for development
# HTTPCACHE_ENABLED = True
# HTTPCACHE_STORAGE = 'scrapy.extensions.httpcache.FilesystemCacheStorage'
```

--------------------------------

### Run Spider with Database Storage Enabled

Source: https://context7.com/opplieam/facebook_page_crawler/llms.txt

Executes the Scrapy spider with database storage enabled, automatically storing scraped data in PostgreSQL. Supports upsert semantics for updating existing records.

```bash
# Crawl and store results in PostgreSQL
scrapy crawl fb_page \
  -a username="your_email@email.com" \
  -a password="your_password" \
  -a page_id="ejeab" \
  -a database=true
```

--------------------------------

### Scrape Multiple Facebook Pages

Source: https://github.com/opplieam/facebook_page_crawler/blob/master/README.md

Command to scrape multiple Facebook pages by providing a comma-separated list of page IDs. Requires username, password, and page IDs.

```bash
scrapy crawl fb_page \
-a username="your_email@email.com" \
-a password="yourpassword" \
-a page_id="ejeab,Viciousant" \
-o output.csv
```

--------------------------------

### FbBaseSpider Authentication and Page Request Logic

Source: https://context7.com/opplieam/facebook_page_crawler/llms.txt

Handles Facebook login using Scrapy's FormRequest and initiates requests for specified Facebook pages. Requires username, password, and page_id as arguments.

```python
from scrapy import Spider
from scrapy.exceptions import CloseSpider
from scrapy.http import FormRequest, Request

class FbBaseSpider(Spider):
    allowed_domains = ['facebook.com']
    start_urls = ['https://www.facebook.com/']

    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)

        # Required: username and password for authentication
        self.username = kwargs.get('username')
        self.password = kwargs.get('password')
        if not self.username or not self.password:
            raise CloseSpider('Please provide username or password')

        # Required: page_id (single or comma-separated list)
        self.page_id = kwargs.get('page_id')
        if not self.page_id:
            raise CloseSpider('Please provide page_id')

        # Support multiple page IDs
        if ',' in self.page_id:
            self.page_id = self.page_id.split(',')
        else:
            self.page_id = [self.page_id]

    def parse(self, response):
        # Submit login form
        yield FormRequest.from_response(
            response=response,
            formid='login_form',
            formdata={
                'email': self.username,
                'pass': self.password
            },
            callback=self.parse_login
        )

    def parse_login(self, response):
        # Request each page after successful login
        for page_id in self.page_id:
            url = f'https://www.facebook.com/{page_id}/posts/'
            yield Request(
                url=url, callback=self.parse_page, meta={'page_id': page_id}
            )
```

--------------------------------

### SQL Schema for Facebook Data Storage

Source: https://context7.com/opplieam/facebook_page_crawler/llms.txt

Defines the SQL CREATE TABLE statements for storing Facebook post data, including posts, image URLs, and comments. Ensures referential integrity with foreign keys.

```sql
-- Posts table: stores main post data
CREATE TABLE IF NOT EXISTS posts (
    post_id VARCHAR(80) PRIMARY KEY,
    page_id VARCHAR(40) NOT NULL,
    page_name VARCHAR(50) NOT NULL,
    post_url VARCHAR(100) NOT NULL,
    post_text text,
    video_url VARCHAR(700),
    comment_count INT,
    reaction_count INT,
    share_count INT
);

-- Image URLs table: multiple images per post
CREATE TABLE IF NOT EXISTS image_urls (
    image_id SERIAL PRIMARY KEY,
    post_id VARCHAR(80) NOT NULL,
    image_url VARCHAR(500) NOT NULL,
    UNIQUE (post_id, image_url),
    FOREIGN KEY (post_id)
        REFERENCES posts (post_id)
        ON UPDATE CASCADE ON DELETE CASCADE
);

-- Comments table: stores comment data with author info
CREATE TABLE IF NOT EXISTS comments (
    comment_id VARCHAR(100) PRIMARY KEY,
    post_id VARCHAR(80) NOT NULL,
    comment_text text,
    author_name VARCHAR(250) NOT NULL,
    author_id VARCHAR(30) NOT NULL,
    author_url VARCHAR(100) NOT NULL,
    FOREIGN KEY (post_id)
        REFERENCES posts (post_id)
        ON UPDATE CASCADE ON DELETE CASCADE
);
```

--------------------------------

### Define FacebookPostItem Schema

Source: https://context7.com/opplieam/facebook_page_crawler/llms.txt

Defines the data structure for scraped Facebook posts using Scrapy's Item and Field classes. Includes fields for page information, post content, media URLs, and engagement metrics.

```python
from scrapy import Item, Field
from scrapy.loader import ItemLoader
from scrapy.loader.processors import TakeFirst, MapCompose, Join, Identity

class FacebookPostItem(Item):
    page_id = Field()         # Facebook page identifier (e.g., "ejeab")
    page_name = Field()       # Display name of the page
    post_id = Field()         # Unique post identifier
    post_url = Field()        # Full URL to the post
    post_text = Field()       # Text content of the post
    image_urls = Field()      # List of image URLs in the post
    video_url = Field()       # Video URL if present
    comment_count = Field()   # Total number of comments
    reaction_count = Field()  # Total reaction count (likes, etc.)
    share_count = Field()     # Number of shares
    comments = Field()        # List of FacebookCommentItem objects
```

--------------------------------

### Define FacebookCommentItem Schema

Source: https://context7.com/opplieam/facebook_page_crawler/llms.txt

Defines the data structure for scraped Facebook comments using Scrapy's Item and Field classes. Includes fields for comment ID, text, author name, and author ID.

```python
from scrapy import Item, Field
from scrapy.loader import ItemLoader
from scrapy.loader.processors import TakeFirst

class FacebookCommentItem(Item):
    comment_id = Field()      # Unique comment identifier
    comment_text = Field()    # The comment content
    author_name = Field()     # Display name of commenter
    author_id = Field()       # Facebook user ID of commenter
    author_url = Field()      # Profile URL of commenter
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.