### Create Scrapy Project Example

Source: https://github.com/oxylabs/scrapy-web-scraping/blob/main/README.md

Example command to create a Scrapy project named 'scrapyproject'.

```bash
scrapy startproject scrapyproject
```

--------------------------------

### Generate Spider Example

Source: https://github.com/oxylabs/scrapy-web-scraping/blob/main/README.md

Example command to generate a spider named 'books' targeting 'books.toscrape.com'.

```bash
scrapy genspider books books.toscrape.com
```

--------------------------------

### Install Scrapy

Source: https://context7.com/oxylabs/scrapy-web-scraping/llms.txt

Install the Scrapy framework using pip. This command enables web scraping capabilities in your Python environment.

```bash
pip install scrapy
```

--------------------------------

### Create and Generate Scrapy Project and Spider

Source: https://context7.com/oxylabs/scrapy-web-scraping/llms.txt

Initialize a new Scrapy project and generate a spider for a specific domain. This sets up the standard directory structure and creates the initial spider file.

```bash
# Create a new Scrapy project
scrapy startproject scrapyproject

# Navigate to the project directory
cd scrapyproject

# Generate a new spider targeting a specific domain
scrapy genspider books books.toscrape.com
```

--------------------------------

### Create Scrapy Project

Source: https://github.com/oxylabs/scrapy-web-scraping/blob/main/README.md

Run this command to create a new Scrapy project. Replace <project_name> with your desired project name.

```bash
scrapy startproject <project_name>
```

--------------------------------

### Run Scrapy Spider and Output Data

Source: https://context7.com/oxylabs/scrapy-web-scraping/llms.txt

Execute spiders from the command line using `scrapy crawl`. Optionally specify an output file with `-o` to save data in formats like CSV or JSON.

```bash
# Run the spider and output to console
scrapy crawl books
```

```bash
# Run the spider and save output to CSV file
scrapy crawl -o out.csv books
```

```bash
# Run the spider and save output to JSON file
scrapy crawl -o out.json books
```

--------------------------------

### Execute Scrapy crawl command

Source: https://github.com/oxylabs/scrapy-web-scraping/blob/main/README.md

Commands to run the spider and optionally export results to a CSV file.

```bash
scrapy crawl books
```

```bash
scrapy crawl -o out.csv books
```

--------------------------------

### Implement BooksSpider class

Source: https://github.com/oxylabs/scrapy-web-scraping/blob/main/README.md

Defines a spider that extracts book titles and prices from product pods and follows next-page links.

```python
 class BooksSpider(scrapy.Spider):
    name = 'books'

    def start_requests(self):
        URL = 'https://books.toscrape.com/'
        yield scrapy.Request(url=URL, callback=self.response_parser)

    def response_parser(self, response):
        for selector in response.css('article.product_pod'):
            yield {
                'title': selector.css('h3 > a::attr(title)').extract_first(),
                'price': selector.css('.price_color::text').extract_first()
            }

        next_page_link = response.css('li.next a::attr(href)').extract_first()
        if next_page_link:
            yield response.follow(next_page_link, callback=self.response_parser)
```

--------------------------------

### Define a Scrapy Spider Class

Source: https://context7.com/oxylabs/scrapy-web-scraping/llms.txt

Create a spider by subclassing `scrapy.Spider`. Implement `start_requests()` to define initial URLs and `response_parser` for parsing responses and handling pagination.

```python
import scrapy

class BooksSpider(scrapy.Spider):
    name = 'books'

    def start_requests(self):
        URL = 'https://books.toscrape.com/'
        yield scrapy.Request(url=URL, callback=self.response_parser)

    def response_parser(self, response):
        for selector in response.css('article.product_pod'):
            yield {
                'title': selector.css('h3 > a::attr(title)').extract_first(),
                'price': selector.css('.price_color::text').extract_first()
            }

        next_page_link = response.css('li.next a::attr(href)').extract_first()
        if next_page_link:
            yield response.follow(next_page_link, callback=self.response_parser)
```

--------------------------------

### Handle Pagination in Scrapy

Source: https://context7.com/oxylabs/scrapy-web-scraping/llms.txt

Implement pagination by finding the next page link using CSS selectors and recursively following it with `response.follow()` until no more pages are found.

```python
def response_parser(self, response):
    # Extract data from current page
    for selector in response.css('article.product_pod'):
        yield {
            'title': selector.css('h3 > a::attr(title)').extract_first(),
            'price': selector.css('.price_color::text').extract_first()
        }

    # Follow pagination link to next page
    next_page_link = response.css('li.next a::attr(href)').extract_first()
    if next_page_link:
        yield response.follow(next_page_link, callback=self.response_parser)
```

--------------------------------

### Generate a New Spider

Source: https://github.com/oxylabs/scrapy-web-scraping/blob/main/README.md

Use this command to generate a new spider for your web scraping target. Replace <spider_name> and <url_domain> accordingly.

```bash
scrapy genspider <spider_name> <url_domain>
```

--------------------------------

### Connect to Scrapy Signals

Source: https://context7.com/oxylabs/scrapy-web-scraping/llms.txt

Register a callback function to the `item_scraped` signal to process each item as it is scraped. This is useful for real-time data handling or aggregation.

```python
from scrapy import signals
from scrapy.signalmanager import dispatcher

def collect_results():
    results = []

    def on_item_scraped(item):
        results.append(item)

    # Register callback for item_scraped signal
    dispatcher.connect(on_item_scraped, signal=signals.item_scraped)

    return results
```

--------------------------------

### Execute Scrapy Spider Programmatically

Source: https://github.com/oxylabs/scrapy-web-scraping/blob/main/README.md

Uses CrawlerProcess to initiate a crawl and a signal dispatcher to collect scraped items into a list for local processing.

```python
import csv
import scrapy
from scrapy import signals
from scrapy.crawler import CrawlerProcess
from scrapy.signalmanager import dispatcher

class BooksSpider(scrapy.Spider):
    name = 'books'

    def start_requests(self):
        URL = 'https://books.toscrape.com/'
        yield scrapy.Request(url=URL, callback=self.response_parser)

    def response_parser(self, response):
        for selector in response.css('article.product_pod'):
            yield {
                'title': selector.css('h3 > a::attr(title)').extract_first(),
                'price': selector.css('.price_color::text').extract_first()
            }

        next_page_link = response.css('li.next a::attr(href)').extract_first()
        if next_page_link:
            yield response.follow(next_page_link, callback=self.response_parser)



def book_spider_result():
    books_results = []

    def crawler_results(item):
        books_results.append(item)

    dispatcher.connect(crawler_results, signal=signals.item_scraped)
    crawler_process = CrawlerProcess()
    crawler_process.crawl(BooksSpider)
    crawler_process.start()
    return books_results


if __name__ == '__main__':
    books_data=book_spider_result()

    keys = books_data[0].keys()
    with open('books_data.csv', 'w', newline='') as output_file_name:
        writer = csv.DictWriter(output_file_name, keys)
        writer.writeheader()
        writer.writerows(books_data)
```

--------------------------------

### Extract Data with CSS Selectors

Source: https://context7.com/oxylabs/scrapy-web-scraping/llms.txt

Use CSS selectors to extract specific data like titles and prices from HTML elements. `extract_first()` retrieves the first matching result.

```python
def response_parser(self, response):
    for selector in response.css('article.product_pod'):
        yield {
            # Extract the 'title' attribute from an <a> tag inside <h3>
            'title': selector.css('h3 > a::attr(title)').extract_first(),
            # Extract text content from element with class 'price_color'
            'price': selector.css('.price_color::text').extract_first()
        }
```

--------------------------------

### Run Scrapy Spider Programmatically

Source: https://context7.com/oxylabs/scrapy-web-scraping/llms.txt

Execute a Scrapy spider from a Python script and collect scraped items using signal dispatchers. This method allows for integration into larger applications or scheduled tasks.

```python
import csv
import scrapy
from scrapy import signals
from scrapy.crawler import CrawlerProcess
from scrapy.signalmanager import dispatcher

class BooksSpider(scrapy.Spider):
    name = 'books'

    def start_requests(self):
        URL = 'https://books.toscrape.com/'
        yield scrapy.Request(url=URL, callback=self.response_parser)

    def response_parser(self, response):
        for selector in response.css('article.product_pod'):
            yield {
                'title': selector.css('h3 > a::attr(title)').extract_first(),
                'price': selector.css('.price_color::text').extract_first()
            }

        next_page_link = response.css('li.next a::attr(href)').extract_first()
        if next_page_link:
            yield response.follow(next_page_link, callback=self.response_parser)


def book_spider_result():
    books_results = []

    def crawler_results(item):
        books_results.append(item)

    # Connect signal handler to collect scraped items
    dispatcher.connect(crawler_results, signal=signals.item_scraped)

    # Create and start crawler process
    crawler_process = CrawlerProcess()
    crawler_process.crawl(BooksSpider)
    crawler_process.start()

    return books_results


if __name__ == '__main__':
    # Run spider and collect results
    books_data = book_spider_result()

    # Export results to CSV file
    keys = books_data[0].keys()
    with open('books_data.csv', 'w', newline='') as output_file_name:
        writer = csv.DictWriter(output_file_name, keys)
        writer.writeheader()
        writer.writerows(books_data)
```

--------------------------------

### Export Data to CSV

Source: https://context7.com/oxylabs/scrapy-web-scraping/llms.txt

Write a list of dictionaries to a CSV file using `csv.DictWriter`. This method automatically handles writing headers based on dictionary keys and then writes each dictionary as a row.

```python
import csv

# Assuming books_data is a list of dictionaries
books_data = [
    {'title': 'A Light in the Attic', 'price': '£51.77'},
    {'title': 'Tipping the Velvet', 'price': '£53.74'},
    {'title': 'Soumission', 'price': '£50.10'}
]

# Write to CSV file
keys = books_data[0].keys()
with open('books_data.csv', 'w', newline='') as output_file:
    writer = csv.DictWriter(output_file, keys)
    writer.writeheader()
    writer.writerows(books_data)

# Output file contents:
# title,price
# A Light in the Attic,£51.77
# Tipping the Velvet,£53.74
# Soumission,£50.10
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.