### Setup Virtual Environment for Django-Robots

Source: https://github.com/jazzband/django-robots/blob/master/docs/index.md

These commands outline the process of creating and activating a Python virtual environment for developing and testing the django-robots project. It ensures a clean and isolated environment for dependencies.

```bash
virtualenv --python 3.10 .venv
. .venv/bin/activate
# or if you use source
source .venv/bin/activate
```

--------------------------------

### Install Django-Robots and Dependencies

Source: https://github.com/jazzband/django-robots/blob/master/docs/index.md

This snippet shows the commands to install the django-robots project in editable mode and its testing dependencies, including Django. This allows for direct modification and testing of the library.

```bash
pip install -e .
pip install -r tests/requirements.txt
pip install django
```

--------------------------------

### Example Generated robots.txt Output

Source: https://context7.com/jazzband/django-robots/llms.txt

This snippet displays example output formats for a generated robots.txt file. It illustrates directives like User-agent, Allow, Disallow, Crawl-delay, Host, and Sitemap, showing variations for different bots and default configurations.

```text
# Example generated robots.txt output:

User-agent: *
Allow: /
Disallow: /admin
Disallow: /private/
Crawl-delay: 10

User-agent: Googlebot
Allow: /
Allow: /media
Disallow: /admin
Crawl-delay: 5

User-agent: Bingbot
Allow: /
Disallow: /admin
Disallow: /media
Crawl-delay: 20

Host: www.example.com
Sitemap: https://www.example.com/sitemap.xml
Sitemap: https://www.example.com/news-sitemap.xml

# When no rules exist, the default permissive output is:
User-agent: *
Disallow:

Host: example.com
Sitemap: http://example.com/sitemap.xml

```

--------------------------------

### Install django-robots using pip

Source: https://github.com/jazzband/django-robots/blob/master/docs/index.md

This command installs the django-robots package from PyPI. It's the standard method for adding the application to your Python environment.

```bash
pip install django-robots
```

--------------------------------

### Install and Configure Django Robots

Source: https://context7.com/jazzband/django-robots/llms.txt

This snippet shows how to install the django-robots package using pip and configure the necessary settings and URL routing in a Django project. It includes adding the 'robots' app to INSTALLED_APPS and including the robots.urls in the main urls.py.

```python
# Install via pip
# pip install django-robots

# settings.py
INSTALLED_APPS = [
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.sites',  # Required
    'django.contrib.sitemaps',
    'robots',  # Add robots app
]

SITE_ID = 1

# urls.py
from django.urls import path, re_path, include

urlpatterns = [
    path('admin/', admin.site.urls),
    re_path(r'^robots\.txt', include('robots.urls')),  # Serve robots.txt
    path('sitemap.xml', sitemap_view, {'sitemaps': sitemaps}, name='sitemap'),
]

# Run migrations after installation
# python manage.py migrate
```

--------------------------------

### Define URL pattern for cached sitemap in Django

Source: https://github.com/jazzband/django-robots/blob/master/docs/index.md

This example demonstrates how to define a URL pattern for a sitemap view in Django's `urls.py`, specifically when the sitemap view is cached. It assigns a name to the URL pattern, which can then be referenced by django-robots using `ROBOTS_SITEMAP_VIEW_NAME`.

```python
from django.urls import re_path
from django.views.decorators.cache import cache_page
from .views import sitemap_view # Assuming sitemap_view is defined elsewhere

urlpatterns = [
    ...
    re_path(r'^sitemap.xml$', cache_page(60)(sitemap_view), {'sitemaps': [...]}, name='cached-sitemap'),
    ...
]
```

--------------------------------

### Fake Initial South Migration for Django-Robots

Source: https://github.com/jazzband/django-robots/blob/master/docs/history.md

This command fakes the initial database migration for the django-robots app when using South. It ensures that South recognizes the initial migration as applied without actually running it, which is useful when integrating South with existing applications or after initial setup.

```bash
python manage.py migrate --fake robots 0001
```

--------------------------------

### Configure Multi-Site Support for robots.txt

Source: https://context7.com/jazzband/django-robots/llms.txt

This Python code illustrates how to leverage Django's sites framework to serve distinct robots.txt files for different domains. It demonstrates creating multiple `Site` objects and associating `Rule` objects with specific sites to manage crawling permissions on a per-domain basis. Ensure `ROBOTS_SITE_BY_REQUEST = True` is set in your settings.

```python
from django.contrib.sites.models import Site
from robots.models import Rule, Url

# Create multiple sites
main_site = Site.objects.get_or_create(
    domain='www.example.com',
    defaults={'name': 'Main Site'}
)[0]

api_site = Site.objects.get_or_create(
    domain='api.example.com',
    defaults={'name': 'API Site'}
)[0]

blog_site = Site.objects.get_or_create(
    domain='blog.example.com',
    defaults={'name': 'Blog Site'}
)[0]

# Create shared URL patterns
url_root = Url.objects.create(pattern="/")
url_admin = Url.objects.create(pattern="/admin")
url_private = Url.objects.create(pattern="/private")

# Main site: allow most crawling
main_rule = Rule.objects.create(robot="*")
main_rule.allowed.add(url_root)
main_rule.disallowed.add(url_admin, url_private)
main_rule.sites.add(main_site)

# API site: block all crawling
api_rule = Rule.objects.create(robot="*")
api_rule.disallowed.add(url_root)  # Disallow everything
api_rule.sites.add(api_site)

# Blog site: allow everything except admin
blog_rule = Rule.objects.create(robot="*", crawl_delay=0.5)
blog_rule.allowed.add(url_root)
blog_rule.disallowed.add(url_admin)
blog_rule.sites.add(blog_site)

# Enable site detection by request host header
# settings.py: ROBOTS_SITE_BY_REQUEST = True

# Now each domain serves its own robots.txt:
# www.example.com/robots.txt  -> allows most URLs
```

--------------------------------

### Configure Robot Access Rules using Rule Model

Source: https://context7.com/jazzband/django-robots/llms.txt

This snippet illustrates how to use the `Rule` model to define access rules for different user agents. It shows creating rules for general access, specific bots like Googlebot and Bingbot, and associating them with allowed/disallowed URL patterns and sites.

```python
from django.contrib.sites.models import Site
from robots.models import Rule, Url

# Get the current site
site = Site.objects.get_current()

# Create URL patterns
url_root = Url.objects.create(pattern="/")
url_admin = Url.objects.create(pattern="/admin")
url_media = Url.objects.create(pattern="/media")
url_api = Url.objects.create(pattern="/api/internal")

# Create a rule for all robots (wildcard *)
rule_all = Rule.objects.create(
    robot="*",
    crawl_delay=1.0  # 1 second between requests
)
rule_all.allowed.add(url_root)
rule_all.disallowed.add(url_admin, url_media, url_api)
rule_all.sites.add(site)

# Create specific rule for Googlebot (more permissive)
rule_google = Rule.objects.create(
    robot="Googlebot",
    crawl_delay=0.5
)
rule_google.allowed.add(url_root, url_media)  # Allow media for Google
rule_google.disallowed.add(url_admin, url_api)
rule_google.sites.add(site)

# Create rule for Bingbot with longer delay
rule_bing = Rule.objects.create(
    robot="Bingbot",
    crawl_delay=2.0
)
rule_bing.allowed.add(url_root)
rule_bing.disallowed.add(url_admin, url_media, url_api)
rule_bing.sites.add(site)

# Query rules and their URLs
for rule in Rule.objects.filter(sites=site):
    print(f"User-agent: {rule.robot}")
    print(f"  Allowed: {rule.allowed_urls()}")
    print(f"  Disallowed: {rule.disallowed_urls()}")
    if rule.crawl_delay:
        print(f"  Crawl-delay: {rule.crawl_delay}")
```

--------------------------------

### Generate robots.txt Programmatically with RuleList View

Source: https://context7.com/jazzband/django-robots/llms.txt

This Python code demonstrates how to programmatically generate the robots.txt content using Django's RequestFactory and the RuleList view. It sets up test data for sites, URLs, and rules, then simulates a request to generate and print the robots.txt output. It also shows how to test the view using Django's test client.

```python
from django.test import RequestFactory
from django.contrib.sites.models import Site
from robots.views import RuleList
from robots.models import Rule, Url

# Setup test data
site = Site.objects.get(domain="example.com")
url_root = Url.objects.create(pattern="/")
url_admin = Url.objects.create(pattern="/admin")

rule = Rule.objects.create(robot="*", crawl_delay=10)
rule.allowed.add(url_root)
rule.disallowed.add(url_admin)
rule.sites.add(site)

# Generate robots.txt programmatically
factory = RequestFactory()
request = factory.get('/robots.txt')

view = RuleList()
view.request = request
view.current_site = site
view.object_list = view.get_queryset()

context = view.get_context_data(object_list=view.object_list)
response = view.render_to_response(context)
response.render()

print(response.content.decode())

# Test via Django test client
from django.test import Client
client = Client()
response = client.get('/robots.txt')
print(response.status_code)  # 200
print(response['Content-Type'])  # text/plain
print(response.content.decode())
```

--------------------------------

### Run Django-Robots Tests

Source: https://github.com/jazzband/django-robots/blob/master/docs/index.md

This command executes the tests for the django-robots library. It sets the PYTHONPATH environment variable and specifies the Django settings module to run the tests with detailed output.

```bash
env PYTHONPATH=. DJANGO_SETTINGS_MODULE=tests.settings django-admin test robots -v2
```

--------------------------------

### Define URL Patterns for robots.txt using Url Model

Source: https://context7.com/jazzband/django-robots/llms.txt

This snippet demonstrates how to use the `Url` model from `robots.models` to define URL patterns for robots.txt rules. It shows creating patterns for specific paths, using wildcards, and how patterns without a leading slash are automatically handled.

```python
from robots.models import Url

# Create URL patterns for robots.txt
url_root = Url.objects.create(pattern="/")
url_admin = Url.objects.create(pattern="/admin")
url_media = Url.objects.create(pattern="/media")
url_private = Url.objects.create(pattern="/private/")

# Wildcard pattern to block all .pdf files
url_pdfs = Url.objects.create(pattern="/*.pdf$")

# Block all URLs under /api/internal/
url_internal_api = Url.objects.create(pattern="/api/internal")

# Pattern without leading slash gets auto-corrected
url_static = Url.objects.create(pattern="static")  # Stored as "/static"
print(url_static.pattern)  # Output: /static

# Query existing URL patterns
all_urls = Url.objects.all()
for url in all_urls:
    print(f"Pattern: {url.pattern}")
```

--------------------------------

### Configure Django-Robots Host Scheme

Source: https://github.com/jazzband/django-robots/blob/master/docs/index.md

This snippet demonstrates how to include the request protocol (http or https) in the 'Host' directive of the robots.txt file by setting ROBOTS_USE_SCHEME_IN_HOST to True. This ensures the correct scheme is used when search engines access your site.

```python
ROBOTS_USE_SCHEME_IN_HOST = True
```

--------------------------------

### Configure robots.txt URL in Django

Source: https://github.com/jazzband/django-robots/blob/master/docs/index.md

This snippet shows how to integrate the django-robots application into your Django project's URL configuration. It maps the '/robots.txt' URL to the robots application's URL patterns.

```python
from django.urls import re_path, include

urlpatterns = [
    re_path(r'^robots\.txt', include('robots.urls')),
]
```

--------------------------------

### Configure Django Robots Settings

Source: https://context7.com/jazzband/django-robots/llms.txt

This snippet shows how to configure the behavior of django-robots through Django's settings.py file. It covers options for enabling/disabling sitemap inclusion and specifying custom sitemap URLs.

```python
# settings.py

# Sitemap Configuration
# ---------------------
# Automatically include sitemap URL (default: True)
ROBOTS_USE_SITEMAP = True

# Specify custom sitemap URLs instead of auto-discovery
ROBOTS_SITEMAP_URLS = [
    'https://www.example.com/sitemap.xml',
    'https://www.example.com/news-sitemap.xml',
]
```

--------------------------------

### Configure robots.txt Settings in Django

Source: https://context7.com/jazzband/django-robots/llms.txt

These settings in your Django project's settings.py file allow you to customize the behavior of the robots.txt generator. You can specify a named view for sitemaps, control the inclusion of the Host directive, set caching timeouts, and enable site detection based on the request host.

```python
ROBOTS_SITEMAP_VIEW_NAME = 'cached-sitemap'

ROBOTS_USE_HOST = True
ROBOTS_USE_SCHEME_IN_HOST = True

ROBOTS_CACHE_TIMEOUT = 60 * 60 * 24  # 86400 seconds

ROBOTS_SITE_BY_REQUEST = True
```

--------------------------------

### Specify custom sitemap URLs in robots.txt

Source: https://github.com/jazzband/django-robots/blob/master/docs/index.md

This setting allows you to define specific URLs for your sitemaps to be included in the robots.txt file. This is an alternative to the automatically discovered sitemap URL.

```python
ROBOTS_SITEMAP_URLS = [
    'http://www.example.com/sitemap.xml',
]
```

--------------------------------

### Generate robots.txt Rules with Django Template

Source: https://github.com/jazzband/django-robots/blob/master/src/robots/templates/robots/rule_list.html

This Django template code iterates through defined rules to construct the robots.txt file. It handles user-agent directives, URL patterns for allow/disallow, crawl delays, and host information. It also includes logic for cases where no rules are defined and for listing sitemap URLs.

```django
{% load l10n %}{% if rules %}{% for rule in rules %}{% ifchanged rule.robot %}{% if not forloop.first %} {% endif %}User-agent: {{ rule.robot }}{% endifchanged %} {% for url in rule.allowed.all %}Allow: {{ url.pattern|safe }} {% endfor %}{% for url in rule.disallowed.all %}Disallow: {{ url.pattern|safe }} {% endfor %}{% if rule.crawl_delay %}Crawl-delay: {% localize off %}{{ rule.crawl_delay|floatformat:'0' }}{% endlocalize %} {% endif %}{% endfor %}{% else %}User-agent: * Disallow: {% endif %} {% if host %}Host: {{ host }}{% endif %} {% for sitemap_url in sitemap_urls %}Sitemap: {{ sitemap_url }} {% endfor %}
```

--------------------------------

### Configure custom sitemap view name for robots.txt

Source: https://github.com/jazzband/django-robots/blob/master/docs/index.md

This setting is used when your sitemap view is wrapped in a decorator or when you use custom sitemap views. It provides the name of the sitemap view to django-robots for correct URL resolution.

```python
ROBOTS_SITEMAP_VIEW_NAME = 'cached-sitemap'
```

--------------------------------

### Customize Django Admin for Robots Rules

Source: https://context7.com/jazzband/django-robots/llms.txt

This Python code shows how to customize the Django admin interface for managing `Rule` and `Url` models. It demonstrates unregistering the default admin and registering a custom `ModelAdmin` with specific `fieldsets`, `list_filter`, `list_display`, `search_fields`, and `filter_horizontal` configurations for efficient rule management.

```python
from django.contrib import admin
from robots.models import Rule, Url
from robots.forms import RuleAdminForm

# Unregister default and create custom admin
admin.site.unregister(Rule)

@admin.register(Rule)
class CustomRuleAdmin(admin.ModelAdmin):
    form = RuleAdminForm
    fieldsets = (
        (None, {'fields': ('robot', 'sites')}),
        ('URL patterns', {'fields': ('allowed', 'disallowed')}),
        ('Advanced options', {
            'classes': ('collapse',),
            'fields': ('crawl_delay',)
        }),
    )
    list_filter = ('sites', 'robot')
    list_display = ('robot', 'allowed_urls', 'disallowed_urls', 'crawl_delay')
    search_fields = ('robot', 'allowed__pattern', 'disallowed__pattern')
    filter_horizontal = ('sites', 'allowed', 'disallowed')

# Access admin at /admin/robots/rule/ and /admin/robots/url/
# Create rules through the admin interface:
# 1. First create Url patterns: /admin/robots/url/add/
# 2. Then create Rules linking patterns to robots: /admin/robots/rule/add/
```

--------------------------------

### Configure Django-Robots Caching

Source: https://github.com/jazzband/django-robots/blob/master/docs/index.md

This snippet illustrates how to set a cache timeout for the robots.txt file generation in seconds. Setting ROBOTS_CACHE_TIMEOUT to a specific value, like 86400 (24 hours), reduces server load by serving a cached version of the file.

```python
ROBOTS_CACHE_TIMEOUT = 60*60*24
```

--------------------------------

### Configure Cached Sitemap Integration in Django

Source: https://context7.com/jazzband/django-robots/llms.txt

This snippet shows how to configure django-robots to find a cached sitemap by its named URL pattern. It involves defining sitemap URLs in `urls.py` and setting `ROBOTS_SITEMAP_VIEW_NAME` in `settings.py` to point to the named sitemap view.

```python
# urls.py
from django.urls import path, re_path, include
from django.views.decorators.cache import cache_page
from django.contrib.sitemaps import views as sitemap_views

sitemaps = {
    'static': StaticViewSitemap,
    'blog': BlogSitemap,
}

urlpatterns = [
    # Cached sitemap with named URL
    path(
        'sitemap.xml',
        cache_page(86400)(sitemap_views.index),
        {'sitemaps': sitemaps},
        name='cached-sitemap'
    ),
    path(
        'sitemap-<section>.xml',
        cache_page(86400)(sitemap_views.sitemap),
        {'sitemaps': sitemaps},
        name='django.contrib.sitemaps.views.sitemap'
    ),

    # robots.txt
    re_path(r'^robots\.txt', include('robots.urls')),
]

# settings.py
# Tell django-robots to use the named sitemap view
ROBOTS_SITEMAP_VIEW_NAME = 'cached-sitemap'

# The generated robots.txt will include:
# Sitemap: https://www.example.com/sitemap.xml

```

--------------------------------

### Configure Django-Robots Host Directive

Source: https://github.com/jazzband/django-robots/blob/master/docs/index.md

This snippet shows how to disable the automatic inclusion of the 'Host' directive in the robots.txt file by setting ROBOTS_USE_HOST to False in Django's settings. This is useful for controlling how search engines identify your main website.

```python
ROBOTS_USE_HOST = False
```

--------------------------------

### Disable automatic sitemap inclusion in robots.txt

Source: https://github.com/jazzband/django-robots/blob/master/docs/index.md

This setting in your Django settings file disables the automatic inclusion of a Sitemap URL in the generated robots.txt file. This is useful if you manage sitemap inclusion manually or do not use the sitemap feature.

```python
ROBOTS_USE_SITEMAP = False
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.