Beautiful Soup

https://github.com/wention/beautifulsoup4
Admin
Beautiful Soup is a Python library for pulling data out of HTML and XML files, providing idiomatic...
Tokens:29,590
Snippets:176
Trust Score:8.2
License:MIT
Update:4 months ago
Show doc for...
Tokens:
Raw
### Install html5lib parser

Source: https://github.com/wention/beautifulsoup4/blob/master/doc/source/index.rst

Installs the html5lib parser, a pure-Python parser that mimics web browser parsing behavior. Installation can be done via apt-get, easy_install, or pip.

```shell
$ apt-get install python-html5lib
```

```shell
$ easy_install html5lib
```

```shell
$ pip install html5lib
```

--------------------------------

### Install Beautiful Soup 4 from source

Source: https://github.com/wention/beautifulsoup4/blob/master/doc/source/index.rst

Installs Beautiful Soup 4 by downloading the source tarball and running the setup.py script. This method is useful if pip or easy_install are not available.

```shell
python setup.py install
```

--------------------------------

### Install Beautiful Soup 4 via pip

Source: https://github.com/wention/beautifulsoup4/blob/master/doc/source/index.rst

Installs the Beautiful Soup 4 library using pip, the standard package installer for Python. This is the recommended method for installing BS4.

```bash
$ pip install beautifulsoup4
```

--------------------------------

### Install Beautiful Soup 4 using pip or easy_install

Source: https://github.com/wention/beautifulsoup4/blob/master/doc/source/index.rst

Installs the beautifulsoup4 package, which is compatible with both Python 2 and Python 3. This is the recommended method for most users.

```shell
$ pip install beautifulsoup4
```

```shell
$ easy_install beautifulsoup4
```

--------------------------------

### Install Beautiful Soup 3 tarball

Source: https://github.com/wention/beautifulsoup4/blob/master/doc/source/index.rst

Downloads and installs Beautiful Soup 3 from a tarball archive. This is an alternative installation method, though not typically recommended for latest versions.

```bash
Download a tarball of Beautiful Soup 3.2.0
<http://www.crummy.com/software/BeautifulSoup/bs3/download/3.x/BeautifulSoup-3.2.0.tar.gz>
```

--------------------------------

### Install Beautiful Soup 3 via easy_install

Source: https://github.com/wention/beautifulsoup4/blob/master/doc/source/index.rst

Installs Beautiful Soup 3 using the easy_install tool. This method is generally superseded by pip.

```bash
$ easy_install BeautifulSoup
```

--------------------------------

### Install lxml parser

Source: https://github.com/wention/beautifulsoup4/blob/master/doc/source/index.rst

Installs the lxml parser, a fast and lenient HTML parser, using system package managers (apt-get), easy_install, or pip. This parser is a common choice for Beautiful Soup.

```shell
$ apt-get install python-lxml
```

```shell
$ easy_install lxml
```

```shell
$ pip install lxml
```

--------------------------------

### Install Beautiful Soup 3 via apt-get

Source: https://github.com/wention/beautifulsoup4/blob/master/doc/source/index.rst

Installs Beautiful Soup 3 using the apt-get package manager. This method is less common for BS4 and may install an older version.

```bash
$ apt-get install python-beautifulsoup
```

--------------------------------

### Convert Python 2 Beautiful Soup code to Python 3

Source: https://github.com/wention/beautifulsoup4/blob/master/doc/source/index.rst

Provides two methods to convert Beautiful Soup 4 code from Python 2 to Python 3: by installing the package with python3 setup.py install or by manually running the 2to3 conversion script.

```shell
python3 setup.py install
```

```shell
2to3-3.2 -w bs4
```

--------------------------------

### Parsing XML with BeautifulSoup

Source: https://github.com/wention/beautifulsoup4/blob/master/doc/source/index.rst

This snippet demonstrates how to parse an XML document using Beautiful Soup. It requires the 'lxml' library to be installed. The second argument to the BeautifulSoup constructor specifies the parser type.

```python
from bs4 import BeautifulSoup

soup = BeautifulSoup(markup, "xml")
```

--------------------------------

### Parse HTML/XML Documents with Beautiful Soup

Source: https://context7.com/wention/beautifulsoup4/llms.txt

Demonstrates parsing HTML and XML documents into navigable tree structures using the BeautifulSoup constructor. It supports automatic parser selection and shows examples for parsing from strings and files.

```python
from bs4 import BeautifulSoup

# Parse HTML document (uses default HTML parser)
html_doc = """
<html><head><title>The Dormouse's story</title></head>
<body>
<p class="title"><b>The Dormouse's story</b></p>
<p class="story">Once upon a time there were three little sisters; and their names were
<a href="http://example.com/elsie" class="sister" id="link1">Elsie</a>,
<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>
</body></html>
"""
soup = BeautifulSoup(html_doc, 'html.parser')

# Parse from file
with open('document.html') as f:
    soup = BeautifulSoup(f, 'lxml')

# Parse XML with explicit XML parser
xml_doc = '<?xml version="1.0"?><root><item id="1">Content</item></root>'
soup = BeautifulSoup(xml_doc, 'xml')

# Output: <title>The Dormouse's story</title>
print(soup.title)
```

--------------------------------

### Unicode Dammit: Specify Encoding Guesses

Source: https://github.com/wention/beautifulsoup4/blob/master/doc/source/index.rst

Shows how to provide a list of suspected encodings to Unicode, Dammit to improve its guessing accuracy. This is useful when you have some prior knowledge about the data's origin. The example demonstrates providing a list of encodings and accessing the resulting Unicode markup and detected encoding.

```python
from bs4 import UnicodeDammit

dammit = UnicodeDammit("Sacr\xe9 bleu!", ["latin-1", "iso-8859-1"])
print(dammit.unicode_markup)
# Sacré bleu!
print(dammit.original_encoding)
# 'latin-1'
```

--------------------------------

### Parse HTML with SoupStrainer using html.parser

Source: https://github.com/wention/beautifulsoup4/blob/master/doc/source/index.rst

Shows how to parse an HTML document using BeautifulSoup with the 'html.parser' and apply a SoupStrainer to extract specific elements. This example filters the document to include only 'a' tags, tags with id 'link2', or short text strings. The output is then prettified for readability. Dependencies: 'bs4' library.

```python
from bs4 import BeautifulSoup, SoupStrainer

html_doc = """
<html><head><title>The Dormouse's story</title></head>

<p class="title"><b>The Dormouse's story</b></p>

<p class="story">Once upon a time there were three little sisters; and their names were
<a href="http://example.com/elsie" class="sister" id="link1">Elsie</a>,
<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>

<p class="story">...</p>
"""

# Parse only 'a' tags
only_a_tags = SoupStrainer("a")
print(BeautifulSoup(html_doc, "html.parser", parse_only=only_a_tags).prettify())

# Parse tags with id='link2'
only_tags_with_id_link2 = SoupStrainer(id="link2")
print(BeautifulSoup(html_doc, "html.parser", parse_only=only_tags_with_id_link2).prettify())

# Parse short strings
def is_short_string(string):
    return len(string) < 10

only_short_strings = SoupStrainer(text=is_short_string)
print(BeautifulSoup(html_doc, "html.parser", parse_only=only_short_strings).prettify())
```

--------------------------------

### Generator conversion from BS3 to BS4

Source: https://github.com/wention/beautifulsoup4/blob/master/doc/source/index.rst

Illustrates the change in generator method names and their conversion to properties in Beautiful Soup 4, adhering to PEP 8 compliance. For example, `parentGenerator()` becomes `parents`.

```python
# Beautiful Soup 3
for parent in tag.parentGenerator():
    ...

# Beautiful Soup 4
for parent in tag.parents:
    ...
```

--------------------------------

### Beautiful Soup: Find All Previous Elements

Source: https://github.com/wention/beautifulsoup4/blob/master/doc/source/index.rst

Iterates over all tags and strings that appear before the current element in the document. Can find all matches or just the first one. This method searches the entire document preceding the starting element.

```python
first_link = soup.a
first_link.find_all_previous("p")
first_link.find_previous("title")
```

--------------------------------

### XML vs. HTML Parsing Differences

Source: https://github.com/wention/beautifulsoup4/blob/master/doc/source/index.rst

Highlights the distinct parsing behaviors of XML and HTML in Beautiful Soup, using an example of an empty self-closing tag `<b />`. XML parsers preserve the tag as is, while HTML parsers may interpret it as an invalid tag and normalize it.

```python
from bs4 import BeautifulSoup

# Parsing as HTML
soup_html = BeautifulSoup("<a><b /></a>")
# print(soup_html)
# Expected output (may vary slightly based on default parser):
# <html><head></head><body><a><b></b></a></body></html>

# Parsing as XML (requires lxml)
soup_xml = BeautifulSoup("<a><b /></a>", "xml")
# print(soup_xml)
# Expected output:
# <?xml version="1.0" encoding="utf-8"?>
# <a><b/></a>
```

--------------------------------

### Python: Find all tags with any ID attribute

Source: https://github.com/wention/beautifulsoup4/blob/master/doc/source/index.rst

This example finds all tags that have an 'id' attribute, regardless of its value. It uses `id=True` to match any tag where the 'id' attribute is present.

```python
soup.find_all(id=True)
```

--------------------------------

### Python: Find tag by ID attribute

Source: https://github.com/wention/beautifulsoup4/blob/master/doc/source/index.rst

This example shows how to locate a specific tag by its 'id' attribute. It searches for an element with the 'id' attribute equal to 'link2'. This is a common way to target unique elements.

```python
soup.find_all(id="link2")
```

--------------------------------

### Using a Custom Function Formatter with Beautiful Soup 4

Source: https://github.com/wention/beautifulsoup4/blob/master/doc/source/index.rst

You can provide a custom function to the formatter argument. Beautiful Soup will call this function for every string and attribute value, allowing complete control over output formatting. This example converts strings to uppercase.

```python
from bs4 import BeautifulSoup

def uppercase(str_input):
    return str_input.upper()

french = "<p>Il a dit &lt;&lt;Sacr&eacute; bleu!&gt;&gt;</p>"
soup = BeautifulSoup(french)
print(soup.prettify(formatter=uppercase))
```

```python
from bs4 import BeautifulSoup

def uppercase(str_input):
    return str_input.upper()

link_soup = BeautifulSoup('<a href="http://example.com/?foo=val1&bar=val2">A link</a>')
print(link_soup.a.prettify(formatter=uppercase))
```

--------------------------------

### Accessing Tag Contents with BeautifulSoup

Source: https://github.com/wention/beautifulsoup4/blob/master/doc/source/index.rst

This example shows how to access the direct children of an HTML tag using the `.contents` attribute in Beautiful Soup. It retrieves a list of all direct child elements and strings within a given tag.

```python
head_tag = soup.head
head_tag.contents
# [<title>The Dormouse's story</title>]

title_tag = head_tag.contents[0]
title_tag.contents
# [u'The Dormouse's story']
```

--------------------------------

### Extracting Single String Content with BeautifulSoup

Source: https://github.com/wention/beautifulsoup4/blob/master/doc/source/index.rst

This example shows how to extract the text content of a tag using the `.string` attribute in Beautiful Soup. This attribute is available when a tag contains only a single NavigableString, otherwise it returns None.

```python
title_tag.string
# u'The Dormouse's story'

head_tag.string
# u'The Dormouse's story'

print(soup.html.string)
# None
```

--------------------------------

### Parse HTML Document with BeautifulSoup in Python

Source: https://github.com/wention/beautifulsoup4/blob/master/README.md

This snippet demonstrates how to parse an HTML document using the BeautifulSoup library in Python. It takes an HTML string as input and outputs a prettified version of the parsed HTML structure. Ensure you have the 'beautifulsoup4' package installed.

```python
from bs4 import BeautifulSoup

html_doc = """
<html><head><title>The Dormouse's story</title></head>
<body>
<p class="title"><b>The Dormouse's story</b></p>

<p class="story">Once upon a time there were three little sisters; and their names were
<a href="http://example.com/elsie" class="sister" id="link1">Elsie</a>,
<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>

<p class="story">...</p>
"""

soup = BeautifulSoup(html_doc)

print(soup.prettify())
```

--------------------------------

### Modifying HTML Tree - Adding Elements with Beautiful Soup

Source: https://context7.com/wention/beautifulsoup4/llms.txt

Provides examples of how to add new HTML elements and strings to a parsed document tree using Beautiful Soup. This includes creating new tags, appending or inserting them at specific locations, and adding attributes to newly created tags.

```python
from bs4 import BeautifulSoup

html = '<div><p>Existing paragraph</p></div>'
soup = BeautifulSoup(html, 'html.parser')

# Create new tag
new_tag = soup.new_tag('a', href='http://example.com')
new_tag.string = 'Link text'

# Append to existing tag
div = soup.find('div')
div.append(new_tag)
# Output: <div><p>Existing paragraph</p><a href="http://example.com">Link text</a></div>
print(div)

# Create and insert at specific position
new_p = soup.new_tag('p')
new_p.string = 'New paragraph'
div.insert(0, new_p)  # Insert at beginning
# Output: New paragraph comes first
print(div)

# Insert before an element
p = soup.find('p')
header = soup.new_tag('h1')
header.string = 'Title'
p.insert_before(header)

# Insert after an element
footer = soup.new_tag('footer')
footer.string = 'Footer'
p.insert_after(footer)

# Create new string
new_string = soup.new_string(' Additional text')
p.append(new_string)

# Add attributes to tag
new_tag['class'] = 'external'
new_tag['target'] = '_blank'

# Complete example
soup2 = BeautifulSoup('<body></body>', 'html.parser')
body = soup2.body
article = soup2.new_tag('article', id='main')
h1 = soup2.new_tag('h1')
h1.string = 'Article Title'
article.append(h1)
body.append(article)
# Output: <body><article id="main"><h1>Article Title</h1></article></body>
print(soup2)

```

--------------------------------

### Using find() to get a single result in Beautiful Soup

Source: https://github.com/wention/beautifulsoup4/blob/master/doc/source/index.rst

Explains the `find()` method in Beautiful Soup, which is used to find only one matching tag in the document, unlike `find_all()` which returns all matches. If no match is found, `find()` returns `None`, whereas `find_all()` returns an empty list. This method is efficient when only a single result is expected.

```python
soup.find_all('title', limit=1)
# Expected output: [<title>The Dormouse's story</title>]

soup.find('title')
# Expected output: <title>The Dormouse's story</title>

print(soup.find("nosuchtag"))
# Expected output: None

soup.find("head").find("title")
# Expected output: <title>The Dormouse's story</title>
```

--------------------------------

### Diagnose HTML Parsing Issues with diagnose()

Source: https://github.com/wention/beautifulsoup4/blob/master/doc/source/index.rst

Utilizes the `diagnose()` function from `bs4.diagnose` to help troubleshoot problems encountered during HTML parsing. It prints a report detailing how different parsers handle a given document and suggests potential solutions, such as installing missing parsers. Dependencies: 'bs4' library. Input: HTML data as a string.

```python
from bs4.diagnose import diagnose

data = open("bad.html").read()
diagnose(data)
```

--------------------------------

### Basic Tree Navigation with Beautiful Soup

Source: https://context7.com/wention/beautifulsoup4/llms.txt

Illustrates how to navigate the parsed HTML/XML tree using property access for tags, accessing parent and sibling elements, and retrieving tag names, string content, and attributes.

```python
from bs4 import BeautifulSoup

html_doc = '<html><head><title>Page Title</title></head><body><p class="text">Hello <b>World</b></p></body></html>'
soup = BeautifulSoup(html_doc, 'html.parser')

# Access tags by name (returns first matching tag)
title_tag = soup.title
# Output: <title>Page Title</title>
print(title_tag)

# Get tag name
# Output: title
print(soup.title.name)

# Get tag text content
# Output: Page Title
print(soup.title.string)

# Navigate to parent
# Output: head
print(soup.title.parent.name)

# Get tag attributes (dictionary-like access)
p_tag = soup.p
# Output: ['text']
print(p_tag['class'])

# Check if attribute exists
has_class = p_tag.has_attr('class')  # True
has_id = p_tag.has_attr('id')  # False
```

--------------------------------

### HTML Parsing with BeautifulSoup

Source: https://context7.com/wention/beautifulsoup4/llms.txt

Demonstrates how to initialize a BeautifulSoup object with different parsers, prioritizing 'lxml' and falling back to 'html.parser' if necessary. This is crucial for handling potentially malformed HTML content effectively.

```python
from bs4 import BeautifulSoup

html = "<html><body><p>Example</p></body></html>"

try:
    soup_auto = BeautifulSoup(html)  # Uses best available parser (lxml > html5lib > html.parser)
except Exception as e:
    print(f"Error during auto-parsing: {e}")
    soup_auto = BeautifulSoup(html, 'html.parser')  # Fallback to html.parser

print(soup_auto.prettify())
```

--------------------------------

### Configuring HTML and XML Parsers

Source: https://context7.com/wention/beautifulsoup4/llms.txt

Demonstrates how to select and configure different parsers for Beautiful Soup, including Python's built-in 'html.parser', the fast and lenient 'lxml', and the standards-compliant 'html5lib'. It also shows how to specify XML parsers.

```python
from bs4 import BeautifulSoup

html = '<html><body><p>Paragraph</p></body></html>'
broken_html = '<html><body><p>Unclosed paragraph<div>Badly nested</p></div></body></html>'

# Use Python's built-in HTML parser (included with Python)
soup_builtin = BeautifulSoup(html, 'html.parser')
# Output: Uses Python's html.parser

# Use lxml HTML parser (fast, lenient, requires lxml package)
soup_lxml = BeautifulSoup(html, 'lxml')
# Output: Uses lxml's HTML parser (fastest)

# Use lxml XML parser (strict, for well-formed XML)
xml = '<?xml version="1.0"?><root><item>Content</item></root>'
soup_xml = BeautifulSoup(xml, 'xml')
# Or use: soup_xml = BeautifulSoup(xml, 'lxml-xml')
# Output: Preserves XML structure and namespaces

# Use html5lib parser (very lenient, standards-compliant, requires html5lib)
soup_html5 = BeautifulSoup(broken_html, 'html5lib')
# Output: Fixes badly nested tags automatically

# Compare parser behavior
print("html.parser:", BeautifulSoup(broken_html, 'html.parser'))
print("lxml:", BeautifulSoup(broken_html, 'lxml'))
print("html5lib:", BeautifulSoup(broken_html, 'html5lib'))

# Specify parser features explicitly
soup_fast = BeautifulSoup(html, features=['html', 'fast'])

# Check which parser was used
print(f"Parser: {soup_lxml.builder.__class__.__name__}")
```

--------------------------------

### Python: Find all 'title' tags

Source: https://github.com/wention/beautifulsoup4/blob/master/doc/source/index.rst

This example shows the basic usage of `find_all()` to retrieve all tags with a specific name. It searches the parsed HTML document for all occurrences of the '<title>' tag.

```python
soup.find_all("title")
```

--------------------------------

### Instantiate BeautifulSoup with html5lib Parser

Source: https://github.com/wention/beautifulsoup4/blob/master/doc/source/index.rst

Demonstrates how to create a BeautifulSoup object using the html5lib parser. This parser is known for its leniency and ability to parse pages like a web browser, creating valid HTML5. However, it is slower than other parsers and requires an external Python dependency.

```Python
from bs4 import BeautifulSoup

soup = BeautifulSoup(markup, "html5lib")
```

--------------------------------

### Navigating HTML Tree - Parents and Siblings with Beautiful Soup

Source: https://context7.com/wention/beautifulsoup4/llms.txt

Illustrates how to navigate the HTML document tree using Beautiful Soup. This includes accessing parent elements, moving to previous and next siblings, and finding all siblings that match specific criteria. It also shows how to find a parent with specific attributes.

```python
from bs4 import BeautifulSoup

html = """
<div class="container">
    <h1>Title</h1>
    <p id="first">First paragraph</p>
    <p id="second">Second paragraph</p>
    <p id="third">Third paragraph</p>
    <div class="nested">
        <span>Nested content</span>
    </div>
</div>
"""

soup = BeautifulSoup(html, 'html.parser')

# Access parent
second_p = soup.find(id='second')
parent = second_p.parent
# Output: div
print(parent.name)

# Navigate to previous sibling
prev_sibling = second_p.previous_sibling.previous_sibling  # Skip whitespace
# Output: first
print(prev_sibling.get('id'))

# Navigate to next sibling
next_sibling = second_p.next_sibling.next_sibling  # Skip whitespace
# Output: third
print(next_sibling.get('id'))

# Find all next siblings
remaining_paragraphs = second_p.find_next_siblings('p')
# Output: 1 paragraph
print(f"{len(remaining_paragraphs)} paragraph")

# Find all previous siblings
previous_paragraphs = second_p.find_previous_siblings('p')
# Output: 1 paragraph
print(f"{len(previous_paragraphs)} paragraph")

# Find parent with specific criteria
span = soup.find('span')
container_div = span.find_parent('div', class_='container')
# Output: container
print(container_div.get('class')[0])

# Iterate through all parents
for parent in span.parents:
    if parent.name:
        print(parent.name)  # Prints: div, div, body, html, [document]

```

--------------------------------

### Importing BeautifulSoup from bs4

Source: https://github.com/wention/beautifulsoup4/blob/master/NEWS.txt

This snippet demonstrates how to import the BeautifulSoup class from the 'bs4' module, which is the standard way to begin using Beautiful Soup 4.

```python
from bs4 import BeautifulSoup
```

--------------------------------

### Iterating Over Stripped Strings with BeautifulSoup

Source: https://github.com/wention/beautifulsoup4/blob/master/doc/source/index.rst

This example uses the `.stripped_strings` generator in Beautiful Soup to iterate over all non-whitespace strings within a tag. It automatically removes leading/trailing whitespace and ignores strings that contain only whitespace.

```python
for string in soup.stripped_strings:
    print(repr(string))
# u"The Dormouse's story"
# u"The Dormouse's story"
# u'Once upon a time there were three little sisters; and their names were'
# u'Elsie'
# u','
# u'Lacie'
# u'and'
# u'Tillie'
# u';\nand they lived at the bottom of a well.'
# u'...' # (truncated for brevity)
```

--------------------------------

### Parse HTML from a File or String with BeautifulSoup

Source: https://github.com/wention/beautifulsoup4/blob/master/doc/source/index.rst

Shows how to initialize a BeautifulSoup object by passing either an open file handle or a string containing HTML content. Beautiful Soup will automatically convert the input to Unicode and parse it using the best available parser (HTML by default).

```Python
from bs4 import BeautifulSoup

soup = BeautifulSoup(open("index.html"))

soup = BeautifulSoup("<html>data</html>")
```

--------------------------------

### Safely Accessing Tag Attributes with BeautifulSoup

Source: https://github.com/wention/beautifulsoup4/blob/master/doc/source/index.rst

This example shows how to safely access tag attributes in Beautiful Soup using the .get() method, similar to Python dictionaries. This prevents KeyError if the attribute is not defined on the tag.

```python
tag.get('attr')
```

--------------------------------

### Use different parsers with Beautiful Soup

Source: https://github.com/wention/beautifulsoup4/blob/master/doc/source/index.rst

Demonstrates how to specify different parsers when creating a BeautifulSoup object. Options include Python's built-in 'html.parser', the 'lxml' parser, and the 'html5lib' parser.

```python
BeautifulSoup(markup, "html.parser")
```

```python
BeautifulSoup(markup, "lxml")
```

```python
BeautifulSoup(markup, ["lxml", "xml"])
```

```python
BeautifulSoup(markup, "xml")
```

--------------------------------

### Navigate next and previous elements with Beautiful Soup

Source: https://github.com/wention/beautifulsoup4/blob/master/doc/source/index.rst

Demonstrates how to access the element immediately following or preceding a given tag. It also shows how to iterate through all subsequent or preceding elements. This is useful for understanding the order of parsed content.

```python
from bs4 import BeautifulSoup

html_doc = """
<html><head><title>The Dormouse's story</title></head>

<p class="title"><b>The Dormouse's story</b></p>

<p class="story">Once upon a time there were three little sisters; and their names were
<a href="http://example.com/elsie" class="sister" id="link1">Elsie</a>,
<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>

<p class="story">...</p>
"""

soup = BeautifulSoup(html_doc)

last_a_tag = soup.find_all('a')[-1]

# Accessing the next element
print(repr(last_a_tag.next_element))

# Accessing the previous element
print(repr(last_a_tag.previous_element))
print(repr(last_a_tag.previous_element.next_element))

# Iterating through next elements
for element in last_a_tag.next_elements:
    print(repr(element))
```

--------------------------------

### Finding next siblings using find_next_siblings() and find_next_sibling() in Beautiful Soup

Source: https://github.com/wention/beautifulsoup4/blob/master/doc/source/index.rst

Demonstrates the `find_next_siblings()` and `find_next_sibling()` methods in Beautiful Soup, which search for subsequent siblings of an element in the document tree. `find_next_siblings()` returns all matching siblings, while `find_next_sibling()` returns the first one found. These are used in conjunction with the `.next_siblings` generator.

```python
# Signature: find_next_siblings(:ref:`name <name>`, :ref:`attrs <attrs>`, :ref:`text <text>`, :ref:`limit <limit>`, :ref:`**kwargs <kwargs>`)
# Signature: find_next_sibling(:ref:`name <name>`, :ref:`attrs <attrs>`, :ref:`text <text>`, :ref:`**kwargs <kwargs>`)

# Example usage would follow, iterating over .next_siblings
```

--------------------------------

### Get Plain String Representation with Beautiful Soup 4

Source: https://github.com/wention/beautifulsoup4/blob/master/doc/source/index.rst

To obtain a plain string representation of a BeautifulSoup object or a Tag, you can use the built-in str() or unicode() functions. str() returns a UTF-8 encoded string, while unicode() returns a Unicode string.

```python
from bs4 import BeautifulSoup

markup = '<a href="http://example.com/">I linked to <i>example.com</i></a>'
soup = BeautifulSoup(markup)

print(str(soup))
```

```python
from bs4 import BeautifulSoup

markup = '<a href="http://example.com/">I linked to <i>example.com</i></a>'
soup = BeautifulSoup(markup)

print(unicode(soup.a))
```

--------------------------------

### Unicode Dammit: Guess Encoding and Convert to Unicode

Source: https://github.com/wention/beautifulsoup4/blob/master/doc/source/index.rst

Illustrates the use of Unicode, Dammit to convert data with unknown encodings into Unicode. It shows how to initialize UnicodeDammit with a byte string and access the decoded Unicode markup and the guessed original encoding. Requires 'chardet' or 'cchardet' for better accuracy.

```python
from bs4 import UnicodeDammit

dammit = UnicodeDammit("Sacr\xc3\xa9 bleu!")
print(dammit.unicode_markup)
# Sacré bleu!
print(dammit.original_encoding)
# 'utf-8'
```

--------------------------------

### SoupStrainer: Parse Only Specific Parts of a Document

Source: https://github.com/wention/beautifulsoup4/blob/master/doc/source/index.rst

Introduces the SoupStrainer class, which allows Beautiful Soup to parse only specific parts of an incoming document, such as all 'a' tags. This improves efficiency by avoiding the parsing of irrelevant content. The example shows how to create a SoupStrainer and pass it to the BeautifulSoup constructor.

```python
from bs4 import BeautifulSoup, SoupStrainer

# Example: only parse 'a' tags
a_tags = SoupStrainer('a')
soup = BeautifulSoup(html_doc, parse_only=a_tags)

# Now soup will only contain 'a' tags
```

--------------------------------

### Simplified Tree Traversal with Shortcuts and Null Handling

Source: https://github.com/wention/beautifulsoup4/blob/master/NEWS.txt

Beautiful Soup 2.0 introduces convenient shortcuts for traversing the parse tree and a 'Null' object to represent non-existent elements. Tag objects can be called like functions to fetch child tags, and specially named attributes can be used to fetch the first child tag of a certain name. 'Null' objects prevent errors when intermediate elements are missing, simplifying code that navigates deep or potentially incomplete trees.

```python
from bs4 import BeautifulSoup

html_doc = """
<html>
  <head>
    <title>My Document Title</title>
  </head>
  <body>
    <p>First paragraph.</p>
    <p>Second paragraph.</p>
    <ul>
      <li>Item 1</li>
      <li>Item 2</li>
    </ul>
  </body>
</html>
"""

soup = BeautifulSoup(html_doc, 'html.parser')

# Using tag as a function shorthand for fetch()
# soup.body('p') is equivalent to soup.body.fetch('p')
paragraphs = soup.body('p')
print(f"Found {len(paragraphs)} paragraphs using tag call.")

# Using special attribute name for first() shorthand (e.g., titleTag)
# soup.htmlTag.titleTag is shorthand for soup.htmlTag.first('title')
title_tag = soup.html.titleTag
if title_tag:
    print(f"Document title: {title_tag.string}")

# Demonstrating Null object for non-existent elements
# If there was no 'meta' tag, soup.head.metaTag would be Null instead of None
# This allows chaining without explicit checks:
non_existent_element = soup.head.nonExistentTag.someOtherTag
print(f"Accessing non-existent element results in: {non_existent_element}")

# Example of previousSibling and nextSibling
first_li = soup.ul.li
print(f"First li next sibling: {first_li.nextSibling.name}") # Should be 'li'
second_li = first_li.nextSibling
print(f"Second li previous sibling: {second_li.previousSibling.name}") # Should be 'li'

# The 'br' tag example from the docs
html_with_br = "<p><ul><li>Foo<br /><li>Bar</ul>"
soup_br = BeautifulSoup(html_with_br, 'html.parser')
foo_li = soup_br.li
print(f"'Foo' li's nextSibling: {foo_li.nextSibling.name}") # Should be 'br'

```

--------------------------------

### Create New Strings and Tags with Beautiful Soup

Source: https://github.com/wention/beautifulsoup4/blob/master/doc/source/index.rst

Introduces the `BeautifulSoup.new_string()` and `BeautifulSoup.new_tag()` methods for programmatically creating new string and tag objects that can then be added to the parse tree.

```python
# Example usage would involve creating strings/tags and then appending them.
# Specific code examples for new_string and new_tag were not provided in the text.
```

--------------------------------

### Pretty Printing and Outputting HTML with BeautifulSoup

Source: https://context7.com/wention/beautifulsoup4/llms.txt

Details how to format and output the parsed HTML document tree using Beautiful Soup. It covers pretty-printing the entire document or specific tags, converting the soup object to a compact string, encoding the output to bytes with specific encodings, extracting inner HTML using `decode_contents()`, and controlling entity formatting.

```python
from bs4 import BeautifulSoup

html = '<html><head><title>Page</title></head><body><div><p>Text</p></div></body></html>'
soup = BeautifulSoup(html, 'html.parser')

# Pretty print entire document
pretty_html = soup.prettify()
# Output: Formatted HTML with indentation
print(pretty_html)
# <html>
#  <head>
#   <title>
#    Page
#   </title>
#  </head>
#  <body>
#   <div>
#    <p>
#     Text
#    </p>
#   </div>
#  </body>
# </html>

# Pretty print specific tag
div = soup.find('div')
pretty_div = div.prettify()
print(pretty_div)

# Convert to string without formatting
compact = str(soup)
# Output: <html><head><title>Page</title></head><body><div><p>Text</p></div></body></html>
print(compact)

# Encode to bytes with specific encoding
encoded = soup.encode('utf-8')
# Output: bytes object
print(type(encoded))  # <class 'bytes'>

# Get only the contents (inner HTML)
contents = div.decode_contents()
# Output: <p>Text</p>
print(contents)

# Format with different encoding
latin1_html = soup.prettify(encoding='latin-1')

# Control entity formatting
html2 = '<p>Café & "Restaurant"</p>'
soup2 = BeautifulSoup(html2, 'html.parser')
# Minimal formatting (only essential entities)
minimal = soup2.encode(formatter='minimal').decode('utf-8')
# HTML formatting (all entities)
with_entities = soup2.encode(formatter='html').decode('utf-8')
# No formatting
no_format = soup2.encode(formatter=None).decode('utf-8')
```

--------------------------------

### Handling Multi-valued Attributes (Python)

Source: https://github.com/wention/beautifulsoup4/blob/master/NEWS.txt

Demonstrates how Beautiful Soup now consistently handles multi-valued attributes, such as 'class', by always returning a list of values, even if there's only one.

```python
from bs4 import BeautifulSoup

html_doc = "<a href='#' class='link active'>Link</a><span class='highlight'>Text</span>"
soup = BeautifulSoup(html_doc, 'html.parser')

link_tag = soup.find('a')
span_tag = soup.find('span')

print(f"Link classes: {link_tag.get('class')}")
print(f"Span classes: {span_tag.get('class')}")
```

--------------------------------

### Specifying Parsers with BeautifulSoup Constructor

Source: https://github.com/wention/beautifulsoup4/blob/master/doc/source/index.rst

Explains how to explicitly tell Beautiful Soup which parser to use when creating a `BeautifulSoup` object. This is done by passing a second argument specifying the markup type (e.g., 'html', 'xml') or the parser library name (e.g., 'lxml', 'html5lib', 'html.parser').

```python
from bs4 import BeautifulSoup

markup_html = "<html><body><p>Hello</p></body></html>"
markup_xml = "<a><b/></a>"

# Default parsing (best available HTML parser)
soup_html_default = BeautifulSoup(markup_html)

# Explicitly specifying HTML parser
soup_html_explicit = BeautifulSoup(markup_html, "html.parser")

# Explicitly specifying XML parser (requires lxml)
soup_xml = BeautifulSoup(markup_xml, "xml")

# Specifying parser library
soup_lxml = BeautifulSoup(markup_html, "lxml")
soup_html5lib = BeautifulSoup(markup_html, "html5lib")
```

--------------------------------

### Navigate HTML Parse Tree with BeautifulSoup in Python

Source: https://github.com/wention/beautifulsoup4/blob/master/README.md

This snippet shows how to navigate the parsed HTML tree obtained from BeautifulSoup. It demonstrates accessing the title, parent elements, attributes, and finding specific tags like 'a' and elements by their ID. This requires a BeautifulSoup object to be already created.

```python
soup.title
soup.title.name
soup.title.string
soup.title.parent.name
soup.p
soup.p['class']
soup.a
soup.find_all('a')
soup.find(id="link3")
```

--------------------------------

### Beautiful Soup Pickling and Copying Support

Source: https://github.com/wention/beautifulsoup4/blob/master/NEWS.txt

Soup objects in Beautiful Soup can now be pickled and copied using `copy.deepcopy`. This allows for serialization and deep copying of parsed document trees, enabling more flexible data handling and persistence.

```python
import pickle
import copy

saved_soup = pickle.dumps(soup_object)
cloned_soup = copy.deepcopy(soup_object)
```

--------------------------------

### Navigating Next and Previous Siblings in BeautifulSoup

Source: https://github.com/wention/beautifulsoup4/blob/master/doc/source/index.rst

Illustrates how to find the immediate next and previous sibling of an element using `.next_sibling` and `.previous_sibling`. Returns None if no sibling exists on the same level.

```python
from bs4 import BeautifulSoup

soup = BeautifulSoup("<a><b>text1</b><c>text2</c></a>", "html.parser")

print(soup.b.next_sibling)
# <c>text2</c>

print(soup.c.previous_sibling)
# <b>text1</b>

print(soup.b.previous_sibling)
# None

print(soup.c.next_sibling)
# None
```

--------------------------------

### Finding parents using find_parents() and find_parent() in Beautiful Soup

Source: https://github.com/wention/beautifulsoup4/blob/master/doc/source/index.rst

Illustrates the use of `find_parents()` and `find_parent()` methods in Beautiful Soup for searching upwards in the document tree. `find_parents()` returns all matching parent tags, while `find_parent()` returns only the first matching parent. These methods are useful for locating an element's ancestors.

```python
a_string = soup.find(text="Lacie")

a_string.find_parents("a")
# Expected output: [<a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>]

a_string.find_parent("p")
# Expected output: <p class="story">Once upon a time there were three little sisters; and their names were
#  <a class="sister" href="http://example.com/elsie" id="link1">Elsie</a>,
#  <a class="sister" href="http://example.com/lacie" id="link2">Lacie</a> and
#  <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>;
#  and they lived at the bottom of a well.</p>

a_string.find_parents("p", class_="title")
# Expected output: []
```

--------------------------------

### Extracting Attributes and Data from HTML with Beautiful Soup

Source: https://context7.com/wention/beautifulsoup4/llms.txt

Demonstrates how to extract various attributes like 'href' and 'class' from HTML tags, find all links and image sources, and access custom data attributes. It also shows how to check for the existence of attributes.

```python
from bs4 import BeautifulSoup

# Assuming soup is a pre-existing BeautifulSoup object
# Example: soup = BeautifulSoup(html_content, 'html.parser')

# Output: {'href': 'http://example.com/page1', 'class': ['external'], 'target': '_blank', 'data-id': '123'}
# print(attrs) # This line is commented out as attrs is not defined in this context

# Extract all URLs from links
all_links = soup.find_all('a')
urls = [link.get('href') for link in all_links]
# Output: ['http://example.com/page1', 'http://example.com/page2']
print(urls)

# Extract image sources
images = soup.find_all('img')
for img in images:
    src = img.get('src')
    alt = img.get('alt', 'No alt text')
    print(f"Image: {src}, Alt: {alt}")

# Access data attributes
div = soup.find('div', attrs={'data-value': True})
data_value = div.get('data-value')  # 'test'
data_count = div.get('data-count')  # '5'

# Check if attribute exists
# Assuming 'link' is a previously found tag
# has_target = link.has_attr('target')  # True
# has_title = link.has_attr('title')  # False

```

--------------------------------

### Using find() vs. find_all() in BeautifulSoup

Source: https://github.com/wention/beautifulsoup4/blob/master/doc/source/index.rst

This snippet clarifies the difference between find() and find_all() in Beautiful Soup. find() returns a single tag or None, while find_all() returns a ResultSet (list) of tags. Use find() when expecting a single result.

```python
tag = soup.find('tag_name')
```

--------------------------------

### Navigating Next and Previous Elements in BeautifulSoup

Source: https://github.com/wention/beautifulsoup4/blob/master/doc/source/index.rst

Explains how to find the element that was parsed immediately before or after the current element using `.previous_element` and `.next_element`. This differs from siblings and can traverse different levels.

```python
from bs4 import BeautifulSoup

soup = BeautifulSoup("<p>Some text<a href=\"#\">Link</a>More text</p>", "html.parser")

a_tag = soup.a
print(a_tag.next_sibling)
# More text

print(a_tag.next_element)
# More text

# Example with the 'three sisters' document structure
soup = BeautifulSoup("<a id=\"link3\">Tillie</a>; and they lived at the bottom of a well.", "html.parser")
last_a_tag = soup.find("a", id="link3")
print(last_a_tag.next_sibling)
# ; and they lived at the bottom of a well.
```

--------------------------------

### Creating a New Tag with BeautifulSoup

Source: https://github.com/wention/beautifulsoup4/blob/master/NEWS.txt

Illustrates the usage of BeautifulSoup.new_tag() to create new HTML/XML tags. The behavior of this method (e.g., self-closing vs. empty tags) adapts to the tree builder used for the original BeautifulSoup object.

```python
# Assuming 'soup' is an existing BeautifulSoup object
new_paragraph = soup.new_tag("p")
```

--------------------------------

### Replace HTML Elements with BeautifulSoup

Source: https://context7.com/wention/beautifulsoup4/llms.txt

Illustrates various methods for replacing HTML elements using Beautiful Soup. This includes replacing a tag with another tag, replacing with plain text, unwrapping tags to keep content, and replacing text content directly. It also shows how to replace an element with multiple new elements by wrapping them in a container.

```python
from bs4 import BeautifulSoup

html = """
<div>
    <p>Paragraph with <b>bold text</b> inside.</p>
    <span class="old">Old content</span>
    <a href="#">Link</a>
</div>
"""
soup = BeautifulSoup(html, 'html.parser')

# Replace tag with another tag
span = soup.find('span')
new_p = soup.new_tag('p', class_='new')
new_p.string = 'New content'
span.replace_with(new_p)
# Output: span is replaced with p
print(soup.find('p', class_='new'))

# Replace with string
link = soup.find('a')
link.replace_with('Plain text')
# Output: <div> contains "Plain text" instead of link

# Unwrap - remove tag but keep contents
html2 = '<p>Text with <b>bold</b> formatting</p>'
soup2 = BeautifulSoup(html2, 'html.parser')
b = soup2.find('b')
b.unwrap()
# Output: <p>Text with bold formatting</p>
print(soup2.p)

# Replace text content
html3 = '<div><p>Old text</p></div>'
soup3 = BeautifulSoup(html3, 'html.parser')
p = soup3.find('p')
p.string = 'New text'
# Output: <p>New text</p>
print(p)

# Replace with multiple elements (wrap in container first)
html4 = '<div><p>Target</p></div>'
soup4 = BeautifulSoup(html4, 'html.parser')
p = soup4.find('p')
container = soup4.new_tag('div')
new_p1 = soup4.new_tag('p')
new_p1.string = 'First'
new_p2 = soup4.new_tag('p')
new_p2.string = 'Second'
container.append(new_p1)
container.append(new_p2)
p.replace_with(container)
# Output: Original p replaced with div containing two new paragraphs
print(soup4)
```

--------------------------------

### Inserting Elements Before/After Another Element

Source: https://github.com/wention/beautifulsoup4/blob/master/NEWS.txt

Shows how to use PageElement.insert_before() and PageElement.insert_after() to position a new element in the parse tree relative to an existing element.

```python
# Assuming 'existing_element' and 'new_element' are PageElements
# Insert new_element before existing_element
existing_element.insert_before(new_element)

# Insert new_element after existing_element
existing_element.insert_after(new_element)
```

--------------------------------

### Find tags using regular expressions with Beautiful Soup

Source: https://github.com/wention/beautifulsoup4/blob/master/doc/source/index.rst

Demonstrates how to use regular expressions to find tags whose names match a given pattern. This allows for more flexible searching based on tag name characteristics.

```python
import re
from bs4 import BeautifulSoup

html_doc = """
<html><head><title>The Dormouse's story</title></head>

<p class="title"><b>The Dormouse's story</b></p>

<p class="story">Once upon a time there were three little sisters; and their names were
<a href="http://example.com/elsie" class="sister" id="link1">Elsie</a>,
<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>

<p class="story">...</p>
"""

soup = BeautifulSoup(html_doc)

# Find tags whose names start with 'b'
for tag in soup.find_all(re.compile("^b")):
    print(tag.name)

# Find tags whose names contain 't'
for tag in soup.find_all(re.compile("t")):
    print(tag.name)
```
Tokens:
Raw