### Define Data for Single Form Field Source: https://github.com/scrapy/form2request/blob/main/docs/usage.md Use the `data` parameter to specify values for form fields. This example shows how to set a value for a single text input. ```python >>> html = b"""
""" >>> selector = Selector(body=html, base_url="https://example.com") >>> form = selector.css("form") >>> form2request(form, {"foo": "bar"}) Request(url='https://example.com?foo=bar', method='GET', headers=[], body=b'') ``` -------------------------------- ### Finding Form by Attribute using XPath Source: https://github.com/scrapy/form2request/blob/main/docs/usage.md Provides an example XPath expression to locate a form element based on one of its attributes, such as 'id' or 'name'. This is useful for targeting specific forms in complex HTML. ```xpath //form[@id="foo"] ``` -------------------------------- ### Extracting HTML Form with Scrapy Response Source: https://github.com/scrapy/form2request/blob/main/docs/usage.md Shows how to get an HTML form object directly from a Scrapy TextResponse object using CSS selectors. This is common when working within a Scrapy project. ```python from scrapy.http import TextResponse html = b"
" response = TextResponse("https://example.com", body=html) form = response.css("form") ``` -------------------------------- ### Converting to Requests Library Format Source: https://github.com/scrapy/form2request/blob/main/docs/usage.md Illustrates how to convert the generated Request object into a format compatible with the Python requests library. This allows for easy sending of the form submission request. ```pycon import requests request = request_data.to_requests() requests.send(request) ``` -------------------------------- ### Extracting HTML Form with lxml Source: https://github.com/scrapy/form2request/blob/main/docs/usage.md Demonstrates how to use the lxml library to parse an HTML string and obtain a form element. This is an alternative to using parsel for form extraction. ```pycon from lxml.html import fromstring html = b"
" root = fromstring(html, base_url="https://example.com") form = root.xpath("//form")[0] ``` -------------------------------- ### Extracting HTML Form with Parsel Source: https://github.com/scrapy/form2request/blob/main/docs/usage.md Demonstrates how to use the parsel library to select an HTML form element from a given HTML body. This is the first step before using form2request. ```pycon from parsel import Selector html = b"
" selector = Selector(body=html, base_url="https://example.com") form = selector.css("form") ``` -------------------------------- ### Basic Form to Request Conversion Source: https://github.com/scrapy/form2request/blob/main/docs/usage.md Convert an HTML form into a basic Request object. The output includes the URL, method, headers, and body of the request. ```python >>> request_data = form2request(form) >>> request_data Request(url='https://example.com?foo=bar', method='GET', headers=[], body=b'') ``` -------------------------------- ### Finding Form by Index using XPath Source: https://github.com/scrapy/form2request/blob/main/docs/usage.md Illustrates how to use an XPath expression to select a form element based on its order of appearance in the HTML document. This is useful when forms lack unique identifiers. ```xpath (//form)[2] ``` -------------------------------- ### Submitting Form with Default Submit Button Source: https://github.com/scrapy/form2request/blob/main/docs/usage.md Submits a form by default, including the name and value of the first submit button found. This is the standard behavior when a submit button is present. ```python >>> html = b"
" >>> selector = Selector(body=html, base_url="https://example.com") >>> form = selector.css("form") >>> form2request(form) Request(url='https://example.com?foo=bar', method='GET', headers=[], body='') ``` -------------------------------- ### Uploading Files with FileField Source: https://github.com/scrapy/form2request/blob/main/docs/usage.md Demonstrates how to upload files using the FileField class within form2request. Ensure the form has enctype="multipart/form-data". ```python >>> from form2request import FileField, form2request >>> html = b""" ...
... ... ...
""" >>> selector = Selector(body=html, base_url="https://example.com") >>> form = selector.css("form") >>> request_data = form2request(form, { ... "description": "quarterly report", ... "attachment": FileField( ... content=b"col1,col2\n1,2\n", ... filename="report.csv", ... content_type="text/csv", ... ), ... }) >>> request_data.method 'POST' >>> request_data.url 'https://example.com' >>> request_data.headers[0][1].startswith("multipart/form-data") True ``` -------------------------------- ### Finding Form by Traversing Parent Elements Source: https://github.com/scrapy/form2request/blob/main/docs/usage.md Demonstrates a Python code snippet that finds a form element by first locating an input element within it and then traversing up the DOM tree to reach the parent form. This is a fallback method when direct form selection is difficult. ```python from lxml.html import fromstring html = b"
" root = fromstring(html, base_url="https://example.com") element = root.xpath('//input[@name="zip_code"]')[0] while True: if element.tag == "form": break element = element.getparent() form = element ``` -------------------------------- ### Define Data for Multiple Same-Name Form Fields Source: https://github.com/scrapy/form2request/blob/main/docs/usage.md When a form has multiple fields with the same `name`, provide an iterable of key-value tuples to `form2request` to specify values for all of them. ```python >>> html = b"""
""" >>> selector = Selector(body=html, base_url="https://example.com") >>> form = selector.css("form") >>> form2request(form, (("foo", "bar"), ("foo", "baz"))) Request(url='https://example.com?foo=bar&foo=baz', method='GET', headers=[], body=b'') ``` -------------------------------- ### Submitting Form with Specific Submit Button Source: https://github.com/scrapy/form2request/blob/main/docs/usage.md Submits a form by explicitly selecting which submit button to use. This is useful when a form has multiple submit buttons and you need to control which one is activated. ```python >>> html = b"
" >>> selector = Selector(body=html, base_url="https://example.com") >>> form = selector.css("form") >>> submit_baz = form.css("[value=baz]") >>> form2request(form, click=submit_baz) Request(url='https://example.com?foo=baz', method='GET', headers=[], body='') ``` -------------------------------- ### Convert Request to Scrapy Request Source: https://github.com/scrapy/form2request/blob/main/docs/usage.md Convert the Request object to a Scrapy Request object, suitable for use with the Scrapy framework (version 1.7.1+). Allows specifying a callback function. ```python >>> request_data.to_scrapy(callback=self.parse) ``` -------------------------------- ### Generating Request Data from Form Source: https://github.com/scrapy/form2request/blob/main/docs/usage.md Shows how to use the form2request function to convert a Parsel form object into a Request object. This object contains the URL, method, headers, and body for a form submission. ```pycon from form2request import form2request request_data = form2request(form) request_data ``` -------------------------------- ### Convert Request to web-poet HttpRequest Source: https://github.com/scrapy/form2request/blob/main/docs/usage.md Convert the Request object to a web-poet HttpRequest object, compatible with web-poet 0.2.0+. This format is useful for web scraping tasks. ```python >>> request_data.to_poet() HttpRequest(url=RequestUrl('https://example.com?foo=bar'), method='GET', headers=, body=b'') ``` -------------------------------- ### Submitting Form Without Submit Button Data Source: https://github.com/scrapy/form2request/blob/main/docs/usage.md Submits a form without including data from any submit button. Use this when the form is submitted programmatically or by means other than clicking a submit button. ```python >>> html = b"
" >>> selector = Selector(body=html, base_url="https://example.com") >>> form = selector.css("form") >>> form2request(form, click=False) Request(url='https://example.com', method='GET', headers=[], body='') ``` -------------------------------- ### Default Behavior with Multiple Submit Buttons Source: https://github.com/scrapy/form2request/blob/main/docs/usage.md When a form has multiple submit buttons, form2request defaults to activating the first one encountered. This behavior can be overridden using the 'click' parameter. ```python >>> html = b"
" >>> selector = Selector(body=html, base_url="https://example.com") >>> form = selector.css("form") >>> form2request(form) Request(url='https://example.com?foo=bar', method='GET', headers=[], body='') ``` -------------------------------- ### Remove Form Field Value Source: https://github.com/scrapy/form2request/blob/main/docs/usage.md To prevent a field's value from being included in the generated request data, set its value to `None` in the `data` parameter. ```python >>> html = b"""
""" >>> selector = Selector(body=html, base_url="https://example.com") >>> form = selector.css("form") >>> form2request(form, {"foo": None}) Request(url='https://example.com', method='GET', headers=[], body=b'') ``` -------------------------------- ### Override Form Attributes Source: https://github.com/scrapy/form2request/blob/main/docs/usage.md Override the method and enctype attributes of an HTML form when converting it to a request object. This is useful for specifying custom request methods or content types. ```python >>> form2request(form, method="POST", enctype="text/plain") Request(url='https://example.com', method='POST', headers=[('Content-Type', 'text/plain')], body=b'foo=bar') ``` === COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.