### Reproduce a crash with specific options Source: https://www.gnu.org/software/wget/manual/html_node/Reporting-Bugs.html Example command to attempt to reproduce a crash with a specific set of Wget options. ```bash wget -rl0 -kKE -t5 --no-proxy http://example.com -o /tmp/log ``` -------------------------------- ### Mirror with link conversion for offline viewing Source: https://www.gnu.org/software/wget/manual/html_node/Advanced-Usage.html Same as the previous mirror example but also converts links in downloaded files to point to local files, enabling offline viewing. ```bash wget --convert-links -r https://www.gnu.org/ -o gnulog ``` -------------------------------- ### Download page prerequisites without server directories Source: https://www.gnu.org/software/wget/manual/html_node/Advanced-Usage.html Same as the previous example but avoids creating the www.example.com/ directory by using -nH and -nd, and saves all files under a download/ subdirectory with -Pdownload. ```bash wget -p --convert-links -nH -nd -Pdownload \ http://www.example.com/dir/page.html ``` -------------------------------- ### Log in and download pages using POST Source: https://www.gnu.org/software/wget/manual/html_node/HTTP-Options.html This example demonstrates how to authenticate with a server using POST data and save cookies, then use those cookies to download restricted pages. ```bash # Log in to the server. This can be done only once. wget --save-cookies cookies.txt \ --post-data 'user=foo&password=bar' \ http://example.com/auth.php # Now grab the page or pages we care about. wget --load-cookies cookies.txt \ -p http://example.com/interesting/article.php ``` -------------------------------- ### Basic Wget invocation syntax Source: https://www.gnu.org/software/wget/manual/html_node/Invoking.html Shows the general command-line syntax for invoking Wget. Use this as the starting point for any download; replace `[option]` and `[URL]` with actual options and URLs. ```bash wget [option]... [URL]... ``` -------------------------------- ### Recursive download command: wget -r http://www.example.com/ Source: https://www.gnu.org/software/wget/manual/html_node/Robot-Exclusion.html This command initiates a recursive download of the example site. Wget first downloads the index page, then requests /robots.txt from the server to determine which paths to avoid. The robots.txt file is loaded only once per server. ```shell wget -r http://www.example.com/ ``` -------------------------------- ### Accept specific file types with -A Source: https://www.gnu.org/software/wget/manual/html_node/Types-of-Files.html Use the -A option to download only files matching specific suffixes or patterns. This example downloads only GIF and JPEG files. ```bash wget -A gif,jpg ``` -------------------------------- ### Combine accept and reject options Source: https://www.gnu.org/software/wget/manual/html_node/Types-of-Files.html Combine -A and -R to fine-tune retrieval. This example downloads files containing 'zelazny' but excludes PostScript files. ```bash wget -A "*zelazny*" -R .ps ``` -------------------------------- ### wget --ignore-tags example Source: https://www.gnu.org/software/wget/manual/html_node/Recursive-Accept_002fReject-Options.html Example command using --ignore-tags to skip certain HTML tags when recursively downloading a page. The source notes that this approach is outdated and recommends --page-requisites for downloading a single page and its requisites. ```shell wget --ignore-tags=a,area -H -k -K -r http://site/document ``` -------------------------------- ### Example of --cut-dirs option effects Source: https://www.gnu.org/software/wget/manual/html_node/Directory-Options.html Shows how different combinations of -nH and --cut-dirs affect the local directory structure when retrieving ftp://ftp.xemacs.org/pub/xemacs/. The output lists the resulting paths for each option set. ```text No options -> ftp.xemacs.org/pub/xemacs/ -nH -> pub/xemacs/ -nH --cut-dirs=1 -> xemacs/ -nH --cut-dirs=2 -> . --cut-dirs=1 -> ftp.xemacs.org/xemacs/ ... ``` -------------------------------- ### Disable FTP Globbing Example Source: https://www.gnu.org/software/wget/manual/html_node/FTP-Options.html Example of using a URL with wildcards for FTP retrieval, which can be disabled using the --no-glob option. ```bash wget ftp://gnjilux.srk.fer.hr/*.msg ``` -------------------------------- ### Proxy URL with Authorization Source: https://www.gnu.org/software/wget/manual/html_node/Proxies.html Example of a proxy URL containing a username and password for authentication at proxy.company.com on port 8001. ```text http://hniksic:mypassword@proxy.company.com:8001/ ``` -------------------------------- ### wget -rH -Dexample.com http://www.example.com/ Source: https://www.gnu.org/software/wget/manual/html_node/Spanning-Hosts.html Recursively download from www.example.com while allowing downloads from any host in the example.com domain. The -H option enables host spanning, and -D limits the recursion to the specified domain. ```bash wget -rH -Dexample.com http://www.example.com/ ``` -------------------------------- ### FTP URL with ASCII Transfer Mode Source: https://www.gnu.org/software/wget/manual/html_node/URL-Format.html Example of specifying the ASCII transfer mode for an FTP URL using the type parameter. ```text ftp://host/directory/file;type=a ``` -------------------------------- ### Create a five-level deep mirror of the GNU web site Source: https://www.gnu.org/software/wget/manual/html_node/Advanced-Usage.html Recursively download the GNU web site with a depth of five levels, preserving the directory structure, with one try per document, and save the log to gnulog. ```bash wget -r https://www.gnu.org/ -o gnulog ``` -------------------------------- ### Download a single page with prerequisites and convert links Source: https://www.gnu.org/software/wget/manual/html_node/Advanced-Usage.html Retrieve one HTML page and all elements needed for display (inline images, external style sheets) using -p, and convert links to local references with --convert-links. The page is saved to www.example.com/dir/page.html. ```bash wget -p --convert-links http://www.example.com/dir/page.html ``` -------------------------------- ### Reject specific file types with -R Source: https://www.gnu.org/software/wget/manual/html_node/Types-of-Files.html Use the -R option to download all files except those matching the provided suffixes or patterns. This example excludes MPEG and AU files. ```bash wget -R mpg,mpeg,au ``` -------------------------------- ### Mirror with link conversion and backup Source: https://www.gnu.org/software/wget/manual/html_node/Very-Advanced-Usage.html Mirrors the site, converts links for local viewing, and backs up original HTML files before conversion. Uses --mirror, --convert-links, and --backup-converted. ```bash wget --mirror --convert-links --backup-converted \ https://www.gnu.org/ -o /home/me/weeklog ``` -------------------------------- ### Retrieve first two levels of an FTP site Source: https://www.gnu.org/software/wget/manual/html_node/Advanced-Usage.html Recursively download the first two levels of wuarchive.wustl.edu, saving files to /tmp. ```bash wget -r -l2 -P/tmp ftp://wuarchive.wustl.edu/ ``` -------------------------------- ### Save server headers with the file Source: https://www.gnu.org/software/wget/manual/html_node/Advanced-Usage.html Use --save-headers to store server headers along with the downloaded file, useful for post-processing. The example also shows viewing the file with more. ```bash wget --save-headers http://www.lycos.com/ more index.html ``` -------------------------------- ### Download with recursion, depth limit, and page requisites Source: https://www.gnu.org/software/wget/manual/html_node/Recursive-Retrieval-Options.html Downloads a page, its links up to 2 hops, and all necessary requisites for those pages. ```bash wget -r -l 2 -p http://site/1.html ``` -------------------------------- ### wget --spider --force-html -i bookmarks.html Source: https://www.gnu.org/software/wget/manual/html_node/Download-Options.html Use Wget in spider mode to check that pages exist without downloading them. The example checks bookmarks from a file. ```shell wget --spider --force-html -i bookmarks.html ``` -------------------------------- ### Short form of mirror with extension adjustment Source: https://www.gnu.org/software/wget/manual/html_node/Very-Advanced-Usage.html Equivalent to the previous command using short options: -m (mirror), -k (convert links), -K (backup converted), -E (adjust extension). ```bash wget -m -k -K -E https://www.gnu.org/ -o /home/me/weeklog ``` -------------------------------- ### Redirect Wget output to wget-log on SIGHUP Source: https://www.gnu.org/software/wget/manual/html_node/Signals.html Start Wget in the background, then send SIGHUP to redirect its output to wget-log. The output shows the message 'SIGHUP received, redirecting output to `wget-log'.'. ```shell $ wget http://www.gnus.org/dist/gnus.tar.gz & ... $ kill -HUP %% SIGHUP received, redirecting output to `wget-log'. ``` -------------------------------- ### Accept files matching a shell pattern with -A Source: https://www.gnu.org/software/wget/manual/html_node/Types-of-Files.html Use shell-like wildcards with the -A option to match specific naming conventions. ```bash wget -A "zelazny*196[0-9]*" ``` -------------------------------- ### Prevent parent directory traversal with --no-parent Source: https://www.gnu.org/software/wget/manual/html_node/Directory_002dBased-Limits.html The --no-parent (or -np) option restricts retrieval to the starting directory hierarchy. Note that a trailing slash in the URL is required for Wget to correctly identify the directory path. ```bash wget -r --no-parent http://somehost/~luzer/my-archive/ ```