GNU Wget

https://www.gnu.org/software/wget/manual/html_node/
Admin
GNU Wget is a free command-line utility for non-interactive downloading of files from the Web using...
Tokens:46,528
Snippets:529
Trust Score:10
Update:5 days ago
Show doc for...
Tokens:
Raw
### Reproduce a crash with specific options

Source: https://www.gnu.org/software/wget/manual/html_node/Reporting-Bugs.html

Example command to attempt to reproduce a crash with a specific set of Wget options.

```bash
wget -rl0 -kKE -t5 --no-proxy http://example.com -o /tmp/log
```

--------------------------------

### Mirror with link conversion for offline viewing

Source: https://www.gnu.org/software/wget/manual/html_node/Advanced-Usage.html

Same as the previous mirror example but also converts links in downloaded files to point to local files, enabling offline viewing.

```bash
wget --convert-links -r https://www.gnu.org/ -o gnulog

```

--------------------------------

### Download page prerequisites without server directories

Source: https://www.gnu.org/software/wget/manual/html_node/Advanced-Usage.html

Same as the previous example but avoids creating the www.example.com/ directory by using -nH and -nd, and saves all files under a download/ subdirectory with -Pdownload.

```bash
wget -p --convert-links -nH -nd -Pdownload \
     http://www.example.com/dir/page.html

```

--------------------------------

### Log in and download pages using POST

Source: https://www.gnu.org/software/wget/manual/html_node/HTTP-Options.html

This example demonstrates how to authenticate with a server using POST data and save cookies, then use those cookies to download restricted pages.

```bash
# Log in to the server.  This can be done only once.
wget --save-cookies cookies.txt \
     --post-data 'user=foo&password=bar' \
     http://example.com/auth.php

# Now grab the page or pages we care about.
wget --load-cookies cookies.txt \
     -p http://example.com/interesting/article.php
```

--------------------------------

### Basic Wget invocation syntax

Source: https://www.gnu.org/software/wget/manual/html_node/Invoking.html

Shows the general command-line syntax for invoking Wget. Use this as the starting point for any download; replace `[option]` and `[URL]` with actual options and URLs.

```bash
wget [option]... [URL]...

```

--------------------------------

### Recursive download command: wget -r http://www.example.com/

Source: https://www.gnu.org/software/wget/manual/html_node/Robot-Exclusion.html

This command initiates a recursive download of the example site. Wget first downloads the index page, then requests /robots.txt from the server to determine which paths to avoid. The robots.txt file is loaded only once per server.

```shell
wget -r http://www.example.com/

```

--------------------------------

### Accept specific file types with -A

Source: https://www.gnu.org/software/wget/manual/html_node/Types-of-Files.html

Use the -A option to download only files matching specific suffixes or patterns. This example downloads only GIF and JPEG files.

```bash
wget -A gif,jpg
```

--------------------------------

### Combine accept and reject options

Source: https://www.gnu.org/software/wget/manual/html_node/Types-of-Files.html

Combine -A and -R to fine-tune retrieval. This example downloads files containing 'zelazny' but excludes PostScript files.

```bash
wget -A "*zelazny*" -R .ps
```

--------------------------------

### wget --ignore-tags example

Source: https://www.gnu.org/software/wget/manual/html_node/Recursive-Accept_002fReject-Options.html

Example command using --ignore-tags to skip certain HTML tags when recursively downloading a page. The source notes that this approach is outdated and recommends --page-requisites for downloading a single page and its requisites.

```shell
wget --ignore-tags=a,area -H -k -K -r http://site/document
```

--------------------------------

### Example of --cut-dirs option effects

Source: https://www.gnu.org/software/wget/manual/html_node/Directory-Options.html

Shows how different combinations of -nH and --cut-dirs affect the local directory structure when retrieving ftp://ftp.xemacs.org/pub/xemacs/. The output lists the resulting paths for each option set.

```text
No options        -> ftp.xemacs.org/pub/xemacs/
-nH               -> pub/xemacs/
-nH --cut-dirs=1  -> xemacs/
-nH --cut-dirs=2  -> .

--cut-dirs=1      -> ftp.xemacs.org/xemacs/
...
```

--------------------------------

### Disable FTP Globbing Example

Source: https://www.gnu.org/software/wget/manual/html_node/FTP-Options.html

Example of using a URL with wildcards for FTP retrieval, which can be disabled using the --no-glob option.

```bash
wget ftp://gnjilux.srk.fer.hr/*.msg
```

--------------------------------

### Proxy URL with Authorization

Source: https://www.gnu.org/software/wget/manual/html_node/Proxies.html

Example of a proxy URL containing a username and password for authentication at proxy.company.com on port 8001.

```text
http://hniksic:mypassword@proxy.company.com:8001/
```

--------------------------------

### wget -rH -Dexample.com http://www.example.com/

Source: https://www.gnu.org/software/wget/manual/html_node/Spanning-Hosts.html

Recursively download from www.example.com while allowing downloads from any host in the example.com domain. The -H option enables host spanning, and -D limits the recursion to the specified domain.

```bash
wget -rH -Dexample.com http://www.example.com/

```

--------------------------------

### FTP URL with ASCII Transfer Mode

Source: https://www.gnu.org/software/wget/manual/html_node/URL-Format.html

Example of specifying the ASCII transfer mode for an FTP URL using the type parameter.

```text
ftp://host/directory/file;type=a
```

--------------------------------

### Create a five-level deep mirror of the GNU web site

Source: https://www.gnu.org/software/wget/manual/html_node/Advanced-Usage.html

Recursively download the GNU web site with a depth of five levels, preserving the directory structure, with one try per document, and save the log to gnulog.

```bash
wget -r https://www.gnu.org/ -o gnulog

```

--------------------------------

### Download a single page with prerequisites and convert links

Source: https://www.gnu.org/software/wget/manual/html_node/Advanced-Usage.html

Retrieve one HTML page and all elements needed for display (inline images, external style sheets) using -p, and convert links to local references with --convert-links. The page is saved to www.example.com/dir/page.html.

```bash
wget -p --convert-links http://www.example.com/dir/page.html

```

--------------------------------

### Reject specific file types with -R

Source: https://www.gnu.org/software/wget/manual/html_node/Types-of-Files.html

Use the -R option to download all files except those matching the provided suffixes or patterns. This example excludes MPEG and AU files.

```bash
wget -R mpg,mpeg,au
```

--------------------------------

### Mirror with link conversion and backup

Source: https://www.gnu.org/software/wget/manual/html_node/Very-Advanced-Usage.html

Mirrors the site, converts links for local viewing, and backs up original HTML files before conversion. Uses --mirror, --convert-links, and --backup-converted.

```bash
wget --mirror --convert-links --backup-converted  \
     https://www.gnu.org/ -o /home/me/weeklog

```

--------------------------------

### Retrieve first two levels of an FTP site

Source: https://www.gnu.org/software/wget/manual/html_node/Advanced-Usage.html

Recursively download the first two levels of wuarchive.wustl.edu, saving files to /tmp.

```bash
wget -r -l2 -P/tmp ftp://wuarchive.wustl.edu/

```

--------------------------------

### Save server headers with the file

Source: https://www.gnu.org/software/wget/manual/html_node/Advanced-Usage.html

Use --save-headers to store server headers along with the downloaded file, useful for post-processing. The example also shows viewing the file with more.

```bash
wget --save-headers http://www.lycos.com/
more index.html

```

--------------------------------

### Download with recursion, depth limit, and page requisites

Source: https://www.gnu.org/software/wget/manual/html_node/Recursive-Retrieval-Options.html

Downloads a page, its links up to 2 hops, and all necessary requisites for those pages.

```bash
wget -r -l 2 -p http://site/1.html
```

--------------------------------

### wget --spider --force-html -i bookmarks.html

Source: https://www.gnu.org/software/wget/manual/html_node/Download-Options.html

Use Wget in spider mode to check that pages exist without downloading them. The example checks bookmarks from a file.

```shell
wget --spider --force-html -i bookmarks.html

```

--------------------------------

### Short form of mirror with extension adjustment

Source: https://www.gnu.org/software/wget/manual/html_node/Very-Advanced-Usage.html

Equivalent to the previous command using short options: -m (mirror), -k (convert links), -K (backup converted), -E (adjust extension).

```bash
wget -m -k -K -E https://www.gnu.org/ -o /home/me/weeklog

```

--------------------------------

### Redirect Wget output to wget-log on SIGHUP

Source: https://www.gnu.org/software/wget/manual/html_node/Signals.html

Start Wget in the background, then send SIGHUP to redirect its output to wget-log. The output shows the message 'SIGHUP received, redirecting output to `wget-log'.'.

```shell
$ wget http://www.gnus.org/dist/gnus.tar.gz &
...
$ kill -HUP %%
SIGHUP received, redirecting output to `wget-log'.

```

--------------------------------

### Accept files matching a shell pattern with -A

Source: https://www.gnu.org/software/wget/manual/html_node/Types-of-Files.html

Use shell-like wildcards with the -A option to match specific naming conventions.

```bash
wget -A "zelazny*196[0-9]*"
```

--------------------------------

### Prevent parent directory traversal with --no-parent

Source: https://www.gnu.org/software/wget/manual/html_node/Directory_002dBased-Limits.html

The --no-parent (or -np) option restricts retrieval to the starting directory hierarchy. Note that a trailing slash in the URL is required for Wget to correctly identify the directory path.

```bash
wget -r --no-parent http://somehost/~luzer/my-archive/
```
Tokens:
Raw