### Install Dependencies for Ubuntu

Source: https://github.com/qianlitp/crawlergo/blob/master/README.md

Installs necessary dependencies for running chrome on Ubuntu. Ensure your chrome version is up-to-date.

```shell
// Ubuntu
apt-get install -yq --no-install-recommends \
     libasound2 libatk1.0-0 libc6 libcairo2 libcups2 libdbus-1-3 \
     libexpat1 libfontconfig1 libgcc1 libgconf-2-4 libgdk-pixbuf2.0-0 libglib2.0-0 libgtk-3-0 libnspr4 \
     libpango-1.0-0 libpangocairo-1.0-0 libstdc++6 libx11-6 libx11-xcb1 libxcb1 libgbm1 \
     libxcursor1 libxdamage1 libxext6 libxfixes3 libxi6 libxrandr2 libxrender1 libxss1 libxtst6 libnss3
```

--------------------------------

### Quick Start Crawling a Website

Source: https://github.com/qianlitp/crawlergo/blob/master/README.md

Initiates a crawl on a specified website using crawlergo. Requires the path to the chromium executable and the target URL. Sets the number of concurrent tabs.

```shell
bin/crawlergo -c /tmp/chromium/chrome -t 10 http://testphp.vulnweb.com/
```

--------------------------------

### Build Crawlergo for Current Platform

Source: https://github.com/qianlitp/crawlergo/blob/master/README.md

Compiles the crawlergo for the current operating system. Ensure you have the necessary build tools installed.

```shell
make build
```

--------------------------------

### Install Dependencies for CentOS 7

Source: https://github.com/qianlitp/crawlergo/blob/master/README.md

Installs necessary dependencies for running chrome on CentOS 7. Ensure your chrome version is up-to-date.

```shell
// CentOS 7
sudo yum install pango.x86_64 libXcomposite.x86_64 libXcursor.x86_64 libXdamage.x86_64 libXext.x86_64 libXi.x86_64 \
     libXtst.x86_64 cups-libs.x86_64 libXScrnSaver.x86_64 libXrandr.x86_64 GConf2.x86_64 alsa-lib.x86_64 atk.x86_64 gtk3.x86_64 \
     ipa-gothic-fonts xorg-x11-fonts-100dpi xorg-x11-fonts-75dpi xorg-x11-utils xorg-x11-fonts-cyrillic xorg-x11-fonts-Type1 xorg-x11-fonts-misc -y

sudo yum update nss -y
```

--------------------------------

### Docker Usage for Crawling

Source: https://github.com/qianlitp/crawlergo/blob/master/README.md

Builds a Docker image for crawlergo and runs a container to crawl a website. This simplifies setup and dependency management.

```shell
git clone https://github.com/Qianlitp/crawlergo
docker build . -t crawlergo
docker run crawlergo http://testphp.vulnweb.com/
```

--------------------------------

### Build Crawlergo for All Platforms

Source: https://github.com/qianlitp/crawlergo/blob/master/README.md

Compiles the crawlergo for all supported operating systems. This command may take longer to complete.

```shell
make build_all
```

--------------------------------

### Calling Crawlergo with Python (JSON Output)

Source: https://github.com/qianlitp/crawlergo/blob/master/README.md

Demonstrates how to invoke crawlergo from a Python script and parse its JSON output. Ensures the output mode is set to 'json' for structured results.

```python
#!/usr/bin/python3
# coding: utf-8

import simplejson
import subprocess


def main():
    target = "http://testphp.vulnweb.com/"
    cmd = ["bin/crawlergo", "-c", "/tmp/chromium/chrome", "-o", "json", target]
    rsp = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
    output, error = rsp.communicate()
	#  "--[Mission Complete]--"  is the end-of-task separator string
    result = simplejson.loads(output.decode().split("--[Mission Complete]--")[1])
    req_list = result["req_list"]
    print(req_list[0])


if __name__ == '__main__':
    main()
```

--------------------------------

### Crawling with Proxy Support

Source: https://github.com/qianlitp/crawlergo/blob/master/README.md

Executes a crawl using a specified proxy server. This is useful for bypassing network restrictions or for privacy.

```shell
bin/crawlergo -c /tmp/chromium/chrome -t 10 --request-proxy socks5://127.0.0.1:7891 http://testphp.vulnweb.com/
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.