### Install Abot Web Crawler via NuGet Package Manager

Source: https://github.com/sjdirect/abot/blob/master/README.md

This command-line snippet shows how to install the Abot web crawler framework using the NuGet Package Manager console. It's the first step to integrate Abot into a .NET project.

```command
PM> Install-Package Abot
```

--------------------------------

### Implement Abot Event Processing Methods

Source: https://github.com/sjdirect/abot/blob/master/README.md

Provides examples of C# methods to process various Abot crawl events. This includes handling page crawl start, completion (checking for errors or content), and reasons for disallowing page or link crawls, allowing for detailed logging and custom actions.

```c#
void crawler_ProcessPageCrawlStarting(object sender, PageCrawlStartingArgs e)
{
	PageToCrawl pageToCrawl = e.PageToCrawl;
	Console.WriteLine($"About to crawl link {pageToCrawl.Uri.AbsoluteUri} which was found on page {pageToCrawl.ParentUri.AbsoluteUri}");
}

void crawler_ProcessPageCrawlCompleted(object sender, PageCrawlCompletedArgs e)
{
	CrawledPage crawledPage = e.CrawledPage;	
	if (crawledPage.HttpRequestException != null || crawledPage.HttpResponseMessage.StatusCode != HttpStatusCode.OK)
		Console.WriteLine($"Crawl of page failed {crawledPage.Uri.AbsoluteUri}");
	else
		Console.WriteLine($"Crawl of page succeeded {crawledPage.Uri.AbsoluteUri}");

	if (string.IsNullOrEmpty(crawledPage.Content.Text))
		Console.WriteLine($"Page had no content {crawledPage.Uri.AbsoluteUri}");

	var angleSharpHtmlDocument = crawledPage.AngleSharpHtmlDocument; //AngleSharp parser
}

void crawler_PageLinksCrawlDisallowed(object sender, PageLinksCrawlDisallowedArgs e)
{
	CrawledPage crawledPage = e.CrawledPage;
	Console.WriteLine($"Did not crawl the links on page {crawledPage.Uri.AbsoluteUri} due to {e.DisallowedReason}");
}

void crawler_PageCrawlDisallowed(object sender, PageCrawlDisallowedArgs e)
{
	PageToCrawl pageToCrawl = e.PageToCrawl;
	Console.WriteLine($"Did not crawl page {pageToCrawl.Uri.AbsoluteUri} due to {e.DisallowedReason}");
}
```

--------------------------------

### C# Abot Web Crawler Basic Usage and Configuration

Source: https://github.com/sjdirect/abot/blob/master/README.md

This C# example illustrates the fundamental usage of the Abot web crawler. It covers initializing logging, configuring a PoliteWebCrawler with MaxPagesToCrawl and MinCrawlDelayPerDomainMilliSeconds, handling the PageCrawlCompleted event, and making a single page request using PageRequester.

```C#
using System;
using System.Threading.Tasks;
using Abot2.Core;
using Abot2.Crawler;
using Abot2.Poco;
using Serilog;

namespace TestAbotUse
{
    class Program
    {
        static async Task Main(string[] args)
        {
            Log.Logger = new LoggerConfiguration()
                .MinimumLevel.Information()
                .WriteTo.Console()
                .CreateLogger();

            Log.Logger.Information("Demo starting up!");

            await DemoSimpleCrawler();
            await DemoSinglePageRequest();
        }

        private static async Task DemoSimpleCrawler()
        {
            var config = new CrawlConfiguration
            {
                MaxPagesToCrawl = 10, //Only crawl 10 pages
                MinCrawlDelayPerDomainMilliSeconds = 3000 //Wait this many millisecs between requests
            };
            var crawler = new PoliteWebCrawler(config);

            crawler.PageCrawlCompleted += PageCrawlCompleted;//Several events available...

            var crawlResult = await crawler.CrawlAsync(new Uri("http://!!!!!!!!YOURSITEHERE!!!!!!!!!.com"));
        }

        private static async Task DemoSinglePageRequest()
        {
            var pageRequester = new PageRequester(new CrawlConfiguration(), new WebContentExtractor());

            var crawledPage = await pageRequester.MakeRequestAsync(new Uri("http://google.com"));
            Log.Logger.Information("{result}", new
            {
                url = crawledPage.Uri,
                status = Convert.ToInt32(crawledPage.HttpResponseMessage.StatusCode)
            });
        }

        private static void PageCrawlCompleted(object sender, PageCrawlCompletedArgs e)
        {
            var httpStatus = e.CrawledPage.HttpResponseMessage.StatusCode;
            var rawPageText = e.CrawledPage.Content.Text;
        }
    }
}
```

--------------------------------

### Register Abot Event Handlers

Source: https://github.com/sjdirect/abot/blob/master/README.md

Shows how to register event handlers for different stages of the Abot crawl process. By attaching methods to these events, developers can execute custom logic at specific points, such as when a page crawl starts, completes, or is disallowed.

```c#
crawler.PageCrawlStarting += crawler_ProcessPageCrawlStarting;
crawler.PageCrawlCompleted += crawler_ProcessPageCrawlCompleted;
crawler.PageCrawlDisallowed += crawler_PageCrawlDisallowed;
crawler.PageLinksCrawlDisallowed += crawler_PageLinksCrawlDisallowed;
```

--------------------------------

### Instantiate PoliteWebCrawler with All Custom Dependencies in C#

Source: https://github.com/sjdirect/abot/blob/master/README.md

Shows how to provide custom implementations for all of PoliteWebCrawler's dependencies through its constructor. This allows for complete control over the crawling process by replacing default utility classes.

```c#
var crawler = new PoliteWebCrawler(
    	new CrawlConfiguration(),
	new YourCrawlDecisionMaker(),
	new YourThreadMgr(), 
	new YourScheduler(), 
	new YourPageRequester(), 
	new YourHyperLinkParser(), 
	new YourMemoryManager(), 
    	new YourDomainRateLimiter(),
	new YourRobotsDotTextFinder());
```

--------------------------------

### Instantiate PoliteWebCrawler with Partial Custom Dependencies in C#

Source: https://github.com/sjdirect/abot/blob/master/README.md

Illustrates how to selectively replace PoliteWebCrawler's default dependencies. Passing 'null' for any parameter in the constructor will instruct the crawler to use its default implementation for that specific dependency.

```c#
var crawler = new PoliteWebCrawler(
	null, 
	null, 
    	null,
    	null,
	new YourPageRequester(), 
	new YourHyperLinkParser(), 
	null,
    	null, 
	null);
```

--------------------------------

### Configure Crawl Decision Callbacks in C#

Source: https://github.com/sjdirect/abot/blob/master/README.md

Demonstrates how to use shorthand delegate methods to add custom crawl decision logic for pages, content, and links without implementing full interfaces. These callbacks are executed after the ICrawlDecisionMaker's initial decision.

```c#
var crawler = new PoliteWebCrawler();

crawler.ShouldCrawlPageDecisionMaker = (pageToCrawl, crawlContext) => 
{
	var decision = new CrawlDecision{ Allow = true };
	if(pageToCrawl.Uri.Authority == "google.com")
		return new CrawlDecision{ Allow = false, Reason = "Dont want to crawl google pages" };
	
	return decision;
};

crawler.ShouldDownloadPageContentDecisionMaker = (crawledPage, crawlContext) =>
{
	var decision = new CrawlDecision{ Allow = true };
	if (!crawledPage.Uri.AbsoluteUri.Contains(".com"))
		return new CrawlDecision { Allow = false, Reason = "Only download raw page content for .com tlds" };

	return decision;
};

crawler.ShouldCrawlPageLinksDecisionMaker = (crawledPage, crawlContext) =>
{
	var decision = new CrawlDecision{ Allow = true };
	if (crawledPage.Content.Bytes.Length < 100)
		return new CrawlDecision { Allow = false, Reason = "Just crawl links in pages that have at least 100 bytes" };

	return decision;
};
```

--------------------------------

### Configure Abot Crawl Settings

Source: https://github.com/sjdirect/abot/blob/master/README.md

Demonstrates how to initialize and set various configuration options for an Abot web crawler, including crawl timeout, maximum concurrent threads, pages to crawl, user agent, and custom extension values. These settings control the fundamental behavior of the crawling process.

```c#
var crawlConfig = new CrawlConfiguration();
crawlConfig.CrawlTimeoutSeconds = 100;
crawlConfig.MaxConcurrentThreads = 10;
crawlConfig.MaxPagesToCrawl = 1000;
crawlConfig.UserAgentString = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36";
crawlConfig.ConfigurationExtensions.Add("SomeCustomConfigValue1", "1111");
crawlConfig.ConfigurationExtensions.Add("SomeCustomConfigValue2", "2222");
etc...
```

--------------------------------

### C# IRobotsDotTextFinder Interface Definition

Source: https://github.com/sjdirect/abot/blob/master/README.md

The `IRobotsDotTextFinder` interface defines the contract for components responsible for locating and parsing `robots.txt` files. Its primary method, `Find`, takes a root URI and returns an `IRobotsDotText` abstraction, enabling crawlers to respect website crawling rules.

```C#
/// <summary>
/// Finds and builds the robots.txt file abstraction
/// </summary>
public interface IRobotsDotTextFinder
{
	/// <summary>
	/// Finds the robots.txt file using the rootUri. 
        /// 
	IRobotsDotText Find(Uri rootUri);
}
```

```APIDOC
IRobotsDotTextFinder Interface:
  Description: Finds and builds the robots.txt file abstraction.
  Methods:
    Find(rootUri: Uri): IRobotsDotText
      Description: Finds the robots.txt file using the rootUri.
      Parameters:
        rootUri:
          Type: Uri
          Description: The root URI of the domain to find robots.txt for.
      Returns:
        Type: IRobotsDotText
        Description: An abstraction of the robots.txt file.
```

--------------------------------

### IMemoryManager Interface for Memory Monitoring (C#)

Source: https://github.com/sjdirect/abot/blob/master/README.md

The IMemoryManager interface provides functionalities for monitoring memory usage within the crawling process. This feature is currently experimental and helps in determining if the process is exceeding specified memory limits or if sufficient memory is available.

```c#
/// <summary>
/// Handles memory monitoring/usage
/// </summary>
public interface IMemoryManager : IMemoryMonitor, IDisposable
{
	/// <summary>
	/// Whether the current process that is hosting this instance is allocated/using above the param value of memory in mb
	/// </summary>
	bool IsCurrentUsageAbove(int sizeInMb);

	/// <summary>
	/// Whether there is at least the param value of available memory in mb
	/// </summary>
	bool IsSpaceAvailable(int sizeInMb);
}
```

--------------------------------

### IPageRequester Interface for Making HTTP Requests (C#)

Source: https://github.com/sjdirect/abot/blob/master/README.md

The IPageRequester interface defines the contract for making raw HTTP requests. It is a fundamental component for fetching web pages during the crawling process, allowing for content download based on URI or a custom decision function.

```c#
public interface IPageRequester : IDisposable
{
	/// <summary>
	/// Make an http web request to the url and download its content
	/// </summary>
	Task<CrawledPage> MakeRequestAsync(Uri uri);

	/// <summary>
	/// Make an http web request to the url and download its content based on the param func decision
	/// </summary>
	Task<CrawledPage> MakeRequestAsync(Uri uri, Func<CrawledPage, CrawlDecision> shouldDownloadContent);
}
```

--------------------------------

### IDomainRateLimiter Interface for Domain-Specific Throttling (C#)

Source: https://github.com/sjdirect/abot/blob/master/README.md

The IDomainRateLimiter interface manages rate limiting on a per-domain basis, ensuring that HTTP requests to a specific domain adhere to a minimum crawl delay. It helps prevent overwhelming target servers during crawling.

```c#
/// <summary>
/// Rate limits or throttles on a per domain basis
/// </summary>
public interface IDomainRateLimiter
{
	/// <summary>
	/// If the domain of the param has been flagged for rate limiting, it will be rate limited according to the configured minimum crawl delay
	/// </summary>
	void RateLimit(Uri uri);

	/// <summary>
	/// Add a domain entry so that domain may be rate limited according the the param minumum crawl delay
	/// </summary>
	void AddDomain(Uri uri, long minCrawlDelayInMillisecs);
}
```

--------------------------------

### Access Custom Objects from Abot Crawl/Page Bag

Source: https://github.com/sjdirect/abot/blob/master/README.md

Demonstrates how to retrieve custom objects from the `CrawlBag` within an event handler and how to add dynamic values to the `PageBag` for individual pages. This allows for page-specific data to be stored and accessed during the crawl.

```c#
void crawler_ProcessPageCrawlStarting(object sender, PageCrawlStartingArgs e)
{
    //Get your Foo instances from the CrawlContext object
    var foo1 = e.CrawlConext.CrawlBag.MyFoo1;
    var foo2 = e.CrawlConext.CrawlBag.MyFoo2;

    //Also add a dynamic value to the PageToCrawl or CrawledPage
    e.PageToCrawl.PageBag.Bar = new Bar();
}
```

--------------------------------

### IHyperLinkParser Interface for HTML Link Parsing (C#)

Source: https://github.com/sjdirect/abot/blob/master/README.md

The IHyperLinkParser interface is designed for extracting hyperlinks from raw HTML content. The default implementation leverages AngleSharp for robust HTML parsing, allowing for CSS-style selectors similar to jQuery.

```c#
/// <summary>
/// Handles parsing hyperlikns out of the raw html
/// </summary>
public interface IHyperLinkParser
{
	/// <summary>
	/// Parses html to extract hyperlinks, converts each into an absolute url
	/// </summary>
	IEnumerable<Uri> GetLinks(CrawledPage crawledPage);
}
```

--------------------------------

### Add Custom Objects to Abot Crawl Bag

Source: https://github.com/sjdirect/abot/blob/master/README.md

Illustrates how to add custom dynamic objects to the `CrawlBag` of an Abot crawler instance. These objects become accessible throughout the crawl context, allowing for persistent data sharing across different crawl stages and pages.

```c#
var crawler crawler = new PoliteWebCrawler();
crawler.CrawlBag.MyFoo1 = new Foo();
crawler.CrawlBag.MyFoo2 = new Foo();
crawler.PageCrawlStarting += crawler_ProcessPageCrawlStarting;
...
```

--------------------------------

### IScheduler Interface for Managing Crawl Queues (C#)

Source: https://github.com/sjdirect/abot/blob/master/README.md

The IScheduler interface is responsible for managing the pages that need to be crawled. It acts as a central component for the crawler to submit found links and retrieve the next pages to process. Custom implementations can be used for distributed crawling scenarios.

```c#
/// <summary>
/// Handles managing the priority of what pages need to be crawled
/// </summary>
public interface IScheduler
{
	/// <summary>
	/// Count of remaining items that are currently scheduled
	/// </summary>
	int Count { get; }

	/// <summary>
	/// Schedules the param to be crawled
	/// </summary>
	void Add(PageToCrawl page);

	/// <summary>
	/// Schedules the param to be crawled
	/// </summary>
	void Add(IEnumerable<PageToCrawl> pages);

	/// <summary>
	/// Gets the next page to crawl
	/// </summary>
	PageToCrawl GetNext();

	/// <summary>
	/// Clear all currently scheduled pages
	/// </summary>
	void Clear();
}
```

--------------------------------

### ICrawlDecisionMaker Interface API Documentation

Source: https://github.com/sjdirect/abot/blob/master/README.md

API definition for the ICrawlDecisionMaker interface, which is responsible for determining whether a page should be crawled, its content downloaded, and its links processed. This interface is central to controlling the crawler's behavior.

```APIDOC
interface ICrawlDecisionMaker:
  ShouldCrawlPage(pageToCrawl: PageToCrawl, crawlContext: CrawlContext) -> CrawlDecision
    Decides whether the page should be crawled
  ShouldCrawlPageLinks(crawledPage: CrawledPage, crawlContext: CrawlContext) -> CrawlDecision
    Decides whether the page's links should be crawled
  ShouldDownloadPageContent(crawledPage: CrawledPage, crawlContext: CrawlContext) -> CrawlDecision
    Decides whether the page's content should be dowloaded
```

--------------------------------

### IThreadManager Interface API Documentation

Source: https://github.com/sjdirect/abot/blob/master/README.md

API definition for the IThreadManager interface, which handles the multithreading aspects of the crawler. It manages concurrent HTTP requests and provides methods for controlling thread execution.

```APIDOC
interface IThreadManager:
  MaxThreads: int
    Max number of threads to use.
  DoWork(action: Action) -> void
    Will perform the action asynchrously on a seperate thread
    action: The action to perform
  HasRunningThreads() -> bool
    Whether there are running threads
  AbortAll() -> void
    Abort all running threads
```

--------------------------------

### Cancel Abot Crawl Operation

Source: https://github.com/sjdirect/abot/blob/master/README.md

Shows how to use a `CancellationTokenSource` to enable cancellation for an asynchronous Abot crawl operation. This mechanism allows for graceful termination of the crawl process, preventing resource leaks and ensuring controlled shutdown.

```c#
CancellationTokenSource cancellationTokenSource = new CancellationTokenSource();

var crawler = new PoliteWebCrawler();
var result = await crawler.CrawlAsync(new Uri("addurihere"), cancellationTokenSource);
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.