### Extract News Content Source: https://github.com/crawlscript/webcollector/blob/master/README.md WebCollector can automatically extract news content from HTML or by URL. Use these methods to get News objects or just the content string. ```java News news = ContentExtractor.getNewsByHtml(html, url); News news = ContentExtractor.getNewsByHtml(html); News news = ContentExtractor.getNewsByUrl(url); String content = ContentExtractor.getContentByHtml(html, url); String content = ContentExtractor.getContentByHtml(html); String content = ContentExtractor.getContentByUrl(url); Element contentElement = ContentExtractor.getContentElementByHtml(html, url); Element contentElement = ContentExtractor.getContentElementByHtml(html); Element contentElement = ContentExtractor.getContentElementByUrl(url); ``` -------------------------------- ### Mounting Plugins to a Crawler Source: https://github.com/crawlscript/webcollector/blob/master/README.md Demonstrates how to set various plugin types for a WebCollector crawler instance. Ensure the correct plugin instances are provided. ```java crawler.setRequester(xxxxx); crawler.setVisitor(xxxxx); crawler.setNextFilter(xxxxx); crawler.setGeneratorFilter(xxxxx); crawler.setExecutor(xxxxx); crawler.setDBManager(xxxxx); ``` -------------------------------- ### Custom Crawler Configuration Source: https://github.com/crawlscript/webcollector/blob/master/README.md Demonstrates how to create and apply a custom configuration to a crawler instance. This ensures that configuration changes only affect the specified crawler and do not alter the default configuration used by other crawlers. Use Configuration.copyDefault() to inherit necessary default settings. ```java Configuration conf = Configuration.copyDefault(); conf.set("test_string_key", "test_string_value"); crawler.getConf().setReadTimeout(1000 * 5); crawler.setConf(conf); crawler.getConf().set("test_int_key", 10); crawler.getConf().setConnectTimeout(1000 * 5); ``` -------------------------------- ### Manual News Crawler Configuration and Execution Source: https://github.com/crawlscript/webcollector/blob/master/README.md This Java code sets up a BreadthCrawler to crawl news articles from the GitHub blog. It demonstrates adding seed URLs, setting crawl depth, configuring thread count, and defining custom extraction logic within the visit method. ```Java import cn.edu.hfut.dmic.webcollector.model.CrawlDatums; import cn.edu.hfut.dmic.webcollector.model.Page; import cn.edu.hfut.dmic.webcollector.plugin.rocks.BreadthCrawler; /** * Crawling news from github news * * @author hu */ public class DemoManualNewsCrawler extends BreadthCrawler { /** * @param crawlPath crawlPath is the path of the directory which maintains * information of this crawler * @param autoParse if autoParse is true,BreadthCrawler will auto extract * links which match regex rules from pag */ public DemoManualNewsCrawler(String crawlPath, boolean autoParse) { super(crawlPath, autoParse); // add 5 start pages and set their type to "list" //"list" is not a reserved word, you can use other string instead this.addSeedAndReturn("https://blog.github.com/").type("list"); for(int pageIndex = 2; pageIndex <= 5; pageIndex++) { String seedUrl = String.format("https://blog.github.com/page/%d/", pageIndex); this.addSeed(seedUrl, "list"); } setThreads(50); getConf().setTopN(100); //enable resumable mode //setResumable(true); } @Override public void visit(Page page, CrawlDatums next) { String url = page.url(); if (page.matchType("list")) { /*if type is "list"*/ /*detect content page by css selector and mark their types as "content"*/ next.add(page.links("h1.lh-condensed>a")).type("content"); }else if(page.matchType("content")) { /*if type is "content"*/ /*extract title and content of news by css selector*/ String title = page.select("h1[class=lh-condensed]").first().text(); String content = page.selectText("div.content.markdown-body"); //read title_prefix and content_length_limit from configuration title = getConf().getString("title_prefix") + title; content = content.substring(0, getConf().getInteger("content_length_limit")); System.out.println("URL:\n" + url); System.out.println("title:\n" + title); System.out.println("content:\n" + content); } } public static void main(String[] args) throws Exception { DemoManualNewsCrawler crawler = new DemoManualNewsCrawler("crawl", false); crawler.getConf().setExecuteInterval(5000); crawler.getConf().set("title_prefix","PREFIX_"); crawler.getConf().set("content_length_limit", 20); /*start crawl with depth of 4*/ crawler.start(4); } } ``` -------------------------------- ### Automatic News Crawler Implementation Source: https://github.com/crawlscript/webcollector/blob/master/README.md This Java code sets up a BreadthCrawler to automatically parse and extract links from GitHub's blog. It defines seed URLs, regex patterns for inclusion and exclusion, and a visit method to extract news titles and content. Use this for crawling structured content from websites where link discovery is automated. ```java import cn.edu.hfut.dmic.webcollector.model.CrawlDatums; import cn.edu.hfut.dmic.webcollector.model.Page; import cn.edu.hfut.dmic.webcollector.plugin.rocks.BreadthCrawler; /** * Crawling news from github news * * @author hu */ public class DemoAutoNewsCrawler extends BreadthCrawler { /** * @param crawlPath crawlPath is the path of the directory which maintains * information of this crawler * @param autoParse if autoParse is true,BreadthCrawler will auto extract * links which match regex rules from pag */ public DemoAutoNewsCrawler(String crawlPath, boolean autoParse) { super(crawlPath, autoParse); /*start pages*/ this.addSeed("https://blog.github.com/"); for(int pageIndex = 2; pageIndex <= 5; pageIndex++) { String seedUrl = String.format("https://blog.github.com/page/%d/", pageIndex); this.addSeed(seedUrl); } /*fetch url like "https://blog.github.com/2018-07-13-graphql-for-octokit/" */ this.addRegex("https://blog.github.com/[0-9]{4}-[0-9]{2}-[0-9]{2}-[^/]+/"); /*do not fetch jpg|png|gif*/ //this.addRegex("-.*\\.(jpg|png|gif).*\); /*do not fetch url contains #*/ //this.addRegex("-.*#.*\); setThreads(50); getConf().setTopN(100); //enable resumable mode //setResumable(true); } @Override public void visit(Page page, CrawlDatums next) { String url = page.url(); /*if page is news page*/ if (page.matchUrl("https://blog.github.com/[0-9]{4}-[0-9]{2}-[0-9]{2}[^/]+/")) { /*extract title and content of news by css selector*/ String title = page.select("h1[class=lh-condensed]").first().text(); String content = page.selectText("div.content.markdown-body"); System.out.println("URL:\n" + url); System.out.println("title:\n" + title); System.out.println("content:\n" + content); /*If you want to add urls to crawl,add them to nextLink*/ /*WebCollector automatically filters links that have been fetched before*/ /*If autoParse is true and the link you add to nextLinks does not match the regex rules,the link will also been filtered.*/ //next.add("http://xxxxxx.com"); } } public static void main(String[] args) throws Exception { DemoAutoNewsCrawler crawler = new DemoAutoNewsCrawler("crawl", true); /*start crawl with depth of 4*/ crawler.start(4); } } ``` -------------------------------- ### Add Detected URLs to CrawlDatum Source: https://github.com/crawlscript/webcollector/blob/master/README.md Demonstrates various ways to add single or multiple detected URLs to the `next` container in WebCollector's visit or execute methods. Includes setting URL type and metadata. ```java //add one detected URL next.add("detected URL"); ``` ```java //add one detected URL and set its type next.add("detected URL", "type"); ``` ```java //add one detected URL next.add(new CrawlDatum("detected URL")); ``` ```java //add detected URLs next.add("detected URL list"); ``` ```java //add detected URLs next.add(("detected URL list","type")); ``` ```java //add detected URLs next.add(new CrawlDatums("detected URL list")); ``` ```java //add one detected URL and return the added URL(CrawlDatum) //and set its key and type next.addAndReturn("detected URL").key("key").type("type"); ``` ```java //add detected URLs and return the added URLs(CrawlDatums) //and set their type and meta info next.addAndReturn("detected URL list").type("type").meta("page_num",10); ``` ```java //add detected URL and return next //and modify the type and meta info of all the CrawlDatums in next, //including the added URL next.add("detected URL").type("type").meta("page_num", 10); ``` ```java //add detected URLs and return next //and modify the type and meta info of all the CrawlDatums in next, //including the added URLs next.add("detected URL list").type("type").meta("page_num", 10); ``` -------------------------------- ### Enable Resumable Crawling Source: https://github.com/crawlscript/webcollector/blob/master/README.md To enable resumable crawling, set the resumable option to true. This prevents the crawler from deleting history, allowing it to resume from where it left off. ```java crawler.setResumable(true); ``` -------------------------------- ### Maven Dependency for WebCollector Source: https://github.com/crawlscript/webcollector/blob/master/README.md Add this dependency to your Maven project to include WebCollector. ```xml cn.edu.hfut.dmic.webcollector WebCollector 2.73-alpha ``` === COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.