### Run SQL Server Docker Container for DotnetSpider Development Source: https://github.com/dotnetcore/dotnetspider/blob/master/README.md Set up a SQL Server 2017 instance using Docker for DotnetSpider's development environment. This command starts a SQL Server container, maps port 1433, accepts the EULA, and sets a SA password. ```Shell docker run --name sqlserver -d -p 1433:1433 --restart always -e 'ACCEPT_EULA=Y' -e 'SA_PASSWORD=1qazZAQ!' mcr.microsoft.com/mssql/server:2017-latest ``` -------------------------------- ### Run MongoDB Docker Container for DotnetSpider Development Source: https://github.com/dotnetcore/dotnetspider/blob/master/README.md Set up a MongoDB instance using Docker, which can be optionally used by DotnetSpider for data storage. This command starts a MongoDB container and maps port 27017. ```Shell docker run --name mongo -d -p 27017:27017 --restart always mongo ``` -------------------------------- ### Run MySQL Docker Container for DotnetSpider Development Source: https://github.com/dotnetcore/dotnetspider/blob/master/README.md Set up a MySQL database instance using Docker for DotnetSpider's development environment. This command starts a MySQL 5.7 container, maps port 3306, and sets a root password for persistent storage. ```Shell docker run --name mysql -d -p 3306:3306 --restart always -e MYSQL_ROOT_PASSWORD=1qazZAQ! mysql:5.7 ``` -------------------------------- ### Run PostgreSQL Docker Container for DotnetSpider Development Source: https://github.com/dotnetcore/dotnetspider/blob/master/README.md Deploy a PostgreSQL database instance using Docker, which can be optionally used by DotnetSpider. This command starts a PostgreSQL container and maps port 5432 with a specified password. ```Shell docker run --name postgres -d -p 5432:5432 --restart always -e POSTGRES_PASSWORD=1qazZAQ! postgres ``` -------------------------------- ### Run HBase Docker Container for DotnetSpider Development Source: https://github.com/dotnetcore/dotnetspider/blob/master/README.md Deploy an HBase instance using Docker, which can be used by DotnetSpider for distributed data storage. This command starts an HBase container and maps several necessary ports. ```Shell docker run -d --restart always --name hbase -p 20550:8080 -p 8085:8085 -p 9090:9090 -p 9095:9095 -p 16010:16010 dajobe/hbase ``` -------------------------------- ### Call `facade` to Get New Market Items in JavaScript Source: https://github.com/dotnetcore/dotnetspider/blob/master/src/DotnetSpider.Sample/cat.html This `facade` call retrieves new market items by interacting with the `/mkt/ajax/getNewItems` server endpoint. It specifies query parameters like page ID, city ID, shop type, category ID, region ID, and an optional keyword to filter the results. ```JavaScript facade({entry:"app-midas/js/mkt.js", data: { query: { pageId: 1, cityId: 1, shopType: 10, categoryId: 113, regionId: 0, keyword: '' }, server: '/mkt/ajax/getNewItems' }}) ``` -------------------------------- ### Run Redis Docker Container for DotnetSpider Development Source: https://github.com/dotnetcore/dotnetspider/blob/master/README.md Deploy a Redis instance using Docker, which can be optionally used by DotnetSpider for caching or message queuing. This command starts a Redis container and maps port 6379. ```Shell docker run --name redis -d -p 6379:6379 --restart always redis ``` -------------------------------- ### Call `facade` for Picture Monitoring in JavaScript Source: https://github.com/dotnetcore/dotnetspider/blob/master/src/DotnetSpider.Sample/cat.html This `facade` call initializes a picture monitoring component. It specifies the page type as 'search-list' and a CSS selector to target image elements within shop listings, enabling tracking or analysis of these images. ```JavaScript facade({entry:"pic-monitor", data: { pageType: 'search-list', selector: '.shop-list .pic img' }}) ``` -------------------------------- ### Initialize Facade with Application Data Source: https://github.com/dotnetcore/dotnetspider/blob/master/src/DotnetSpider.Sample/cat.html This JavaScript call initializes a 'facade' object or function, likely a central entry point for an application, providing it with initial data including user ID, IP address, city ID, and an empty keyword. ```JavaScript facade({entry:"app-midas", data: { query: { userId: 0, userip: '222.189.82.62', cityId: 1, keyword: ''}}}) ``` -------------------------------- ### Call `facade` for Main Search with Shop IDs in JavaScript Source: https://github.com/dotnetcore/dotnetspider/blob/master/src/DotnetSpider.Sample/cat.html This snippet uses the `facade` function for a main search operation. It provides a city ID and a comprehensive list of specific shop IDs to filter and narrow down the search results, likely for displaying items from particular vendors. ```JavaScript facade({entry:"app-main-search", data: { cityId: 1, shopIDs: [ "YtPSMfqXJZp5Nr8c","isZAHlfHjag4yBFJ","G9v8TaIXsnVa1okY","k8V02S89mrjhvI5B","laWQv9hlwPqupYrl","k6qxl4KqAs2Q56CM","FqQZQKoSWt85FsMM","G6MIuSR2Af8MiWfE","H9qBVK3Jlk3GXNMK","l4D63BhAZT3VDSnM","Ha59pbYtfGVTmt1D","k3jGXLNuh0Bku9qf","k7aanAoYGwkSEx2v","Eel12KSAP6WLMiJD","FNawPmC2lRTrhLH4",] }}) ``` -------------------------------- ### Configure Myget Feed for DotnetSpider Beta Packages Source: https://github.com/dotnetcore/dotnetspider/blob/master/README.md To access the latest beta packages of DotnetSpider, add this Myget feed configuration to your project's NuGet sources. This allows NuGet to resolve and download pre-release versions of the library. ```HTML ``` -------------------------------- ### Call `facade` for Search Ad Suggestions in JavaScript Source: https://github.com/dotnetcore/dotnetspider/blob/master/src/DotnetSpider.Sample/cat.html This snippet invokes the `facade` function to fetch suggested advertisements. It targets the `/searchads/ajax/suggestads` endpoint and includes query parameters such as region, channel, category, page ID, and a list of shops to exclude from suggestions. ```JavaScript facade({entry:"app-searchads/js/suggestads.js", data: { query: { regionId: 0, channelId: 10, categoryId: 113, pageId: 1, cookieid: '', multiregionIds: '', adShops: [], notAdShops: [ "YtPSMfqXJZp5Nr8c", "isZAHlfHjag4yBFJ", "G9v8TaIXsnVa1okY", "k8V02S89mrjhvI5B", "laWQv9hlwPqupYrl", "k6qxl4KqAs2Q56CM", "FqQZQKoSWt85FsMM", "G6MIuSR2Af8MiWfE", "H9qBVK3Jlk3GXNMK", "l4D63BhAZT3VDSnM", "Ha59pbYtfGVTmt1D", "k3jGXLNuh0Bku9qf", "k7aanAoYGwkSEx2v", "Eel12KSAP6WLMiJD", "FNawPmC2lRTrhLH4" ], }, server: '/searchads/ajax/suggestads' }}) ``` -------------------------------- ### Integrate Baidu Analytics Tracking Script Source: https://github.com/dotnetcore/dotnetspider/blob/master/src/DotnetSpider.Sample/cat.html This JavaScript snippet initializes the Baidu Analytics tracking code (`_hmt`) and dynamically inserts the `hm.js` script into the document, enabling website traffic monitoring. ```JavaScript var _hmt = _hmt || []; (function() { var hm = document.createElement("script"); hm.src = "https://hm.baidu.com/hm.js?602b80cf8079ae6591966cc70a3940e7"; var s = document.getElementsByTagName("script")[0]; s.parentNode.insertBefore(hm, s); })(); ``` -------------------------------- ### Configure Neuron JavaScript Module Loader Source: https://github.com/dotnetcore/dotnetspider/blob/master/src/DotnetSpider.Sample/cat.html This JavaScript code configures the Neuron module loader, specifying the base path for modules, a list of already loaded modules with their versions, and a dependency graph for various JavaScript libraries and application components. ```JavaScript neuron.config({path: "//www.dpfile.com/mod", loaded:["app-midas@0.6.12","app-midas@0.6.12/js/mkt.js","app-main-search@1.3.0","pic-monitor@0.1.3","cookie@0.2.0","css-shop-tag@1.0.1","map-iframe-dp@0.1.1","main-authbox@1.0.9","css-rating@1.0.2","midas-log@0.0.3","request@0.2.8","css-shop-tag@0.1.3","tpl@0.2.1","switch@0.1.0","hippo@1.2.28","placeholder@0.1.1","easy-login@0.4.41","cookie@0.1.2","mbox@1.0.4","request@1.0.8","mix@1.0.1","fx@2.0.0","attributes@1.4.1","class@2.0.5","lang@1.0.1","hexrgb@0.0.1","jquery@1.9.2","underscore@1.5.3","events@1.0.5","clone@0.1.13","util@1.0.5","json@1.0.1"], graph:{"1":["0.1.3"],"3":["1.9.2"],"6":["1.0.1"],"5":["1.0.5",{"json@~1.0.0":6}],"4":["1.0.5",{"util@~1.0.0":5}],"7":["0.0.1"],"8":["1.0.1"],"2":["2.0.0",{"jquery@~1.9.2":3,"events@~1.0.5":4,"hexrgb@~0.0.1":7,"lang@~1.0.0":8,"util@~1.0.4":5}],"9":["1.2.28"],"10":["0.0.3",{"jquery@^1.9.2":3}],"12":["1.0.2"],"14":["0.1.13",{"util@~1.0.2":5}],"15":["1.5.3"],"13":["2.0.5",{"clone@~0.1.11":14,"events@~1.0.0":4,"underscore@~1.5.2":15,"util@~1.0.0":5}],"11":["0.2.8",{"json@~1.0.0":6,"lang@~1.0.0":8,"asset@~1.0.0":12,"class@~2.0.0":13}],"17":["1.4.1",{"clone@~0.1.11":14,"util@~1.0.0":5}],"18":["1.0.1"],"16":["0.1.0",{"jquery@~1.9.2":3,"fx@~2.0.0":2,"events@~1.0.5":4,"attributes@~1.4.0":17,"clone@~0.1.12":14,"mix@~1.0.1":18,"underscore@~1.5.3":15,"util@~1.0.4":5}],"19":["0.2.1"],"0":["0.6.12",{"css-shop-tag@~0.1.0":1,"fx@^2.0.0":2,"hippo@~1.2.2":9,"jquery@~1.9.2":3,"json@~1.0.1":6,"midas-log@~0.0.0":10,"request@~0.2.4":11,"switch@~0.1.0":16,"tpl@~0.2.1":19}],"21":["0.2.0"],"22":["1.0.2"],"23":["1.0.1"],"25":["0.1.2"],"26":["1.0.4",{"class@~2.0.5":13,"jquery@~1.9.2":3,"lang@~1.0.0":8}],"27":["1.0.8",{"class@~2.0.0":13,"asset@~1.0.0":12,"json@~1.0.0":6,"lang@~1.0.0":8}],"28":["0.4.41"],"29":["0.1.1",{"jquery@~1.9.2":3}],"24":["1.0.9",{"jquery@~1.9.2":3,"cookie@~0.1.0":25,"mbox@~1.0.1":26,"request@~1.0.7":27,"easy-login@~0.4.39":28,"placeholder@~0.1.1":29}],"30":["0.1.1",{"jquery@~1.9.2":3}],"20":["1.3.0",{"class@^2.0.5":13,"cookie@~0.2.0":21,"css-rating@^1.0.0":22,"css-shop-tag@^1.0.1":23,"hippo@^1.2.28":9,"jquery@~1.9.2":3,"main-authbox@^1.0.0":24,"map-iframe-dp@~0.1.0":30,"mbox@^1.0.0":26,"request@~0.2.0":11,"tpl@~0.2.1":19}],"31":["0.1.3"],"_":{"app-midas@*":0,"app-main-search@*":20,"pic-monitor@*":31}}}) ``` -------------------------------- ### Run RabbitMQ Docker Container for DotnetSpider Development Source: https://github.com/dotnetcore/dotnetspider/blob/master/README.md Deploy a RabbitMQ message broker with management plugin using Docker. This is used by DotnetSpider for message queuing, mapping various ports and setting default user credentials. ```Shell docker run -d --restart always --name rabbimq -p 4369:4369 -p 5671-5672:5671-5672 -p 25672:25672 -p 15671-15672:15671-15672 \ -e RABBITMQ_DEFAULT_USER=user -e RABBITMQ_DEFAULT_PASS=password \ rabbitmq:3-management ``` -------------------------------- ### Implement Configurable Entity Spider for Cnblogs News in C# Source: https://github.com/dotnetcore/dotnetspider/blob/master/README.md This C# code defines an 'EntitySpider' subclass, extending DotnetSpider's 'Spider' to crawl Cnblogs news. It demonstrates how to initialize the spider, add data flows for parsing 'CnblogsEntry' entities, and add initial requests. The 'CnblogsEntry' class showcases how to define the structure of extracted data using attributes for selectors, global value selectors, and formatters. ```C# [DisplayName("博客园爬虫")] public class EntitySpider( IOptions options, DependenceServices services, ILogger logger) : Spider(options, services, logger) { public static async Task RunAsync() { var builder = Builder.CreateDefaultBuilder(options => { options.Speed = 1; }); builder.UseSerilog(); builder.IgnoreServerCertificateError(); await builder.Build().RunAsync(); } protected override async Task InitializeAsync(CancellationToken stoppingToken = default) { AddDataFlow>(); AddDataFlow(GetDefaultStorage); await AddRequestsAsync( new Request( "https://news.cnblogs.com/n/page/1", new Dictionary { { "网站", "博客园" } })); } [Schema("cnblogs", "news")] [EntitySelector(Expression = ".//div[@class='news_block']", Type = SelectorType.XPath)] [GlobalValueSelector(Expression = ".//a[@class='current']", Name = "类别", Type = SelectorType.XPath)] [GlobalValueSelector(Expression = "//title", Name = "Title", Type = SelectorType.XPath)] [FollowRequestSelector(Expressions = ["//div[@class='pager']"])] public class CnblogsEntry : EntityBase { protected override void Configure() { HasIndex(x => x.Title); HasIndex(x => new { x.WebSite, x.Guid }, true); } public int Id { get; set; } [Required] [StringLength(200)] [ValueSelector(Expression = "类别", Type = SelectorType.Environment)] public string Category { get; set; } [Required] [StringLength(200)] [ValueSelector(Expression = "网站", Type = SelectorType.Environment)] public string WebSite { get; set; } [StringLength(200)] [ValueSelector(Expression = "Title", Type = SelectorType.Environment)] [ReplaceFormatter(NewValue = "", OldValue = " - 博客园")] public string Title { get; set; } [StringLength(40)] [ValueSelector(Expression = "GUID", Type = SelectorType.Environment)] public string Guid { get; set; } [ValueSelector(Expression = ".//h2[@class='news_entry']/a")] public string News { get; set; } [ValueSelector(Expression = ".//h2[@class='news_entry']/a/@href")] public string Url { get; set; } [ValueSelector(Expression = ".//div[@class='entry_summary']")] [TrimFormatter] public string PlainText { get; set; } [ValueSelector(Expression = "DATETIME", Type = SelectorType.Environment)] public DateTime CreationTime { get; set; } } } ``` -------------------------------- ### Detect and Prompt for Internet Explorer Browser Update Source: https://github.com/dotnetcore/dotnetspider/blob/master/src/DotnetSpider.Sample/cat.html This JavaScript function checks if the user's browser is Internet Explorer and if its version is less than 9. If true, it dynamically injects HTML elements to display a prompt encouraging the user to download a modern browser like Chrome or Firefox. ```JavaScript (function(){var h=navigator.userAgent;var i=navigator.appName;var b=i.indexOf("Microsoft Internet Explorer")!==-1;if(!b){return false}var d=/MSIE (\d+).0/g;var e=d.exec(h);if(e&&e.length&&e[1]<9){var j='
×
';var f=document.createElement("div");f.id="browser-update-ie";f.className="browser-update-ie";f.innerHTML=j;document.body.appendChild(f);var a=document.documentElement.clientWidth||document.body.clientWidth;var c=document.getElementById("browser-ie-con").offsetWidth;var g=(a-c)/2;document.getElementById("browser-ie-con").style.left=g+"px";document.getElementById("browser-close").attachEvent("onclick",function(){document.getElementById("browser-update-ie").style.display="none"},false)}})(); ``` -------------------------------- ### Run Docker Remote API Proxy for Mac Development Source: https://github.com/dotnetcore/dotnetspider/blob/master/README.md Set up a `socat` container to expose the Docker daemon socket as a TCP port (2376) on macOS. This allows remote Docker clients to connect to the local Docker daemon. ```Shell docker run -d --restart always --name socat -v /var/run/docker.sock:/var/run/docker.sock -p 2376:2375 bobrik/socat TCP4-LISTEN:2375,fork,reuseaddr UNIX-CONNECT:/var/run/docker.sock ``` -------------------------------- ### Redis Configuration for DotnetSpider Scheduler Source: https://github.com/dotnetcore/dotnetspider/blob/master/README.md This snippet provides recommended Redis configuration settings, specifically 'timeout 0' and 'tcp-keepalive 60', which are necessary when using the Redis scheduler with DotnetSpider to prevent connection issues. ```Redis Config timeout 0 tcp-keepalive 60 ``` === COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.