### Bash: Cheburcheck Systemd Automated Scanning Source: https://context7.com/lowderplay/cheburcheck/llms.txt This section guides users on setting up automated regular scans using Systemd on Debian-based systems. It covers installing the .deb package, configuring the API key, checking the status of the timer units for different scan frequencies, viewing logs using `journalctl`, manually starting a service, and locating the saved reports. ```bash # Установка .deb пакета sudo apt install ./cheburchecker_0.1.2_amd64.deb # Настройка API-ключа echo "AGENCY_KEY=your_api_key_here" | sudo tee /etc/default/cheburchecker # Проверка статуса таймеров systemctl status cheburchecker.100k.timer # каждые 6 часов, 100k доменов systemctl status cheburchecker.1kk.timer # ежедневно, 1M доменов # Просмотр логов journalctl -u cheburchecker@100000.service -f journalctl -u cheburchecker@1000000.service -f # Ручной запуск сервиса sudo systemctl start cheburchecker@100000.service # Отчёты сохраняются в: ls /var/log/cheburchecker/ # report-100000.csv # report-1000000.csv ``` -------------------------------- ### Bash Environment Variables for Cheburcheck Web Server Source: https://context7.com/lowderplay/cheburcheck/llms.txt Configures the Cheburcheck web server using environment variables. Includes mandatory variables like DATABASE_URL and optional settings for database intervals, CDN sources, and RKN lists. Assumes `cargo run --release` is used to start the server. ```bash # Обязательные переменные export DATABASE_URL="postgres://user:password@localhost/cheburcheck" # Опциональные переменные export DATABASE_INTERVAL_SECONDS=21600 # интервал обновления БД (6 часов) # Кастомные источники данных export CDN_SOURCE="https://example.com/cdn-list.csv" export RKN_NETS="https://example.com/blocked-nets.lst" export RKN_DOMAINS="https://example.com/blocked-domains.lst" # Запуск сервера cargo run --release ``` -------------------------------- ### Bash: Cheburcheck Reporter CLI Advanced Configuration Source: https://context7.com/lowderplay/cheburcheck/llms.txt This section details advanced configuration options for the Cheburcheck Reporter CLI. It includes parameters for testing plain HTTP connections, sending junk data for DPI testing, using custom IP servers, employing a single domain for all requests (fake mode), and sending reports to a custom endpoint. A combined example shows aggressive scanning settings. ```bash # Тестирование через plain HTTP (без TLS) cheburchecker -H -c 10000 output.csv # Отправка junk-данных (для тестирования DPI) cheburchecker -x -c 10000 output.csv # Использование кастомного IP-сервера cheburchecker -i 1.2.3.4 -P test.bin output.csv # Использование одного домена для всех запросов (fake-режим) cheburchecker -f google.com -c 1000 output.csv # Отправка на кастомный endpoint cheburchecker -a "https://custom-server.com/api/report" -k API_KEY # Комбинированный пример: агрессивное сканирование cheburchecker \ -c 500000 \ -p 3000 \ -t 3 \ -r 1 \ -v block \ -k YOUR_API_KEY ``` -------------------------------- ### Whitelist API: Get Distribution Histogram Source: https://context7.com/lowderplay/cheburcheck/llms.txt Endpoint to retrieve statistics on domain distribution by rank in JSON format. ```APIDOC ## GET /whitelist/histogram ### Description Endpoint to retrieve statistics on domain distribution by rank in JSON format. ### Method GET ### Endpoint `/whitelist/histogram` ### Parameters #### Query Parameters - **limit** (integer) - Optional - The maximum number of entries to return. - **filter** (boolean) - Optional - If true, filters the results. ### Response #### Success Response (200) Returns a JSON array of objects, where each object represents a bin and its count: - **bin** (integer) - The rank bin. - **count** (integer) - The number of domains in that bin. ### Request Example ```bash # Basic histogram request curl "https://cheburcheck.ru/whitelist/histogram" # With a limit curl "https://cheburcheck.ru/whitelist/histogram?limit=50000" # With filtering and limit curl "https://cheburcheck.ru/whitelist/histogram?filter=true&limit=100000" ``` ### Response Example ```json [ {"bin": 0, "count": 1234}, {"bin": 1, "count": 5678} ] ``` ``` -------------------------------- ### Get Whitelist Distribution Histogram (Whitelist API) Source: https://context7.com/lowderplay/cheburcheck/llms.txt Endpoint to retrieve statistics on domain distribution by rank in JSON format. Supports basic requests, limiting results, and filtering. ```bash # Базовый запрос гистограммы curl "https://cheburcheck.ru/whitelist/histogram" # С ограничением количества curl "https://cheburcheck.ru/whitelist/histogram?limit=50000" # С фильтрацией curl "https://cheburcheck.ru/whitelist/histogram?filter=true&limit=100000" # Пример ответа: # [ # {"bin": 0, "count": 1234}, # {"bin": 1, "count": 5678}, # ... # ] ``` -------------------------------- ### Upload Scan Report (Agency API) Source: https://context7.com/lowderplay/cheburcheck/llms.txt Endpoint for uploading mass scan results from Cheburcheck Reporter. Requires API key authorization and data transmission in MessagePack format. Includes example report structure and success response. ```bash # Загрузка отчёта с API-ключом curl -X POST "https://cheburcheck.ru/agency/report" \ -H "Content-Type: application/msgpack" \ -H "Authorization: Bearer YOUR_API_KEY" \ --data-binary @report.msgpack # Структура отчёта AgencyReport (в JSON для понимания): # { # "version": "0.1.2", # "config": { # "http": false, # "tx_junk": false, # "ip": "5.78.7.195", # "path": "100MB.bin", # "retry_count": 2, # "timeout_secs": 5, # "probe_count": 1000 # }, # "data": { # "google.com": "ok", # "blocked-site.com": "blocked", # "error-domain.com": "connect_error" # } # } # Успешный ответ: # {"ok": true, "id": 42} ``` -------------------------------- ### Work with Target for Domain, IP, and URL Representation (Rust) Source: https://context7.com/lowderplay/cheburcheck/llms.txt Illustrates how to create Target instances from various inputs (domain, IPv4, IPv6, URL) and retrieve their readable type or convert them back to queryable strings. It also shows how to resolve a domain's IP addresses. ```rust use querying::target::Target; use querying::resolver::Resolver; #[tokio::main] async fn main() { // Автоматическое определение типа цели let domain = Target::from("example.com"); let ipv4 = Target::from("192.168.1.1"); let ipv6 = Target::from("2001:db8::1"); let from_url = Target::from("https://example.com/path?query=1"); // Получение читаемого типа println!("{}", domain.readable_type()); // "Домен" println!("{}", ipv4.readable_type()); // "IPv4-адрес" println!("{}", ipv6.readable_type()); // "IPv6-адрес" // Преобразование обратно в строку запроса println!("{}", domain.to_query()); // "example.com" println!("{}", ipv4.to_query()); // "192.168.1.1" // Резолвинг через DoH let resolver = Resolver::new().await; match domain.resolve(&resolver).await { Ok(ips) => { for ip in ips { println!("Resolved IP: {}", ip); } } Err(e) => eprintln!("Resolve error: {:?}", e), } } ``` -------------------------------- ### Create and Use Checker for Domain/IP Checks (Rust) Source: https://context7.com/lowderplay/cheburcheck/llms.txt Demonstrates how to initialize the Checker, download and update blocklists, and perform checks on domains. It handles different check outcomes, including blocked status and errors. ```rust use querying::{Checker, Check, CheckVerdict, CheckError}; use querying::target::Target; #[tokio::main] async fn main() { // Создание нового экземпляра Checker let checker = Checker::new().await; // Загрузка всех баз данных match Checker::download_all().await { Ok(bases) => { checker.update_all(bases).await; println!("Базы данных обновлены: {:?}", checker.last_update()); } Err(e) => eprintln!("Ошибка загрузки: {}", e), } // Проверка домена let target = Target::from("example.com"); match checker.check(target).await { Ok(Check { verdict: CheckVerdict::Clear, geo, ips, rkn_subnets }) => { println!("Домен не заблокирован"); println!("IP-адреса: {:?}", ips); println!("Геолокация: {:?}", geo); } Ok(Check { verdict: CheckVerdict::Blocked { rkn_domain, cdn_provider_subnets }, .. }) => { println!("Домен заблокирован!"); if let Some(domain) = rkn_domain { println!("Найден в реестре РКН: {}", domain); } for (provider, subnets) in cdn_provider_subnets { println!("CDN провайдер {}: {:?}", provider, subnets); } } Err(CheckError::NotFound) => println!("Домен не найден (NXDOMAIN)"), Err(e) => eprintln!("Ошибка проверки: {:?}", e), } // Получение статистики println!("Всего доменов в базе: {}", checker.total_domains().await); println!("Всего IPv4 адресов: {}", checker.total_v4s().await); } ``` -------------------------------- ### TOML Configuration for Rocket Web Server Source: https://context7.com/lowderplay/cheburcheck/llms.txt Sets up the Rocket web server using a TOML configuration file. Defines the default address and port, and specifies the database connection URL for the 'cheburcheck' database. ```toml [default] address = "0.0.0.0" port = 8000 [default.databases.cheburcheck] url = "postgres://localhost/cheburcheck" ``` -------------------------------- ### Rust: Manage IP and Domain Blocklists Source: https://context7.com/lowderplay/cheburcheck/llms.txt This Rust code demonstrates how to use the `lists` module to manage CDN, Russian blacklist (RuBlacklist) of IPs and domains. It covers loading data from CSV and byte slices, checking for IP and domain containment, and retrieving statistics. Dependencies include the `querying` crate. ```rust use querying::lists::{CdnList, RuBlacklist, NetworkRecord}; use std::collections::VecDeque; use std::net::IpAddr; fn main() { // Работа с CDN списком let mut cdn_list = CdnList::new(); // Загрузка из CSV (provider,cidr,region) let csv_data = b"provider,cidr,region Cloudflare,104.16.0.0/12, Google,8.8.8.0/24,US"; cdn_list.update(&csv_data[..]).unwrap(); // Проверка IP на принадлежность к CDN let ip: IpAddr = "104.16.1.1".parse().unwrap(); if let Some(record) = cdn_list.contains(&ip) { println!("IP принадлежит CDN: {} ({})", record.provider, record.cidr); } // Работа со списком РКН let mut ru_blacklist = RuBlacklist::new(); // Загрузка IP-подсетей и доменов let nets = b"192.168.1.0/24\n10.0.0.0/8"; let domains = b"blocked-site.ru\nbanned-domain.com"; let custom = b"custom-blocked.ru"; ru_blacklist.update(&nets[..], &domains[..], &custom[..]).unwrap(); // Проверка домена if let Some(matched) = ru_blacklist.contains_domain("sub.blocked-site.ru") { println!("Домен заблокирован, совпадение: {}", matched); } // Проверка IP let ip: IpAddr = "192.168.1.100".parse().unwrap(); if let Some(subnet) = ru_blacklist.contains_ip(&ip) { println!("IP в заблокированной подсети: {}", subnet); } // Статистика println!("Доменов в базе: {}", ru_blacklist.domain_count); println!("IPv4 подсетей: {}", ru_blacklist.v4_count()); } ``` -------------------------------- ### View Service Logs (Bash) Source: https://github.com/lowderplay/cheburcheck/blob/master/reporter/README.md This command retrieves logs for the Cheburcheck service units using journalctl. It displays logs for specific service instances based on the provided domain count. ```bash journalctl -u cheburchecker@100000.service journalctl -u cheburchecker@1000000.service ``` -------------------------------- ### Bash: Cheburcheck Reporter CLI Basic Usage Source: https://context7.com/lowderplay/cheburcheck/llms.txt This section covers the basic command-line usage of the Cheburcheck Reporter CLI tool. It shows how to save results to a CSV file, send results to the Cheburcheck Agency using an API key, specify the number of domains to scan, and adjust the number of parallel requests and timeouts. Ensure `ulimit -n` is sufficiently high for increased parallel requests. ```bash # Сохранение результатов в CSV файл cheburchecker output.csv # Отправка результатов в Cheburcheck Agency cheburchecker -k YOUR_API_KEY # Сканирование с указанием количества доменов cheburchecker -c 50000 output.csv # Использование большего числа параллельных запросов # (убедитесь что ulimit -n достаточно высокий) cheburchecker -p 2000 -c 100000 output.csv # Сканирование с повышенным таймаутом cheburchecker -t 10 -r 3 output.csv # Отображение результатов в консоли cheburchecker -v all output.csv cheburchecker -v block output.csv # только заблокированные cheburchecker -v error output.csv # только ошибки ``` -------------------------------- ### Whitelist API: Export Full Whitelist Source: https://context7.com/lowderplay/cheburcheck/llms.txt Endpoint to export the whitelist of domains available despite blockages. Data is cached for 24 hours. ```APIDOC ## GET /whitelist/full.csv ### Description Endpoint to export the full whitelist of domains, including their rank and last check date. Data is cached for 24 hours. ### Method GET ### Endpoint `/whitelist/full.csv` ### Response #### Success Response (200) Returns a CSV file with the following columns: - **domain** (string) - The domain name. - **rank** (integer) - The rank of the domain. - **last_ok** (string) - The date and time of the last successful check in ISO 8601 format. ### Request Example ```bash curl "https://cheburcheck.ru/whitelist/full.csv" -o whitelist_full.csv ``` ### Response Example ```csv domain,rank,last_ok google.com,1,2024-01-15T12:00:00Z youtube.com,2,2024-01-15T11:30:00Z ``` ``` -------------------------------- ### Whitelist API: Export Domain List Source: https://context7.com/lowderplay/cheburcheck/llms.txt Endpoint to export only the list of whitelisted domain names. Data is cached for 24 hours. ```APIDOC ## GET /whitelist/domains.csv ### Description Endpoint to export only the list of whitelisted domain names. Data is cached for 24 hours. ### Method GET ### Endpoint `/whitelist/domains.csv` ### Response #### Success Response (200) Returns a CSV file with a single column: - **domain** (string) - The domain name. ### Request Example ```bash curl "https://cheburcheck.ru/whitelist/domains.csv" -o domains.csv ``` ### Response Example ```csv google.com youtube.com facebook.com ``` ``` -------------------------------- ### Export Whitelist to CSV (Whitelist API) Source: https://context7.com/lowderplay/cheburcheck/llms.txt Endpoints to export a whitelist of domains that are accessible despite blocklists. Data is cached for 24 hours. Supports full export with rank and last check date, or just domain names. ```bash # Полный экспорт с рейтингом и датой последней проверки curl "https://cheburcheck.ru/whitelist/full.csv" -o whitelist_full.csv # Пример содержимого full.csv: # domain,rank,last_ok # google.com,1,2024-01-15T12:00:00Z # youtube.com,2,2024-01-15T11:30:00Z # Только список доменов curl "https://cheburcheck.ru/whitelist/domains.csv" -o domains.csv # Пример содержимого domains.csv: # google.com # youtube.com # facebook.com ``` -------------------------------- ### Check Timer Status (Bash) Source: https://github.com/lowderplay/cheburcheck/blob/master/reporter/README.md This command allows you to check the status of the Cheburcheck timer units. It requires systemctl to be available on the system. ```bash systemctl status cheburchecker.100k.timer systemctl status cheburchecker.1kk.timer ``` -------------------------------- ### DNS Resolver via DoH for Domain Name Resolution (Rust) Source: https://context7.com/lowderplay/cheburcheck/llms.txt Shows how to use the Resolver module to perform DNS lookups using Quad9's DNS-over-HTTPS service. This bypasses local DNS restrictions and handles potential resolution errors like NXDOMAIN. ```rust use querying::resolver::{Resolver, ResolveError}; #[tokio::main] async fn main() { // Создание резолвера (использует Quad9 DoH) let resolver = Resolver::new().await; // Резолвинг домена в IP-адреса (IPv4 и IPv6) match resolver.lookup_ips("google.com").await { Ok(ips) => { for ip in ips { println!("IP: {}", ip); } } Err(ResolveError::NxDomain) => { println!("Домен не существует"); } Err(ResolveError::Other(e)) => { eprintln!("Ошибка резолвера: {}", e); } } } ``` -------------------------------- ### Check Domain or IP Address (Web API) Source: https://context7.com/lowderplay/cheburcheck/llms.txt Endpoint to check a domain or IP address against blocklists. Returns an HTML page with results, including blocked subnets, CDN providers, and geolocation. Supports domains, IPv4, IPv6, and URLs. ```bash # Проверка домена curl "https://cheburcheck.ru/check?target=example.com" # Проверка IPv4-адреса curl "https://cheburcheck.ru/check?target=8.8.8.8" # Проверка IPv6-адреса curl "https://cheburcheck.ru/check?target=2001:4860:4860::8888" # Проверка URL (извлекает домен автоматически) curl "https://cheburcheck.ru/check?target=https://example.com/path" ``` -------------------------------- ### Web API: Check Domain or IP Address Source: https://context7.com/lowderplay/cheburcheck/llms.txt Endpoint to check a domain or IP address against blocklists. Returns an HTML page with check results, including information about blocked subnets, CDN providers, and geolocation. ```APIDOC ## GET /check ### Description Endpoint to check a domain or IP address against blocklists. Returns an HTML page with check results. ### Method GET ### Endpoint `/check` ### Parameters #### Query Parameters - **target** (string) - Required - The domain name, IPv4 address, IPv6 address, or URL to check. ### Request Example ```bash # Check a domain curl "https://cheburcheck.ru/check?target=example.com" # Check an IPv4 address curl "https://cheburcheck.ru/check?target=8.8.8.8" # Check an IPv6 address curl "https://cheburcheck.ru/check?target=2001:4860:4860::8888" # Check a URL (domain is extracted automatically) curl "https://cheburcheck.ru/check?target=https://example.com/path" ``` ### Response #### Success Response (200) Returns an HTML page with the check results. #### Response Example (HTML content) ``` -------------------------------- ### Rust Data Structures for Agency Reports Source: https://context7.com/lowderplay/cheburcheck/llms.txt Defines data structures for reporting between Reporter and Agency API. Includes configuration for the reporter, evidence of scan results, and the overall agency report structure. Serializes the report into MessagePack format. ```rust use reports::{AgencyReport, ReporterConfig, Evidence}; use std::collections::HashMap; use std::net::IpAddr; fn main() { // Конфигурация репортера let config = ReporterConfig { http: false, // использовать HTTPS tx_junk: false, // не отправлять junk-данные ip: "5.78.7.195".parse().unwrap(), path: "100MB.bin".to_string(), retry_count: 2, timeout_secs: 5, probe_count: 1000, }; // Результаты сканирования let mut data = HashMap::new(); data.insert("google.com".to_string(), Evidence::Ok); data.insert("blocked-site.ru".to_string(), Evidence::Blocked); data.insert("timeout-domain.com".to_string(), Evidence::ConnectError); data.insert("error-domain.net".to_string(), Evidence::Error); // Формирование отчёта let report = AgencyReport { version: "0.1.2".to_string(), config, data, }; // Сериализация в MessagePack для отправки let msgpack_data = rmp_serde::to_vec(&report).unwrap(); println!("Report size: {} bytes", msgpack_data.len()); // Форматирование Evidence для логов println!("{}", Evidence::Ok); // "ok" println!("{}", Evidence::Blocked); // "blocked" println!("{}", Evidence::ConnectError); // "connect_error" println!("{}", Evidence::Error); // "unknown_error" } ``` -------------------------------- ### Submit Feedback (Web API) Source: https://context7.com/lowderplay/cheburcheck/llms.txt Endpoint for users to provide feedback on check results, indicating if a blocked resource is working on their network. Requires a UUID obtained from a previous check. ```bash # Сообщить что ресурс работает (UUID получается из результата проверки) curl -X POST "https://cheburcheck.ru/feedback/550e8400-e29b-41d4-a716-446655440000/true" # Сообщить что ресурс не работает curl -X POST "https://cheburcheck.ru/feedback/550e8400-e29b-41d4-a716-446655440000/false" ``` -------------------------------- ### Service Healthcheck (Web API) Source: https://context7.com/lowderplay/cheburcheck/llms.txt Endpoint to verify the service's operational status. Returns HTTP 200 OK if databases are loaded, or 500 if the service is still initializing. ```bash # Проверка здоровья сервиса curl -i "https://cheburcheck.ru/healthcheck" # Ожидаемый ответ при готовности: # HTTP/1.1 200 OK # OK # Ответ при загрузке баз данных: # HTTP/1.1 500 Internal Server Error # LOADING DATABASES ``` -------------------------------- ### Web API: Submit Feedback Source: https://context7.com/lowderplay/cheburcheck/llms.txt Endpoint for submitting user feedback on check results. Allows users to report if a blocked resource is working on their network. ```APIDOC ## POST /feedback/{uuid}/{status} ### Description Endpoint for submitting user feedback on check results. Allows users to report if a blocked resource is working on their network. ### Method POST ### Endpoint `/feedback/{uuid}/{status}` ### Parameters #### Path Parameters - **uuid** (string) - Required - The unique identifier obtained from a previous check result. - **status** (boolean) - Required - `true` if the resource is working, `false` if it is not. ### Request Example ```bash # Report that the resource is working curl -X POST "https://cheburcheck.ru/feedback/550e8400-e29b-41d4-a716-446655440000/true" # Report that the resource is not working curl -X POST "https://cheburcheck.ru/feedback/550e8400-e29b-41d4-a716-446655440000/false" ``` ``` -------------------------------- ### Agency API: Upload Scan Report Source: https://context7.com/lowderplay/cheburcheck/llms.txt Endpoint for uploading mass scan results from Cheburcheck Reporter. Requires API key authorization. Data is transmitted in MessagePack format. ```APIDOC ## POST /agency/report ### Description Endpoint for uploading mass scan results from Cheburcheck Reporter. Requires API key authorization. Data is transmitted in MessagePack format. ### Method POST ### Endpoint `/agency/report` ### Parameters #### Request Body - **Content-Type** (string) - Must be `application/msgpack`. - **Authorization** (string) - Required - Bearer token with your API key (e.g., `Bearer YOUR_API_KEY`). - **Data** (binary) - The scan report in MessagePack format. ### Request Example ```bash curl -X POST "https://cheburcheck.ru/agency/report" \ -H "Content-Type: application/msgpack" \ -H "Authorization: Bearer YOUR_API_KEY" \ --data-binary @report.msgpack ``` ### Request Body Structure (Conceptual JSON representation) ```json { "version": "0.1.2", "config": { "http": false, "tx_junk": false, "ip": "5.78.7.195", "path": "100MB.bin", "retry_count": 2, "timeout_secs": 5, "probe_count": 1000 }, "data": { "google.com": "ok", "blocked-site.com": "blocked", "error-domain.com": "connect_error" } } ``` ### Response #### Success Response (200) - **ok** (boolean) - Indicates if the report was successfully processed. - **id** (integer) - The ID assigned to the uploaded report. #### Response Example ```json {"ok": true, "id": 42} ``` ``` -------------------------------- ### Web API: Healthcheck Service Source: https://context7.com/lowderplay/cheburcheck/llms.txt Endpoint to check the service's operational status. Returns 200 OK if databases are loaded, or 500 if the service is still initializing. ```APIDOC ## GET /healthcheck ### Description Endpoint to check the operational status of the service. Returns 200 OK if databases are loaded, or 500 if the service is still initializing. ### Method GET ### Endpoint `/healthcheck` ### Response #### Success Response (200) - **Status** (string) - "OK" when the service is ready. #### Error Response (500) - **Status** (string) - "LOADING DATABASES" when the service is initializing. ### Request Example ```bash # Check service health curl -i "https://cheburcheck.ru/healthcheck" ``` ### Response Example ``` # Expected response when ready: HTTP/1.1 200 OK OK # Response when loading databases: HTTP/1.1 500 Internal Server Error LOADING DATABASES ``` ``` === COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.