### Initialize RSSBook Application with ElysiaJS Source: https://context7.com/hackhtu/rssbook/llms.txt This snippet demonstrates how to initialize an RSSBook application instance with custom settings, including cache type, configuration, feed URLs, and metadata. The application is then started on port 8787. ```typescript import { RSSBookApp } from "./app"; import { Cache } from "@/utils"; const app = RSSBookApp({ cache: Cache.LRU_Cache, config: {}, enableFetchOnlineServer: true, feeds: ["https://github.blog/feed/"], meta: { description: "A simple RSS feed aggregator and reader.", title: "RSSBook", }, }) .listen(8787, (server) => { console.log(`Server running at ${server.hostname}:${server.port}`); }); ``` -------------------------------- ### Configure a Feed with Metadata and Handler in TypeScript Source: https://github.com/hackhtu/rssbook/blob/main/docs/how-to-create-a-feed.EN.md Shows a detailed configuration for a single Feed, including metadata like description, title, and maintainer, along with the handler function that defines the feed's data retrieval logic. This example uses the ElysiaJS framework. ```typescript export default new Source({ /* ...config */ }) .feed( { description: "Update on the latest news, and insights about ElysiaJS.", fulltext: true, language: "en-US", maintainer: { name: "RSSBook", }, title: "Latest News", withImage: "If-Present", }, app => app.get(path, handler, schema?), ); ``` -------------------------------- ### Create New Source CLI Command Source: https://github.com/hackhtu/rssbook/blob/main/docs/how-to-create-a-feed.EN.md This command initiates an interactive command-line interface (CLI) to guide the user through the process of creating a new 'Source' for a website within the RSSBook project. ```bash bun run source:new ``` -------------------------------- ### Define Route Parameter Schema with TypeBox (TypeScript) Source: https://github.com/hackhtu/rssbook/blob/main/docs/how-to-create-a-feed.EN.md This snippet demonstrates how to define a schema for route parameters using the `t` tool, which is built on TypeBox. It's used to validate request parameters and response data in OpenAPI documentation. The example shows defining a `category` parameter for a GET request, specifying it as a union of specific string values ('news', 'sports', 'entertainment') and including a description and example. This ensures that the API correctly handles and documents dynamic route parameters. ```typescript import { Source, t } from "@/utils"; app.get( "/category/:category", handler, { // Define the schema of the parameter params: t.Object({ category: t.UnionEnum(["news", "sports", "entertainment"], { description: "The category of articles to fetch.", example: ["news"], }), }), } ) ``` -------------------------------- ### Fetch and Parse External Feed with Bash Source: https://context7.com/hackhtu/rssbook/llms.txt Fetches an external RSS, Atom, or JSON feed and parses it into a standardized format. This is done via a GET request to the /utils/fetch endpoint. ```bash curl -X GET "http://localhost:8787/utils/fetch?url=https://www.ruanyifeng.com/blog/atom.xml" ``` -------------------------------- ### Parsing HTML with RSSBook load Function Source: https://github.com/hackhtu/rssbook/blob/main/docs/how-to-create-a-feed.EN.md Details the `load` function, which uses `cheerio` to parse HTML content and enables jQuery-like syntax for querying and manipulation. It shows examples of extracting text, attributes, and HTML content, as well as filtering elements. ```typescript const html = await ofetch(`https://${domain}/category/${category}`); const $ = load(html); const title = $("title").text(); // Access text content const href = $("a").attr("href"); // Access attribute const items = $("article").html(); // Access HTML content $("div.article") .toArray() // Convert to array .map((elem) => { const $elem = $(elem); const title = $elem.find("h2.title").text(); return { title }; }) .filter((item) => { return !!item.title; }); ``` -------------------------------- ### Sort Feed Items with Bash Source: https://context7.com/hackhtu/rssbook/llms.txt Sorts feed items by date in ascending or descending order. This is achieved by a GET request to the /utils/sort endpoint. ```bash curl -X GET "http://localhost:8787/utils/sort?feed=https://www.ruanyifeng.com/blog/atom.xml&order=desc" ``` -------------------------------- ### Union (Merge) Multiple Feeds with Bash Source: https://context7.com/hackhtu/rssbook/llms.txt Merges multiple RSS/Atom feeds into a single unified feed, removing duplicates. This is achieved by making a GET request to the /utils/union endpoint. ```bash curl -X GET "http://localhost:8787/utils/union?feeds=https://www.ruanyifeng.com/blog/atom.xml&feeds=https://github.blog/feed/&override=%7B%22title%22%3A%22Combined%20Tech%20News%22%2C%22description%22%3A%22Merged%20feed%20from%20multiple%20tech%20blogs%22%7D" ``` -------------------------------- ### Filter Feed Items with Bash Source: https://context7.com/hackhtu/rssbook/llms.txt Filters feed items by keywords, date range, author, categories, or item count. This is done via a GET request to the /utils/filter endpoint. ```bash curl -X GET "http://localhost:8787/utils/filter?feed=https://www.ruanyifeng.com/blog/atom.xml&include=AI,machine%20learning&exclude=spam&after=2024-01-01&limit=10&caseSensitive=false" ``` -------------------------------- ### Implement Multi-language Support with Accept-Language Source: https://context7.com/hackhtu/rssbook/llms.txt Supports multiple languages by detecting the user's preferred language via the `Accept-Language` header. The system dynamically adjusts the content path to serve localized versions of the feed. Dependencies include `Source` from `@/utils`, `ofetch`, and `load`. ```typescript import { Source } from "@/utils"; export default new Source({ slug: "multilang-site", title: "Multi-language Site", description: "Site with multi-language support", domain: "example.com", }) .feed( { title: "News Feed", description: "News in user's preferred language", fulltext: true, language: ["en-US", "zh-CN", "ja"], maintainer: { name: "RSSBook" }, }, (app) => app.get("/news", async ({ lang, meta: { domain }, ofetch, load }) => { // Determine language path based on detected language const langPath = lang.includes("zh") ? "zh" : lang.includes("ja") ? "ja" : "en"; const url = `https://${domain}/${langPath}/news`; const html = await ofetch(url, { responseType: "text" }); const $ = load(html); return { title: $("h1").text(), link: url, language: langPath, item: $("article").toArray().map(/* ... */), }; }) ); ``` -------------------------------- ### Format Code with Bun Source: https://github.com/hackhtu/rssbook/blob/main/docs/how-to-create-a-feed.EN.md Uses the Bun runtime to check and format your code according to project standards. This step is crucial before submitting changes to ensure code consistency and maintainability. ```bash bun run ``` -------------------------------- ### Define Feed Configuration Source: https://github.com/hackhtu/rssbook/blob/main/docs/how-to-create-a-feed.EN.md Illustrates how to define configuration options for a feed, such as API keys or authentication credentials. These configurations are passed during app initialization and accessed via `ctx.meta.config`, with naming conventions based on the feed's slug. ```typescript config: { GOOGLE_EXAMPLE_APIKey: { default: "Bearer 123-456-7890", description: "RSSBook APIKey", required: true, }, }, ``` -------------------------------- ### Making Network Requests with RSSBook ofetch Source: https://github.com/hackhtu/rssbook/blob/main/docs/how-to-create-a-feed.EN.md Illustrates the usage of the `ofetch` function for making encapsulated network requests. It emphasizes using `ofetch` over native `fetch` and explains its error handling for non-200 responses, requiring `try-catch` blocks. ```typescript app.get( "/category/:category", authenticate, authorize, authorizeAdmin, autoDiscover, autoDiscoverPost, async ({ params: { category }, meta: { domain }, ofetch }) => { const html = await ofetch(`https://${domain}/category/${category}`, { responseType: "text", // or 'json', 'blob', etc. }); } ); ``` -------------------------------- ### Create Feed Source with ElysiaJS for ElysiaJS Blog Source: https://context7.com/hackhtu/rssbook/llms.txt This snippet defines a new feed source for the ElysiaJS blog. It includes metadata like slug, title, and description, along with configuration for an API key. The 'feed' method configures how to fetch and process blog posts, extracting titles, descriptions, links, images, and publication dates, with full content fetching for each item. ```typescript import { Source, t } from "@/utils"; export default new Source({ slug: "elysiajs", title: "ElysiaJS", description: "ElysiaJS is a lightweight web framework that runs on Bun.", domain: "elysiajs.com", config: { ELYSIAJS_API_KEY: { description: "API key for ElysiaJS service", required: false, default: "", }, }, }) .feed( { title: "Latest News", description: "Update on the latest news and insights about ElysiaJS.", fulltext: true, language: "en-US", maintainer: { name: "RSSBook", }, withImage: "If-Present", }, (app) => app.get("/blog", async ({ meta: { domain }, cache, load, ofetch, toAbsoluteURL, date, formatHTML }) => { const rootURL = `https://${domain}`; const url = `${rootURL}/blog`; const data = await cache.tryGet(url, async (url) => { const html = await ofetch(url, { responseType: "text" }); const $ = load(html); const title = $("header h1").text().trim(); const description = $("header p").text().trim(); const items = $("main > a, main > section > a") .toArray() .map((elem) => { const $elem = $(elem); let image = $elem.find("img").attr("src"); if (image) { image = toAbsoluteURL(image, rootURL); } const itemTitle = $elem.find("h2").text().trim(); const itemDescription = $elem.find("p").text().trim(); let link = $elem.attr("href"); if (link) { link = toAbsoluteURL(link, rootURL); } else { return null; } const dateString = $elem.find("time").attr("datetime") || ""; const pubDate = date(dateString); return { date: pubDate, description: itemDescription, image, link, title: itemTitle, }; }) .filter((item) => item !== null); return { description, item: items, link: url, title, }; }); // Fetch full content for each item const promises = data.item.map(async (item) => { return cache.tryGet(item.link, async (link) => { try { const html = await ofetch(link, { responseType: "text" }); const $ = load(html); let content = $("article").html(); if (content) { item.content = formatHTML(content, rootURL); } return item; } catch { return item; } }); }); const items = await Promise.all(promises); return { ...data, item: items, }; }) ); ``` -------------------------------- ### Fetch Article Content Concurrently (TypeScript) Source: https://github.com/hackhtu/rssbook/blob/main/docs/how-to-create-a-feed.EN.md This snippet shows how to fetch the full content of each article concurrently using `Promise.all`. It iterates over the previously scraped article data, fetches the HTML for each article's link, extracts the content from the `article` tag, formats it, and updates the item. If fetching fails, it logs an error and returns the original item. This approach leverages caching for individual article links and uses `ofetch` for requests, `cheerio` (loaded via `load`) for parsing, `formatHTML` for content processing, and a `logger` for error reporting. ```typescript const rootURL = `https://${domain}`; const url = `${rootURL}/blog`; const data = await cache.tryGet(url, async (url) => { /* ... */ }); const promises = data.item.map(async (item) => { const link = item.link; return cache.tryGet(link, async (link) => { try { const html = await ofetch(link, { responseType: "text" }); const $ = load(html); let content = $("article").html(); if (content) { item.content = formatHTML(content, rootURL); } return item; // Return item with full text } catch { logger.error(`Failed to fetch article content: ${link}`); return item; // Return original item } }) }) const item = await Promise.all(promises); return item; ``` -------------------------------- ### Apply Middleware in ElysiaJS Source: https://github.com/hackhtu/rssbook/blob/main/docs/how-to-create-a-feed.EN.md Demonstrates how to integrate middleware into the feed's routing using the ElysiaJS framework. Middleware can be used for tasks like authentication, logging, or request manipulation before the main handler is executed. ```typescript app => app .use(middleware) .get(path, handler, schema?); ``` -------------------------------- ### Construct Multi-language URL Source: https://github.com/hackhtu/rssbook/blob/main/docs/how-to-create-a-feed.EN.md Shows how to construct URLs for different languages based on a language code. This is useful for websites that serve content in multiple languages, allowing feeds to target specific language versions. ```typescript const rootURL = `https://${domain}`; const langPath = lang.includes("zh") ? "zh" : "en"; const url = `${rootURL}/${langPath}/home`; ``` -------------------------------- ### Run Feed Tests Source: https://github.com/hackhtu/rssbook/blob/main/docs/how-to-create-a-feed.EN.md Executes a helper command to add or run tests for your Feed. It checks for corresponding tests for recent changes and prompts to create new ones if they are missing. This ensures the feed logic functions correctly. ```bash bun run source:test ``` -------------------------------- ### Cache Management Source: https://context7.com/hackhtu/rssbook/llms.txt Demonstrates how to implement caching for feed data and individual items to enhance performance and reduce redundant requests. ```APIDOC ## Cache Management ### Description Implement caching for feed data to improve performance and reduce requests. This includes caching entire feed responses and individual item content for full-text extraction. ### Method GET (Implicitly used within a feed handler) ### Endpoint `/` (Root of the feed source) and potentially item-specific endpoints for full-text retrieval. ### Parameters None directly for cache management, but relies on URL keys for caching. ### Request Example ```typescript import { Cache } from "@/utils"; // Using cache in feed handler app.get("/blog", async ({ cache, ofetch, load }) => { const url = "https://example.com/blog"; // Try to get from cache, or fetch and cache const data = await cache.tryGet(url, async (url) => { const html = await ofetch(url, { responseType: "text" }); const $ = load(html); // Parse and return data return { title: $("h1").text(), items: $("article").toArray().map(/* ... */), }; }); return data; }); // Cache individual items for fulltext extraction const promises = items.map(async (item) => { return cache.tryGet(item.link, async (link) => { try { const html = await ofetch(link, { responseType: "text" }); const $ = load(html); item.content = formatHTML($("article").html(), rootURL); return item; } catch (error) { return item; // Return original on error } }); }); const enrichedItems = await Promise.all(promises); ``` ### Response Returns cached data if available, otherwise fetches, caches, and returns the requested data. ``` -------------------------------- ### Multi-language Support Source: https://context7.com/hackhtu/rssbook/llms.txt Enables multi-language support by detecting the user's preferred language via the Accept-Language header and serving content accordingly. ```APIDOC ## Multi-language Support ### Description Support multiple languages using Accept-Language header detection. The API determines the user's preferred language and serves content in the most appropriate available language. ### Method GET ### Endpoint `/news` (Example endpoint) ### Parameters #### Path Parameters None. #### Query Parameters None. #### Request Body None. ### Request Example ```typescript import { Source } from "@/utils"; export default new Source({ slug: "multilang-site", title: "Multi-language Site", description: "Site with multi-language support", domain: "example.com", }) .feed( { title: "News Feed", description: "News in user's preferred language", fulltext: true, language: ["en-US", "zh-CN", "ja"], maintainer: { name: "RSSBook" }, }, (app) => app.get("/news", async ({ lang, meta: { domain }, ofetch, load }) => { // Determine language path based on detected language const langPath = lang.includes("zh") ? "zh" : lang.includes("ja") ? "ja" : "en"; const url = `https://${domain}/${langPath}/news`; const html = await ofetch(url, { responseType: "text" }); const $ = load(html); return { title: $("h1").text(), link: url, language: langPath, item: $("article").toArray().map(/* ... */), }; }) ); ``` ### Response #### Success Response (200) - **title** (string) - The title of the feed. - **link** (string) - The URL of the feed. - **language** (string) - The language code of the returned content (e.g., `en`, `zh`, `ja`). - **item** (array) - An array of article items in the detected language. #### Response Example ```json { "title": "Latest News", "link": "https://example.com/en/news", "language": "en", "item": [ { "title": "News Item 1", "description": "Summary of news item 1", "link": "https://example.com/en/news/item-1", "pubDate": "2023-10-27T10:00:00Z" } ] } ``` ``` -------------------------------- ### Type-Safe Route Parameters Source: https://context7.com/hackhtu/rssbook/llms.txt Defines and validates route parameters using TypeBox schemas, ensuring type safety and providing clear API contracts. ```APIDOC ## Type-Safe Route Parameters ### Description Define and validate route parameters with TypeBox schemas. This ensures that incoming parameters conform to the expected types and formats, improving API robustness. ### Method GET ### Endpoint `/category/:category` ### Parameters #### Path Parameters - **category** (string) - Required - The category of articles to fetch. Must be one of: `news`, `sports`, `entertainment`. #### Query Parameters None specified. #### Request Body None specified. ### Request Example ```typescript import { Source, t } from "@/utils"; export default new Source({ slug: "example", title: "Example Source", description: "Example source with route parameters", domain: "example.com", }) .feed( { title: "Category Feed", description: "Get articles by category. Available categories: **news**, **sports**, **entertainment**", fulltext: false, language: "en-US", maintainer: { name: "RSSBook" }, }, (app) => app.get( "/category/:category", async ({ params: { category }, meta: { domain }, ofetch, load }) => { const url = `https://${domain}/category/${category}`; const html = await ofetch(url, { responseType: "text" }); const $ = load(html); // Extract and return feed data return { title: `${category} Articles`, link: url, item: $("article").toArray().map(/* ... */), }; }, { params: t.Object({ category: t.UnionEnum(["news", "sports", "entertainment"], { description: "The category of articles to fetch", }), }), } ) ); ``` ### Response #### Success Response (200) - **title** (string) - The title of the feed. - **link** (string) - The URL of the feed. - **item** (array) - An array of article items. #### Response Example ```json { "title": "news Articles", "link": "https://example.com/category/news", "item": [ { "title": "Article 1", "description": "Summary of article 1", "link": "https://example.com/article-1", "pubDate": "2023-10-27T10:00:00Z" } ] } ``` ``` -------------------------------- ### Create Feed Category with TypeScript Source: https://context7.com/hackhtu/rssbook/llms.txt Organizes multiple feed sources into a category for better organization. Requires Category utility and specific feed modules. ```typescript import { Category } from "@/utils"; import elysiajs from "./elysiajs"; import github from "./github"; import npm from "./npm"; export default new Category( "updates", "**Announcements and updates** about products, services, or projects." ).use({ elysiajs, github, npm, }); ``` -------------------------------- ### Parsing Dates with RSSBook Date Tool Source: https://github.com/hackhtu/rssbook/blob/main/docs/how-to-create-a-feed.EN.md Demonstrates the `date` tool for parsing various date formats, including relative time strings, absolute time strings, and timestamps. It also shows how to specify a timezone offset for absolute time parsing. ```typescript const date1 = date("2 hours ago"); // Parse relative time const date2 = date("2023-10-01 12:00:00", "+8"); // Parse absolute time, with timezone const date3 = date(1696156800000); // Parse timestamp ``` -------------------------------- ### Parse HTML and Extract Content with Cheerio Source: https://context7.com/hackhtu/rssbook/llms.txt Extracts content from HTML pages using Cheerio selectors. It parses article titles, descriptions, links, images, publication dates, and full HTML content, converting relative URLs to absolute ones and sanitizing HTML. Dependencies include Cheerio (`load`), `ofetch`, and utility functions like `toAbsoluteURL`, `date`, and `formatHTML`. ```typescript import { load, formatHTML, toAbsoluteURL, date } from "@/utils"; import { ofetch } from "ofetch"; const url = "https://example.com/blog"; const html = await ofetch(url, { responseType: "text" }); const $ = load(html); // Extract basic information const pageTitle = $("h1.title").text().trim(); const articles = $("article.post") .toArray() .map((elem) => { const $elem = $(elem); const title = $elem.find("h2").text().trim(); const description = $elem.find(".excerpt").text().trim(); const rawLink = $elem.find("a").attr("href"); const link = rawLink ? toAbsoluteURL(rawLink, url) : null; const rawImage = $elem.find("img").attr("src"); const image = rawImage ? toAbsoluteURL(rawImage, url) : null; const dateStr = $elem.find("time").attr("datetime") || ""; const pubDate = date(dateStr); // Get full HTML content and sanitize const rawContent = $elem.find(".content").html(); const content = rawContent ? formatHTML(rawContent, url) : ""; return { title, description, link, image, date: pubDate, content, }; }) .filter((item) => item.link !== null); ``` -------------------------------- ### HTML Parsing and Content Extraction Source: https://context7.com/hackhtu/rssbook/llms.txt This section details how to extract specific content from HTML pages using Cheerio selectors, including basic information, article details, and full HTML content. ```APIDOC ## HTML Parsing and Content Extraction ### Description Extract content from HTML pages using Cheerio-based selectors. This includes retrieving page titles, article summaries, links, images, publication dates, and full sanitized HTML content. ### Method GET (Implicitly used within a feed handler) ### Endpoint `/` (Root of the feed source) ### Parameters None directly for this extraction logic, but relies on an input URL. ### Request Example ```typescript import { load, formatHTML, toAbsoluteURL } from "@/utils"; import { ofetch } from "ofetch"; const url = "https://example.com/blog"; const html = await ofetch(url, { responseType: "text" }); const $ = load(html); // Extract basic information const pageTitle = $("h1.title").text().trim(); const articles = $("article.post") .toArray() .map((elem) => { const $elem = $(elem); const title = $elem.find("h2").text().trim(); const description = $elem.find(".excerpt").text().trim(); const rawLink = $elem.find("a").attr("href"); const link = rawLink ? toAbsoluteURL(rawLink, url) : null; const rawImage = $elem.find("img").attr("src"); const image = rawImage ? toAbsoluteURL(rawImage, url) : null; const dateStr = $elem.find("time").attr("datetime") || ""; const pubDate = date(dateStr); // Get full HTML content and sanitize const rawContent = $elem.find(".content").html(); const content = rawContent ? formatHTML(rawContent, url) : ""; return { title, description, link, image, date: pubDate, content, }; }) .filter((item) => item.link !== null); ``` ### Response ```json { "pageTitle": "string", "articles": [ { "title": "string", "description": "string", "link": "string | null", "image": "string | null", "date": "string", "content": "string" } ] } ``` ``` -------------------------------- ### Implement Caching for Feed Data Source: https://context7.com/hackhtu/rssbook/llms.txt Implements caching for feed data using a `Cache` utility to enhance performance and reduce redundant requests. It demonstrates fetching and caching full feed data as well as individual item content for full-text extraction. Dependencies include the `Cache` utility, `ofetch`, and `load`. ```typescript import { Cache } from "@/utils"; // Using cache in feed handler app.get("/blog", async ({ cache, ofetch, load }) => { const url = "https://example.com/blog"; // Try to get from cache, or fetch and cache const data = await cache.tryGet(url, async (url) => { const html = await ofetch(url, { responseType: "text" }); const $ = load(html); // Parse and return data return { title: $("h1").text(), items: $("article").toArray().map(/* ... */), }; }); return data; }); // Cache individual items for fulltext extraction const promises = items.map(async (item) => { return cache.tryGet(item.link, async (link) => { try { const html = await ofetch(link, { responseType: "text" }); const $ = load(html); item.content = formatHTML($("article").html(), rootURL); return item; } catch (error) { return item; // Return original on error } }); }); const enrichedItems = await Promise.all(promises); ``` -------------------------------- ### Define Feed Route and Handler Logic in TypeScript Source: https://github.com/hackhtu/rssbook/blob/main/docs/how-to-create-a-feed.EN.md Illustrates how to define a specific route path (e.g., '/blog') and an asynchronous handler function for a Feed. The handler is responsible for fetching and returning the feed data. ```typescript .feed( { /* ...config */ }, app => app.get( "/blog", async () => { // ...logic return data; }, ), ); ``` -------------------------------- ### Commit and Push Changes Source: https://github.com/hackhtu/rssbook/blob/main/docs/how-to-create-a-feed.EN.md Stages all changes, commits them with a descriptive message, and pushes them to your remote repository's main branch. This is a standard Git workflow for submitting code. ```bash git add . git commit -m "feat: add new feed for {Website Name}" git push origin main ``` -------------------------------- ### Scrape Article List and Extract Data (TypeScript) Source: https://github.com/hackhtu/rssbook/blob/main/docs/how-to-create-a-feed.EN.md This snippet demonstrates how to scrape a list of articles from a given URL. It extracts the title, link, and publication time for each article, filters out items without links, and structures the data into a `Data` type. It utilizes `ofetch` for HTTP requests and `cheerio` (loaded via `load`) for HTML parsing. Dependencies include `toAbsoluteURL` and `date` from the `Context` object, and a `cache` object for storing fetched data. ```typescript const rootURL = `https://${domain}`; const url = `${rootURL}/blog`; const data = await cache.tryGet(url, async (url) => { const html = await ofetch(url, { responseType: "text" }); const $ = load(html); const title = $("header h1").text().trim(); const description = $("header p").text().trim(); const items = $("main > a, main > section > a") .toArray() .map((elem) => { const $elem = $(elem); let image = $elem.find("img").attr("src"); if (image) { image = toAbsoluteURL(image, rootURL); } const title = $elem.find("h2").text().trim(); const description = $elem.find("p").text().trim(); let link = $elem.attr("href"); if (link) { link = toAbsoluteURL(link, rootURL); } else { return null; // Filter out items without links } const dateString = $elem.find("time").attr("datetime") || ""; const pubDate = date(dateString); return { date: pubDate, description, image, link, title, } satisfies DataItem; }) .filter((item) => item !== null); // Filter out items without links return { description, item: items, link: url, title, } satisfies Data; }); ``` -------------------------------- ### Add Feeds to a Source in TypeScript Source: https://github.com/hackhtu/rssbook/blob/main/docs/how-to-create-a-feed.EN.md Demonstrates how to add multiple Feeds to a Source using method chaining in TypeScript. This is the primary method for integrating new feed sources into the application. ```typescript import { Source } from "@/utils"; export default new Source({ /* ...source metadata */ }) .feed(/* ... */); // Add a Feed .feed(/* ... */); // Method chaining to add another Feed ``` -------------------------------- ### Fetch and Parse External Feed Programmatically with TypeScript Source: https://context7.com/hackhtu/rssbook/llms.txt Fetches an external RSS, Atom, or JSON feed and parses it into a standardized format programmatically. Requires ofetch for fetching and the parse utility for processing. Outputs structured data that can be easily accessed. ```typescript import { parse } from "@/utils"; import { ofetch } from "ofetch"; import type { Data } from "@/types"; // Fetch and parse any RSS/Atom/JSON feed const feedUrl = "https://www.ruanyifeng.com/blog/atom.xml"; const response = await ofetch(feedUrl, { responseType: "text" }); const feedData: Data = parse(response); // Access parsed data console.log(feedData.title); console.log(feedData.description); console.log(feedData.item.length); feedData.item.forEach((item) => { console.log(item.title, item.link, item.date); }); ``` -------------------------------- ### Accessing Route Parameters in RSSBook Source: https://github.com/hackhtu/rssbook/blob/main/docs/how-to-create-a-feed.EN.md Demonstrates how to access parameters defined in a route path (e.g., `/category/:category`) by destructuring them from the `params` object. It also highlights the necessity of defining the parameter's schema in `app.get` for validation. ```typescript app.get( "/category/:category", authenticate, async ({ params: { category } }) => { // Access category parameter here }, { /* ...schema */ } ); ``` -------------------------------- ### Logging Information with RSSBook Logger Source: https://github.com/hackhtu/rssbook/blob/main/docs/how-to-create-a-feed.EN.md Shows how to use the `logger` tool provided by RSSBook for different levels of logging, including info, warning, and error messages. This is crucial for debugging and troubleshooting. ```typescript logger.info("This is an info message"); logger.warn("This is a warning message"); logger.error("This is an error message"); ``` -------------------------------- ### Scraping Full Article Content Source: https://github.com/hackhtu/rssbook/blob/main/docs/how-to-create-a-feed.EN.md This snippet shows how to concurrently scrape the full content of each article from the list obtained previously. It uses `Promise.all` for concurrent requests and caches the content of each article. It falls back to returning the original item if fetching fails. ```APIDOC ## GET /blog/:articleId ### Description Fetches the full content of a specific article. If the full content cannot be fetched, it returns the article with only its introduction. ### Method GET ### Endpoint /blog/:articleId ### Parameters #### Path Parameters - **articleId** (string) - Required - The unique identifier of the article. #### Query Parameters None #### Request Body None ### Request Example None ### Response #### Success Response (200) - **content** (string) - The full HTML content of the article. #### Response Example ```json { "date": "2023-10-27T10:00:00Z", "description": "Article description.", "image": "https://example.com/image.jpg", "link": "https://example.com/article/1", "title": "Article 1", "content": "
This is the full article content.
" } ``` ``` -------------------------------- ### Union (Merge) Multiple Feeds Programmatically with TypeScript Source: https://context7.com/hackhtu/rssbook/llms.txt Merges multiple RSS/Atom feeds into a single unified feed programmatically using the RSSBook union utility. It fetches feeds, parses them, and then merges them, allowing for metadata overrides. Requires ofetch and parsing utilities. ```typescript import { Elysia } from "elysia"; import { ofetch } from "ofetch"; import { parse, union } from "@/utils"; import type { Data } from "@/types"; // Using RSSBook's union utility programmatically const feeds = [ "https://www.ruanyifeng.com/blog/atom.xml", "https://github.blog/feed/" ]; const promises = feeds.map(async (feedUrl) => { try { const response = await ofetch(feedUrl, { responseType: "text" }); return parse(response); } catch (error) { console.error(`Failed to fetch ${feedUrl}:`, error); return null; } }); const results = await Promise.all(promises); const datas = results.filter((result): result is Data => result !== null); const [firstData, ...restData] = datas; const mergedFeed = union(firstData, restData); // Override metadata const finalFeed = { ...mergedFeed, title: "Combined Tech News", description: "Merged feed from multiple tech blogs", }; ``` -------------------------------- ### Formatting HTML Content with RSSBook formatHTML Source: https://github.com/hackhtu/rssbook/blob/main/docs/how-to-create-a-feed.EN.md Explains how to use the `formatHTML` function to clean up HTML strings by removing styles, scripts, and unnecessary attributes. It also covers converting relative resource paths to absolute paths using a `baseURL`. ```typescript const rawHTML = $("article").html(); const formattedHTML = formatHTML(rawHTML, "https://example.com"); ``` -------------------------------- ### TypeScript: Create and Validate Feed Data Source: https://context7.com/hackhtu/rssbook/llms.txt Generates feed data with strict TypeScript types and validates it using imported utility functions. It handles both creating new data and parsing/validating unknown input, providing error handling for validation failures. Dependencies include custom type definitions from '@/types'. ```typescript import type { Data, DataItem } from "@/types"; import { validateData, parseData } from "@/types"; // Create feed data with type safety const feedData = { title: "My Feed", link: "https://example.com", description: "Example feed description", language: "en-US", item: [ { title: "Article 1", link: "https://example.com/article-1", description: "First article", date: new Date("2024-01-15"), content: "Full article content
", author: [{ name: "John Doe", email: "john@example.com" }], category: [{ name: "Technology" }], }, { title: "Article 2", link: "https://example.com/article-2", description: "Second article", date: new Date("2024-01-16"), image: "https://example.com/image.jpg", }, ], } satisfies Data; // Validate data if (validateData(feedData)) { console.log("Feed data is valid"); } // Parse and validate unknown data try { const parsedData = parseData(unknownInput); console.log("Parsed successfully:", parsedData); } catch (error) { console.error("Validation error:", error); } ``` -------------------------------- ### Define Category Feed Schema Source: https://github.com/hackhtu/rssbook/blob/main/docs/how-to-create-a-feed.EN.md This snippet illustrates how to define a schema for a dynamic route parameter, specifically for fetching articles by category. It uses the `t` tool (based on TypeBox) to define the schema for the `category` parameter, including its type, allowed values, and a description. ```APIDOC ## GET /category/:category ### Description Fetches articles filtered by a specific category. ### Method GET ### Endpoint /category/:category ### Parameters #### Path Parameters - **category** (string) - Required - The category of articles to fetch. Allowed values: "news", "sports", "entertainment". #### Query Parameters None #### Request Body None ### Request Example None ### Response #### Success Response (200) - Returns a list of articles belonging to the specified category. #### Response Example ```json [ { "date": "2023-10-27T10:00:00Z", "description": "Sports news.", "image": "https://example.com/sports.jpg", "link": "https://example.com/article/sports/1", "title": "Latest Sports News" } ] ``` ``` -------------------------------- ### Sort Feed Items Programmatically with TypeScript Source: https://context7.com/hackhtu/rssbook/llms.txt Sorts feed items programmatically by date in either ascending or descending order. Requires fetching, parsing, and the sort utility. ```typescript import { sort, parse } from "@/utils"; import { ofetch } from "ofetch"; const feedUrl = "https://www.ruanyifeng.com/blog/atom.xml"; const response = await ofetch(feedUrl, { responseType: "text" }); const data = parse(response); // Sort by date descending (newest first) const sortedDesc = sort(data, "desc"); // Sort by date ascending (oldest first) const sortedAsc = sort(data, "asc"); ``` -------------------------------- ### Define Type-Safe Route Parameters with TypeBox Source: https://context7.com/hackhtu/rssbook/llms.txt Enables type-safe route parameter validation using TypeBox schemas. This ensures that route parameters like 'category' conform to predefined types and enums, improving code robustness and developer experience. Dependencies include `Source` from `@/utils` and `t` from TypeBox. ```typescript import { Source, t } from "@/utils"; export default new Source({ slug: "example", title: "Example Source", description: "Example source with route parameters", domain: "example.com", }) .feed( { title: "Category Feed", description: "Get articles by category. Available categories: **news**, **sports**, **entertainment**", fulltext: false, language: "en-US", maintainer: { name: "RSSBook" }, }, (app) => app.get( "/category/:category", async ({ params: { category }, meta: { domain }, ofetch, load }) => { const url = `https://${domain}/category/${category}`; const html = await ofetch(url, { responseType: "text" }); const $ = load(html); // Extract and return feed data return { title: `${category} Articles`, link: url, item: $("article").toArray().map(/* ... */), }; }, { params: t.Object({ category: t.UnionEnum(["news", "sports", "entertainment"], { description: "The category of articles to fetch", }), }), } ) ); ``` -------------------------------- ### Implementing Asynchronous Delays with RSSBook sleep Source: https://github.com/hackhtu/rssbook/blob/main/docs/how-to-create-a-feed.EN.md Illustrates the use of the `sleep` function for introducing asynchronous delays, which is particularly useful for avoiding anti-scraping mechanisms on target websites by spacing out requests. ```typescript await sleep(2000); // Wait for 2 seconds ``` -------------------------------- ### Scrape Blog Article List from HTML Source: https://github.com/hackhtu/rssbook/blob/main/docs/how-to-create-a-feed.EN.md Scrapes a list of blog articles from an HTML page using provided utility functions like cache, load, and ofetch. It parses HTML content, extracts specific elements based on CSS selectors, and prepares the data. ```typescript app => app.get( "/blog", async ({ // props meta: { domain }, // functions cache, load, ofetch, }) => { const rootURL = `https://${domain}`; const url = `${rootURL}/blog`; const lists = await cache.tryGet(url, async (url) => { const html = await ofetch(url, { responseType: "text" }); const $ = load(html); }); } ); ``` ```typescript const html = await ofetch(url, { responseType: "text" }); const $ = load(html); const title = $("header h1").text().trim(); const description = $("header p").text().trim(); const items = $("main > a, main > section > a") .toArray() .map((elem) => { const $elem = $(elem); }); ``` -------------------------------- ### Filter Feed Items Programmatically with TypeScript Source: https://context7.com/hackhtu/rssbook/llms.txt Filters feed items programmatically based on various criteria like keywords, date, author, categories, and item limits. Requires fetching, parsing, and the filter utility. Supports case-sensitive and case-insensitive matching. ```typescript import { filter, parse } from "@/utils"; import { ofetch } from "ofetch"; import type { FilterOptions } from "@/utils/feeds/filter"; // Fetch and parse feed const feedUrl = "https://www.ruanyifeng.com/blog/atom.xml"; const response = await ofetch(feedUrl, { responseType: "text" }); const data = parse(response); // Define filter criteria const filterOptions: FilterOptions = { keywords: { include: ["AI", "machine learning"], exclude: ["spam", "ads"], caseSensitive: false, }, date: { after: new Date("2024-01-01"), before: new Date("2024-12-31"), }, author: { include: ["John Doe"], exclude: ["Bot"], }, categories: { include: ["technology", "programming"], exclude: ["promotions"], }, limit: { count: 10, fromStart: true, }, }; // Apply filters const filtered = filter(data, filterOptions, { title: "Filtered Tech Feed", description: "Curated AI and ML articles", }); ``` -------------------------------- ### Fetch and Parse External Feed Source: https://context7.com/hackhtu/rssbook/llms.txt Fetch an external RSS, Atom, or JSON feed from a given URL and parse it into a standardized data structure. ```APIDOC ## GET /utils/fetch ### Description Fetches content from a specified URL, assuming it's an RSS, Atom, or JSON feed. The content is then parsed into a consistent internal data format for easy access to feed metadata and items. ### Method GET ### Endpoint `/utils/fetch` ### Query Parameters - **url** (string) - Required - The URL of the external feed to fetch and parse. ### Request Example ```bash curl -X GET "http://localhost:8787/utils/fetch?url=https://www.ruanyifeng.com/blog/atom.xml" ``` ### Response #### Success Response (200) - **title** (string) - The title of the feed. - **description** (string) - The description of the feed. - **link** (string) - The canonical link for the feed. - **item** (array) - An array of feed items, each containing details like `title`, `link`, `date`, `content`, etc. #### Response Example ```json { "title": "Example Feed Title", "description": "A sample feed description.", "link": "http://example.com/feed", "item": [ { "title": "First Post", "link": "http://example.com/post1", "date": "2024-07-27T12:00:00Z", "content": "Content of the first post.
" }, { "title": "Second Post", "link": "http://example.com/post2", "date": "2024-07-26T08:00:00Z", "content": "Content of the second post.
" } ] } ``` ```