### Full Server Lifecycle Example Source: https://github.com/bluesky-social/feed-generator/blob/main/_autodocs/api-reference/FeedGenerator.md Demonstrates a complete server setup and lifecycle using environment variables for configuration. This includes initializing dotenv, creating the FeedGenerator, starting it, and logging its address. ```typescript import dotenv from 'dotenv' import FeedGenerator from './server' async function runServer() { dotenv.config() const server = FeedGenerator.create({ port: parseInt(process.env.FEEDGEN_PORT || '3000'), listenhost: process.env.FEEDGEN_LISTENHOST || 'localhost', hostname: process.env.FEEDGEN_HOSTNAME || 'example.com', sqliteLocation: process.env.FEEDGEN_SQLITE_LOCATION || ':memory:', subscriptionEndpoint: process.env.FEEDGEN_SUBSCRIPTION_ENDPOINT || 'wss://bsky.network', publisherDid: process.env.FEEDGEN_PUBLISHER_DID || 'did:example:alice', subscriptionReconnectDelay: parseInt(process.env.FEEDGEN_SUBSCRIPTION_RECONNECT_DELAY || '3000'), serviceDid: process.env.FEEDGEN_SERVICE_DID || 'did:web:example.com', }) await server.start() console.log(`Feed generator running at http://${server.cfg.listenhost}:${server.cfg.port}`) } runServer().catch(console.error) ``` -------------------------------- ### FeedGenerator Start Example Source: https://github.com/bluesky-social/feed-generator/blob/main/_autodocs/api-reference/FeedGenerator.md Example of starting the FeedGenerator server and handling potential errors. Logs the server port upon successful startup. ```typescript try { const httpServer = await feedGenerator.start() console.log(`Server running on port ${feedGenerator.cfg.port}`) } catch (err) { console.error('Failed to start feed generator:', err) } ``` -------------------------------- ### Example Config Object Source: https://github.com/bluesky-social/feed-generator/blob/main/_autodocs/types.md Provides an example of how to instantiate the Config type. This demonstrates the expected values for each configuration field. ```typescript const config: Config = { port: 3000, listenhost: 'localhost', hostname: 'example.com', sqliteLocation: ':memory:', subscriptionEndpoint: 'wss://bsky.network', serviceDid: 'did:web:example.com', publisherDid: 'did:example:alice', subscriptionReconnectDelay: 3000, } ``` -------------------------------- ### Configuration: Service DID and Hostname for did:web Source: https://github.com/bluesky-social/feed-generator/blob/main/_autodocs/api-reference/WellKnown.md Example TypeScript configurations demonstrating the correct setup for `serviceDid` and `hostname` when using `did:web`. Incorrect configurations will result in a 404 Not Found response. ```typescript // ✅ Works FEEDGEN_SERVICE_DID=did:web:feed.example.com // ❌ Returns 404 FEEDGEN_SERVICE_DID=did:plc:z7r4qhxemkxd7fh... ``` ```typescript // ✅ Works (serviceDid ends with hostname) FEEDGEN_HOSTNAME=feed.example.com FEEDGEN_SERVICE_DID=did:web:feed.example.com // ❌ Returns 404 (mismatch) FEEDGEN_HOSTNAME=example.com FEEDGEN_SERVICE_DID=did:web:feed.example.com ``` -------------------------------- ### Setting up .env file Source: https://github.com/bluesky-social/feed-generator/blob/main/_autodocs/configuration.md Command to copy the example .env file to .env for local configuration. ```bash cp .env.example .env ``` -------------------------------- ### DID Document Request Example Source: https://github.com/bluesky-social/feed-generator/blob/main/_autodocs/endpoints.md Example HTTP GET request to the `/.well-known/did.json` endpoint for retrieving the service's DID document. ```http GET /.well-known/did.json HTTP/1.1 Host: example.com ``` -------------------------------- ### Development Configuration Example (.env) Source: https://github.com/bluesky-social/feed-generator/blob/main/_autodocs/configuration.md An example .env file for setting up the feed generator in a development environment, using an in-memory database. ```bash # .env FEEDGEN_HOSTNAME=localhost FEEDGEN_SERVICE_DID=did:web:localhost FEEDGEN_PORT=3000 FEEDGEN_LISTENHOST=localhost FEEDGEN_SQLITE_LOCATION=:memory: FEEDGEN_SUBSCRIPTION_ENDPOINT=wss://bsky.network FEEDGEN_PUBLISHER_DID=did:example:alice FEEDGEN_SUBSCRIPTION_RECONNECT_DELAY=3000 ``` -------------------------------- ### Describe Feed Generator Request Example Source: https://github.com/bluesky-social/feed-generator/blob/main/_autodocs/endpoints.md Example HTTP GET request to the `describeFeedGenerator` endpoint. ```http GET /xrpc/app.bsky.feed.describeFeedGenerator HTTP/1.1 Host: example.com ``` -------------------------------- ### DID:plc Configuration Example Source: https://github.com/bluesky-social/feed-generator/blob/main/_autodocs/configuration.md Example of setting FEEDGEN_SERVICE_DID and FEEDGEN_PUBLISHER_DID for did:plc configuration. This is recommended for long-lived services. ```bash FEEDGEN_SERVICE_DID=did:plc:z7r4qhxemkxd7fh... FEEDGEN_PUBLISHER_DID=did:plc:my_user_did_here ``` -------------------------------- ### JWT Authentication Header Example Source: https://github.com/bluesky-social/feed-generator/blob/main/_autodocs/endpoints.md Example of how to pass JWT tokens for authentication in the `Authorization` header. ```text Authorization: Bearer eyJhbGciOiJFUzI1NksiLCJ0eXAiOiJKV1QifQ... ``` -------------------------------- ### Production Configuration Example (.env.production) Source: https://github.com/bluesky-social/feed-generator/blob/main/_autodocs/configuration.md An example .env.production file for configuring the feed generator in a production environment, using a persistent database and listening on all interfaces. ```bash # .env.production FEEDGEN_HOSTNAME=feeds.myservice.com FEEDGEN_SERVICE_DID=did:web:feeds.myservice.com FEEDGEN_PORT=3000 FEEDGEN_LISTENHOST=0.0.0.0 FEEDGEN_SQLITE_LOCATION=/data/feed.db FEEDGEN_SUBSCRIPTION_ENDPOINT=wss://bsky.network FEEDGEN_PUBLISHER_DID=did:plc:my_user_did_here FEEDGEN_SUBSCRIPTION_RECONNECT_DELAY=5000 ``` -------------------------------- ### Configure Feed Generator Environment Source: https://github.com/bluesky-social/feed-generator/blob/main/_autodocs/README.md Copies the example environment file and provides an example of essential configuration variables for the feed generator. ```bash cp .env.example .env ``` ```env FEEDGEN_HOSTNAME=localhost FEEDGEN_SERVICE_DID=did:web:localhost FEEDGEN_PORT=3000 FEEDGEN_LISTENHOST=localhost FEEDGEN_SQLITE_LOCATION=:memory: FEEDGEN_SUBSCRIPTION_ENDPOINT=wss://bsky.network FEEDGEN_PUBLISHER_DID=did:example:alice FEEDGEN_SUBSCRIPTION_RECONNECT_DELAY=3000 ``` -------------------------------- ### Example Request/Response for getFeedSkeleton Source: https://github.com/bluesky-social/feed-generator/blob/main/_autodocs/endpoints.md A concrete example of an HTTP GET request to the getFeedSkeleton endpoint and its corresponding JSON response. ```http GET /xrpc/app.bsky.feed.getFeedSkeleton?feed=at://did:example:alice/app.bsky.feed.generator/whats-alf&limit=10 HTTP/1.1 Host: example.com ``` ```json { "feed": [ { "post": "at://did:example:alice/app.bsky.feed.post/1" }, { "post": "at://did:example:bob/app.bsky.feed.post/2" } ], "cursor": "1683643619000" } ``` -------------------------------- ### Server Setup for Feed Generation Source: https://github.com/bluesky-social/feed-generator/blob/main/_autodocs/api-reference/Methods.md Shows how to integrate feed generation and description handlers into the server setup using `FeedGenerator.create()` and `app.use()`. ```typescript const ctx: AppContext = { db, didResolver, cfg } feedGeneration(server, ctx) describeGenerator(server, ctx) app.use(server.xrpc.router) ``` -------------------------------- ### FeedGenerator Start Method Source: https://github.com/bluesky-social/feed-generator/blob/main/_autodocs/api-reference/FeedGenerator.md Use the `start` method to run the HTTP server, apply database migrations, and begin the firehose subscription. This method resolves once the server is listening. ```typescript async start(): Promise ``` -------------------------------- ### Run Feed Generator Quickstart Source: https://github.com/bluesky-social/feed-generator/blob/main/go/README.md Steps to run the feed generator locally and fetch feed items using curl. ```bash cp .env.example .env # Edit .env with your values go run . # In another terminal, get some feed items with: curl -s "http://localhost:3000/xrpc/app.bsky.feed.getFeedSkeleton?feed=at://did:example:alice/app.bsky.feed.generator/whats-alf&limit=10" | jq . ``` -------------------------------- ### Feed Generation Example Registration Source: https://github.com/bluesky-social/feed-generator/blob/main/_autodocs/api-reference/Methods.md Example of how to import and register the feed generation handler with the XRPC server. ```typescript import feedGeneration from './methods/feed-generation' const ctx: AppContext = { db, didResolver, cfg } feedGeneration(server, ctx) // Now GET /xrpc/app.bsky.feed.getFeedSkeleton is registered ``` -------------------------------- ### FeedGenerator Initialization Example Source: https://github.com/bluesky-social/feed-generator/blob/main/_autodocs/api-reference/FeedGenerator.md Example of creating a FeedGenerator instance using the `create` method with specific configuration options. This sets up the server with custom parameters. ```typescript import FeedGenerator from './server' const server = FeedGenerator.create({ port: 3000, listenhost: 'localhost', hostname: 'example.com', sqliteLocation: '/tmp/feed.db', subscriptionEndpoint: 'wss://bsky.network', publisherDid: 'did:example:alice', subscriptionReconnectDelay: 3000, serviceDid: 'did:web:example.com', }) ``` -------------------------------- ### Describe Generator Example Registration Source: https://github.com/bluesky-social/feed-generator/blob/main/_autodocs/api-reference/Methods.md Example of how to import and register the describe generator handler with the XRPC server. ```typescript import describeGenerator from './methods/describe-generator' const ctx: AppContext = { db, cfg } describeGenerator(server, ctx) // Now GET /xrpc/app.bsky.feed.describeFeedGenerator is registered ``` -------------------------------- ### Feed Generator .env File Example Source: https://github.com/bluesky-social/feed-generator/blob/main/_autodocs/configuration.md An example .env file demonstrating how to configure the feed generator with various settings including server, database, and firehose subscription details. ```env # Server configuration FEEDGEN_HOSTNAME=localhost FEEDGEN_SERVICE_DID=did:web:localhost FEEDGEN_PORT=3000 FEEDGEN_LISTENHOST=localhost # Database FEEDGEN_SQLITE_LOCATION=:memory: # Firehose subscription FEEDGEN_SUBSCRIPTION_ENDPOINT=wss://bsky.network FEEDGEN_PUBLISHER_DID=did:example:alice FEEDGEN_SUBSCRIPTION_RECONNECT_DELAY=3000 ``` -------------------------------- ### Example Configuration: Using did:web Source: https://github.com/bluesky-social/feed-generator/blob/main/_autodocs/api-reference/WellKnown.md Demonstrates setting up the `FEEDGEN_HOSTNAME` and `FEEDGEN_SERVICE_DID` environment variables for `did:web` resolution. Includes the expected request and 200 OK response. ```bash # .env FEEDGEN_HOSTNAME=feeds.example.com FEEDGEN_SERVICE_DID=did:web:feeds.example.com ``` ```text GET https://feeds.example.com/.well-known/did.json → 200 OK { "@context": ["https://www.w3.org/ns/did/v1"], "id": "did:web:feeds.example.com", "service": [{ "id": "#bsky_fg", "type": "BskyFeedGenerator", "serviceEndpoint": "https://feeds.example.com" }] } ``` -------------------------------- ### GET Request for app.bsky.feed.describeFeedGenerator Source: https://github.com/bluesky-social/feed-generator/blob/main/_autodocs/endpoints.md Example GET request to the describeFeedGenerator endpoint. This endpoint does not require any query parameters or authentication. ```http GET /xrpc/app.bsky.feed.describeFeedGenerator ``` -------------------------------- ### DID:web Configuration Example Source: https://github.com/bluesky-social/feed-generator/blob/main/_autodocs/configuration.md Example of setting the FEEDGEN_HOSTNAME environment variable for did:web configuration. The service DID is automatically derived from the hostname. ```bash FEEDGEN_HOSTNAME=feed.example.com # Resolves to: did:web:feed.example.com # Well-known endpoint: https://feed.example.com/.well-known/did.json ``` -------------------------------- ### FeedGenerator.start Source: https://github.com/bluesky-social/feed-generator/blob/main/_autodocs/api-reference/FeedGenerator.md Starts the HTTP server and begins listening for incoming requests. This method also runs database migrations and starts the firehose subscription. ```APIDOC ## Instance Method: start Starts the HTTP server and begins listening for incoming requests. Runs database migrations and starts the firehose subscription. ```typescript async start(): Promise ``` **Returns** `Promise` — Resolves to the Node.js HTTP server instance once it is listening. **Behavior** - Executes all pending database migrations - Starts the firehose subscription with automatic reconnection - Listens on the configured port and hostname - Waits for the server 'listening' event before resolving **Throws** - May throw if migrations fail - May throw if the server fails to bind to the specified port **Example** ```typescript try { const httpServer = await feedGenerator.start() console.log(`Server running on port ${feedGenerator.cfg.port}`) } catch (err) { console.error('Failed to start feed generator:', err) } ``` ``` -------------------------------- ### Testing Get Feed Skeleton (Authenticated) Source: https://github.com/bluesky-social/feed-generator/blob/main/_autodocs/api-reference/Methods.md Use curl with an Authorization header to test the `app.bsky.feed.getFeedSkeleton` endpoint with authentication. Includes example query parameters. ```bash curl -H "Authorization: Bearer YOUR_JWT_TOKEN" \ 'http://localhost:3000/xrpc/app.bsky.feed.getFeedSkeleton?feed=at://...&limit=30' ``` -------------------------------- ### Example Configuration: Using did:plc (Returns 404) Source: https://github.com/bluesky-social/feed-generator/blob/main/_autodocs/api-reference/WellKnown.md Illustrates the configuration for `did:plc` and the resulting 404 Not Found response for the well-known endpoint. When `did:plc` is used, the DID document is fetched from the PLC directory instead. ```bash # .env FEEDGEN_HOSTNAME=feeds.example.com FEEDGEN_SERVICE_DID=did:plc:z7r4qhxemkxd7fh... ``` ```text GET https://feeds.example.com/.well-known/did.json → 404 Not Found ``` ```text GET https://plc.directory/did:plc:z7r4qhxemkxd7fh... ``` -------------------------------- ### AT-URI Example Source: https://github.com/bluesky-social/feed-generator/blob/main/_autodocs/README.md Illustrates the format of an AT-URI, which is used to address resources within the AT Protocol. It breaks down the components: DID, collection, and rkey. ```text at://DID/collection/rkey Example: at://did:example:alice/app.bsky.feed.generator/whats-alf - DID: Account that published the record - collection: Type of record (e.g., `app.bsky.feed.generator`) - rkey: Record key (algorithm identifier) ``` -------------------------------- ### Install Feed Generator Dependencies Source: https://github.com/bluesky-social/feed-generator/blob/main/_autodocs/README.md Installs the necessary dependencies for the feed generator project using yarn. ```bash cd /workspace/home/feed-generator yarn install ``` -------------------------------- ### Database Query Examples Source: https://github.com/bluesky-social/feed-generator/blob/main/_autodocs/README.md Demonstrates common database operations including reading, inserting, updating, and deleting records using Kysely. ```typescript import { Database } from './db' // Read const posts = await db .selectFrom('post') .selectAll() .where('indexedAt', '>', twoHoursAgo) .orderBy('indexedAt', 'desc') .limit(50) .execute() // Insert await db .insertInto('post') .values({ uri, cid, indexedAt }) .onConflict((oc) => oc.doNothing()) .execute() // Update await db .updateTable('sub_state') .set({ cursor }) .where('service', '=', service) .execute() // Delete await db .deleteFrom('post') .where('uri', 'in', urisToDelete) .execute() ``` -------------------------------- ### Describe Feed Generator Response Example Source: https://github.com/bluesky-social/feed-generator/blob/main/_autodocs/endpoints.md Example JSON response from the `describeFeedGenerator` endpoint, including the service DID and feed URIs. ```json { "did": "did:web:example.com", "feeds": [ { "uri": "at://did:example:alice/app.bsky.feed.generator/whats-alf" } ] } ``` -------------------------------- ### Development Run Command Source: https://github.com/bluesky-social/feed-generator/blob/main/_autodocs/configuration.md Command to start the feed generator service after configuring it for development. ```bash yarn start # 🤖 running feed generator at http://localhost:3000 ``` -------------------------------- ### GET Request for app.bsky.feed.getFeedSkeleton Source: https://github.com/bluesky-social/feed-generator/blob/main/_autodocs/endpoints.md Example GET request to the getFeedSkeleton endpoint, specifying the feed URI and limit. Authentication is optional but required for personalized feeds. ```http GET /xrpc/app.bsky.feed.getFeedSkeleton?feed=at://...&limit=30[&cursor=...] Authorization: Bearer (optional) ``` -------------------------------- ### Example: Personalized Feed Handler (Auth Required) Source: https://github.com/bluesky-social/feed-generator/blob/main/_autodocs/api-reference/Auth.md Demonstrates a feed handler that requires authentication for personalized content. It uses `validateAuth` to get the requester's DID and then fetches user-specific data. ```typescript // This feed filters for posts from the requester's follows export default function(server: Server, ctx: AppContext) { server.app.bsky.feed.getFeedSkeleton(async ({ params, req }) => { const requesterDid = await validateAuth( req, ctx.cfg.serviceDid, ctx.didResolver, ) const feed = await getFollowingFeed(ctx.db, requesterDid, params) return { encoding: 'application/json', body: feed } }) } ``` -------------------------------- ### Configure DidResolver Source: https://github.com/bluesky-social/feed-generator/blob/main/_autodocs/api-reference/Auth.md Shows how to configure the `DidResolver` for fetching signing keys to verify JWT signatures. This example uses the official DID Placeholder service and an in-memory cache for performance. ```typescript import { DidResolver, MemoryCache } from '@atproto/identity' const didCache = new MemoryCache() const didResolver = new DidResolver({ plcUrl: 'https://plc.directory', didCache, }) ``` -------------------------------- ### Cursor Format Recommendations Source: https://github.com/bluesky-social/feed-generator/blob/main/_autodocs/api-reference/Algorithms.md Provides examples of recommended cursor formats for pagination, emphasizing the use of compound values for uniqueness and stability. ```typescript // Good: timestamp + CID ensures uniqueness and stability cursor = `${timestamp}::${cid}` // Acceptable: just timestamp if posts are spaced out cursor = timestamp.toString() // Bad: limit offset (can miss posts if content deleted/hidden) cursor = offset.toString() ``` -------------------------------- ### Algorithm for Checking User Follows Source: https://github.com/bluesky-social/feed-generator/blob/main/_autodocs/api-reference/Algorithms.md An example algorithm that queries posts from users followed by the requester. Requires implementing follow tracking in the database. ```typescript export const handler = async ( ctx: AppContext, params: QueryParams, ): Promise => { // Note: You would need to implement follow tracking in your database // and validate the requesting user first const userFollows = await ctx.db .selectFrom('follow') .select('targetDid') .where('userDid', '=', requesterDid) .execute() const followDids = userFollows.map((f) => f.targetDid) const feed = await ctx.db .selectFrom('post') .selectAll() .where('author', 'in', followDids) .orderBy('indexedAt', 'desc') .limit(params.limit) .execute() return { feed: feed.map((p) => ({ post: p.uri })) } ``` -------------------------------- ### Docker Deployment Configuration Source: https://github.com/bluesky-social/feed-generator/blob/main/_autodocs/README.md Defines a Dockerfile for building a feed generator image and provides example commands for running the container with environment variables. ```dockerfile FROM node:18 WORKDIR /app COPY . . RUN yarn install --production ENV FEEDGEN_LISTENHOST=0.0.0.0 ENV FEEDGEN_PORT=3000 ENV FEEDGEN_SQLITE_LOCATION=/data/feed.db EXPOSE 3000 CMD ["yarn", "start"] ``` ```bash docker build -t feed-generator . docker run \ -e FEEDGEN_HOSTNAME=feeds.example.com \ -e FEEDGEN_SERVICE_DID=did:web:feeds.example.com \ -e FEEDGEN_PUBLISHER_DID=did:plc:... -v /data:/data \ -p 3000:3000 \ feed-generator ``` -------------------------------- ### Firehose Subscription Recovery Example Source: https://github.com/bluesky-social/feed-generator/blob/main/_autodocs/api-reference/FirehoseSubscription.md Demonstrates how the run() method automatically handles connection errors and reconnections. Includes a try-catch block to log event processing failures and manage connection loss. ```typescript // The run() method automatically handles reconnection: try { for await (const evt of this.sub) { this.handleEvent(evt).catch((err) => { console.error('Failed to process event:', err) }) } } catch (err) { console.error('Connection lost, reconnecting...', err) setTimeout(() => this.run(delayMs), delayMs) } ``` -------------------------------- ### Dockerfile for Feed Generator Source: https://github.com/bluesky-social/feed-generator/blob/main/_autodocs/configuration.md This Dockerfile sets up the environment for the feed generator, including installing dependencies and setting environment variables for runtime configuration. ```dockerfile FROM node:18 WORKDIR /app COPY . . RUN yarn install ENV FEEDGEN_LISTENHOST=0.0.0.0 ENV FEEDGEN_PORT=3000 ENV FEEDGEN_SQLITE_LOCATION=/data/feed.db EXPOSE 3000 CMD ["yarn", "start"] ``` -------------------------------- ### FirehoseSubscriptionBase.run Method Source: https://github.com/bluesky-social/feed-generator/blob/main/_autodocs/api-reference/FirehoseSubscription.md Starts listening to the firehose and processes events indefinitely with automatic reconnection. Reconnects after the specified delay if disconnected. ```typescript async run(subscriptionReconnectDelay: number): Promise ``` ```typescript const firehose = new FirehoseSubscription(db, 'wss://bsky.network') await firehose.run(3000) // Reconnect every 3 seconds if disconnected ``` -------------------------------- ### Custom Firehose Implementation Example Source: https://github.com/bluesky-social/feed-generator/blob/main/_autodocs/api-reference/FirehoseSubscription.md Extends FirehoseSubscriptionBase to create a custom feed algorithm. Overrides handleEvent to filter posts based on specific topics and insert them into a database. ```typescript import { FirehoseSubscriptionBase, getOpsByType } from './util/subscription' class TopicalFeedSubscription extends FirehoseSubscriptionBase { async handleEvent(evt: RepoEvent) { if (!isCommit(evt)) return const ops = await getOpsByType(evt) // Filter for posts about specific topics const topicalPosts = ops.posts.creates.filter((create) => { return ( create.record.text.includes('climate') || create.record.text.includes('sustainability') ) }) const rows = topicalPosts.map((create) => ({ uri: create.uri, cid: create.cid, indexedAt: new Date().toISOString(), })) if (rows.length > 0) { await this.db.insertInto('post').values(rows).execute() } } } ``` -------------------------------- ### Handler Response Format with Headers Source: https://github.com/bluesky-social/feed-generator/blob/main/_autodocs/api-reference/Methods.md Example of a handler response including custom HTTP headers like 'Cache-Control'. ```typescript return { encoding: 'application/json', body: { ... }, headers: { 'Cache-Control': 'public, max-age=60', }, } ``` -------------------------------- ### Configure AT Protocol Firehose Subscription Endpoint Source: https://github.com/bluesky-social/feed-generator/blob/main/_autodocs/configuration.md Set the WebSocket URL for the AT Protocol firehose. Examples are provided for the official Bluesky firehose and a local testing sandbox. ```bash # Official Bluesky firehose FEEDGEN_SUBSCRIPTION_ENDPOINT=wss://bsky.network # Local sandbox for testing FEEDGEN_SUBSCRIPTION_ENDPOINT=ws://localhost:7000 ``` -------------------------------- ### Algorithm Registry Example Source: https://github.com/bluesky-social/feed-generator/blob/main/_autodocs/api-reference/Algorithms.md Demonstrates how to register feed algorithm handler functions using their shortnames. The `algos` object maps shortnames to their corresponding `AlgoHandler` implementations. ```typescript const algos: Record = { [whatsAlf.shortname]: whatsAlf.handler, } export default algos ``` -------------------------------- ### Response Body for app.bsky.feed.getFeedSkeleton Source: https://github.com/bluesky-social/feed-generator/blob/main/_autodocs/endpoints.md Example JSON response body for the getFeedSkeleton endpoint, containing a list of posts and a pagination cursor. ```json { "feed": [ { "post": "at://did:example:alice/app.bsky.feed.post/123" }, { "post": "at://did:example:bob/app.bsky.feed.post/456", "reason": { "$type": "app.bsky.feed.defs#skeletonReasonRepost", "repost": "at://did:example:carol/app.bsky.feed.repost/789" } } ], "cursor": "1683643619000" } ``` -------------------------------- ### Run Feed Generator Server Source: https://github.com/bluesky-social/feed-generator/blob/main/README.md Starts the feed generator server. The server will listen on port 3000 by default, or as defined in the .env file. You can observe the firehose output in the console. ```bash yarn start ``` -------------------------------- ### Test Well-Known Endpoint with curl Source: https://github.com/bluesky-social/feed-generator/blob/main/_autodocs/api-reference/WellKnown.md Provides a command-line example using `curl` to test the functionality of the well-known DID endpoint. This helps verify that the endpoint is accessible and returning the correct response. ```bash # Test the well-known endpoint curl -i https://example.com/.well-known/did.json ``` -------------------------------- ### Response Body for app.bsky.feed.describeFeedGenerator Source: https://github.com/bluesky-social/feed-generator/blob/main/_autodocs/endpoints.md Example JSON response body for the describeFeedGenerator endpoint, detailing the service DID and a list of available feeds. ```json { "did": "did:web:example.com", "feeds": [ { "uri": "at://did:example:alice/app.bsky.feed.generator/whats-alf" }, { "uri": "at://did:example:alice/app.bsky.feed.generator/climate-feed" } ] } ``` -------------------------------- ### FirehoseSubscriptionBase.run Source: https://github.com/bluesky-social/feed-generator/blob/main/_autodocs/api-reference/FirehoseSubscription.md Starts listening to the firehose and processes events indefinitely, with automatic reconnection capabilities. It updates the cursor every 20 events and schedules reconnections on failure. ```APIDOC ## FirehoseSubscriptionBase.run ### Description Starts listening to the firehose and processes events indefinitely with automatic reconnection. It iterates over incoming events, calls `handleEvent()`, updates the cursor every 20 events, and schedules reconnection attempts on connection failure. ### Parameters #### Path Parameters - **subscriptionReconnectDelay** (number) - Yes - Milliseconds to wait before reconnecting after disconnection ### Returns `Promise` ### Request Example ```typescript const firehose = new FirehoseSubscription(db, 'wss://bsky.network') await firehose.run(3000) // Reconnect every 3 seconds if disconnected ``` ``` -------------------------------- ### AtUri Class Definition and Example Source: https://github.com/bluesky-social/feed-generator/blob/main/_autodocs/types.md Provides static methods for creating and instance properties for parsing AT-URIs. Use for manipulating AT Protocol resource identifiers. ```typescript class AtUri { static make(did: string, collection: string, rkey: string): AtUri readonly did: string readonly hostname: string // Alias for did readonly collection: string readonly rkey: string toString(): string } // Example: const feedUri = new AtUri('at://did:example:alice/app.bsky.feed.generator/whats-alf') console.log(feedUri.did) // 'did:example:alice' console.log(feedUri.collection) // 'app.bsky.feed.generator' console.log(feedUri.rkey) // 'whats-alf' ``` -------------------------------- ### Initialize Subscription Cursor Source: https://github.com/bluesky-social/feed-generator/blob/main/_autodocs/api-reference/Database.md Initializes a new subscription cursor for a given service in the 'sub_state' table with a starting value of 0. Uses 'onConflict((oc) => oc.doNothing())' to prevent overwriting existing entries. ```typescript await db .insertInto('sub_state') .values({ service: 'wss://bsky.network', cursor: 0, }) .onConflict((oc) => oc.doNothing()) .execute() ``` -------------------------------- ### Configure SQLite Database Location Source: https://github.com/bluesky-social/feed-generator/blob/main/_autodocs/configuration.md Specify the path for the SQLite database file. Use ':memory:' for in-memory testing or a file path for persistent storage. The example shows both file-based and in-memory configurations. ```bash # File-based (recommended for production) FEEDGEN_SQLITE_LOCATION=/data/feed.db # In-memory (testing only) FEEDGEN_SQLITE_LOCATION=:memory: ``` -------------------------------- ### Describe Generator Response Format Source: https://github.com/bluesky-social/feed-generator/blob/main/_autodocs/api-reference/Methods.md Example JSON response format for the describeFeedGenerator handler, including the service DID and a list of available feed URIs. ```json { "did": "did:web:example.com", "feeds": [ { "uri": "at://did:example:alice/app.bsky.feed.generator/whats-alf" }, { "uri": "at://did:example:alice/app.bsky.feed.generator/another-feed" } ] } ``` -------------------------------- ### Example: Generic Feed Handler (No Auth Needed) Source: https://github.com/bluesky-social/feed-generator/blob/main/_autodocs/api-reference/Auth.md Shows a feed handler that does not require authentication, suitable for generic feeds that provide the same content to all users. This simplifies the handler by omitting the `validateAuth` call. ```typescript // This feed returns the same posts for all users export default function(server: Server, ctx: AppContext) { server.app.bsky.feed.getFeedSkeleton(async ({ params }) => { const feed = await getTopPosts(ctx.db, params) return { encoding: 'application/json', body: feed } }) } ``` -------------------------------- ### Example: Using validateAuth in a Feed Handler Source: https://github.com/bluesky-social/feed-generator/blob/main/_autodocs/api-reference/Auth.md Demonstrates how to use the `validateAuth` function within a feed handler to authenticate a requester before providing personalized results. Ensure the `requesterDid` is used for personalized content. ```typescript import express from 'express' import { validateAuth } from './auth' import { AppContext } from './config' export default function(server: Server, ctx: AppContext) { server.app.bsky.feed.getFeedSkeleton(async ({ params, req }) => { // Validate that the requester is authenticated const requesterDid = await validateAuth( req, ctx.cfg.serviceDid, ctx.didResolver, ) // Now use the requesterDid to provide personalized results const feed = await getPersonalizedFeed(ctx.db, requesterDid, params) return { encoding: 'application/json', body: feed, } }) } ``` -------------------------------- ### FirehoseSubscriptionBase Constructor Source: https://github.com/bluesky-social/feed-generator/blob/main/_autodocs/api-reference/FirehoseSubscription.md Initializes a FirehoseSubscriptionBase object, establishing a connection to the firehose, setting up message validation, and loading the stored cursor from the database to resume from the last processed event. ```APIDOC ## FirehoseSubscriptionBase Constructor ### Description Initializes a Subscription object that connects to the firehose, sets up message validation against the AT Protocol lexicon, and loads the stored cursor from the database to resume from the last processed event. ### Parameters #### Path Parameters - **db** (Database) - Required - Kysely database for storing subscription cursor state - **service** (string) - Required - Service endpoint URL (e.g., 'wss://bsky.network') ``` -------------------------------- ### Get Feed Skeleton Source: https://github.com/bluesky-social/feed-generator/blob/main/_autodocs/INDEX.md Retrieves a paginated list of posts for a given feed. This is the primary endpoint for fetching feed content. ```APIDOC ## GET /xrpc/app.bsky.feed.getFeedSkeleton ### Description Retrieves a paginated list of posts for a given feed. This is the primary endpoint for fetching feed content. ### Method GET ### Endpoint /xrpc/app.bsky.feed.getFeedSkeleton ### Parameters #### Query Parameters - **feed** (string) - Required - The URI of the feed to retrieve. - **limit** (integer) - Optional - The maximum number of posts to return. - **cursor** (string) - Optional - A cursor for fetching the next page of results. ### Response #### Success Response (200) - **feed** (SkeletonFeedPost[]) - An array of feed posts. - **cursor** (string) - Optional - A cursor for fetching the next page of results. ``` -------------------------------- ### Run Feed Generator Server with Docker Source: https://github.com/bluesky-social/feed-generator/blob/main/README.md Builds and runs the feed generator server using Docker. This provides a containerized environment for the application. ```bash docker build -t feed-generator . docker run feed-generator ``` -------------------------------- ### Get Current Subscription Cursor Source: https://github.com/bluesky-social/feed-generator/blob/main/_autodocs/api-reference/Database.md Selects the current subscription cursor for a specific service from the 'sub_state' table. Returns undefined if no state is found. ```typescript const state = await db .selectFrom('sub_state') .selectAll() .where('service', '=', 'wss://bsky.network') .executeTakeFirst() console.log(`Cursor: ${state?.cursor ?? 'not set'}`) ``` -------------------------------- ### FirehoseSubscription Constructor Source: https://github.com/bluesky-social/feed-generator/blob/main/_autodocs/api-reference/FirehoseSubscription.md Initializes a FirehoseSubscription object, which is a concrete implementation that filters for ALF-related posts and stores them in the database. ```APIDOC ## FirehoseSubscription Constructor ### Description Initializes a FirehoseSubscription object, which is a concrete implementation that filters for ALF-related posts and stores them in the database. It inherits all functionalities from FirehoseSubscriptionBase. ### Parameters #### Path Parameters - **db** (Database) - Required - Kysely database for storing subscription cursor state - **service** (string) - Required - Service endpoint URL (e.g., 'wss://bsky.network') ``` -------------------------------- ### Set FEEDGEN_LISTENHOST Environment Variable Source: https://github.com/bluesky-social/feed-generator/blob/main/_autodocs/configuration.md Configure the IP address or hostname to bind the server to. Use '0.0.0.0' to accept connections from any address, which is recommended for Docker deployments. ```bash FEEDGEN_LISTENHOST=0.0.0.0 ``` -------------------------------- ### Configure Firehose Reconnection Delay Source: https://github.com/bluesky-social/feed-generator/blob/main/_autodocs/configuration.md Specify the delay in milliseconds before attempting to reconnect to the firehose if the connection is lost. Examples show a 3-second and a 10-second delay. ```bash # 3 second delay FEEDGEN_SUBSCRIPTION_RECONNECT_DELAY=3000 # 10 second delay FEEDGEN_SUBSCRIPTION_RECONNECT_DELAY=10000 ``` -------------------------------- ### SQL CREATE TABLE for Post Source: https://github.com/bluesky-social/feed-generator/blob/main/_autodocs/api-reference/Database.md Defines the SQL schema for the 'post' table, specifying columns and constraints. ```sql CREATE TABLE post ( uri VARCHAR PRIMARY KEY, cid VARCHAR NOT NULL, indexedAt VARCHAR NOT NULL ) ``` -------------------------------- ### GET /.well-known/did.json Source: https://github.com/bluesky-social/feed-generator/blob/main/_autodocs/api-reference/WellKnown.md Returns the DID document for the feed generator service. This endpoint is crucial for `did:web` resolution, allowing other services to discover and interact with the feed generator. ```APIDOC ## GET /.well-known/did.json ### Description Returns the DID document for the feed generator service. Required for `did:web` resolution. ### Method GET ### Endpoint /.well-known/did.json ### Parameters No query parameters or request body. ### Response #### Success Response (200 OK) - **@context** (string[]) - Required - Standard DID context URLs - **id** (string) - Required - The service's DID - **service** (object[]) - Required - Array of service endpoints - **service[].id** (string) - Required - Fragment identifier for this service - **service[].type** (string) - Required - Service type identifier - **service[].serviceEndpoint** (string) - Required - URL where XRPC endpoints are available #### Response Example ```json { "@context": ["https://www.w3.org/ns/did/v1"], "id": "did:web:example.com", "service": [ { "id": "#bsky_fg", "type": "BskyFeedGenerator", "serviceEndpoint": "https://example.com" } ] } ``` #### Error Response (404 Not Found) Returned if `serviceDid` is `did:plc` or other non-web DID. ``` -------------------------------- ### FirehoseSubscriptionBase Constructor Source: https://github.com/bluesky-social/feed-generator/blob/main/_autodocs/api-reference/FirehoseSubscription.md Initializes a Subscription object that connects to the firehose, sets up message validation, and loads the stored cursor from the database. ```typescript constructor( public db: Database, public service: string, ) ``` -------------------------------- ### Accessing Request Context in Algorithms Source: https://github.com/bluesky-social/feed-generator/blob/main/_autodocs/api-reference/Algorithms.md Demonstrates how to access the request context, including the database instance, configuration, and DID resolver, within an algorithm handler. ```typescript ctx.db // Database instance for querying posts ctx.cfg // Configuration (hostname, serviceDid, publisherDid) ctx.didResolver // DID resolver for validating users ``` -------------------------------- ### Recommended Cursor Format Source: https://github.com/bluesky-social/feed-generator/blob/main/README.md A recommended format for the pagination cursor, combining a timestamp and a CID for uniqueness. ```text 1683654690921::bafyreia3tbsfxe3cc75xrxyyn6qc42oupi73fxiox76prlyi5bpx7hr72u ``` -------------------------------- ### Registering a New Algorithm Source: https://github.com/bluesky-social/feed-generator/blob/main/_autodocs/api-reference/Algorithms.md Shows how to import and register a new feed generator algorithm in the main `src/algos/index.ts` file. ```typescript import * as climate from './climate' const algos: Record = { [whatsAlf.shortname]: whatsAlf.handler, [climate.shortname]: climate.handler, // Add new algorithm } ``` -------------------------------- ### Feed Generator Main Entry Point Configuration Source: https://github.com/bluesky-social/feed-generator/blob/main/_autodocs/configuration.md Loads environment variables using dotenv and configures the FeedGenerator with various settings, providing defaults for missing values. ```typescript import dotenv from 'dotenv' import FeedGenerator from './server' const run = async () => { dotenv.config() const hostname = maybeStr(process.env.FEEDGEN_HOSTNAME) ?? 'example.com' const serviceDid = maybeStr(process.env.FEEDGEN_SERVICE_DID) ?? `did:web:${hostname}` const server = FeedGenerator.create({ port: maybeInt(process.env.FEEDGEN_PORT) ?? 3000, listenhost: maybeStr(process.env.FEEDGEN_LISTENHOST) ?? 'localhost', sqliteLocation: maybeStr(process.env.FEEDGEN_SQLITE_LOCATION) ?? ':memory:', subscriptionEndpoint: maybeStr(process.env.FEEDGEN_SUBSCRIPTION_ENDPOINT) ?? 'wss://bsky.network', publisherDid: maybeStr(process.env.FEEDGEN_PUBLISHER_DID) ?? 'did:example:alice', subscriptionReconnectDelay: maybeInt(process.env.FEEDGEN_SUBSCRIPTION_RECONNECT_DELAY) ?? 3000, hostname, serviceDid, }) await server.start() } const maybeStr = (val?: string) => val || undefined const maybeInt = (val?: string) => { if (!val) return undefined const int = parseInt(val, 10) return isNaN(int) ? undefined : int } ``` -------------------------------- ### Error Handling for AuthRequiredError Source: https://github.com/bluesky-social/feed-generator/blob/main/_autodocs/api-reference/Auth.md Provides an example of how to catch and handle `AuthRequiredError` and other JWT verification errors when calling `validateAuth`. This ensures appropriate HTTP responses (e.g., 401 Unauthorized) are returned. ```typescript try { const userDid = await validateAuth(req, serviceDid, didResolver) } catch (err) { if (err instanceof AuthRequiredError) { // Return 401 Unauthorized return { status: 401, message: 'Authentication required', } } // Other JWT verification errors return { status: 401, message: 'Invalid authentication', } } ``` -------------------------------- ### Feed Generator - All Interfaces Networking Source: https://github.com/bluesky-social/feed-generator/blob/main/_autodocs/configuration.md Configuration for accepting connections from any network interface, recommended for Docker and Kubernetes deployments. Ensures the service is accessible externally. ```bash FEEDGEN_LISTENHOST=0.0.0.0 FEEDGEN_HOSTNAME=example.com ``` -------------------------------- ### Catching Database Constraint Errors Source: https://github.com/bluesky-social/feed-generator/blob/main/_autodocs/errors.md Shows how to catch specific database errors, like SQLite constraint violations, and handle them differently from other database errors. This example ignores duplicate key errors. ```typescript try { await db.insertInto('post').values(posts).execute() } catch (err: any) { if (err.code === 'SQLITE_CONSTRAINT') { console.log('Duplicate key, ignoring') } else { throw err } } ``` -------------------------------- ### FeedGenerator.create Source: https://github.com/bluesky-social/feed-generator/blob/main/_autodocs/api-reference/FeedGenerator.md Factory method to construct and initialize a FeedGenerator instance with all dependencies. It sets up the Express application, database, firehose subscription, and registers XRPC handlers for feed generation and description endpoints. ```APIDOC ## Static Method: create Factory method to construct and initialize a FeedGenerator instance with all dependencies. ```typescript static create(cfg: Config): FeedGenerator ``` **Parameters** | Parameter | Type | Required | Default | Description | |-----------|------|----------|---------|-------------| | cfg | Config | yes | — | Configuration object with port, hostname, database location, and subscription settings | **Returns** `FeedGenerator` — A new FeedGenerator instance with initialized app, database, firehose, and XRPC server with registered handlers for `app.bsky.feed.getFeedSkeleton` and `app.bsky.feed.describeFeedGenerator` endpoints. **Behavior** - Creates an Express application - Initializes SQLite database at the specified location - Sets up a FirehoseSubscription for monitoring the AT Protocol firehose - Creates a DID resolver with memory cache for JWT verification - Registers an XRPC server with payload limits (100KB JSON, 100KB text, 5MB blobs) - Wires up feed generation and feed description handlers - Mounts the well-known DID endpoint **Example** ```typescript import FeedGenerator from './server' const server = FeedGenerator.create({ port: 3000, listenhost: 'localhost', hostname: 'example.com', sqliteLocation: '/tmp/feed.db', subscriptionEndpoint: 'wss://bsky.network', publisherDid: 'did:example:alice', subscriptionReconnectDelay: 3000, serviceDid: 'did:web:example.com', }) ``` ``` -------------------------------- ### SQL CREATE TABLE for SubState Source: https://github.com/bluesky-social/feed-generator/blob/main/_autodocs/api-reference/Database.md Defines the SQL schema for the 'sub_state' table, including service endpoint and cursor. ```sql CREATE TABLE sub_state ( service VARCHAR PRIMARY KEY, cursor INTEGER NOT NULL ) ``` -------------------------------- ### Project Structure Overview Source: https://github.com/bluesky-social/feed-generator/blob/main/_autodocs/README.md This snippet outlines the directory and file structure of the feed-generator project. It shows the organization of source code, scripts, and configuration files. ```tree feed-generator/ ├── src/ │ ├── index.ts # Entry point │ ├── server.ts # FeedGenerator class │ ├── config.ts # Config and AppContext types │ ├── auth.ts # JWT validation │ ├── well-known.ts # DID document endpoint │ ├── subscription.ts # Firehose event handler │ ├── algos/ │ │ ├── index.ts # Algorithm registry │ │ └── whats-alf.ts # Example algorithm │ ├── db/ │ │ ├── index.ts # Database creation │ │ ├── schema.ts # Database types │ │ └── migrations.ts # Schema migrations │ ├── methods/ │ │ ├── feed-generation.ts # getFeedSkeleton handler │ │ └── describe-generator.ts # describeFeedGenerator handler │ ├── util/ │ │ └── subscription.ts # Firehose utilities │ └── lexicon/ # Generated AT Protocol types ├── scripts/ │ ├── publishFeedGen.ts # Publish feed to network │ └── unpublishFeedGen.ts # Unpublish feed ├── package.json ├── tsconfig.json └── .env.example ``` -------------------------------- ### Interacting with SQLite Database Source: https://github.com/bluesky-social/feed-generator/blob/main/_autodocs/errors.md Provides commands for interacting with the feed generator's SQLite database using the sqlite3 CLI. This includes listing tables, querying data, and checking subscription states. ```bash # Using sqlite3 CLI sqlite3 feed.db # List tables sqlite3 feed.db ".tables" # Query posts sqlite3 feed.db "SELECT COUNT(*) FROM post;" # Check subscription state sqlite3 feed.db "SELECT * FROM sub_state;" ``` -------------------------------- ### Apply Database Migrations Source: https://github.com/bluesky-social/feed-generator/blob/main/_autodocs/api-reference/Database.md Asynchronously applies all pending database migrations to ensure the schema is up to date. This function requires an initialized Kysely database instance. ```typescript import { createDb, migrateToLatest } from './db' const db = createDb(':memory:') await migrateToLatest(db) // Schema is now initialized ``` -------------------------------- ### Whats-Alf Algorithm Handler Implementation Source: https://github.com/bluesky-social/feed-generator/blob/main/_autodocs/api-reference/Algorithms.md Provides the implementation for the 'whats-alf' feed algorithm. This handler fetches posts containing 'alf', sorts them by index time, and supports cursor-based pagination. ```typescript export const handler = async (ctx: AppContext, params: QueryParams) => { let builder = ctx.db .selectFrom('post') .selectAll() .orderBy('indexedAt', 'desc') .orderBy('cid', 'desc') .limit(params.limit) // Pagination using cursor (unix timestamp) if (params.cursor) { const timeStr = new Date(parseInt(params.cursor, 10)).toISOString() builder = builder.where('post.indexedAt', '<', timeStr) } const res = await builder.execute() // Convert rows to feed skeleton const feed = res.map((row) => ({ post: row.uri, })) // Generate cursor for next page let cursor: string | undefined const last = res.at(-1) if (last) { cursor = new Date(last.indexedAt).getTime().toString(10) } return { cursor, feed } } ``` -------------------------------- ### Describe Generator Algorithm Discovery Source: https://github.com/bluesky-social/feed-generator/blob/main/_autodocs/api-reference/Methods.md Discovers available algorithms by reading the 'algos' registry and constructs AT-URIs for each. ```typescript const feeds = Object.keys(algos).map((shortname) => ({ uri: AtUri.make( ctx.cfg.publisherDid, 'app.bsky.feed.generator', shortname, ).toString(), })) ``` -------------------------------- ### FeedGenerator Server Architecture Source: https://github.com/bluesky-social/feed-generator/blob/main/_autodocs/README.md Illustrates the core components of the FeedGenerator server, including its Express app, database, firehose subscription, DID resolver, and XRPC server. ```text FeedGenerator.create(config) ├── Express app ├── SQLite database (via Kysely) ├── Firehose subscription ├── DID resolver └── XRPC server ├── app.bsky.feed.getFeedSkeleton handler ├── app.bsky.feed.describeFeedGenerator handler └── /.well-known/did.json endpoint ``` -------------------------------- ### createDb Source: https://github.com/bluesky-social/feed-generator/blob/main/_autodocs/api-reference/Database.md Factory function that creates a Kysely database instance connected to a SQLite database. It supports both file paths and in-memory databases for testing. ```APIDOC ## Function: createDb ### Description Factory function that creates a Kysely database instance connected to a SQLite database. ### Parameters #### Path Parameters - **location** (string) - Required - Path to SQLite database file or `:memory:` for in-memory database ### Returns - **Database** — A Kysely instance typed to the DatabaseSchema. ### Example ```typescript import { createDb } from './db' const db = createDb('/tmp/feed.db') const db = createDb(':memory:') // For testing ``` ``` -------------------------------- ### Ensuring Directory Exists for SQLite Source: https://github.com/bluesky-social/feed-generator/blob/main/_autodocs/errors.md Ensure the directory for the SQLite database file exists and is writable. Alternatively, use an in-memory database for testing purposes. ```bash # Ensure directory exists mkdir -p /data # Or use in-memory for testing FEEDGEN_SQLITE_LOCATION=:memory: ``` -------------------------------- ### Algorithm Error Handling Source: https://github.com/bluesky-social/feed-generator/blob/main/_autodocs/api-reference/Algorithms.md Illustrates how to implement error handling within an algorithm by wrapping the logic in a try-catch block and logging any errors. ```typescript export const handler = async ( ctx: AppContext, params: QueryParams, ): Promise => { try { // Algorithm logic } catch (err) { console.error('Algorithm execution failed:', err) throw err // Will be caught by XRPC handler } } ```