### Install X Algorithm Skills for Claude Code Source: https://github.com/cloudai-x/x-algo-skills/blob/main/README.md Installs the X algorithm skills for Claude Code. This involves copying the skill files to either a project-specific or global skills directory. Ensure the path to your project's .claude/skills directory is correct for project-scoped installation. ```bash # Project-scoped (recommended) cp -r x-algo-* /path/to/your/project/.claude/skills/ # Or global cp -r x-algo-* ~/.claude/skills/ ``` -------------------------------- ### X Recommendation Pipeline Stages Source: https://context7.com/cloudai-x/x-algo-skills/llms.txt Provides an overview of the 8-stage X recommendation pipeline, from initial query hydration and candidate retrieval through scoring, filtering, and final selection. It includes examples of source enablement logic and a top-K selection strategy. ```rust // Complete pipeline flow // Stage 1: Query Hydration // - Load user features, action history, socialgraph // - Load muted keywords, subscriptions, Bloom filters // Stage 2: Sources (Candidate Retrieval) // Thunder Source - In-network posts from followed accounts served_type: Some(pb::ServedType::ForYouInNetwork) // Phoenix Source - Out-of-network ML retrieval fn enable(&self, query: &ScoredPostsQuery) -> bool { !query.in_network_only // Disabled for "Following" tab } served_type: Some(pb::ServedType::ForYouPhoenixRetrieval) // Stage 3: Candidate Hydration // - Fetch tweet text, author data, media metadata // - Run visibility filtering // Stage 4: Pre-Score Filtering // AgeFilter, DropDuplicatesFilter, VFFilter, // AuthorSocialgraphFilter, CoreDataHydrationFilter // Stage 5: Scoring Pipeline // PhoenixScorer → WeightedScorer → AuthorDiversityScorer → OONScorer // Stage 6: Selection pub struct TopKScoreSelector; impl Selector for TopKScoreSelector { fn score(&self, candidate: &PostCandidate) -> f64 { candidate.score.unwrap_or(f64::NEG_INFINITY) } fn size(&self) -> Option { Some(params::TOP_K_CANDIDATES_TO_SELECT) } } // Stage 7: Post-Score Filtering // DedupConversationFilter, RetweetDeduplicationFilter, // PreviouslySeenPostsFilter, MutedKeywordFilter, SelfTweetFilter // Stage 8: Side Effects // Impression logging, analytics, cache updates ``` -------------------------------- ### Convert Logits to Probabilities (Python) Source: https://github.com/cloudai-x/x-algo-skills/blob/main/x-algo-ml/SKILL.md Demonstrates how to convert the model's output logits (log-probabilities) into actual probabilities using the exponential function. This is a standard operation for converting log-probabilities. ```python probability = exp(log_prob) ``` -------------------------------- ### Phoenix Model Configuration Source: https://github.com/cloudai-x/x-algo-skills/blob/main/x-algo-ml/SKILL.md Defines the configuration parameters for the Phoenix ML model, including transformer settings, embedding dimensions, and sequence lengths. This structure is essential for initializing and training the recommendation model. ```python from dataclasses import dataclass @dataclass class TransformerConfig: # Placeholder for transformer configuration pass @dataclass class HashConfig: num_user_hashes: int = 2 # Hash user ID 2 ways num_item_hashes: int = 2 # Hash post ID 2 ways num_author_hashes: int = 2 # Hash author ID 2 ways @dataclass class PhoenixModelConfig: model: TransformerConfig # Grok-1 based transformer emb_size: int # Embedding dimension D num_actions: int # 18 action types history_seq_len: int = 128 # User history length candidate_seq_len: int = 32 # Candidates per batch product_surface_vocab_size: int = 16 # Where post was seen hash_config: HashConfig # Hash embedding config ``` -------------------------------- ### Phoenix ML Model Configuration (Python) Source: https://context7.com/cloudai-x/x-algo-skills/llms.txt Defines the configuration for the Phoenix ML model, a two-stage transformer architecture for predicting engagement probabilities. It includes settings for the base transformer model, embedding dimensions, number of actions, sequence lengths, and hashing configurations for user, item, and author IDs. ```python # phoenix/recsys_model.py @dataclass class PhoenixModelConfig: model: TransformerConfig # Grok-1 based transformer emb_size: int # Embedding dimension D num_actions: int # 18 action types history_seq_len: int = 128 # User history length candidate_seq_len: int = 32 # Candidates per batch product_surface_vocab_size: int = 16 # Where post was seen hash_config: HashConfig # Hash embedding config @dataclass class HashConfig: num_user_hashes: int = 2 # Hash user ID 2 ways num_item_hashes: int = 2 # Hash post ID 2 ways num_author_hashes: int = 2 # Hash author ID 2 ways ``` -------------------------------- ### Bloom Filter Deduplication Logic (Rust) Source: https://github.com/cloudai-x/x-algo-skills/blob/main/x-algo-filters/SKILL.md Explains the use of Bloom filters in the PreviouslySeenPostsFilter for efficient 'seen' post tracking. It covers client-side entry creation, server-side filter reconstruction, probabilistic 'may_contain()' checks, and fallback to explicit ID checks. ```rust - Client sends Bloom filter entries with request - Server reconstructs filters via `BloomFilter::from_entry` - Uses `may_contain()` (probabilistic) for fast lookup - Falls back to explicit `seen_ids` for definitive checks ``` -------------------------------- ### Conditional Filter Enabling Logic (Rust) Source: https://github.com/cloudai-x/x-algo-skills/blob/main/x-algo-filters/SKILL.md Demonstrates conditional filter enabling, specifically for the PreviouslyServedPostsFilter. This filter is designed to only run during pagination requests, as indicated by the 'is_bottom_request' flag in the ScoredPostsQuery. ```rust // PreviouslyServedPostsFilter only runs on pagination fn enable(&self, query: &ScoredPostsQuery) -> bool { query.is_bottom_request } ``` -------------------------------- ### PostCandidate Data Structure - Rust Source: https://github.com/cloudai-x/x-algo-skills/blob/main/x-algo-pipeline/SKILL.md The PostCandidate structure holds all relevant information about a potential post to be displayed in the feed. It includes identifiers, text content, scores from various stages, source information, and conversation context. ```rust pub struct PostCandidate { pub tweet_id: i64, pub author_id: u64, pub tweet_text: String, pub in_reply_to_tweet_id: Option, pub retweeted_tweet_id: Option, pub retweeted_user_id: Option, pub phoenix_scores: PhoenixScores, // ML predictions pub weighted_score: Option, // After WeightedScorer pub score: Option, // Final score pub served_type: Option, pub in_network: Option, pub ancestors: Vec, pub video_duration_ms: Option, pub visibility_reason: Option, pub subscription_author_id: Option, // ... } ``` -------------------------------- ### Filter Previously Seen Posts using Bloom Filters and ID Lists (Rust) Source: https://context7.com/cloudai-x/x-algo-skills/llms.txt Efficiently filters posts a user has already viewed using Bloom filters for probabilistic checks and explicit ID lists for exact matches. It partitions candidate posts into those removed (seen) and those kept. ```rust // home-mixer/filters/previously_seen_posts_filter.rs let (removed, kept) = candidates.into_iter().partition(|c| { get_related_post_ids(c).iter().any(|&post_id| { // Check explicit seen IDs list query.seen_ids.contains(&post_id) // Check Bloom filters (probabilistic but fast) || bloom_filters .iter() .any(|filter| filter.may_contain(post_id)) }) }); // Client sends Bloom filter entries with request // Server reconstructs via BloomFilter::from_entry // may_contain() provides fast probabilistic lookup ``` -------------------------------- ### Transformer Forward Pass with Candidate Isolation (Python) Source: https://github.com/cloudai-x/x-algo-skills/blob/main/x-algo-ml/SKILL.md This method orchestrates the transformer model's forward pass. It builds the combined input embeddings, applies the candidate isolation mask during the transformer computation, extracts candidate outputs, and projects them to action logits. The `candidate_start_offset` parameter is crucial for the attention masking. ```python def __call__(self, batch, recsys_embeddings) -> RecsysModelOutput: # 1. Build combined embeddings embeddings, padding_mask, candidate_start = self.build_inputs(batch, recsys_embeddings) # 2. Pass through transformer (with candidate isolation mask) model_output = self.model( embeddings, padding_mask, candidate_start_offset=candidate_start, # For attention masking ) # 3. Extract candidate outputs out_embeddings = layer_norm(model_output.embeddings) candidate_embeddings = out_embeddings[:, candidate_start:, :] # 4. Project to action logits logits = jnp.dot(candidate_embeddings, unembeddings) # Shape: [B, num_candidates, num_actions] return RecsysModelOutput(logits=logits) ``` -------------------------------- ### Generate Product Surface Embeddings (Python) Source: https://github.com/cloudai-x/x-algo-skills/blob/main/x-algo-ml/SKILL.md This utility function converts single-hot encoded product surface inputs into dense embeddings. It utilizes a standard embedding lookup table, creating embeddings for different engagement surfaces like home feed, search, etc. ```python def _single_hot_to_embeddings(self, input, vocab_size, emb_size, name): # Standard embedding lookup table embedding_table = hk.get_parameter(name, [vocab_size, emb_size]) input_one_hot = jax.nn.one_hot(input, vocab_size) return jnp.dot(input_one_hot, embedding_table) ``` -------------------------------- ### RecsysBatch Input Structure Source: https://github.com/cloudai-x/x-algo-skills/blob/main/x-algo-ml/SKILL.md Defines the structure for input data batches used in the recommendation system. It includes user identification, engagement history, and candidate information required for scoring. ```python from typing import NamedTuple from jax.Array import ArrayLike # Assuming jax.Array for ArrayLike class RecsysBatch(NamedTuple): # User identification user_hashes: ArrayLike # [B, num_user_hashes] # User engagement history history_post_hashes: ArrayLike # [B, S, num_item_hashes] history_author_hashes: ArrayLike # [B, S, num_author_hashes] history_actions: ArrayLike # [B, S, num_actions] history_product_surface: ArrayLike # [B, S] # Candidates to score candidate_post_hashes: ArrayLike # [B, C, num_item_hashes] candidate_author_hashes: ArrayLike # [B, C, num_author_hashes] candidate_product_surface: ArrayLike # [B, C] ``` -------------------------------- ### Phoenix Source (Out-of-Network) - Rust Source: https://github.com/cloudai-x/x-algo-skills/blob/main/x-algo-pipeline/SKILL.md The Phoenix source uses ML-based retrieval with user embeddings to find relevant posts from the entire corpus. It's enabled for 'For You' feeds but disabled for 'Following' tabs, indicated by the 'ForYouPhoenixRetrieval' served type. ```rust // home-mixer/sources/phoenix_source.rs fn enable(&self, query: &ScoredPostsQuery) -> bool { !query.in_network_only // Disabled for "Following" tab } served_type: Some(pb::ServedType::ForYouPhoenixRetrieval) ``` -------------------------------- ### PhoenixScorer Implementation - Rust Source: https://github.com/cloudai-x/x-algo-skills/blob/main/x-algo-pipeline/SKILL.md The PhoenixScorer is responsible for calling the Phoenix ML model to predict engagement probabilities for posts. It generates 'phoenix_scores' which include 18 different action probabilities. ```rust // home-mixer/scorers/phoenix_scorer.rs // Calls Phoenix ML to predict engagement probabilities ``` -------------------------------- ### Calculate Weighted Score for Posts (Rust) Source: https://context7.com/cloudai-x/x-algo-skills/llms.txt Computes a single ranking score for a post by combining 18 engagement predictions with configurable weights. It applies weights to various scores, handles negative engagement signals, and adjusts the final score. Dependencies include `PostCandidate` and `PhoenixScores` structs, along with predefined weight constants. ```rust // home-mixer/scorers/weighted_scorer.rs fn compute_weighted_score(candidate: &PostCandidate) -> f64 { let s: &PhoenixScores = &candidate.phoenix_scores; let vqv_weight = Self::vqv_weight_eligibility(candidate); let combined_score = Self::apply(s.favorite_score, p::FAVORITE_WEIGHT) + Self::apply(s.reply_score, p::REPLY_WEIGHT) + Self::apply(s.retweet_score, p::RETWEET_WEIGHT) + Self::apply(s.photo_expand_score, p::PHOTO_EXPAND_WEIGHT) + Self::apply(s.click_score, p::CLICK_WEIGHT) + Self::apply(s.profile_click_score, p::PROFILE_CLICK_WEIGHT) + Self::apply(s.vqv_score, vqv_weight) + Self::apply(s.share_score, p::SHARE_WEIGHT) + Self::apply(s.share_via_dm_score, p::SHARE_VIA_DM_WEIGHT) + Self::apply(s.share_via_copy_link_score, p::SHARE_VIA_COPY_LINK_WEIGHT) + Self::apply(s.dwell_score, p::DWELL_WEIGHT) + Self::apply(s.quote_score, p::QUOTE_WEIGHT) + Self::apply(s.quoted_click_score, p::QUOTED_CLICK_WEIGHT) + Self::apply(s.dwell_time, p::CONT_DWELL_TIME_WEIGHT) + Self::apply(s.follow_author_score, p::FOLLOW_AUTHOR_WEIGHT) + Self::apply(s.not_interested_score, p::NOT_INTERESTED_WEIGHT) // Negative + Self::apply(s.block_author_score, p::BLOCK_AUTHOR_WEIGHT) // Negative + Self::apply(s.mute_author_score, p::MUTE_AUTHOR_WEIGHT) // Negative + Self::apply(s.report_score, p::REPORT_WEIGHT); // Negative Self::offset_score(combined_score) } // VQV (Video Quality View) only applies to videos above minimum duration fn vqv_weight_eligibility(candidate: &PostCandidate) -> f64 { if candidate.video_duration_ms.is_some_and(|ms| ms > p::MIN_VIDEO_DURATION_MS) { p::VQV_WEIGHT } else { 0.0 // No VQV contribution for short videos or non-videos } } // Score offset handles negative combined scores fn offset_score(combined_score: f64) -> f64 { if p::WEIGHTS_SUM == 0.0 { combined_score.max(0.0) } else if combined_score < 0.0 { (combined_score + p::NEGATIVE_SCORES_SUM) / p::WEIGHTS_SUM * p::NEGATIVE_SCORES_OFFSET } else { combined_score + p::NEGATIVE_SCORES_OFFSET } } ``` -------------------------------- ### Apply Decay Multiplier for Author Diversity (Rust) Source: https://context7.com/cloudai-x/x-algo-skills/llms.txt Calculates a decay multiplier to penalize consecutive posts from the same author, promoting feed variety. The multiplier decreases with each subsequent post from an author, based on a decay factor and a floor value. Configuration parameters include `AUTHOR_DIVERSITY_DECAY` and `AUTHOR_DIVERSITY_FLOOR`. ```rust // home-mixer/scorers/author_diversity_scorer.rs fn multiplier(&self, position: usize) -> f64 { // First post from author: full score (multiplier = 1.0) // Second post: score × decay_factor // Third post: score × decay_factor² // ... (1.0 - self.floor) * self.decay_factor.powf(position as f64) + self.floor } // Example calculation: // decay_factor = 0.7, floor = 0.1 // Post 1 from @user: multiplier = 1.0 // Post 2 from @user: multiplier = (1-0.1) * 0.7^1 + 0.1 = 0.73 // Post 3 from @user: multiplier = (1-0.1) * 0.7^2 + 0.1 = 0.54 // Parameters: AUTHOR_DIVERSITY_DECAY, AUTHOR_DIVERSITY_FLOOR ``` -------------------------------- ### Filter Ineligible Subscription Posts (Rust) Source: https://github.com/cloudai-x/x-algo-skills/blob/main/x-algo-filters/SKILL.md The IneligibleSubscriptionFilter removes subscription-only posts from authors the user is not subscribed to. It partitions posts based on whether they are subscription posts and if the user is subscribed to the author of that subscription post. Non-subscription posts are always kept. ```rust // home-mixer/filters/ineligible_subscription_filter.rs let (kept, removed) = candidates.into_iter().partition(|candidate| { match candidate.subscription_author_id { Some(author_id) => subscribed_user_ids.contains(&author_id), None => true, // Not a subscription post, keep it } }); ``` -------------------------------- ### Transformer Forward Pass with Candidate Isolation Source: https://context7.com/cloudai-x/x-algo-skills/llms.txt Implements the forward pass for a transformer model used in recommendations. It builds input embeddings, applies a specialized attention mask for candidate isolation, and projects the output to action logits. Dependencies include JAX (jnp) and NumPy-like array operations. ```python def __call__(self, batch, recsys_embeddings) -> RecsysModelOutput: # 1. Build combined embeddings: [User] + [History] + [Candidates] embeddings, padding_mask, candidate_start = self.build_inputs(batch, recsys_embeddings) # 2. Pass through transformer with candidate isolation mask model_output = self.model( embeddings, padding_mask, candidate_start_offset=candidate_start, # Enables isolation masking ) # 3. Extract candidate outputs only out_embeddings = layer_norm(model_output.embeddings) candidate_embeddings = out_embeddings[:, candidate_start:, :] # 4. Project to action logits logits = jnp.dot(candidate_embeddings, unembeddings) # Output shape: [B, num_candidates, num_actions] return RecsysModelOutput(logits=logits) ``` -------------------------------- ### User Embedding Reduction Source: https://github.com/cloudai-x/x-algo-skills/blob/main/x-algo-ml/SKILL.md Demonstrates how user hash embeddings are combined and projected to a fixed dimension D. This process is crucial for creating a unified user representation from multiple hash embeddings. ```python import jax.numpy as jnp def block_user_reduce(user_embeddings: jnp.ndarray, proj_mat_1: jnp.ndarray, num_user_hashes: int, D: int) -> tuple[jnp.ndarray, jnp.ndarray]: # user_embeddings shape: [B, num_user_hashes, D] B = user_embeddings.shape[0] # Combine hash embeddings: [B, num_user_hashes, D] -> [B, 1, num_user_hashes * D] user_embedding_combined = user_embeddings.reshape((B, 1, num_user_hashes * D)) # Project down to embedding dimension D: [B, 1, num_user_hashes * D] -> [B, 1, D] user_embedding = jnp.dot(user_embedding_combined, proj_mat_1) # Create a dummy padding mask for demonstration user_padding_mask = jnp.ones((B, 1, 1)) return user_embedding, user_padding_mask ``` -------------------------------- ### Define PostCandidate Data Structure in Rust Source: https://context7.com/cloudai-x/x-algo-skills/llms.txt Defines the PostCandidate struct in Rust, which encapsulates all data related to a tweet candidate throughout the recommendation pipeline. It includes identifiers, relationship information, ML scores, and metadata for processing and ranking. ```rust pub struct PostCandidate { // Core identifiers pub tweet_id: i64, pub author_id: u64, pub tweet_text: String, // Relationship data pub in_reply_to_tweet_id: Option, pub retweeted_tweet_id: Option, pub retweeted_user_id: Option, pub ancestors: Vec, // Conversation context // ML predictions and scores pub phoenix_scores: PhoenixScores, // Raw ML predictions pub weighted_score: Option, // After WeightedScorer pub score: Option, // Final ranking score // Source and network info pub served_type: Option, // Thunder or Phoenix source pub in_network: Option, // User follows author? // Media and visibility pub video_duration_ms: Option, // For VQV eligibility pub visibility_reason: Option, pub subscription_author_id: Option, } ``` -------------------------------- ### Thunder Source (In-Network) - Rust Source: https://github.com/cloudai-x/x-algo-skills/blob/main/x-algo-pipeline/SKILL.md The Thunder source retrieves recent posts from accounts a user follows. It queries the Thunder service and includes conversation context, returning posts specifically for the 'For You In-Network' served type. ```rust // home-mixer/sources/thunder_source.rs // Posts from accounts the user follows served_type: Some(pb::ServedType::ForYouInNetwork) ``` -------------------------------- ### Apply Safety Visibility Filtering (Rust) Source: https://context7.com/cloudai-x/x-algo-skills/llms.txt Applies safety-based filtering using a visibility filtering service to remove policy-violating content. It checks a 'reason' field, dropping posts if it indicates a safety violation or any other filtered reason. ```rust // home-mixer/filters/vf_filter.rs fn should_drop(reason: &Option) -> bool { match reason { Some(FilteredReason::SafetyResult(safety_result)) => { matches!(safety_result.action, Action::Drop(_)) } Some(_) => true, // Other filtered reasons also drop None => false, // No filter reason = keep } } // Safety violations include: spam, abuse, policy violations // Visibility filtering runs during candidate hydration ``` -------------------------------- ### X Algorithm Engagement Signals Reference (Rust) Source: https://context7.com/cloudai-x/x-algo-skills/llms.txt Defines the `PhoenixScores` struct in Rust, which represents the 18 engagement action types and 1 continuous metric tracked by the X algorithm. These signals are used by the Phoenix ML model to predict user behavior and calculate a weighted score for post ranking. Higher scores indicate a higher probability of engagement, while negative signals decrease the overall score. ```rust pub struct PhoenixScores { // Positive engagement signals pub favorite_score: Option, pub reply_score: Option, pub retweet_score: Option, pub quote_score: Option, pub share_score: Option, pub share_via_dm_score: Option, pub share_via_copy_link_score: Option, pub follow_author_score: Option, // Engagement metrics pub photo_expand_score: Option, pub click_score: Option, pub profile_click_score: Option, pub vqv_score: Option, pub dwell_score: Option, pub quoted_click_score: Option, // Negative signals (decrease score) pub not_interested_score: Option, pub block_author_score: Option, pub mute_author_score: Option, pub report_score: Option, // Continuous metric pub dwell_time: Option, } // Example: Interpreting engagement scores // A favorite_score of 0.15 means 15% predicted chance the user will like the post // Higher scores = more likely engagement // Negative signals have negative weights in final scoring ``` -------------------------------- ### VQV Video Engagement Eligibility Check (Rust) Source: https://github.com/cloudai-x/x-algo-skills/blob/main/x-algo-scoring/SKILL.md Determines if the Video Quality View (VQV) engagement score should be applied. It checks if the video duration in milliseconds is present and exceeds a minimum threshold defined by `MIN_VIDEO_DURATION_MS`. If eligible, it returns the `VQV_WEIGHT`; otherwise, it returns 0.0. ```rust fn vqv_weight_eligibility(candidate: &PostCandidate) -> f64 { if candidate .video_duration_ms .is_some_and(|ms| ms > p::MIN_VIDEO_DURATION_MS) { p::VQV_WEIGHT } else { 0.0 // No VQV contribution for short videos or non-videos } } ``` -------------------------------- ### TopKScoreSelector Implementation - Rust Source: https://github.com/cloudai-x/x-algo-skills/blob/main/x-algo-pipeline/SKILL.md The TopKScoreSelector is a generic selector implementation that keeps the top K posts based on their final calculated score. It defines how to extract the score from a candidate and specifies the number of candidates to select. ```rust // home-mixer/selectors/top_k_score_selector.rs pub struct TopKScoreSelector; impl Selector for TopKScoreSelector { fn score(&self, candidate: &PostCandidate) -> f64 { candidate.score.unwrap_or(f64::NEG_INFINITY) } fn size(&self) -> Option { Some(params::TOP_K_CANDIDATES_TO_SELECT) } } ``` -------------------------------- ### Calculate Author Diversity Multiplier in Rust Source: https://github.com/cloudai-x/x-algo-skills/blob/main/x-algo-scoring/SKILL.md Calculates a multiplier to penalize multiple posts from the same author, promoting content variety. The multiplier is based on the post's position and configured decay factors. ```rust // From home-mixer/scorers/author_diversity_scorer.rs fn multiplier(&self, position: usize) -> f64 { // First post from author: full score // Second post: score × decay_factor // Third post: score × decay_factor² (1.0 - self.floor) * self.decay_factor.powf(position as f64) + self.floor } ``` -------------------------------- ### Calculate Weighted Engagement Score (Rust) Source: https://github.com/cloudai-x/x-algo-skills/blob/main/x-algo-scoring/SKILL.md Computes the weighted engagement score for a post by summing the products of weights and predicted probabilities for 18 user actions. It uses a PhoenixScores struct and applies predefined weights, with a special condition for VQV engagement. The final score is then offset. ```rust fn compute_weighted_score(candidate: &PostCandidate) -> f64 { let s: &PhoenixScores = &candidate.phoenix_scores; let vqv_weight = Self::vqv_weight_eligibility(candidate); let combined_score = Self::apply(s.favorite_score, p::FAVORITE_WEIGHT) + Self::apply(s.reply_score, p::REPLY_WEIGHT) + Self::apply(s.retweet_score, p::RETWEET_WEIGHT) + Self::apply(s.photo_expand_score, p::PHOTO_EXPAND_WEIGHT) + Self::apply(s.click_score, p::CLICK_WEIGHT) + Self::apply(s.profile_click_score, p::PROFILE_CLICK_WEIGHT) + Self::apply(s.vqv_score, vqv_weight) + Self::apply(s.share_score, p::SHARE_WEIGHT) + Self::apply(s.share_via_dm_score, p::SHARE_VIA_DM_WEIGHT) + Self::apply(s.share_via_copy_link_score, p::SHARE_VIA_COPY_LINK_WEIGHT) + Self::apply(s.dwell_score, p::DWELL_WEIGHT) + Self::apply(s.quote_score, p::QUOTE_WEIGHT) + Self::apply(s.quoted_click_score, p::QUOTED_CLICK_WEIGHT) + Self::apply(s.dwell_time, p::CONT_DWELL_TIME_WEIGHT) + Self::apply(s.follow_author_score, p::FOLLOW_AUTHOR_WEIGHT) + Self::apply(s.not_interested_score, p::NOT_INTERESTED_WEIGHT) + Self::apply(s.block_author_score, p::BLOCK_AUTHOR_WEIGHT) + Self::apply(s.mute_author_score, p::MUTE_AUTHOR_WEIGHT) + Self::apply(s.report_score, p::REPORT_WEIGHT); Self::offset_score(combined_score) } ``` -------------------------------- ### Visibility Filtering Logic (Rust) Source: https://github.com/cloudai-x/x-algo-skills/blob/main/x-algo-filters/SKILL.md The VFFilter (Visibility Filtering) implements safety-based filtering using an external service. It determines whether to drop a post based on the 'reason' provided, specifically checking for 'SafetyResult' actions like 'Drop'. Any other reason also results in the post being dropped. ```rust // home-mixer/filters/vf_filter.rs fn should_drop(reason: &Option) -> bool { match reason { Some(FilteredReason::SafetyResult(safety_result)) => { matches!(safety_result.action, Action::Drop(_)) } Some(_) => true, None => false, } } ``` -------------------------------- ### Combine History Features for Transformer Input (Python) Source: https://github.com/cloudai-x/x-algo-skills/blob/main/x-algo-ml/SKILL.md This function concatenates various historical features (post, author, actions, product surface) and projects them to a lower dimension for transformer input. It returns the combined embedding and a padding mask. ```python def block_history_reduce(...): # Concatenate all features, project to D post_author_embedding = jnp.concatenate([ history_post_embeddings_reshaped, history_author_embeddings_reshaped, history_actions_embeddings, history_product_surface_embeddings, ], axis=-1) history_embedding = jnp.dot(post_author_embedding, proj_mat_3) return history_embedding, history_padding_mask ``` -------------------------------- ### Encode Action Embeddings (Python) Source: https://github.com/cloudai-x/x-algo-skills/blob/main/x-algo-ml/SKILL.md This function encodes user actions by converting multi-hot action vectors into signed embeddings. It transforms the 0/1 representation to -1/+1, indicating 'didn't do action' vs 'did action', and then projects these signed vectors. ```python def _get_action_embeddings(self, actions): # actions: [B, S, num_actions] multi-hot vector actions_signed = (2 * actions - 1) # 0→-1, 1→+1 action_emb = jnp.dot(actions_signed, action_projection) return action_emb ``` -------------------------------- ### WeightedScorer Logic - Rust Source: https://github.com/cloudai-x/x-algo-skills/blob/main/x-algo-pipeline/SKILL.md The WeightedScorer combines predicted action probabilities from the PhoenixScorer into a single 'weighted_score'. This is calculated as the sum of each action's probability multiplied by its corresponding weight. ```rust // home-mixer/scorers/weighted_scorer.rs // Combines probabilities into single score weighted_score = Σ(weight × P(action)) ``` -------------------------------- ### Filter Posts by Age using Snowflake ID (Rust) Source: https://context7.com/cloudai-x/x-algo-skills/llms.txt Removes posts older than a specified maximum age to maintain feed freshness. It leverages the timestamp embedded within Snowflake IDs to determine the post's age and filters out those exceeding the `max_age` threshold. The `max_age` is typically set to 24-48 hours. ```rust // home-mixer/filters/age_filter.rs pub struct AgeFilter { pub max_age: Duration, // Typically 24-48 hours } fn is_within_age(&self, tweet_id: i64) -> bool { snowflake::duration_since_creation_opt(tweet_id) .map(|age| age <= self.max_age) .unwrap_or(false) } // Posts with Snowflake IDs encode their creation timestamp // Filter removes any post exceeding max_age threshold ``` -------------------------------- ### Rust: AgeFilter - Remove Old Posts Source: https://github.com/cloudai-x/x-algo-skills/blob/main/x-algo-filters/SKILL.md The AgeFilter removes posts older than a configured maximum age. It uses the creation timestamp embedded in Snowflake IDs to determine post age. Dependencies include the `Duration` type and a `snowflake` utility for timestamp extraction. ```rust pub struct AgeFilter { pub max_age: Duration, } fn is_within_age(&self, tweet_id: i64) -> bool { snowflake::duration_since_creation_opt(tweet_id) .map(|age| age <= self.max_age) .unwrap_or(false) } ``` -------------------------------- ### Rust: PreviouslySeenPostsFilter - Filter Seen Posts with Bloom Filters Source: https://github.com/cloudai-x/x-algo-skills/blob/main/x-algo-filters/SKILL.md This filter removes posts that the user has already viewed. It utilizes Bloom filters for efficient probabilistic checking and also considers an explicit list of seen IDs provided by the client. The function partitions candidate posts into 'removed' and 'kept' based on whether they match seen criteria. ```rust let (removed, kept) = candidates.into_iter().partition(|c| { get_related_post_ids(c).iter().any(|&post_id| { query.seen_ids.contains(&post_id) || bloom_filters .iter() .any(|filter| filter.may_contain(post_id)) }) }); ``` -------------------------------- ### Filter Posts from Blocked or Muted Authors (Rust) Source: https://github.com/cloudai-x/x-algo-skills/blob/main/x-algo-filters/SKILL.md The AuthorSocialgraphFilter removes posts from authors that the user has blocked or muted. It checks if the author's ID is present in the user's muted or blocked lists. Posts from such authors are moved to the 'removed' list. ```rust // home-mixer/filters/author_socialgraph_filter.rs let muted = viewer_muted_user_ids.contains(&author_id); let blocked = viewer_blocked_user_ids.contains(&author_id); if muted || blocked { removed.push(candidate); } ``` -------------------------------- ### Rust: DropDuplicatesFilter - Deduplicate by Tweet ID Source: https://github.com/cloudai-x/x-algo-skills/blob/main/x-algo-filters/SKILL.md This filter performs simple deduplication of posts based on their tweet IDs within the candidate set. It maintains a `HashSet` of seen IDs and only keeps a candidate if its ID has not been encountered before. ```rust let mut seen_ids = HashSet::new(); for candidate in candidates { if seen_ids.insert(candidate.tweet_id) { kept.push(candidate); } else { removed.push(candidate); } } ``` -------------------------------- ### Filter Posts with Failed Data Hydration (Rust) Source: https://github.com/cloudai-x/x-algo-skills/blob/main/x-algo-filters/SKILL.md The CoreDataHydrationFilter removes posts that failed to hydrate required data, such as missing author IDs or empty tweet text. It partitions the candidates, keeping only those with a valid author ID (non-zero) and non-empty trimmed tweet text. ```rust // home-mixer/filters/core_data_hydration_filter.rs let (kept, removed) = candidates .into_iter() .partition(|c| c.author_id != 0 && !c.tweet_text.trim().is_empty()); ``` -------------------------------- ### Filter Posts with Muted Keywords (Rust) Source: https://github.com/cloudai-x/x-algo-skills/blob/main/x-algo-filters/SKILL.md The MutedKeywordFilter removes posts containing keywords that the user has muted. It tokenizes the tweet text and uses a matcher to check for the presence of any muted keywords. If a match is found, the post is filtered out. ```rust // home-mixer/filters/muted_keyword_filter.rs let tweet_text_token_sequence = self.tokenizer.tokenize(&candidate.tweet_text); if matcher.matches(&tweet_text_token_sequence) { removed.push(candidate); // Matches muted keywords } ``` -------------------------------- ### Adjust Scores for Out-of-Network Posts (Rust) Source: https://context7.com/cloudai-x/x-algo-skills/llms.txt Modifies the score of posts from authors not followed by the user. It applies a weight factor (`OON_WEIGHT_FACTOR`) to reduce the score of out-of-network posts, ensuring a balance between content from followed accounts and recommendations. The `in_network` boolean indicates if the post is from a followed account. ```rust // home-mixer/scorers/oon_scorer.rs let updated_score = c.score.map(|base_score| match c.in_network { Some(false) => base_score * p::OON_WEIGHT_FACTOR, // Reduced weight for out-of-network _ => base_score, // Full weight for in-network (followed accounts) }); // OON_WEIGHT_FACTOR < 1.0 means out-of-network posts need higher // engagement predictions to compete with followed account posts ``` -------------------------------- ### Deduplicate Conversation Threads (Rust) Source: https://context7.com/cloudai-x/x-algo-skills/llms.txt Filters posts to keep only the highest-scored post per conversation thread. It identifies conversation roots by finding the minimum ancestor ID or using the post's own ID if no ancestors exist. Posts from the same conversation are grouped, and only the top-scoring one is retained. ```rust // home-mixer/filters/dedup_conversation_filter.rs fn get_conversation_id(candidate: &PostCandidate) -> u64 { // Conversation root = minimum ancestor ID, or self if no ancestors candidate .ancestors .iter() .copied() .min() .unwrap_or(candidate.tweet_id as u64) } // Groups candidates by conversation_id // Keeps only the highest-scored post per conversation // Other posts from same conversation are filtered out ``` -------------------------------- ### Normalize Weighted Score in Rust Source: https://github.com/cloudai-x/x-algo-skills/blob/main/x-algo-scoring/SKILL.md Applies normalization to a computed weighted score. This function is part of the score normalization process, with its implementation details located in `util/score_normalizer.rs` and excluded from open source. ```rust let weighted_score = Self::compute_weighted_score(c); let normalized_weighted_score = normalize_score(c, weighted_score); ``` -------------------------------- ### PostCandidate Score Fields Definition in Rust Source: https://github.com/cloudai-x/x-algo-skills/blob/main/x-algo-scoring/SKILL.md Defines the structure for `PostCandidate`, including fields for `weighted_score` (after initial scoring) and `score` (final score after all stages). ```rust pub struct PostCandidate { pub weighted_score: Option, // After WeightedScorer pub score: Option, // Final score after all scorers // ... } ``` -------------------------------- ### Score Offset Logic for Ranking (Rust) Source: https://github.com/cloudai-x/x-algo-skills/blob/main/x-algo-scoring/SKILL.md Adjusts the calculated combined score to ensure proper ranking, especially for negative scores. If the sum of weights is zero, it ensures the score is non-negative. Negative combined scores are scaled using `NEGATIVE_WEIGHTS_SUM` and `NEGATIVE_SCORES_OFFSET`, while positive scores simply have the offset added. ```rust fn offset_score(combined_score: f64) -> f64 { if p::WEIGHTS_SUM == 0.0 { combined_score.max(0.0) } else if combined_score < 0.0 { // Negative scores get scaled offset (combined_score + p::NEGATIVE_WEIGHTS_SUM) / p::WEIGHTS_SUM * p::NEGATIVE_SCORES_OFFSET } else { // Positive scores just add offset combined_score + p::NEGATIVE_SCORES_OFFSET } } ``` -------------------------------- ### Rust: PreviouslyServedPostsFilter - Filter Posts Served in Session Source: https://github.com/cloudai-x/x-algo-skills/blob/main/x-algo-filters/SKILL.md The PreviouslyServedPostsFilter removes posts that have already been presented to the user within the current session, typically for pagination or infinite scroll scenarios. It checks if the post ID exists in the `served_ids` list from the request, and is only enabled for bottom requests. ```rust fn enable(&self, query: &ScoredPostsQuery) -> bool { query.is_bottom_request // Only for pagination requests } // Checks served_ids from request get_related_post_ids(c).iter().any(|id| query.served_ids.contains(id)) ``` -------------------------------- ### Adjust Score for Out-of-Network Posts in Rust Source: https://github.com/cloudai-x/x-algo-skills/blob/main/x-algo-scoring/SKILL.md Modifies the score of posts originating from accounts the user does not follow. Out-of-network posts receive a reduced weight factor. ```rust // From home-mixer/scorers/oon_scorer.rs let updated_score = c.score.map(|base_score| match c.in_network { Some(false) => base_score * p::OON_WEIGHT_FACTOR, // Reduced weight _ => base_score, // Full weight for in-network }); ``` -------------------------------- ### Filter Posts by Author Socialgraph (Muted/Blocked) (Rust) Source: https://context7.com/cloudai-x/x-algo-skills/llms.txt Removes posts from authors that the user has blocked or muted. It checks the author's ID against lists of muted and blocked user IDs, filtering out posts if the author is present in either list. ```rust // home-mixer/filters/author_socialgraph_filter.rs let muted = viewer_muted_user_ids.contains(&author_id); let blocked = viewer_blocked_user_ids.contains(&author_id); if muted || blocked { removed.push(candidate); } // Both blocked and muted authors are completely filtered // User socialgraph is loaded during query hydration stage ``` -------------------------------- ### Extract Phoenix Scores from Action Predictions (Rust) Source: https://github.com/cloudai-x/x-algo-skills/blob/main/x-algo-engagement/SKILL.md This Rust function, `extract_phoenix_scores`, takes `ActionPredictions` as input and maps specific action names (like favorite, reply, retweet) to their corresponding predicted probabilities. It's a core part of processing raw predictions into usable scores. ```rust fn extract_phoenix_scores(&self, p: &ActionPredictions) -> PhoenixScores { PhoenixScores { favorite_score: p.get(ActionName::ServerTweetFav), reply_score: p.get(ActionName::ServerTweetReply), retweet_score: p.get(ActionName::ServerTweetRetweet), // ... maps each action to its probability } } ``` -------------------------------- ### OONScorer Logic - Rust Source: https://github.com/cloudai-x/x-algo-skills/blob/main/x-algo-pipeline/SKILL.md The OONScorer (Out-of-Network Scorer) adjusts the scores of posts that are not from the user's network. It increases their score by multiplying it with an 'OON_WEIGHT_FACTOR' to balance in-network and out-of-network content. ```rust // home-mixer/scorers/oon_scorer.rs // Adjusts out-of-network post scores if !in_network: score *= OON_WEIGHT_FACTOR ``` -------------------------------- ### Remove User's Own Tweets (Rust) Source: https://github.com/cloudai-x/x-algo-skills/blob/main/x-algo-filters/SKILL.md The SelfTweetFilter removes posts authored by the viewing user from their 'For You' feed. It partitions the candidate posts into two groups: those authored by the viewer and those authored by others. This ensures the feed prioritizes content from other users. ```rust // home-mixer/filters/self_tweet_filter.rs let viewer_id = query.user_id as u64; let (kept, removed) = candidates .into_iter() .partition(|c| c.author_id != viewer_id); ``` -------------------------------- ### AuthorDiversityScorer Logic - Rust Source: https://github.com/cloudai-x/x-algo-skills/blob/main/x-algo-pipeline/SKILL.md The AuthorDiversityScorer adjusts post scores to promote author variety in the feed. It penalizes multiple posts from the same author using a decay-based multiplier that depends on the post's position. ```rust // home-mixer/scorers/author_diversity_scorer.rs // Penalizes multiple posts from same author multiplier = (1 - floor) × decay^position + floor ``` -------------------------------- ### Filter Retweets to Show Original Content Only (Rust) Source: https://context7.com/cloudai-x/x-algo-skills/llms.txt Prevents displaying the same underlying post multiple times by differentiating between original posts and retweets. It uses a set to track seen tweet IDs, ensuring that only the first occurrence of a post (either original or retweet) is kept. ```rust // home-mixer/filters/retweet_deduplication_filter.rs match candidate.retweeted_tweet_id { Some(retweeted_id) => { // This is a retweet - check if we've seen the original if seen_tweet_ids.insert(retweeted_id) { kept.push(candidate); // First time seeing this content } else { removed.push(candidate); // Duplicate content } } None => { // Original post - mark as seen seen_tweet_ids.insert(candidate.tweet_id as u64); kept.push(candidate); } } ``` -------------------------------- ### Filter Result Structure Definition (Rust) Source: https://github.com/cloudai-x/x-algo-skills/blob/main/x-algo-filters/SKILL.md Defines the generic FilterResult struct used by all filters. This structure contains two vectors: 'kept' for candidates that passed the filter and 'removed' for candidates that were filtered out. ```rust pub struct FilterResult { pub kept: Vec, // Candidates that passed pub removed: Vec, // Candidates that were filtered out } ``` -------------------------------- ### Deduplicate Posts by Conversation ID (Rust) Source: https://github.com/cloudai-x/x-algo-skills/blob/main/x-algo-filters/SKILL.md The DedupConversationFilter keeps only the highest-scored post per conversation thread. It identifies conversations by finding the minimum ancestor ID or the post's own ID if no ancestors exist. This ensures that within a single conversation, only the most relevant post is presented. ```rust // home-mixer/filters/dedup_conversation_filter.rs fn get_conversation_id(candidate: &PostCandidate) -> u64 { // Conversation root = minimum ancestor ID, or self if no ancestors candidate .ancestors .iter() .copied() .min() .unwrap_or(candidate.tweet_id as u64) } // Keeps highest score per conversation_id ```