Donald Trump’s inauguration marked a shift in power dynamics, emphasizing a government "for the people." The slogan "America First" was introduced, focusing on job creation, border control, and economic revitalization. A commitment to "Buy American & Hire American" was established as a core principle of the administration.
Trump engaged with industry leaders, reinforcing his commitment to job creation and economic growth. The administration began to address controversial topics such as sanctuary cities and healthcare reform. Public sentiment reflected optimism regarding economic prospects, with the stock market showing positive trends.
Continued emphasis on "America First" with a focus on job creation and economic revitalization. The administration highlighted its commitment to border control and national security. Public demonstrations and protests were acknowledged as part of the democratic process.
The focus on domestic job creation leads to a revitalized manufacturing sector, resulting in significant economic growth and reduced unemployme ``` -------------------------------- ### Initialize BERTrend Object with a Topic Model Source: https://github.com/rte-france/bertrend/blob/main/getting_started/bertrend_quickstart.ipynb Creates an instance of the BERTrend class, passing a pre-configured BERTopicModel object. BERTrend relies on these topic models to perform its trend analysis, especially for retrospective tasks where multiple models are trained on different time slices of data. ```python # Basic creation of the object and parametrization # BERTrend uses several topic models; therefore, it is necessary to pass a topic_model object as a reference bertrend = BERTrend(topic_model=topic_model) ``` -------------------------------- ### Install BERTrend Package from PyPI Source: https://github.com/rte-france/bertrend/blob/main/README.md Provides the command to install the BERTrend package directly from the Python Package Index (PyPI). This is the simplest method for users who want to use the pre-built distribution of the package. ```bash pip install bertrend ``` -------------------------------- ### Retrieve Embedding Model Name Source: https://github.com/rte-france/bertrend/blob/main/getting_started/bertrend_quickstart.ipynb Retrieves the name of the embedding model currently being used by the initialized EmbeddingService. This can be useful for logging or verifying the model configuration. ```python embedding_model_name = embedding_service.embedding_model_name ``` -------------------------------- ### Iterate Through Input Data for BERTrend Analysis (Python) Source: https://github.com/rte-france/bertrend/blob/main/getting_started/bertrend_quickstart.ipynb This script iterates through a list of input data files, extracts a timestamp from each filename, and calls the `process_new_data` function to perform prospective trend analysis for each data slice. ```python for data_file in input_data: timestamp = pd.Timestamp(data_file.name.split("_")[0]) display(process_new_data(data_file, timestamp)) ``` -------------------------------- ### Initialize and Use EmbeddingService Source: https://github.com/rte-france/bertrend/blob/main/getting_started/bertrend_quickstart.ipynb Configures and initializes an EmbeddingService for generating text embeddings. It specifies the service URL and client secret, then uses the service to embed the 'text' column of a DataFrame. This process involves sending texts to a remote embedding service. ```python df = df.head(1000) client_secret = "bd76aa472dd91aed4a56bf1935dbb802583c119824380d8567086579c0ef3324" # embedding_service_cfg = {"local": False, "url":"https://localhost:6464"} embedding_service_cfg = { "local": False, "url": "https://10.132.5.44:6464", "client_secret": client_secret, } embedding_service = EmbeddingService(**embedding_service_cfg) embeddings, token_strings, token_embeddings = embedding_service.embed( texts=df["text"], ) ``` -------------------------------- ### Generate Topic Description using BERTopic Source: https://github.com/rte-france/bertrend/blob/main/getting_started/bertrend_quickstart.ipynb Generates a textual description for a specific topic within a BERTopic model. This function takes the topic model, topic number, filtered documents, and a language code as input. It leverages LLM utilities to create a coherent description. ```python desc = generate_topic_description( topic_model=selected_topic_model, topic_number=5, filtered_docs=df, language_code="en", ) ``` -------------------------------- ### Example Configuration File (TOML) Source: https://github.com/rte-france/bertrend/blob/main/bertrend_apps/prospective_demo/docs/automated_report_generation.md A complete example of a TOML configuration file for a Bertrend model, including settings for model, analysis, and reporting. ```toml [model_config] granularity = 7 window_size = 7 language = "en" split_by_paragraph = true [analysis_config] topic_evolution = true evolution_scenarios = true multifactorial_analysis = true [report_config] auto_send = true email_recipients = [ "team-lead@company.com", "analyst@company.com" ] report_title = "Weekly AI & Technology Trends Report" max_emerging_topics = 3 max_strong_topics = 5 ``` -------------------------------- ### Run Topic Analysis Demo (Bash) Source: https://github.com/rte-france/bertrend/blob/main/docs/demos.md Navigates to the Topic Analysis Demo directory and runs the Streamlit application. Requires Streamlit to be installed. ```bash cd bertrend/demos/topic_analysis streamlit run app.py ``` -------------------------------- ### Python OpenAI Client API Call for Content Generation Source: https://github.com/rte-france/bertrend/blob/main/getting_started/bertrend_quickstart.ipynb This Python code snippet demonstrates a call to the OpenAI API using the `openai_client` library to generate content based on a historical conversation. It logs the API response, including the generated content and metadata like chat completion ID and choices. This function is part of a larger LLM utility. ```python API returned: ChatCompletion(id='chatcmpl-BH8i9vBvO8xD7z4Fcx7mZocOJMFIN', choices=[Choice(finish_reason='stop', index=0, logprobs=None, message=ChatCompletionMessage(content='```html
Donald Trump’s inauguration marked a shift in power dynamics, emphasizing a government "for the people." The slogan "America First" was introduced, focusing on job creation, border control, and economic revitalization. A commitment to "Buy American & Hire American" was established as a core principle of the administration.
Trump engaged with industry leaders, reinforcing his commitment to job creation and economic growth. The administration began to address controversial topics such as sanctuary cities and healthcare reform. Public sentiment reflected optimism regarding economic prospects, with the stock market showing positive trends.
Continued emphasis on "America First" with a focus on job creation and economic revitalization. The administration highlighted its commitment to border control and national security. Public demonstrations and protests were acknowledged as part of the democratic process.
The focus on domestic job creation leads to a revitalized manufacturing sector, resulting in significant economic growth and reduced unemployme ``` -------------------------------- ### Calculate Signal Popularity in BERTrend Source: https://github.com/rte-france/bertrend/blob/main/getting_started/bertrend_quickstart.ipynb Calculates the signal popularity within the BERTrend model. This function often requires a `window_size` parameter to define the period over which popularity is assessed. The output typically includes information about the saved model location. ```python bertrend.calculate_signal_popularity() window_size = 30 ``` -------------------------------- ### Run Weak Signals Demo (Bash) Source: https://github.com/rte-france/bertrend/blob/main/docs/demos.md Navigates to the Weak Signals Demo directory and runs the Streamlit application. Requires Streamlit to be installed. ```bash cd bertrend/demos/weak_signals streamlit run app.py ``` -------------------------------- ### Process New Data with BERTrend (Python) Source: https://github.com/rte-france/bertrend/blob/main/getting_started/bertrend_quickstart.ipynb Processes a new slice of data for trend analysis. It attempts to restore existing BERTrend models, loads and prepares the new data, generates embeddings, trains new topic models, and merges them with existing ones. It then calculates signal popularity and classifies weak and strong signals based on a specified window size. ```python def process_new_data(data_slice_path: Path, timestamp: pd.Timestamp): logger.debug(f"Processing new data: {data_slice_path}") # Restore previous models try: bertrend = BERTrend.restore_model(BERTREND_MODELS_PATH) except: logger.warning("Cannot restore previous models, creating new one") bertrend = BERTrend(topic_model=BERTopicModel()) # Read data df = load_data(data_slice_path, language="French") df = split_data(df) text = df[TEXT_COLUMN] # Embed new data embeddings, token_strings, token_embeddings = embedding_service.embed( texts=text, ) # Create topic model for new data bertrend.train_topic_models( {timestamp: df}, embeddings=embeddings, embedding_model=embedding_model_name ) logger.info(f"BERTrend processed {len(bertrend.doc_groups)} time periods") # Save models bertrend.save_model(models_path=BERTREND_MODELS_PATH) if len(bertrend.doc_groups) < 2: return None # Compute popularities bertrend.calculate_signal_popularity() # classify last signals noise_topics_df, weak_signal_topics_df, strong_signal_topics_df = ( bertrend.classify_signals(window_size, timestamp) ) # TODO: save dfs if weak_signal_topics_df.empty: return None wt = weak_signal_topics_df["Topic"] logger.info(f"Weak topics: {wt}") wt_list = [] for topic in wt: topic_model = bertrend.restore_topic_model(timestamp) desc = generate_topic_description( topic_model=topic_model, topic_number=topic, filtered_docs=df, language_code="fr", ) wt_list.append({ "timestamp": timestamp, "topic": topic, "title": desc.title, "description": desc.description, }) return pd.DataFrame(wt_list) ``` -------------------------------- ### Download and Load Historical Tweet Data Source: https://github.com/rte-france/bertrend/blob/main/getting_started/bertrend_quickstart.ipynb Downloads a CSV file containing Donald Trump's tweets from a specified GitHub URL using a shell command. It then loads this data into a pandas DataFrame, specifying the separator, quote character, and skipping initial spaces. This step is crucial for preparing the dataset for trend analysis. ```python # Here some Trump tweets from: https://github.com/MarkHershey/CompleteTrumpTweetsArchive/blob/master/data/realDonaldTrump_in_office.csv #!wget "https://raw.githubusercontent.com/MarkHershey/CompleteTrumpTweetsArchive/refs/heads/master/data/realDonaldTrump_in_office.csv" df = pd.read_csv( "realDonaldTrump_in_office.csv", sep=",", quotechar='"', skipinitialspace=True ) ``` -------------------------------- ### Run Main BERTrend Application Container Source: https://github.com/rte-france/bertrend/blob/main/docs/docker.md Starts the main BERTrend application container, exposing demo ports and mounting a volume for persistent data. It configures API keys, embedding service URLs, and user IDs, with GPU support enabled via `--gpus all`. ```bash docker run --gpus all \ -p 8501:8501 -p 8502:8502 -p 8503:8503 \ -v /path/to/bertrend/data:/bertrend \ -e OPENAI_API_KEY=your_key \ -e OPENAI_BASE_URL=your_endpoint \ -e EMBEDDING_SERVICE_URL=https://your-embedding-server:6464 \ -e HOST_UID=$(id -u) -e HOST_GID=$(id -g) \ bertrend:latest ``` -------------------------------- ### BERTrend Main Application Volume Mount Example Source: https://github.com/rte-france/bertrend/blob/main/docs/docker.md Illustrates how to mount a local directory to the `/bertrend` path inside the main BERTrend Docker container. This ensures that data generated or used by BERTrend is persisted on the host system. ```bash -v /path/on/host:/bertrend ``` -------------------------------- ### Running BERTrend Streamlit Demos Source: https://context7.com/rte-france/bertrend/llms.txt This section describes how to launch and utilize the interactive Streamlit demos for BERTrend, specifically for Topic Analysis and Weak Signals. It provides command-line instructions for launching the demos and outlines the key features of each application. It also includes an example of programmatic usage for data loading and embedding components. ```bash # Launching the Topic Analysis Demo cd bertrend/demos/topic_analysis CUDA_VISIBLE_DEVICES=0 streamlit run app.py # Launching the Weak Signals Demo cd bertrend/demos/weak_signals CUDA_VISIBLE_DEVICES=0 streamlit run app.py ``` ```python from bertrend.demos.demos_utils.data_loading_component import ( display_data_loading_component ) from bertrend.demos.demos_utils.embed_documents_component import ( embed_documents_component ) import streamlit as st # Example of programmatic usage within a Streamlit app: # st.title("BERTrend Demo Components") # # st.header("Data Loading") # display_data_loading_component() # # st.header("Document Embedding") # # Assuming 'documents' is a list of strings and 'model_name' is specified # # documents, model_name = embed_documents_component(...) # st.write("Run embed_documents_component to get embeddings.") ``` -------------------------------- ### Generate Topic Summary from LLM API Response Source: https://github.com/rte-france/bertrend/blob/main/getting_started/bertrend_quickstart.ipynb This snippet demonstrates how to extract and parse a JSON response from an LLM API call, specifically for generating a title and description for a given topic. It handles the JSON parsing and ensures the data is structured correctly for further use. This function is part of the LLM utilities within the BERTrend project. ```python from bertrend.llm_utils.openai_client import generate_from_history import json # Assuming 'history' is a pre-defined list of messages for the API call # The actual content of 'history' would depend on the specific conversation context history = [ {"role": "system", "content": "You are a helpful assistant that summarizes topics."}, {"role": "user", "content": "Summarize the following topic: ..."} ] # Call the API to generate the response api_response = generate_from_history(history) # Extract the content from the API response content = api_response.choices[0].message.content # Parse the JSON content to get the title and description topic_data = json.loads(content) title = topic_data.get("title") description = topic_data.get("description") print(f"Topic Title: {title}") print(f"Topic Description: {description}") ``` -------------------------------- ### Train Topic Models with BERTrend (Python) Source: https://github.com/rte-france/bertrend/blob/main/getting_started/bertrend_quickstart.ipynb Trains topic models using the `bertrend.train_topic_models` function. This function requires grouped data, the name of the embedding model, and the corresponding embeddings. It handles the iterative training process and fitting the underlying BERTopic model, including steps like dimensionality reduction and clustering. ```python bertrend.train_topic_models( grouped_data=grouped_data, embedding_model=embedding_model_name, embeddings=embeddings, ) ``` -------------------------------- ### Configure BERTopic Model with Custom Parameters Source: https://github.com/rte-france/bertrend/blob/main/getting_started/bertrend_quickstart.ipynb Defines a configuration string in TOML format to customize the BERTopic model. It allows overriding default settings for global parameters, BERTopic itself, UMAP, HDBSCAN, CountVectorizer, ClassTfidfTransformer, and MaximalMarginalRelevance. This enables fine-tuning topic modeling for specific use cases, such as setting the language to English and adjusting representation models. ```python # Topic model with default parameters - each parameter of BERTopic can be modified from the constructor or can be read from a configuration file # overrides the default config to use English config = """ # Default configuration file to be used for topic model # Global parameters [global] language = "English" # BERTopic parameters: https://maartengr.github.io/BERTopic/api/bertopic.html#bertopic._bertopic.BERTopic.__init__ [bertopic_model] top_n_words = 10 verbose = true representation_model = ["MaximalMarginalRelevance"] # KeyBERTInspired, OpenAI zeroshot_topic_list = [] zeroshot_min_similarity = 0 # UMAP parameters: https://umap-learn.readthedocs.io/en/latest/api.html [umap_model] n_neighbors = 5 n_components = 5 min_dist = 0.0 metric = "cosine" random_state = 42 # HDBSCAN parameters: https://hdbscan.readthedocs.io/en/latest/api.html [hdbscan_model] min_cluster_size = 5 min_samples = 5 metric = "euclidean" cluster_selection_method = "eom" prediction_data = true # CountVectorizer: https://scikit-learn.org/1.5/modules/generated/sklearn.feature_extraction.text.CountVectorizer.html [vectorizer_model] ngram_range = [1, 1] stop_words = true # If true, will check `language` parameter and load associated stopwords file min_df = 2 # ClassTfidfTransformer: https://maartengr.github.io/BERTopic/api/ctfidf.html [ctfidf_model] bm25_weighting = false reduce_frequent_words = true # MaximalMarginalRelevance: https://maartengr.github.io/BERTopic/api/representation/mmr.html [mmr_model] diversity = 0.3 # Reduce outliers: https://maartengr.github.io/BERTopic/api/bertopic.html#bertopic._bertopic.BERTopic.reduce_outliers [reduce_outliers] strategy = "c-tf-idf" " topic_model = BERTopicModel(config) ``` -------------------------------- ### Embedding Server Volume Mount Example Source: https://github.com/rte-france/bertrend/blob/main/docs/docker.md Demonstrates mounting a local directory to the Hugging Face cache path within the embedding server container. This strategy prevents redundant model downloads across container instances, optimizing storage and startup time. ```bash -v /path/to/huggingface/cache:/root/.cache/huggingface ``` -------------------------------- ### Run Integration Tests with Pytest Source: https://github.com/rte-france/bertrend/blob/main/bertrend/tests/README_integration_tests.md This command executes all tests marked with the 'integration' marker in the specified file, providing verbose output. Ensure BERTrend is installed with its dependencies. ```bash pytest -m integration bertrend/tests/test_integration.py -v ``` -------------------------------- ### Test Email Sending Configuration (Python) Source: https://github.com/rte-france/bertrend/blob/main/docs/mail_configuration.md This script demonstrates how to test the configured email sending functionality. It imports necessary functions from 'bertrend_apps.common.mail_utils' and sends a test email. It shows how to obtain credentials for Gmail OAuth and format the email content. ```python from bertrend_apps.common.mail_utils import get_credentials, send_email # For Gmail OAuth credentials = get_credentials() send_email( credentials=credentials, subject="BERTrend Test Email", recipients=["your-test-email@example.com"], content="