FinBERT

https://github.com/yya518/finbert
Admin
A Pretrained BERT Model for Financial Communications. https://arxiv.org/abs/2006.08097
Tokens:2,372
Snippets:16
Trust Score:7.5
License:Apache-2.0
Update:1 year ago
Show doc for...
Tokens:
Raw
### Python: Set Up FinBERT Environment and Verify Transformers Version

Source: https://github.com/yya518/finbert/blob/master/FinBERT-demo.ipynb

This snippet initializes the Python environment by importing necessary classes from the `transformers` library, such as `BertTokenizer`, `BertForSequenceClassification`, and `pipeline`. It also includes a check to display the installed `transformers` library version, ensuring compatibility.

```Python
from transformers import BertTokenizer, BertForSequenceClassification, pipeline
```

```Python
# tested in transformers==4.18.0 
import transformers
transformers.__version__
```

--------------------------------

### Define Training Arguments and Initialize Hugging Face Trainer for FinBERT in Python

Source: https://github.com/yya518/finbert/blob/master/finetune.ipynb

This snippet defines a `compute_metrics` function to calculate accuracy during evaluation. It then configures `TrainingArguments` including output directory, evaluation/save strategy, learning rate, batch sizes, epochs, and weight decay. Finally, it initializes the `Trainer` with the model, arguments, datasets, and the custom metric function, then starts the training process.

```Python
def compute_metrics(eval_pred):
    predictions, labels = eval_pred
    predictions = np.argmax(predictions, axis=1)
    return {'accuracy' : accuracy_score(predictions, labels)}

args = TrainingArguments(
        output_dir = 'temp/',
        evaluation_strategy = 'epoch',
        save_strategy = 'epoch',
        learning_rate=2e-5,
        per_device_train_batch_size=32,
        per_device_eval_batch_size=32,
        num_train_epochs=5,
        weight_decay=0.01,
        load_best_model_at_end=True,
        metric_for_best_model='accuracy',
)

trainer = Trainer(
        model=model,                         # the instantiated 🤗 Transformers model to be trained
        args=args,                  # training arguments, defined above
        train_dataset=dataset_train,         # training dataset
        eval_dataset=dataset_val,            # evaluation dataset
        compute_metrics=compute_metrics
)

trainer.train()
```

--------------------------------

### Verify PyTorch and Hugging Face Transformers Versions in Python

Source: https://github.com/yya518/finbert/blob/master/finetune.ipynb

This code snippet checks the installed versions of PyTorch and Hugging Face Transformers. It's important for ensuring compatibility with the provided notebook and specific model functionalities, as the code was tested with `transformers==4.18.0` and `pytorch==1.7.1`.

```Python
# tested in transformers==4.18.0, pytorch==1.7.1 
import torch
import transformers
torch.__version__, transformers.__version__
```

--------------------------------

### Load FinBERT Pretrained Model and Tokenizer from Hugging Face in Python

Source: https://github.com/yya518/finbert/blob/master/finetune.ipynb

This snippet loads the `BertForSequenceClassification` model and `BertTokenizer` from the 'yiyanghkust/finbert-pretrain' path on Hugging Face. The model is initialized with `num_labels=3` to match the expected output classes for the sentiment task. This step prepares the core components for fine-tuning.

```Python
model = BertForSequenceClassification.from_pretrained('yiyanghkust/finbert-pretrain',num_labels=3)
tokenizer = BertTokenizer.from_pretrained('yiyanghkust/finbert-pretrain')
```

--------------------------------

### Import Core Libraries for FinBERT Fine-tuning in Python

Source: https://github.com/yya518/finbert/blob/master/finetune.ipynb

This snippet imports essential libraries including `numpy`, `pandas`, `torch`, `sklearn`, and `transformers` components like `BertTokenizer`, `Trainer`, and `BertForSequenceClassification`. These are crucial for data manipulation, model training, and evaluation in NLP tasks.

```Python
import numpy as np
import pandas as pd 
from transformers import BertTokenizer, Trainer, BertForSequenceClassification, TrainingArguments
from datasets import Dataset
import torch
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score
```

--------------------------------

### Load Custom Financial Dataset with Pandas in Python

Source: https://github.com/yya518/finbert/blob/master/finetune.ipynb

This snippet demonstrates loading a custom CSV dataset named 'analysttone.csv' into a Pandas DataFrame. Users should replace 'analysttone.csv' with their own dataset path. The `head()` method is used to display the first few rows for initial inspection.

```Python
df = pd.read_csv('analysttone.csv') ## use your own customized dataset
df.head()
```

--------------------------------

### Check GPU Availability for PyTorch in Python

Source: https://github.com/yya518/finbert/blob/master/finetune.ipynb

This snippet checks if a CUDA-enabled GPU is available for PyTorch operations. Utilizing a GPU significantly accelerates model training and inference, especially for large-scale datasets, as noted in the original documentation.

```Python
torch.cuda.is_available()
```

--------------------------------

### Prepare Hugging Face Datasets for FinBERT Fine-tuning in Python

Source: https://github.com/yya518/finbert/blob/master/finetune.ipynb

This snippet converts Pandas DataFrames into Hugging Face `Dataset` objects for training, validation, and testing. It then tokenizes the 'sentence' column using the loaded FinBERT tokenizer, applying truncation, padding, and a maximum length of 128. Finally, it sets the format to 'torch' and specifies the required columns for model input.

```Python
dataset_train = Dataset.from_pandas(df_train)
dataset_val = Dataset.from_pandas(df_val)
dataset_test = Dataset.from_pandas(df_test)

dataset_train = dataset_train.map(lambda e: tokenizer(e['sentence'], truncation=True, padding='max_length', max_length=128), batched=True)
dataset_val = dataset_val.map(lambda e: tokenizer(e['sentence'], truncation=True, padding='max_length', max_length=128), batched=True)
dataset_test = dataset_test.map(lambda e: tokenizer(e['sentence'], truncation=True, padding='max_length' , max_length=128), batched=True)

dataset_train.set_format(type='torch', columns=['input_ids', 'token_type_ids', 'attention_mask', 'label'])
dataset_val.set_format(type='torch', columns=['input_ids', 'token_type_ids', 'attention_mask', 'label'])
dataset_test.set_format(type='torch', columns=['input_ids', 'token_type_ids', 'attention_mask', 'label'])
```

--------------------------------

### Save Fine-tuned FinBERT Model in Python

Source: https://github.com/yya518/finbert/blob/master/finetune.ipynb

This snippet saves the fine-tuned FinBERT model to the specified directory 'finbert-sentiment/'. This allows for easy loading and deployment of the trained model without needing to retrain it, making it ready for inference or further use.

```Python
trainer.save_model('finbert-sentiment/')
```

--------------------------------

### Split Dataset into Training, Validation, and Test Sets in Python

Source: https://github.com/yya518/finbert/blob/master/finetune.ipynb

This snippet uses `sklearn.model_selection.train_test_split` to divide the dataset into training, testing, and validation sets. Stratified splitting based on the 'label' column ensures an even distribution of classes across splits, which is crucial for balanced model training and evaluation. The shapes of the resulting DataFrames are printed for verification.

```Python
df_train, df_test, = train_test_split(df, stratify=df['label'], test_size=0.1, random_state=42)
df_train, df_val = train_test_split(df_train, stratify=df_train['label'],test_size=0.1, random_state=42)
print(df_train.shape, df_test.shape, df_val.shape)
```

--------------------------------

### Evaluate Fine-tuned FinBERT Model on Test Set in Python

Source: https://github.com/yya518/finbert/blob/master/finetune.ipynb

This snippet sets the fine-tuned model to evaluation mode (`model.eval()`) and then uses the `trainer.predict()` method on the `dataset_test` to obtain performance metrics. This step provides an unbiased assessment of the model's generalization ability on unseen data.

```Python
model.eval()
trainer.predict(dataset_test).metrics
```

--------------------------------

### Apply FinBERT for Financial Sentiment Analysis in Python

Source: https://github.com/yya518/finbert/blob/master/README.md

This Python code snippet demonstrates how to use the pre-trained FinBERT-Sentiment model from Huggingface's `transformers` library to perform financial sentiment analysis. It shows the process of loading the tokenizer and model, tokenizing input sentences, making predictions, and interpreting the results as 'neutral', 'positive', or 'negative' sentiments. This snippet requires the `transformers` and `numpy` libraries.

```Python
from transformers import BertTokenizer, BertForSequenceClassification
import numpy as np

finbert = BertForSequenceClassification.from_pretrained('yiyanghkust/finbert-tone',num_labels=3)
tokenizer = BertTokenizer.from_pretrained('yiyanghkust/finbert-tone')

sentences = ["there is a shortage of capital, and we need extra financing", 
             "growth is strong and we have plenty of liquidity", 
             "there are doubts about our finances", 
             "profits are flat"]

inputs = tokenizer(sentences, return_tensors="pt", padding=True)
outputs = finbert(**inputs)[0]

labels = {0:'neutral', 1:'positive',2:'negative'}
for idx, sent in enumerate(sentences):
    print(sent, '----', labels[np.argmax(outputs.detach().numpy()[idx])])
    
'''
there is a shortage of capital, and we need extra financing ---- negative
growth is strong and we have plenty of liquidity ---- positive
there are doubts about our finances ---- negative
profits are flat ---- neutral
'''
```

--------------------------------

### Python: Perform FinBERT FLS Classification

Source: https://github.com/yya518/finbert/blob/master/FinBERT-demo.ipynb

This section demonstrates the use of the FinBERT-FLS model to classify financial texts as Specific-FLS, Non-specific FLS, or Not-FLS. It loads the 'yiyanghkust/finbert-fls' model and tokenizer, then applies a text classification pipeline to identify forward-looking statements in input sentences.

```Python
finbert = BertForSequenceClassification.from_pretrained('yiyanghkust/finbert-fls',num_labels=3)
tokenizer = BertTokenizer.from_pretrained('yiyanghkust/finbert-fls')
```

```Python
nlp = pipeline("text-classification", model=finbert, tokenizer=tokenizer)
results = nlp(['we expect the age of our fleet to enhance availability and reliability due to reduced downtime for repairs.',
               'on an equivalent unit of production basis, general and administrative expenses declined 24 percent from 1994 to $.67 per boe.',
               'we will continue to assess the need for a valuation allowance against deferred tax assets considering all available evidence obtained in future reporting periods.'])
```

```Python
results
```

--------------------------------

### Python: Perform FinBERT Sentiment Analysis

Source: https://github.com/yya518/finbert/blob/master/FinBERT-demo.ipynb

This section demonstrates how to use the FinBERT-Sentiment model to classify the sentiment of financial texts. It loads the pretrained 'yiyanghkust/finbert-tone' model and tokenizer, then uses a text classification pipeline to predict sentiment (Positive, Neutral, or Negative) for given input sentences.

```Python
finbert = BertForSequenceClassification.from_pretrained('yiyanghkust/finbert-tone',num_labels=3)
tokenizer = BertTokenizer.from_pretrained('yiyanghkust/finbert-tone')
```

```Python
nlp = pipeline("text-classification", model=finbert, tokenizer=tokenizer)
results = nlp(['growth is strong and we have plenty of liquidity.', 
               'there is a shortage of capital, and we need extra financing.',
              'formulation patents might protect Vasotec to a limited extent.'])
```

```Python
results
```

--------------------------------

### Python: Perform FinBERT ESG Classification

Source: https://github.com/yya518/finbert/blob/master/FinBERT-demo.ipynb

This section illustrates how to apply the FinBERT-ESG model for classifying financial texts into Environmental, Social, Governance, or None categories. It loads the 'yiyanghkust/finbert-esg' model and tokenizer, then uses a text classification pipeline to analyze input sentences related to ESG factors.

```Python
finbert = BertForSequenceClassification.from_pretrained('yiyanghkust/finbert-esg',num_labels=4)
tokenizer = BertTokenizer.from_pretrained('yiyanghkust/finbert-esg')
```

```Python
nlp = pipeline("text-classification", model=finbert, tokenizer=tokenizer)
results = nlp(['Managing and working to mitigate the impact our operations have on the environment is a core element of our business.',
               'Rhonda has been volunteering for several years for a variety of charitable community programs.',
               'Cabot\'s annual statements are audited annually by an independent registered public accounting firm.',
               'As of December 31, 2012, the 2011 Term Loan had a principal balance of $492.5 million.'])
```

```Python
results
```

--------------------------------

### Drop Missing Values from Dataset in Python

Source: https://github.com/yya518/finbert/blob/master/finetune.ipynb

This snippet cleans the DataFrame by dropping rows where either the 'sentence' or 'label' columns have missing values. This ensures data integrity before proceeding with model training and evaluation.

```Python
df = df.dropna(subset=['sentence', 'label']) ## drop missing values
```
Tokens:
Raw