Anomaly detection with Transformers involves using transformer-based models, such as BERT or GPT, to identify outliers or anomalies in time series data. One popular approach is to use the transformer model to learn the patterns in the time series data and then use a thresholding method to identify data points that deviate significantly from these patterns.
In this example, we'll use the PyTorch library along with the Transformers library to create a simple anomaly detection model using BERT. We'll use a publicly available time series dataset from the Numenta Anomaly Benchmark (NAB) for demonstration purposes.
Make sure you have the necessary libraries installed:
pip install torch transformers numpy pandas matplotlib
Here's the Python code for the anomaly detection example:
import torch
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from transformers import BertTokenizer, BertForSequenceClassification
# Load the NAB dataset (or any other time series dataset)
# Replace 'nyc_taxi.csv' with your dataset filename or URL
data = pd.read_csv('https://raw.githubusercontent.com/numenta/NAB/master/data/realKnownCause/nyc_taxi.csv')
time_series = data['value'].values
# Normalize the time series data
mean, std = time_series.mean(), time_series.std()
time_series = (time_series - mean) / std
# Define the window size for each input sequence
window_size = 10
# Prepare the input sequences and labels
sequences = []
labels = []
for i in range(len(time_series) - window_size):
seq = time_series[i:i+window_size]
sequences.append(seq)
labels.append(1 if time_series[i+window_size] > 3 * std else 0) # Threshold-based anomaly labeling
# Convert sequences and labels to tensors
sequences = torch.tensor(sequences)
labels = torch.tensor(labels)
# Load the BERT tokenizer and model
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model = BertForSequenceClassification.from_pretrained('bert-base-uncased')
# Tokenize the sequences and pad them to the same length
inputs = tokenizer.batch_encode_plus(
sequences.tolist(),
add_special_tokens=True,
padding=True,
truncation=True,
max_length=window_size,
return_tensors='pt'
)
# Perform the anomaly detection with BERT
outputs = model(**inputs, labels=labels.unsqueeze(1))
loss = outputs.loss
logits = outputs.logits
probabilities = torch.sigmoid(logits).squeeze().detach().numpy()
# Plot the original time series and the anomaly scores
plt.figure(figsize=(12, 6))
plt.plot(data['timestamp'], time_series, label='Original Time Series')
plt.plot(data['timestamp'][window_size:], probabilities, label='Anomaly Scores', color='red')
plt.xlabel('Timestamp')
plt.ylabel('Value')
plt.legend()
plt.title('Anomaly Detection with Transformers')
plt.show()
This code loads the NYC taxi dataset from the Numenta Anomaly Benchmark (NAB), normalizes the data, and creates sequences of fixed window sizes. The model then learns to classify each sequence as an anomaly or not, using threshold-based labeling. The anomaly scores are plotted on top of the original time series data.
Note that this is a simplified example, and more sophisticated anomaly detection models and techniques can be used in practice. Additionally, fine-tuning the model on a specific anomaly dataset may improve its performance. However, this example should give you a starting point for anomaly detection with Transformers on time series data.
