Sparse Transformers is another variant of the transformer architecture, proposed in the research paper titled "Sparse Transformers" by Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever, published in 2019. The main goal of Sparse Transformers is to improve memory efficiency in deep learning models, particularly for tasks involving long sequences.
Traditional transformers have a quadratic self-attention complexity, which means that the computational cost increases with the square of the sequence length. This poses a significant challenge when dealing with long sequences, such as in natural language processing tasks or other sequence-to-sequence problems. Sparse Transformers address this challenge by introducing several key components:
1. **Fixed Pattern Masking**: Instead of having every token attend to every other token, Sparse Transformers use a fixed pattern mask that limits the attention to a small subset of tokens. This reduces the number of computations required during attention and helps make the model more memory-efficient.
2. **Re-parametrization of Attention**: Sparse Transformers re-parametrize the attention mechanism using a set of learnable parameters, enabling the model to learn which tokens should be attended to for specific tasks. This approach allows the model to focus on relevant tokens and ignore irrelevant ones, further reducing memory consumption.
3. **Localized Attention**: To improve efficiency even further, Sparse Transformers adopt localized attention, where each token only attends to a nearby neighborhood of tokens within the sequence. This local attention helps in capturing short-range dependencies efficiently while keeping computational costs low.
By incorporating these design choices, Sparse Transformers achieve a substantial reduction in memory requirements and computational complexity compared to standard transformers. This efficiency is particularly advantageous when processing long sequences, as the model can handle much larger inputs without running into memory constraints.
Sparse Transformers have demonstrated competitive performance on various tasks, including language modeling, machine translation, and image generation. They have shown that with appropriate structural modifications, transformers can be made more memory-efficient and can handle much longer sequences than previously possible.
It's essential to note that both Reformer and Sparse Transformers tackle the issue of memory efficiency in transformers but do so through different approaches. Reformer utilizes reversible residual layers and locality-sensitive hashing attention, while Sparse Transformers use fixed pattern masking, re-parametrization of attention, and localized attention to achieve similar goals. The choice between the two depends on the specific requirements of the task and the available computational resources.
