Welcome to the repository for ECG-JEPA 2.0, the enhanced version of the original ECG-JEPA. This repository showcases advancements in the model, offering better performance and efficiency tailored to 12-lead ECG analysis.
- Larger Dataset: ECG-JEPA 2.0 is trained on an expanded dataset, providing better generalizability and robustness.
- Optimized Training and Inference: By leveraging flash attention, the pretraining process is approximately 25–30% faster, enabling quicker development cycles and lower computational costs.
- Enhanced Positional Encoding: The model now incorporates a new positional encoding scheme specifically designed for ECG datasets, improving the representation of lead and temporal information.
We have identified the presence of attention noise in ECG-JEPA, a phenomenon similar to what has been observed in large language models (LLMs). To mitigate this issue, we are currently exploring the use of Differential Transformers (arXiv:2410.05258). If this approach demonstrates improved performance, we plan to adopt Differential Transformers in future updates.
→ Differential transformer backbone collapses, which is a common problem in non-contrastive SSL frameworks.
Stay tuned for further updates and improvements!