Scaling if context window size has been a problem but now good potential of solutions using mamba.

HyenaDNA is one to look at wrt DNA.

And here are some other interesting links from Erik Garrison—a leader in pangenomics.

https://hazyresearch.stanford.edu/blog/2023-06-29-hyena-dna

https://github.com/instadeepai/nucleotide-transformer

https://dl.acm.org/doi/pdf/10.1145/3535508.3545512

https://github.com/dnbaker/bioseq

https://huggingface.co/AIRI-Institute/gena-lm-bert-base

https://discuss.huggingface.co/t/dna-long-sequence-tokenizat...