At least for ClickBench, the remaining bottleneck is memory bandwidth. I think that's mostly going to be solved by better data representations than SIMD.
One of the biggest wins was our hash table implementation. Depending on the cardinality of the data, it switches between design that is optimized for L2 cache vs something that is outside of L2.
TigerBeetle has static allocation for their known workload.
SIMD is more useful for heavy parallel analytical workloads. There are probably returns from SIMD even for index updates on transactional inserts. SIMD is faster for: vectorized index comparisons, Vectorized Constraint Validation, Multi-Column SIMD Hashing, Masked Bitmaps for Nullable Fields, gather and scatter, string processing, casting to numeric types.
IIUC FWIW Cerebras' does not have L2 cache.
TIL Vortex is Zero-Copy compatible with Arrow;
vortex-data/vortex: An extensible, state-of-the-art framework for columnar compression, and the fastest FOSS columnar file format. Formerly at @spiraldb, now an Incubation Stage project at LFAI&Data, part of the Linux Foundation. https://github.com/vortex-data/vortex
Vortex also has Segment Profiling and Adaptive Encoding.
Comments
At least for ClickBench, the remaining bottleneck is memory bandwidth. I think that's mostly going to be solved by better data representations than SIMD.
One of the biggest wins was our hash table implementation. Depending on the cardinality of the data, it switches between design that is optimized for L2 cache vs something that is outside of L2.
TigerBeetle has static allocation for their known workload.
SIMD is more useful for heavy parallel analytical workloads. There are probably returns from SIMD even for index updates on transactional inserts. SIMD is faster for: vectorized index comparisons, Vectorized Constraint Validation, Multi-Column SIMD Hashing, Masked Bitmaps for Nullable Fields, gather and scatter, string processing, casting to numeric types.
IIUC FWIW Cerebras' does not have L2 cache.
TIL Vortex is Zero-Copy compatible with Arrow;
Vortex also has Segment Profiling and Adaptive Encoding.
"What is Vortex? Columnar File Format Explained" https://spice.ai/learn/vortex