SIMD execution in Comet

[ajeyabsfujitsu]

In the architecture diagram of Comet documentation, it mentions that there is SIMD accelerated columnar execution enabled. On this front I have a few queries.

1. Is the SIMD query execution enabled only for certain operations like Join?
2. In Comet's codebase there is no presence of SIMD related keywords (read SVE,Neon for ARM and AVX,SSE for x86). Rust also does not support SVE currently. So how is SIMD being enabled?
3. Even when a search of SIMD is done, the only result that shows up is for SIMD-adhler. Is this the only place where SIMD related code is added?

[andygrove]

@ajeyabsfujitsu Sorry, I did not see this discussion until now.

DataFusion uses SIMD by leveraging the Rust compiler's auto-vectorization capabilities. Instead of manually writing SIMD intrinsics, it relies on the compiler to optimize data-parallel operations (like filtering, projections, and aggregations) during code generation. This approach ensures portability across architectures while still benefiting from hardware acceleration.

Comet uses DataFusion for native query exection and therefore benefits from this SIMD support.