thank you for sharing! The core of memory management is abstracted away. All of the query execution logic is unaware of the source of memory pointer. That said we are still learning and really appreciate your feedback. There are some places where we could not beat aggregation of julia, but the delta wasn't very big. This could have been down to mapped memory. We will definitely try things with direct memory too!
The databases I implemented experimented with various ways to compile queries. Turns out that the JVM can run quite fast. I feel like LLVM (Julia) is likely to be able to be better for throughput and definitely better for predictability of performance.
If you know layouts and sizes, then your generated code can run really fast - using SIMD and not checking bounds is a win.
Hugepages would greatly reduce pagetable bookkeeping, but obviously may magnify writes. Wish I could have tried that!
Our best performance currently is in C++ code and LLVM is something we are considering using to compute expressions, such as predicates and select clause items. This is most likely to be way faster than what we can currently do in Java. What I would like to know if LLVM can optimize all the way to AVX512?
We also need to experiment with hugepages. The beauty is that if read and write are separated - there is no issue with writes. They can still use 4k pages!
