Deep dive in Spark code generation
Date:
Apache Spark introduced code generation in 2.0 to make queries up to 100x faster. Of course, there is also a dark side of code generation: which are the problems it brings? How to address them? Is it production ready? Let’s take a journey through the evolution of Spark code generation from its birth to the current status, with a glance to the future improvements.