Spark Performance Tuning – Part 1 | Data Exposed

This week’s Data Exposed show welcomes back Maxim Lukiyanov to kick off a 4-part series on Spark performance tuning with Spark 2.x. Maxim is a Senior PM on the big data HDInsight team and is in the studio today to present Part 1 of his 4-part series.

[03:00] – Spark Use Cases as it pertains to performance.

[06:00] – Performance Demos and why performance matters. Query graphs and query performance depending on query optimization.

[11:00] – Spark 2.x project Tungsten overview, which includes: [13:45] Catalyst query optimizer,  [15:20] Direct Memory management, [18:10] WholeStage Code generation.