The dev playground

Spark is used daily in data engineering for large-scale distributed data processing. I’m building Spark’s core execution architecture in Go to explore the systems behind distributed computation and learn through implementation. The work will be divided into three sessions, each with a different focus. Before starting the project, it helps to understand Spark’s core components and how they relate to one another. RDDs are Spark’s fundamental abstraction. We can think of an RDD as a metadata container carrying three main pieces of information: ...

Introduction Hi, I’m Wen — a software engineer with three years of experience building data platforms and large-scale data systems in production. I specialize in Spark, data lakes, data streaming, validation, and visualization. I’ve worked across the full data pipeline, from ingesting real-time factory streaming data and batch processing to storing it in data warehouses, building Spark batch jobs, and surfacing aggregated data for Tesla’s cell manufacturing yield and financial reporting. ...