View source for Learning Big Data

[[Category:Hacking]]

Registry of Open Data on AWS[https://registry.opendata.aws/]

Q: There are lot open datasets to play with here

A: Yeah - you can learn Spark and maybe help cure cancer at the same time. :)

Q: Could you also please help me with the most relevant skills I need to focus on for Data Engineering career path

A: Sure - the Open Source tools are from The Hadoop Ecosystem. Here's a Coursera course[https://www.coursera.org/lecture/introduction-to-big-data-with-spark-hadoop/hadoop-ecosystem-g7y7n] that will give you an introduction,

The main components we use are Spark[https://en.wikipedia.org/wiki/Apache_Spark], Parquet[https://en.wikipedia.org/wiki/Apache_Parquet], and Hive[https://en.wikipedia.org/wiki/Apache_Hive].

We use AWS implementations of Spark called EMR[https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-what-is-emr.html] and Glue[https://aws.amazon.com/glue/], and the AWS implementation of Hive called Glue Data Catalog[https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-hive-metastore-glue.html].

Then we store the data in S3[https://aws.amazon.com/s3/].

The most challenging thing to learn is how to performance optimize joins between billion row datasets. When you get to the point where you can identify small changes in queries that make job runs go from hours to minutes, you'll know you're starting to see the problem.

You'll want to look into things like window functions, broadcast joins, and partitioning strategies.

Also the effects of things like sort order in stored data and coalesce, repartition, and part files sizes.

Those things should get you far enough down the path that you'll see lots of other roads you can explore. From there you can choose your own adventure according to how you'd like to develop your career.