Learning Big Data - Revision history

RobertBushman at 20:36, 27 September 2021

2021-09-27T20:36:28Z

← Older revision		Revision as of 20:36, 27 September 2021
Line 5:		Line 5:
	Q: There are lot open datasets to play with here		Q: There are lot open datasets to play with here

	A: Yeah - you can learn Spark and maybe help cure cancer at the same time. :~~slightly_smiling_face:~~		A: Yeah - you can learn Spark and maybe help cure cancer at the same time. :)

	Q: Could you also please help me with the most relevant skills I need to focus on for Data Engineering career path		Q: Could you also please help me with the most relevant skills I need to focus on for Data Engineering career path

RobertBushman at 20:36, 27 September 2021

2021-09-27T20:36:00Z

← Older revision		Revision as of 20:36, 27 September 2021
Line 1:		Line 1:
	[Category:Hacking]		[[Category:Hacking]]

	Registry of Open Data on AWS[https://registry.opendata.aws/]		Registry of Open Data on AWS[https://registry.opendata.aws/]

RobertBushman at 20:35, 27 September 2021

2021-09-27T20:35:41Z

← Older revision		Revision as of 20:35, 27 September 2021
Line 1:		Line 1:
			[Category:Hacking]

	Registry of Open Data on AWS[https://registry.opendata.aws/]		Registry of Open Data on AWS[https://registry.opendata.aws/]

RobertBushman at 20:34, 27 September 2021

2021-09-27T20:34:58Z

← Older revision		Revision as of 20:34, 27 September 2021
Line 1:		Line 1:

	Registry of Open Data on AWS:		Registry of Open Data on AWS[https://registry.opendata.aws/]
	https://registry.opendata.aws/

	Q: There are lot open datasets to play with here		Q: There are lot open datasets to play with here

RobertBushman at 20:34, 27 September 2021

2021-09-27T20:34:30Z

← Older revision		Revision as of 20:34, 27 September 2021
Line 24:		Line 24:

	Those things should get you far enough down the path that you'll see lots of other roads you can explore. From there you can choose your own adventure according to how you'd like to develop your career.		Those things should get you far enough down the path that you'll see lots of other roads you can explore. From there you can choose your own adventure according to how you'd like to develop your career.

	~~<references />~~

RobertBushman at 20:34, 27 September 2021

2021-09-27T20:34:22Z

← Older revision		Revision as of 20:34, 27 September 2021
Line 9:		Line 9:
	Q: Could you also please help me with the most relevant skills I need to focus on for Data Engineering career path		Q: Could you also please help me with the most relevant skills I need to focus on for Data Engineering career path

	A: Sure - the Open Source tools are from The Hadoop Ecosystem. Here's a Coursera course~~<ref name="Coursera Intro to Hadoop">~~[https://www.coursera.org/lecture/introduction-to-big-data-with-spark-hadoop/hadoop-ecosystem-g7y7n]~~</ref>~~ that will give you an introduction,		A: Sure - the Open Source tools are from The Hadoop Ecosystem. Here's a Coursera course[https://www.coursera.org/lecture/introduction-to-big-data-with-spark-hadoop/hadoop-ecosystem-g7y7n] that will give you an introduction,

	The main components we use are Spark[https://en.wikipedia.org/wiki/Apache_Spark], Parquet[https://en.wikipedia.org/wiki/Apache_Parquet], and Hive[https://en.wikipedia.org/wiki/Apache_Hive].		The main components we use are Spark[https://en.wikipedia.org/wiki/Apache_Spark], Parquet[https://en.wikipedia.org/wiki/Apache_Parquet], and Hive[https://en.wikipedia.org/wiki/Apache_Hive].

RobertBushman at 20:32, 27 September 2021

2021-09-27T20:32:53Z

← Older revision		Revision as of 20:32, 27 September 2021
Line 9:		Line 9:
	Q: Could you also please help me with the most relevant skills I need to focus on for Data Engineering career path		Q: Could you also please help me with the most relevant skills I need to focus on for Data Engineering career path

	A: Sure - the Open Source tools are from The Hadoop Ecosystem. Here's a Coursera course[https://www.coursera.org/lecture/introduction-to-big-data-with-spark-hadoop/hadoop-ecosystem-g7y7n] that will give you an introduction,		A: Sure - the Open Source tools are from The Hadoop Ecosystem. Here's a Coursera course<ref name="Coursera Intro to Hadoop">[https://www.coursera.org/lecture/introduction-to-big-data-with-spark-hadoop/hadoop-ecosystem-g7y7n]</ref> that will give you an introduction,

	The main components we use are Spark[https://en.wikipedia.org/wiki/Apache_Spark], Parquet[https://en.wikipedia.org/wiki/Apache_Parquet], and Hive[https://en.wikipedia.org/wiki/Apache_Hive].		The main components we use are Spark[https://en.wikipedia.org/wiki/Apache_Spark], Parquet[https://en.wikipedia.org/wiki/Apache_Parquet], and Hive[https://en.wikipedia.org/wiki/Apache_Hive].

RobertBushman at 20:31, 27 September 2021

2021-09-27T20:31:40Z

← Older revision		Revision as of 20:31, 27 September 2021
Line 24:		Line 24:

	Those things should get you far enough down the path that you'll see lots of other roads you can explore. From there you can choose your own adventure according to how you'd like to develop your career.		Those things should get you far enough down the path that you'll see lots of other roads you can explore. From there you can choose your own adventure according to how you'd like to develop your career.

			<references />

RobertBushman at 20:31, 27 September 2021

2021-09-27T20:31:07Z

← Older revision		Revision as of 20:31, 27 September 2021
Line 10:		Line 10:

	A: Sure - the Open Source tools are from The Hadoop Ecosystem. Here's a Coursera course[https://www.coursera.org/lecture/introduction-to-big-data-with-spark-hadoop/hadoop-ecosystem-g7y7n] that will give you an introduction,		A: Sure - the Open Source tools are from The Hadoop Ecosystem. Here's a Coursera course[https://www.coursera.org/lecture/introduction-to-big-data-with-spark-hadoop/hadoop-ecosystem-g7y7n] that will give you an introduction,

	~~https://www.coursera.org/lecture/introduction-to-big-data-with-spark-hadoop/hadoop-ecosystem-g7y7n~~

	The main components we use are Spark[https://en.wikipedia.org/wiki/Apache_Spark], Parquet[https://en.wikipedia.org/wiki/Apache_Parquet], and Hive[https://en.wikipedia.org/wiki/Apache_Hive].		The main components we use are Spark[https://en.wikipedia.org/wiki/Apache_Spark], Parquet[https://en.wikipedia.org/wiki/Apache_Parquet], and Hive[https://en.wikipedia.org/wiki/Apache_Hive].

RobertBushman: Created page with " Registry of Open Data on AWS: https://registry.opendata.aws/ Q: There are lot open datasets to play with here A: Yeah - you can learn Spark and maybe help cure cancer at th..."

2021-09-27T20:30:54Z

Created page with " Registry of Open Data on AWS: https://registry.opendata.aws/ Q: There are lot open datasets to play with here A: Yeah - you can learn Spark and maybe help cure cancer at th..."

New page

Registry of Open Data on AWS:
https://registry.opendata.aws/

Q: There are lot open datasets to play with here

A: Yeah - you can learn Spark and maybe help cure cancer at the same time. :slightly_smiling_face:

Q: Could you also please help me with the most relevant skills I need to focus on for Data Engineering career path

A: Sure - the Open Source tools are from The Hadoop Ecosystem. Here's a Coursera course[https://www.coursera.org/lecture/introduction-to-big-data-with-spark-hadoop/hadoop-ecosystem-g7y7n] that will give you an introduction,

https://www.coursera.org/lecture/introduction-to-big-data-with-spark-hadoop/hadoop-ecosystem-g7y7n

The main components we use are Spark[https://en.wikipedia.org/wiki/Apache_Spark], Parquet[https://en.wikipedia.org/wiki/Apache_Parquet], and Hive[https://en.wikipedia.org/wiki/Apache_Hive].

We use AWS implementations of Spark called EMR[https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-what-is-emr.html] and Glue[https://aws.amazon.com/glue/], and the AWS implementation of Hive called Glue Data Catalog[https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-hive-metastore-glue.html].

Then we store the data in S3[https://aws.amazon.com/s3/].

The most challenging thing to learn is how to performance optimize joins between billion row datasets. When you get to the point where you can identify small changes in queries that make job runs go from hours to minutes, you'll know you're starting to see the problem.

You'll want to look into things like window functions, broadcast joins, and partitioning strategies.

Also the effects of things like sort order in stored data and coalesce, repartition, and part files sizes.

Those things should get you far enough down the path that you'll see lots of other roads you can explore. From there you can choose your own adventure according to how you'd like to develop your career.