Category:Parquet - Revision history

RobertBushman: RobertBushman moved page Category:Parquert to Category:Parquet: Typo

2023-10-14T23:41:33Z

RobertBushman moved page Category:Parquert to Category:Parquet: Typo

← Older revision

Revision as of 23:41, 14 October 2023

RobertBushman: /* Partitioning */

2023-09-14T13:00:52Z

Partitioning

← Older revision		Revision as of 13:00, 14 September 2023
Line 4:		Line 4:

	== Partitioning ==		== Partitioning ==
	Partitioning your data well means each part file within a partition will have fewer rows ~~that do not need~~ to be ~~filtered~~.		Partitioning your data well means each part file within a partition will have fewer rows, a greater percentage of them will be relevant to your query, and fewer accesses to external datasets will be required.

	Reducing the row length of a collection of rows means you can include more columns in the same size Parquet part file.		Reducing the row length of a collection of rows means you can include more columns or a larger share of the total dataset in the same size Parquet part file.

	~~Assuming this aligns well~~ with your ~~queries~~, ~~this means increasing~~ the ~~number~~ of		Having more columns means you can do more complex statistics without joining against other datasets. Increasing the share of the total dataset means you can get the same job done with fewer task nodes.

			But the cost is that are splitting up the dataset. If your query mismatches the index, you lose all the benefit of indexing and may introduce problems memory footprint for a single task.

RobertBushman: Created page with "Category:Hacking = Overview = Parquet is a file format that collects schema, columnar data, and columnar metadata in a partitioned collection of files. The partitioned files allow many processes to work together simultaneously, the columnar data enables fast aggregate values for data analytics, and the columnar metadata ensures each process has the information it needs to access the fields quickly. == Partitioning == Partitioning your data well means each part file..."

2023-09-14T12:44:25Z

Created page with "Category:Hacking = Overview = Parquet is a file format that collects schema, columnar data, and columnar metadata in a partitioned collection of files. The partitioned files allow many processes to work together simultaneously, the columnar data enables fast aggregate values for data analytics, and the columnar metadata ensures each process has the information it needs to access the fields quickly. == Partitioning == Partitioning your data well means each part file..."

New page

[[Category:Hacking]]
= Overview =
Parquet is a file format that collects schema, columnar data, and columnar metadata in a partitioned collection of files. The partitioned files allow many processes to work together simultaneously, the columnar data enables fast aggregate values for data analytics, and the columnar metadata ensures each process has the information it needs to access the fields quickly.

== Partitioning ==
Partitioning your data well means each part file within a partition will have fewer rows that do not need to be filtered.

Reducing the row length of a collection of rows means you can include more columns in the same size Parquet part file.

Assuming this aligns well with your queries, this means increasing the number of