Category:AWS - Revision history

RobertBushman: /* Sync Parquet */

2023-09-24T00:51:04Z

Sync Parquet

← Older revision		Revision as of 00:51, 24 September 2023
Line 30:		Line 30:
	</pre>		</pre>

	=== Sync Parquet ===		==== Sync Parquet ====
	<pre>		<pre>
	aws s3 sync ../data/reddit/parquet s3://iterative-chaos/cyphernews/harvest/reddit/parquet --size-only --delete		aws s3 sync ../data/reddit/parquet s3://iterative-chaos/cyphernews/harvest/reddit/parquet --size-only --delete
	</pre>		</pre>

RobertBushman: /* Sync the Discussion JSONs */

2023-09-24T00:50:57Z

Sync the Discussion JSONs

← Older revision		Revision as of 00:50, 24 September 2023
Line 25:		Line 25:
	</pre>		</pre>

	=== Sync the Discussion JSONs ===		==== Sync the Discussion JSONs ====
	<pre>		<pre>
	aws s3 sync ../data/reddit/json/discussion s3://iterative-chaos/cyphernews/harvest/reddit/json/discussion --size-only --delete		aws s3 sync ../data/reddit/json/discussion s3://iterative-chaos/cyphernews/harvest/reddit/json/discussion --size-only --delete

RobertBushman: /* Sync the Link List JSONs */

2023-09-24T00:50:48Z

Sync the Link List JSONs

← Older revision		Revision as of 00:50, 24 September 2023
Line 18:		Line 18:
	</pre>		</pre>

	=== Sync the Link List JSONs ===		=== Sync Cyphernews ===
			==== the Link List JSONs ====
	<pre>		<pre>
	aws s3 sync ../data/reddit/json/link_list s3://iterative-chaos/cyphernews/harvest/reddit/json/link_list --size-only --delete		aws s3 sync ../data/reddit/json/link_list s3://iterative-chaos/cyphernews/harvest/reddit/json/link_list --size-only --delete

RobertBushman: /* Sync the Link List JSONs */

2023-09-24T00:40:53Z

Sync the Link List JSONs

← Older revision		Revision as of 00:40, 24 September 2023
Line 21:		Line 21:
	<pre>		<pre>
	aws s3 sync ../data/reddit/json/link_list s3://iterative-chaos/cyphernews/harvest/reddit/json/link_list --size-only --delete		aws s3 sync ../data/reddit/json/link_list s3://iterative-chaos/cyphernews/harvest/reddit/json/link_list --size-only --delete
			aws s3 sync ../data/reddit/json/archive_link_list s3://iterative-chaos/cyphernews/harvest/reddit/json/archive_link_list --size-only --delete
	</pre>		</pre>

RobertBushman: /* Sync the Link List JSONs */

2023-09-24T00:39:58Z

Sync the Link List JSONs

@@ Line 21: / Line 21: @@
 <pre>
 aws s3 sync ../data/reddit/json/link_list s3://iterative-chaos/cyphernews/harvest/reddit/json/link_list --size-only --delete
 </pre>

RobertBushman at 00:37, 24 September 2023

2023-09-24T00:37:49Z

← Older revision		Revision as of 00:37, 24 September 2023
Line 16:		Line 16:
	<pre>		<pre>
	aws --profile iterative-chaos s3 sync dedupe s3://iterative-chaos/cyphernews/harvest/reddit/parquet/dedupe --size-only --delete		aws --profile iterative-chaos s3 sync dedupe s3://iterative-chaos/cyphernews/harvest/reddit/parquet/dedupe --size-only --delete
			</pre>

			=== Sync the Link List JSONs ===
			<pre>
			aws s3 sync ../data/reddit/json/link_list s3://iterative-chaos/cyphernews/harvest/reddit/json/link_list --size-only --delete
	</pre>		</pre>

RobertBushman: /* S3 Sync */

2023-09-24T00:33:05Z

S3 Sync

← Older revision		Revision as of 00:33, 24 September 2023
Line 13:		Line 13:
	* Ensure you've properly configured your AWS CLI with the necessary access rights to perform the s3 sync operation.		* Ensure you've properly configured your AWS CLI with the necessary access rights to perform the s3 sync operation.
	* The aws s3 sync command by default won't delete files in the destination that are not present in the source. However, if you add the --delete option, it would delete files from the S3 bucket th		* The aws s3 sync command by default won't delete files in the destination that are not present in the source. However, if you add the --delete option, it would delete files from the S3 bucket th

			<pre>
			aws --profile iterative-chaos s3 sync dedupe s3://iterative-chaos/cyphernews/harvest/reddit/parquet/dedupe --size-only --delete
			</pre>

RobertBushman: /* S3 Sync */

2023-09-23T04:49:20Z

S3 Sync

← Older revision		Revision as of 04:49, 23 September 2023
Line 2:		Line 2:
	= CLI =		= CLI =
	== S3 Sync ==		== S3 Sync ==
	<pre>aws s3 sync <local_directory> s3://<bucket_name>/<optional_prefix> --exact-timestamps --size-only</pre>		<pre>aws --profile monkey-banana s3 sync <local_directory> s3://<bucket_name>/<optional_prefix> --exact-timestamps --size-only</pre>

	Here's a breakdown of the options:		Here's a breakdown of the options:

RobertBushman: Created page with "Category:Hacking = CLI = == S3 Sync ==

aws s3 sync  s3:/// --exact-timestamps --size-only

Here's a breakdown of the options: * : The local directory you want to sync with S3. * s3:///: The destination S3 bucket and an optional prefix (like a folder). * --exact-timestamps: By default, aws s3 sync uses the LastModified time to determine whether an S3 object is the..."

2023-09-23T04:48:48Z

Created page with "Category:Hacking = CLI = == S3 Sync == <pre>aws s3 sync <local_directory> s3://<bucket_name>/<optional_prefix> --exact-timestamps --size-only</pre> Here's a breakdown of the options: * <local_directory>: The local directory you want to sync with S3. * s3://<bucket_name>/<optional_prefix>: The destination S3 bucket and an optional prefix (like a folder). * --exact-timestamps: By default, aws s3 sync uses the LastModified time to determine whether an S3 object is the..."

New page

[[Category:Hacking]]
= CLI =
== S3 Sync ==
<pre>aws s3 sync <local_directory> s3://<bucket_name>/<optional_prefix> --exact-timestamps --size-only</pre>

Here's a breakdown of the options:
* <local_directory>: The local directory you want to sync with S3.
* s3://<bucket_name>/<optional_prefix>: The destination S3 bucket and an optional prefix (like a folder).
* --exact-timestamps: By default, aws s3 sync uses the LastModified time to determine whether an S3 object is the same as a local file. If the times are not the same but the sizes of the files are the same, AWS CLI will consider them to be the same and will not replace the file. The --exact-timestamps option changes this behavior to consider files as different if their LastModified times are different.
* --size-only: Use this option to make the comparison based on the size of the files only, and not the last modified timestamp. This can be useful if timestamps might differ but the content hasn't changed.

Note:
* Ensure you've properly configured your AWS CLI with the necessary access rights to perform the s3 sync operation.
* The aws s3 sync command by default won't delete files in the destination that are not present in the source. However, if you add the --delete option, it would delete files from the S3 bucket th