CypherResilience - Revision history

RobertBushman: /* Delete Parquet */

2024-03-09T20:40:15Z

Delete Parquet

← Older revision		Revision as of 20:40, 9 March 2024
Line 147:		Line 147:
	=== Delete Parquet ===		=== Delete Parquet ===
	<pre>		<pre>
	rsync -~~avz~~ --size-only --delete www.iterativechaos.com:/opt/cypherpunk/data/reddit/parquet/ /opt/cypherpunk/data/reddit/parquet/		rsync -navz --size-only --delete www.iterativechaos.com:/opt/cypherpunk/data/reddit/parquet/ /opt/cypherpunk/data/reddit/parquet/
	</pre>		</pre>

RobertBushman: /* Delete All Other JSON */

2024-03-09T20:40:08Z

Delete All Other JSON

← Older revision		Revision as of 20:40, 9 March 2024
Line 141:		Line 141:
	=== Delete All Other JSON ===		=== Delete All Other JSON ===
	<pre>		<pre>
	rsync -~~avz~~ --size-only www.iterativechaos.com:/opt/cypherpunk/data/reddit/json/archive_link_list/ /opt/cypherpunk/data/reddit/json/archive_link_list/		rsync -navz --size-only www.iterativechaos.com:/opt/cypherpunk/data/reddit/json/archive_link_list/ /opt/cypherpunk/data/reddit/json/archive_link_list/
	rsync -~~avz~~ --size-only --delete www.iterativechaos.com:/opt/cypherpunk/data/reddit/json/link_list/ /opt/cypherpunk/data/reddit/json/link_list/		rsync -navz --size-only --delete www.iterativechaos.com:/opt/cypherpunk/data/reddit/json/link_list/ /opt/cypherpunk/data/reddit/json/link_list/
	</pre>		</pre>

RobertBushman: /* Sync */

2024-03-09T20:38:54Z

Sync

← Older revision		Revision as of 20:38, 9 March 2024
Line 130:		Line 130:
	rsync -avz --size-only www.iterativechaos.com:/opt/cypherpunk/data/reddit/ /opt/cypherpunk/data/reddit/		rsync -avz --size-only www.iterativechaos.com:/opt/cypherpunk/data/reddit/ /opt/cypherpunk/data/reddit/
	</pre>		</pre>
	=== ~~Pull~~ Discussions ===		=== Delete Discussions ===
	This has to be done separately because the remote harvester host will not have the full discussion archive. Remote will have the current and previous month, which can be pulled with --delete.		This has to be done separately because the remote harvester host will not have the full discussion archive. Remote will have the current and previous month, which can be pulled with --delete.

Line 139:		Line 139:
	</pre>		</pre>

	=== ~~Pull~~ All Other JSON ===		=== Delete All Other JSON ===
	<pre>		<pre>
	rsync -avz --size-only www.iterativechaos.com:/opt/cypherpunk/data/reddit/json/archive_link_list/ /opt/cypherpunk/data/reddit/json/archive_link_list/		rsync -avz --size-only www.iterativechaos.com:/opt/cypherpunk/data/reddit/json/archive_link_list/ /opt/cypherpunk/data/reddit/json/archive_link_list/
Line 145:		Line 145:
	</pre>		</pre>

	=== ~~Pull Compacted~~ Parquet ===		=== Delete Parquet ===
	<pre>		<pre>
	rsync -avz --size-only --delete www.iterativechaos.com:/opt/cypherpunk/data/reddit/parquet~~/compacted_raw~~/ /opt/cypherpunk/data/reddit/parquet~~/compacted_raw~~/		rsync -avz --size-only --delete www.iterativechaos.com:/opt/cypherpunk/data/reddit/parquet/ /opt/cypherpunk/data/reddit/parquet/
	</pre>		</pre>

RobertBushman: /* Data Redundancy */

2024-03-09T20:13:27Z

Data Redundancy

← Older revision		Revision as of 20:13, 9 March 2024
Line 124:		Line 124:
	\|}		\|}

	== Sync (~~simplified~~) ==		== Sync ==
			'''''Execute On Local (Backup) Host'''''
	=== Pull All The New Stuff ===		=== Pull All The New Stuff ===
	<pre>		<pre>
	rsync -avz --size-only www.iterativechaos.com:/opt/cypherpunk/data/reddit/ /opt/cypherpunk/data/reddit/		rsync -avz --size-only www.iterativechaos.com:/opt/cypherpunk/data/reddit/ /opt/cypherpunk/data/reddit/
	</pre>		</pre>
	~~=== Check What's Gettin' Deleted ===~~
	~~<pre>~~
	~~rsync -navz --size-only --delete www.iterativechaos.com:/opt/cypherpunk/data/reddit/ /opt/cypherpunk/data/reddit/ \| tee /opt/cypherpunk/data/deletes.log~~
	~~</pre>~~
	~~=== Fix whatever's wrong. ===~~
	~~== Sync (Old Version) ==~~
	~~'''''Execute On Local (Backup) Host'''''~~
	=== Pull Discussions ===		=== Pull Discussions ===
	This has to be done separately because the remote harvester host will not have the full discussion archive. Remote will have the current and previous month, which can be pulled with --delete.		This has to be done separately because the remote harvester host will not have the full discussion archive. Remote will have the current and previous month, which can be pulled with --delete.

RobertBushman: /* Do the delete */

2024-03-09T20:12:36Z

Do the delete

← Older revision		Revision as of 20:12, 9 March 2024
Line 134:		Line 134:
	</pre>		</pre>
	=== Fix whatever's wrong. ===		=== Fix whatever's wrong. ===
	~~=== Do the delete ===~~
	~~<pre>~~
	~~rsync -avz --size-only --delete www.iterativechaos.com:/opt/cypherpunk/data/reddit/ /opt/cypherpunk/data/reddit/ \| tee /opt/cypherpunk/data/deletes.log~~
	~~</pre>~~

	== Sync (Old Version) ==		== Sync (Old Version) ==
	'''''Execute On Local (Backup) Host'''''		'''''Execute On Local (Backup) Host'''''

RobertBushman: /* Sync */

2024-03-09T20:02:39Z

Sync

@@ Line 124: / Line 124: @@
 |}
-== Sync ==
+== Sync (simplified) ==
 '''''Execute On Local (Backup) Host'''''
 === Pull Discussions ===

RobertBushman: /* Execution */

2024-03-09T19:47:03Z

Execution

← Older revision		Revision as of 19:47, 9 March 2024
Line 60:		Line 60:
	# remove the active month files from compacted		# remove the active month files from compacted
	find compacted_raw/ -path 'created_year=2023/created_month=10/' -type f \| xargs rm		find compacted_raw/ -path 'created_year=2023/created_month=10/' -type f \| xargs rm
			# make the dirs in compacted_raw
			find raw/ -path '*created_year=2024/created_month=2' -type d \| awk '{print "mkdir compacted_"$1}'
	# Move the active month files from raw to compacted_raw		# Move the active month files from raw to compacted_raw
	find raw -path 'created_year=2023/created_month=10/' -type f \| awk '{print "mv "$1" compacted_"$1}'		find raw -path 'created_year=2023/created_month=10/' -type f \| awk '{print "mv "$1" compacted_"$1}'

RobertBushman: /* JSON, Daily */

2024-03-09T19:02:17Z

JSON, Daily

← Older revision		Revision as of 19:02, 9 March 2024
Line 157:		Line 157:
	==== JSON, Daily ====		==== JSON, Daily ====
	<pre>		<pre>
	time aws s3 sync --size-only --delete /opt/cypherpunk/data/reddit/json/ s3://iterative-chaos/cyphernews/harvest/reddit/json/		time aws s3 sync --size-only --delete --dryrun /opt/cypherpunk/data/reddit/json/ s3://iterative-chaos/cyphernews/harvest/reddit/json/
	</pre>		</pre>

	==== Parquet Compacted, Weekly ====		==== Parquet Compacted, Weekly ====
	<pre>		<pre>
	time aws s3 sync --size-only --delete --dryrun /opt/cypherpunk/data/reddit/parquet/compacted_raw/ s3://iterative-chaos/cyphernews/harvest/reddit/parquet/compacted_raw/		time aws s3 sync --size-only --delete --dryrun /opt/cypherpunk/data/reddit/parquet/compacted_raw/ s3://iterative-chaos/cyphernews/harvest/reddit/parquet/compacted_raw/
	</pre>		</pre>

RobertBushman: /* Parquet Compacted, Weekly */

2024-03-09T19:02:05Z

Parquet Compacted, Weekly

← Older revision		Revision as of 19:02, 9 March 2024
Line 161:		Line 161:
	==== Parquet Compacted, Weekly ====		==== Parquet Compacted, Weekly ====
	<pre>		<pre>
	time aws s3 sync --size-only --delete /opt/cypherpunk/data/reddit/parquet/compacted_raw/ s3://iterative-chaos/cyphernews/harvest/reddit/parquet/compacted_raw/		time aws s3 sync --size-only --delete --dryrun /opt/cypherpunk/data/reddit/parquet/compacted_raw/ s3://iterative-chaos/cyphernews/harvest/reddit/parquet/compacted_raw/
	</pre>		</pre>

RobertBushman: /* gigabrick backup */

2024-03-09T18:19:54Z

gigabrick backup

← Older revision		Revision as of 18:19, 9 March 2024
Line 145:		Line 145:

	=== gigabrick backup ===		=== gigabrick backup ===
			<pre>
			ssh 192.186.0.8
			lsblk
			udisksctl mount -b /dev/sdb2
			</pre>
	<pre>		<pre>
	rsync -navz --size-only --delete /opt/cypherpunk/data/reddit/ 192.168.0.8:/media/bob/backup/cypherpunk/data/reddit/		rsync -navz --size-only --delete /opt/cypherpunk/data/reddit/ 192.168.0.8:/media/bob/backup/cypherpunk/data/reddit/