CypherReddit - Revision history

RobertBushman: /* Data Processing Process */

2023-10-13T15:46:30Z

Data Processing Process

← Older revision		Revision as of 15:46, 13 October 2023
Line 31:		Line 31:
	* 2023-08-26: Overnight buildup: 3,649 (bug)		* 2023-08-26: Overnight buildup: 3,649 (bug)
	* 2023-08-27: overnight buildup: 369		* 2023-08-27: overnight buildup: 369
			* Note: The mutation rate of the discussion is faster than the mutation rate of the story summary. Therefore, the regeneration rate of the discussion tree should be decoupled from the regeneration rate of the story summary.
			** Cocicil: Since the regeneration rate of the discussion tree is different from the regeneration rate of the summary, the content store may benefit from being updated at a different rate. (eg: if using static HTML files, those might be generated by different processes, or the summary might be static HTML while the discussion could live in a Nostr relay or graph database).

	= Interesting Subreddits =		= Interesting Subreddits =

RobertBushman: /* Data Processing Process */

2023-10-13T15:06:06Z

Data Processing Process

← Older revision		Revision as of 15:06, 13 October 2023
Line 24:		Line 24:
	## Writes the list to archive/gpt_stories_{edition_id}.json		## Writes the list to archive/gpt_stories_{edition_id}.json
	# $ python step_06_gpt_to_html.py		# $ python step_06_gpt_to_html.py
	## Loads the 10 most recent gpt_stories_*.json		## Loads the 10 most recent archive/gpt_stories_*.json
	## generates the news.html file.		## generates the html/news.html file.
	# $ rsync -avz html/summary /var/www/html/		# $ rsync -avz html/summary /var/www/html/
	# $ cp html/news.html /var/www/html/		# $ cp html/news.html /var/www/html/

RobertBushman: /* Data Processing Process */

2023-10-13T15:05:29Z

Data Processing Process

← Older revision		Revision as of 15:05, 13 October 2023
Line 2:		Line 2:

	= Data Processing Process =		= Data Processing Process =
	# $ python ~~get_link_lists~~.py # ~~daily~~		# $ python step_01_get_link_lists.py
	## ~~Needs data archival / backup.~~		## Pulls 24h top 100 every 4 hours
	##~~# Doing a brute-force full backup right now~~, ~~might be sufficient for the time being~~.		## Pulls 7d top 1,000 every 24 hours
	###~~# $ scp -i key.pem reddit-link-list.tar~~.bz2 ~~admin@www.iterativechaos.com~~:./		## Stores as json.bz2, in pages of 100 links.
	##~~# Can do just week/day going forward~~, ~~but that will quickly get~~ slow.		# $ python step_02_parquet_link_list.py
	### ~~Storing~~ the files ~~as .json~~.~~bz2 would be a big improvement~~		## Union all of json.bz2 rows into Parquet (ie: raw layer).
	# $ python ~~parquet_link_list~~.py		## Uses compaction, gets slow after about a week without a compaction run.
	## ~~(done) Make this iterate~~ and ~~do a full refresh each time~~		# $ python step_03_dedupe_link_list_parquet.py
	## ~~Reconsider full refresh when processing time goes over~~ a ~~minute (currently runs~~ in ~~12 seconds (not sure if that's real or user))~~		## Takes the most recent entry for each link_id from the raw layer.
	## ~~Old Version~~:		## Writes to parquet subreddit/day files.
	### $ python ~~parquet_link_list~~.py ~~../data/reddit/link_list/day/science/ ../data/reddit/parquet/link_list/day/science/~~		# $ python step_04_get_discussions.py
	# ~~$ python dedupe_link_list_parquet~~.py		## Finds discussions above a threshold of comments and upvotes.
	## ~~(done) Make this iterate and do a full refresh~~ each ~~time~~		## Downloads the discussion and stores as json.bz2.
	## ~~Reconsider full refresh when processing time goes over a minute (currently runs in 2 seconds (real))~~		## Skips download if a file exists and no more than 20% increase in comments.
	# ~~$ python get_discussions~~.py		## TODO: should do a "final download" after a week (or whatever).
	## ~~Make this check~~ the ~~existing download, harvest timestamp, num_comments~~		# $ python step_05_discussion_to_gpt.py
	# $ python ~~discussion_to_gpt~~.py		## Finds the top couple discussions in a set of subreddits.
	## ~~Make this iterate and do a full refresh each time~~		## Pulls the top N tokens worth of comments from each discussion.
	## ~~Reconsider full refresh when processing time goes over a minute~~		## Sends to GPT to summarize.
	# ~~hit ChatGPT with output~~		## Writes each summary to html/summary/{link-id}.html
	#~~# Change this to API call to ChatGPT~~		## Writes the list to archive/gpt_stories_{edition_id}.json
	~~## Make this check the existing generate, harvest versus generate timestamp~~		# $ python step_06_gpt_to_html.py
	~~## Add data archival~~ / ~~backup~~		## Loads the 10 most recent gpt_stories_*.json
			## generates the news.html file.
			# $ rsync -avz html/summary /var/www/html/
			# $ cp html/news.html /var/www/html/

	* 2023-08-26: Overnight buildup: 3,649 (bug)		* 2023-08-26: Overnight buildup: 3,649 (bug)

RobertBushman at 17:06, 5 October 2023

2023-10-05T17:06:53Z

Show changes

RobertBushman: Created page with "Category:CypherTech"

2023-10-05T17:06:02Z

Created page with "Category:CypherTech"

New page

[[Category:CypherTech]]