Rsync support!

Posted
Comments 0

What? Rsync for what?

I’ve had a request from Beaches (he hangs out in our IRC channel and has been on one or two of the shitshows) to be able to download all episodes at once.

While you can certainly script a python scraper using urllib and lxml or something to parse our RSS feeds, that certainly shouldn’t be necessary!

So I’ve opened a public rsync share of the episode media/signatures and exported versions of the shownotes. Note that it’s capped at 4 active connections at a time- if this becomes a more popular method, I can raise this cap. For the time being, if it conks out, just try again in a bit. :) We also host a mirror of the e-zine PoC||GTFO.

$ rsync rsync://rsync.sysadministrivia.com/
┌─────────────────────────────────────────────────────────────────────┐
│ ____ _   _ ____ ____ ___  _  _ _ _  _ _ ____ ___ ____ _ _  _ _ ____ │
│ [__   \_/  [__  |__| |  \ |\/| | |\ | | [__   |  |__/ | |  | | |__| │
│ ___]   |   ___] |  | |__/ |  | | | \| | ___]  |  |  \ |  \/  | |  | │
│                                                                     │
└─────────────────────────────────────────────────────────────────────┘

		______ _______   ___   _ _____ 
		| ___ /  ___\ \ / | \ | /  __ \
		| |_/ \ `--. \ V /|  \| | /  \/
		|    / `--. \ \ / | . ` | |    
		| |\ \/\__/ / | | | |\  | \__/\
		\_| \_\____/  \_/ \_| \_/\____/

                            ..... 
		   _     ___  ___  __ __ __    
		  /_\   / __\/ __\/__/ _/ _\   
		 //_\\ / /  / /  /_\ \ \\ \    
		/  _  / /__/ /__//__ _\ _\ \   
		\_/ \_\____\____\__/ \__\__/   

         	   ___  __   _       __ _____  __  ___ 
         	  / _ \/__\ /_\   /\ \ /__   \/__\/   \
         	 / /_\/ \////_\\ /  \/ / / /\/_\ / /\ /
         	/ /_\/ _  /  _  / /\  / / / //__/ /_// 
         	\____\/ \_\_/ \_\_\ \/  \/  \__/___,'  

--

https://sysadministrivia.com/news/RSYNC
─────────────────────────────────────────────────────────────────────

archive        	Audio, GPG signatures, and extras for all episodes
notes          	Static versions of all episode notes (updated nightly)
pocorgtfo      	A mirror of PoC||GTFO (https://www.pocogtfo.com/)
$ rsync --no-motd rsync://rsync.sysadministrivia.com/archive
drwxr-xr-x          4,096 2017/01/30 23:57:43 .
drwxr-xr-x          4,096 2017/01/25 20:21:21 S0
drwxr-xr-x          4,096 2017/01/25 20:24:55 S1
drwxr-xr-x          4,096 2017/02/25 18:35:57 S2
drwxrwxr-x          4,096 2017/02/02 08:28:13 extras
$ rsync --no-motd rsync://sysadministrivia.com/notes
drwxr-xr-x          4,096 2017/03/05 00:59:03 .
drwxr-xr-x          4,096 2017/03/05 00:59:28 html
drwxr-xr-x          4,096 2017/03/05 00:59:28 markdown
drwxr-xr-x          4,096 2017/03/05 00:59:28 pdf

(NOTE: you can use rsync with the --no-motd to skip the banner, but it won’t list the rsync modules – “shares” – if you do!)

How do I use it?

Episodes

If, for example, you’d want to download all of S1 (note that this includes the episodes in mp3 AND ogg format, as well as the GPG signatures), and save them locally to /srv/http/sysadmin_mirror, you’d do:

rsync -a rsync://rsync.sysadministrivia.com/archive/S1 /srv/http/sysadmin_mirror/.

or

rsync -a rsync.sysadministrivia.com::archive/S1 /srv/http/sysadmin_mirror/.

Shownotes

I also archive static copies of the shownotes in various formats- PDF, HTML, and Markdown. Note that they may be a little mangled from the live versions. They are freshly regenerated every night (morning?) at midnight UTC.

If you want all notes but only in Markdown format and saved to /srv/http/sysadmin_mirror, you’d do:

rsync -a rsync://rsync.sysadmininistrivia.com/notes/markdown /srv/http/sysadmin_mirror/.

or

rsync -a rsync.sysadmininistrivia.com::notes/markdown /srv/http/sysadmin_mirror/.

PoC||GTFO

If you want to simply grab all issues and download them to /srv/http/pocgtfo_mirror,

rsync -a rsync://rsync.sysadministrivia.com/pocgtfo/*.pdf /srv/http/pocgtfo_mirror/.

or

rsync -a rsync.sysadministrivia.com::pocgtfo/*.pdf /srv/http/pocgtfo_mirror/.

Author
Categories HOWTOs, Pondering

Comments

There are currently no comments on this article.

Comment...

Enter your comment below. Fields marked * are required. You must preview your comment before submitting it.