Releasing tget

Hey there lads! 👋

I just released Torget (a.k.a. tget). What is it, you may ask? Well, it’s a, erm… a Tor http downloader… ASTONISHING, I know. As always I was supposed to release this months ago, but got busy with other things…

But wait! tget is not just your typical downloader, it comes with a tiny twist! Instead of allowing downloads of multiple files over one Tor connection, to speed things up, it establishes multiple Tor circuits and assigns the downloads to each instance.

Why? Because while using Tor is super easy, it is also notoriously slow, as the theorical speed is based on the slowest node in the circuit. In practice, this means that most Tor connections fluctuate between a few Kb/s to 1Mb/s.

The idea behind it, is simple: let’s spawn more connections, so we have more (theoretical) bandwidth! This way we can download more leaks, moar quicker.

And, I mean, it kind of works. While downloading many files over Tor, using tget results in being quite faster compared to a single circuit with parallel downloads. To be totally fair, this isn’t a novel technique or anything, but I struggled to find a tool to allow me to do the above easily, so tget was born!

Here is a quick and lazy overview showcasing its current features:

Torget is a Tor aware file downloader which uses multiple Tor instances to try to use all available bandwidth.
Made by thatsn0tmysite (aka n0tme) | Blog: https://thatsn0tmy.site

Usage:
tget [flags] <url|file> [...url|file]

Flags:
--concurrency int concurrency level (default 10)
-c, --conf string .torrc template file to use
--continue attempt to continue a previously interrupted download
-b, --cookies string cookie(s) to include in all requests
-d, --data string body of request to send
-f, --follow-redirect follow HTTP redirects
-F, --from-file download from files instead of urls
-H, --header strings header(s) to include in all requests
-h, --help help for tget
--host string host running Tor (default "127.0.0.1")
-n, --instances int number of Tor instances to use (default 5)
--keep-alive do not close Tor instances when done
-l, --log-path string path to save logs at
-X, --method string HTTP method to use (default "GET")
-o, --out-path string path to save downloaded files in (default "/home/user")
-O, --ovewrite overwrite file(s) if they already exist
-p, --ports uints ports to for Tor to listen on (default [])
-R, --reuse-instances do not spawn new instances, assume they are already open (implies --keep-alive)
-S, --socks-version string socks version to use (default "socks5")
--test-domain string website to use while testing if Tor is up (default "https://thatsn0tmy.site")
-T, --timeout int max time to wait for Tor before canceling (0: no timeout)
-t, --tor-path string path to Tor binary (default "/usr/bin/tor")
-k, --unsafe-tls skip TLS certificates validation
-U, --useragent string useraget to use when sending requests (default "tget/v0.3")
-v, --verbose be (very) verbose

Benchmarks

To showcase the results i prepared this totally accurate benchmark, acquired by forcing my friends to run tget over and over until data was in my favor (like the best researchers):

The downloaded files for the tests were:

https://ash-speed.hetzner.com/100MB.bin
https://proof.ovh.net/files/10Mb.dat
https://link.testfile.org/60MB
https://www.torproject.org/dist/torbrowser/13.0.16/tor-browser-windows-x86_64-portable-13.0.16.exe
https://archive.org/download/theultimatedoom_doom2_doom.wad/Doom%202.iso

All the poor souls that helped me test performed the download with the following, very complex and sophisticated, script for non Tor:

time bash -c 'cat urls.txt | parallel -j 5 wget {} && exit'

And ran the following to download the same files with tget, assuming 5 circuits are already running (to avoid counting the Tor instances bootstrap time):

tget -F urls.txt --ports 9050,9060,9070,9080,9090 -R -o /tmp

Anyway, based on the very accurate dataset we can observe an average of 16.89% speedup in the total download time. I think this is quite nice!

NOTE keep in mind that between tests circuits might change and may produce different results, depending on the bandwidth of each one.

That was it, feel free to take tget for a ride, test it, break it and contribute, code is as usual ugly as hell and on github. Go get them dataleaks pirates, 🏴‍☠️ arghhh!