code.ivysaur.me

nyaascrape

a scraper and ui for maintaining a mirror of NyaaTorrents.

Tags: anime, torrent, scraper

Scraper

The command-line tool can be used to build up an sqlite3 database of most information hosted on Nyaa. It updates an existing sqlite3 database or creates a new one. To bring your database up-to-date, the --update option will determine the most recent torrents available and set the download-chunk options appropriately. It would be suitable to run this via cron.

Usage: nyaascrape [options]
Options:
  -b, --batches {number}         Number of batches to download
  -c, --count {number}           Number of torrents to download per batch
  --database {filename}          Select output file
  --get-latest-id                Find the most recent Torrent ID on site
  --help                         Display this message
  --no-torrents                  Download entries only, no torrent files
  -s, --start {number}           Torrent ID to start processing from
  --sukebei                      Download from sukebei instead of www
  --test-file {file}             Test parse local file containing nyaa html
  -q                             Quiet, suppress output
  --update                       Automatically set -s/-c/-b options to get all
                                   the latest torrent entries
  --url {http://.../}            Set custom nyaa URL

Web Interface

The web interface allows browse, search, and torrent downloading using an interface modelled on the live NyaaTorrents site. However, the following issues make it a little unsuitable for public / high-traffic access:

• It shamelessly uses nyaa's own images and layout
• It works directly from the unindexed sqlite DB, so searches may be very slow against a full dump

Changelog

2014-10-05: r13

• Feature: Automatically set range parameters to bring database up-to-date (--update option)
• Feature: Set custom nyaa URL (--url option)
• Feature: Display most recent ID on site (--get-latest-id option)
• Sort usage-help alphabetically
• Remove site statistics from usage-help display
• ⬇️ nyaascrape_r13.7z (66.28 KiB)

2014-10-04: r6

• Initial public release
• ⬇️ nyaascrape_r6.7z (65.91 KiB)

See Also

• Magnyaa, similar project http://root.suumitsu.eu/wiki/doku.php?id=php:magnyaa
• Fork with MySQL support by "odangomoe": https://github.com/odangomoe/nyaascrape/ ; changes are listed at https://github.com/odangomoe/nyaascrape/compare/ce7ff4e5a...6a2e49d/