Thanks for the response, that's good to know.

Yeah I'm interested only in the `.ass` files. A one-off of the `.xz` files would help a lot, but if increasing the scraping rate is not an issue I think I should be able to manage by myself without bothering you, especially since the worst case is still "only" 138GB even when counting files that are not subtitles.

When I start scraping I'll include the repo link and contact info in the User-Agent header just in case.

Also, once I have the incremental downloads working I'd like to upload to Nyaa from time to time, in case someone else also wants to use this data.

I didn't notice anything unusual around the time you posted this - would you be able to clarify further, for example did new entries appear on the home page but no links were generated?

seems like the site stopped uploading new entries

I don't mind scraping anything here, as long as you're not too aggressive with it.
One request per minute sounds very conservative - going faster shouldn't be an issue.

If you're going after attachments, I suggest targeting the .xz files, as there won't be duplication there (the .7z packages are just the .xz files joined together).

I don't know if you're after the subtitle text only, or the fonts/chapters etc as well.
I could create a one-off batch of the .xz files. You'd obviously have to deal with updates subsequent to that.
At time of writing, all .xz files total 138GB. If you only want subtitle text, the size would be much less than that.

I have a question. There's already a page with AnimeTosho database dumps, and it has a lot of useful metadata.

But if I want to make a search engine (open source) for anime subtitles, am I allowed to use the subtitle attachment files from Anime Tosho?

While you make it easy to download the subtitle files, I'm reluctant to just start doing it because that would be nasty on the server bandwidth. Even if I limit downloads to something like 1 request per minute, it would probably take several months at least (I'm just eyeballing here, haven't even done the math), so incremental updates would probably only be realistic after an initial bulk import.

So that's why I'm not sure if this use case is allowed at all. And I'm asking here because I couldn't find the answer about this in the FAQ or the dbexport page.

If it *is* allowed, is there a better way to do an initial bulk download of the attachment files that doesn't put too much strain on the Anime Tosho servers? E.g. a torrent, an Internet Archive link, etc (I couldn't find either with a quick search).

If bulk downloads like this are *not* allowed, would it be possible to mention that in the FAQ or the dbexport page?

Thanks.

Nah, you're fine.

No, but I prefer anonymous uploads.

I personally don't really believe in "unlimited storage", which I find typically means "an undisclosed storage limit" (same with "never deleted" promises).