@cancel In the few times I did it in the past I used wget --mirror with a few tweaked parameters for directory traversal and domain-spanning.
@cancel It can only follow HTML code, naturally, it'll follow all hyperlinks regardless of data type.
@mathr @blindcoder They're on the same domain. It looks like either wget requires the file extensions be added to a list of accepted file extensions, or also that robots.txt be ignored. I can do the latter, but I'm not sure how to do the former, because there are many varied file extensions I want to back up. Is there a way to wildcard it?
@csepp Yeah, I'm trying that now... I set to pretty conservative (1 connection, 1 request per second at most) and the site blocked me :/ really not good for preservation... why would you set your server up to do that
@cancel Oof. Well, I thiiink httrack is smart enough to retry them later, but in any case, that sucks.
@cancel Damn. Then I'm out of ideas. Archive Team's wget fork might have some kinda countermeasures, or swarm capability, or something? Maybe? Or if they don't block Tor, you could use that to get a fresh IP... idk.
@cancel Oh, in that case I think there are scripts for downloading from them and stripping the additional WBM frontend stuff. I wish they'd just let people download their WARCs though.
wgethas some nifty options for this: https://www.gnu.org/software/wget/manual/html_node/Advanced-Usage.html#Advanced-Usage
@cancel I have just used wget in the past. It's pretty simple to have it recursively download everything linked on a page. Don't remember the flags right now, but finding a guide would not be too difficult
@cancel ah i just saw the date -- it got tossed over my timeline a few mins ago so i thought it was recent aaa
@cancel Ugh, I've been wanting to download docs, references and guides (like CSS or the i3wm guide) in forever, but have yet to find a good way. All I got so far is a bunch of folders I can't use.
Revel in the marvels of the universe. We are a collective of forward-thinking individuals who strive to better ourselves and our surroundings through constant creation. We express ourselves through music, art, games, and writing. We also put great value in play. A warm welcome to any like-minded people who feel these ideals resonate with them.