Follow

Lord, I had no idea running s3cmd (to set the visibility of all assets) and rclone (to send over all assets to the new bucket) would take so dang long. But we're real close to swapping to Spaces entirely, folks! :D

@somnius Great news! Yeah, bulk commands through s3cmd tend to be very slow, I understand your pain...

I think it's because you're bottlenecking file operations through making HTTPS requests using all sorts of APIs, though I'm not sure since I'm not a dev by trade (maybe you have more insight?).

Anyway, thanks again for the hard work! :D

@raelzero The main difficulty seems to be that running s3cmd recursively doesn't seem to also make it run on multiple threads—rclone at least does multithreaded operations. s3cmd takes so long because it's just one process running on all 500k of the files we have here and dang it takes a while.

I split the work a bit last night so that at least I ran multiple terminal instances of it, and that seemed to get things done way faster!

Sign in to participate in the conversation
Merveilles

Revel in the marvels of the universe. We are a collective of forward-thinking individuals who strive to better ourselves and our surroundings through constant creation. We express ourselves through music, art, games, and writing. We also put great value in play. A warm welcome to any like-minded people who feel these ideals resonate with them. Check out our Patreon to see our donations.