Hrm. Woke up to almost ~20k enqueued sidekiq processes this morning because even though I made a specific process with 500 threads to handle the "default" queue, it seems to only run about 50 threads for it at a time. The "push" and "pull" queues don't have this problem, and use all 200 threads completely when needed.

Does anyone know why this is?

In the meantime, I split the 1 process running the default queue into 3 processes, each with 100 threads. Seems to be running more threads overall when split up like this and our enqueued items are going down again!

One more update: I silenced the largest servers that we federate with so that reduces our overall traffic. Was hoping I didn't have to do that if possible, but we were still seeing some increased load even with all the adjustments I've made so far.

So! We'll keep going like this, and if people feel passionately about not silencing those instances we can try turning that off. For now, things seem to be working.

Follow

I'll give it one more day, and when things are fixed for sure, I'll update our server's documentation to include all the fixes I did to scale up with the influx of new users and instances to the fediverse. I know many other masto admins are experiencing similar issues and have all solved these in slightly different ways—hopefully my solution can help folks know what to do!

· · Web · 1 · 2 · 9

@somnius that's a great idea. Thank you so much for your work!

Sign in to participate in the conversation
Merveilles

Revel in the marvels of the universe. We are a collective of forward-thinking individuals who strive to better ourselves and our surroundings through constant creation. We express ourselves through music, art, games, and writing. We also put great value in play. A warm welcome to any like-minded people who feel these ideals resonate with them.