Hrm. Woke up to almost ~20k enqueued sidekiq processes this morning because even though I made a specific process with 500 threads to handle the "default" queue, it seems to only run about 50 threads for it at a time. The "push" and "pull" queues don't have this problem, and use all 200 threads completely when needed.
Does anyone know why this is? #mastoadmin
One more update: I silenced the largest servers that we federate with so that reduces our overall traffic. Was hoping I didn't have to do that if possible, but we were still seeing some increased load even with all the adjustments I've made so far.
So! We'll keep going like this, and if people feel passionately about not silencing those instances we can try turning that off. For now, things seem to be working.
I'll give it one more day, and when things are fixed for sure, I'll update our server's documentation to include all the fixes I did to scale up with the influx of new users and instances to the fediverse. I know many other masto admins are experiencing similar issues and have all solved these in slightly different ways—hopefully my solution can help folks know what to do!
@somnius Does "silenced" mean basically we don't see posts from mastodon.social and similar scale instances (so mostly mstdn.* and mastodon.*) anymore? Don't get notifications about them? They show up, but deprioritized (would be a weird definition of silenced, but nothing surprises me in modern tech anymore...)
@klardotsh When an instance is "silenced", it means their posts don't show up by default in our Federated feed, but interactions between users are still possible between our instances. It applies an instance-wide "limit" to all their accounts.
So by default, Gargon's posts don't show up in any feeds until I follow him. Then I see his posts and can send him messages. More detailed explanation here: https://docs.joinmastodon.org/admin/moderation/
@klardotsh If you follow anyone from there, or if any of them follow folks from here, you'll still see their content, get notifications, and all that jazz, however :3
Revel in the marvels of the universe. We are a collective of forward-thinking individuals who strive to better ourselves and our surroundings through constant creation. We express ourselves through music, art, games, and writing. We also put great value in play. A warm welcome to any like-minded people who feel these ideals resonate with them.