Hrm. Woke up to almost ~20k enqueued sidekiq processes this morning because even though I made a specific process with 500 threads to handle the "default" queue, it seems to only run about 50 threads for it at a time. The "push" and "pull" queues don't have this problem, and use all 200 threads completely when needed.

Does anyone know why this is?

In the meantime, I split the 1 process running the default queue into 3 processes, each with 100 threads. Seems to be running more threads overall when split up like this and our enqueued items are going down again!

Follow

One more update: I silenced the largest servers that we federate with so that reduces our overall traffic. Was hoping I didn't have to do that if possible, but we were still seeing some increased load even with all the adjustments I've made so far.

So! We'll keep going like this, and if people feel passionately about not silencing those instances we can try turning that off. For now, things seem to be working.

· · Web · 2 · 1 · 11

I'll give it one more day, and when things are fixed for sure, I'll update our server's documentation to include all the fixes I did to scale up with the influx of new users and instances to the fediverse. I know many other masto admins are experiencing similar issues and have all solved these in slightly different ways—hopefully my solution can help folks know what to do!

@somnius that's a great idea. Thank you so much for your work!

@somnius Does "silenced" mean basically we don't see posts from mastodon.social and similar scale instances (so mostly mstdn.* and mastodon.*) anymore? Don't get notifications about them? They show up, but deprioritized (would be a weird definition of silenced, but nothing surprises me in modern tech anymore...)

@klardotsh When an instance is "silenced", it means their posts don't show up by default in our Federated feed, but interactions between users are still possible between our instances. It applies an instance-wide "limit" to all their accounts.

So by default, Gargon's posts don't show up in any feeds until I follow him. Then I see his posts and can send him messages. More detailed explanation here: docs.joinmastodon.org/admin/mo

@klardotsh If you follow anyone from there, or if any of them follow folks from here, you'll still see their content, get notifications, and all that jazz, however :3

@somnius @klardotsh how does that interact with local users boosting toots from the silenced instance? Sorry if this has already been asked

@reynir @klardotsh Hm! It seems like I can still boost posts from Gargron and it shows up on my feed, lemme see if it shows up on another account

@klardotsh @reynir Yup, can see that post just fine from another account! So boosts will appear on other folks' timelines still :3

@somnius @klardotsh awesome! I've been hesitant with silencing because I don't want to break boosts.

Sign in to participate in the conversation
Merveilles

Revel in the marvels of the universe. We are a collective of forward-thinking individuals who strive to better ourselves and our surroundings through constant creation. We express ourselves through music, art, games, and writing. We also put great value in play. A warm welcome to any like-minded people who feel these ideals resonate with them.