Hey folks
This is a heads up that I will be performing some maintenance and hardware upgrades on our database this Saturday.
We are currently experiencing several spikes throughout the day which cause our database to become overloaded - this results in degraded performance for many users. The spikes are happening due to a combination of continued growth of the database, some expensive periodic scheduled tasks which Lemmy runs, and fluctuating traffic patterns. Some of this can be optimized on the code level in the future, but it seems that the best way to deal with it right now is to add some additional resources to our database server.
I am intending to switch to slightly different hardware in this upgrade, and will be unable to make this switch without downtime, so unfortunately lemm.ee will be unavailable for the duration.
As our database has grown quite a bit, cloning it will most likely take a few hours, so I expect the downtime to last 2-3 hours. Sorry for the inconvenience, I am hopeful that it will be worth it and that this upgrade will significantly reduce some of our recent long page load times!
Edit: upgrade complete!
I have now migrated the lemm.ee database from the original DigitalOcean managed database service to a dedicated server on Hetzner.
As part of this migration, I have also moved all of our Lemmy servers from the DigitalOcean cloud to Hetzner’s Cloud. I always want the servers to be as close as possible to the database, in order to keep latencies low. At the same time, I am very interested in having the ability to dynamically spin up and down servers as needed, so a cloud-type solution is really ideal for that. Fortunately, Hetzner allows connecting cloud servers to their dedicated servers through a private network, so we are able to take advantage of a powerful dedicated server for the database, while retaining the flexibility of the cloud approach for the rest of our servers. I’m really happy with the solution now.
In terms of results, I am already seeing far better page load times and far less resource use on the new hardware, so I think the migration has been a success. I will keep monitoring things and tuning as necessary.
Thank you for your work on lemm.ee! I was actually looking to post somewhere to see if there was some slowness or if it was just me, looks like you answered my question.
No worries about the downtime. If that’s what’s needed to get back to 0.18 levels of performance, go for it! Thanks for all the work y’all do.
Tänan teid selle hea kogukonna eest
Aye, hope ye do well…
In other words, lemm.ee will be down starting at UTC 2024-01-06T10:00 for 2+ hours. Convert to your local time via https://www.utctime.net/utc-time-zone-converter if you want.
Thank you sunaurus, for the heads up and for the ongoing maintenance you do!
I prefer using timeanddate.com since it has a fixed time URL.
So I wasn’t going crazy, the long loads times were real. Glad to see upgrades coming soon!
Although the long page load times have been real, you still might be going crazy. Definitely a possibility on Lemmy (づ￣ ³￣)づ
My experience has been downgraded since the latest Lemmy update, all these caveats/workarounds are pretty much welcome in order to get this in a stable state again!
Going to PostgreSQL 16?
Yes!
This is complete, correct? I believe the site feels significantly faster. Have you ever thought about doing a community with detailed write-ups of your trials and tribulations? It will probably be a good resource to others setting up their own federation. I know I’d read the shit out of it.
Yep, it’s complete!
I try to write summaries of updates whenever we have them in this community (I edited the OP here as well, btw). In general I’ve also written some Lemmy documentation about scaling, but I think there are actually very few instances which have similar problems to lemm.ee, as the vast majority of instances are far smaller. So probably my experiences are not super useful for most people 😃
Tänan teid selle hea kogukonna eest
What are the current specs and what are you going to?
He won’t tell
Ah thought he had the old specs posted
I’m pretty sure I remember seeing the specs when he moved the server behind Cloudflare. It was put behind their CDN because we kept getting DDoS attacks. It’s fairly nice if I recall correctly.
I thought so as well. I thought he had an architectural diagram but I can’t find them. So I must be wrong
You’re not necessarily wrong. You just can’t find it at the moment. I have trouble finding information all the time and that’s a big chunk of my job.
Currently the database is a managed DigitalOcean Postgres instance, but I am going to migrate it to a 32 thread 128 gb RAM dedicated server.
It’s something that I’ve been hoping we won’t need, as the managed database service has allowed me to not worry about patches, backups, etc (they took care of all of that automatically). Unfortunately it is clear now that further upgrades are just too costly on that service & actually the amount of configuration and tuning I would like to do is simply not possible there. So seems like right now, moving to the dedicated server for the database is the only option.
Makes sense. I appreciate the reply back. I use to work on highly available fault tolerant systems. So I’m always interested how much hardware/etc to run something.
Sounds like some serious optimization needs to be done or there are a lot more transaction than I was expecting.
Thanks for the heads up and for the upgrade! Good luck
What hardware? Why is the server admin keeping everything secret?
Because he has no obligation at all not to
Yeah, he has no obligation to post changes or even maintain this server as well. Would it mean he cares for the community? No. This admin sucks balls and you can’t change my mind.
He’s keeping it secret probably because our server looks like this:
It literally doesn’t matter if people know or not. As long as things run smoothly, it’s fine.
It literally doesn’t matter if people know or not. As long as things run smoothly, it’s fine.
Except it doesn’t
How did you get access to my webcam?
lol! damn hacker kids!
Just rooty toot your ass on to another instance then. Or spin up your own.
Having a good stable server is seen as luxury today smh.
Yes, things like Lemmy are literally a luxury.
Meta is luxury too 😂
As stated, spin up your own, show us how it’s done. You get no compensation, and you better fucking get it right.
Is this a threat? I’ve been administrating several servers in my free time over the past few years. You know nothing kid.
Yeah it’s a threat, make your own flawless instance or I’ll make fun of you on the internet.
Pft
Thanks for the update and information. You really set an example for communication and level headedness in decision making. Thanks for providing this space for us.
It’s flying now! Thanks for the upgrade @sunaurus@lemm.ee.
Np, thanks so much for all you do!
Nice, I’ll still be asleep then and the update should be done by the time I wake up. Good luck!
