Hi all, we wanted to share some information about the downtime we had in late December.

Long story short, the hard drive for the main server’s disk filled up. By the time our infrastructure people were available to free space, we had another issue: pictrs, our image backend, was refusing to start.

Turns out that once pictrs detects it’s unable to start, it assumes it’s a good time to attempt a database upgrade. The drive still being full, this didn’t work, and it actually broke the images database.

To worsen things, this is when we discovered we didn’t have regular, automated image backups in place. The site was put together in a very short amount of time and this means that we have some infrastructural issues – this is one of them.

The last backup we had was from August 2021. This means that any image uploaded the site between August 3rd 2021 and December 28th 2021 are, unfortunately, lost.

We have since implemented a backup system for our images and the image database, and now take daily automated backups of them alongside the server database backups we already take. On that note, we identified a performance issue with our server’s database backup system that lead to reduced performance on the site as it was running, so we also pushed a change that means y’all should see less site instability from here on out.

As we continue to work on moving back to Lemmy, we hope to also continue to make improvements to Hexbear’s underlying infrastructure.

Thanks for reading all, and viva la Hexbear! :hexbear-logo: