[Resolved] Degraded performance on the Patina Project app and patinaproject.com

Resolved - Oct 6, 5:57 UTC

This incident is resolved. The Patina Project app and patinaproject.com have worked normally since Oct 1 at 14:59 UTC. Our database now adds capacity on its own when activity goes up, which helps prevent a repeat. Your photos and submissions are safe, and no data was lost.

A small number of photos still don't load because of a problem with the file itself. For example, some HEIC photos are saved with a .jpg name, and some TIFF files are larger than 50 MB. We're tracking these as separate bug reports. If one of your photos still doesn't load, please create a bug report.

Monitoring - Oct 1, 15:23 UTC

Most services have recovered as of 14:59 UTC. Photos, including those submitted for the Thunderdome, are loading again. Your photos and submissions are safe, and no data has been lost. We are monitoring closely and will post another update once we are sure everything is stable.

Investigating - Oct 1, 14:24 UTC

We are investigating elevated activity on our production database. Database reported 97.5% CPU usage on a primary database node, which may cause delays when loading some photos. We are reviewing database queries and system capacity. Your photos and submissions remain safe, and we will provide another update when we know more.

Update - Oct 1, 14:17 UTC

We have shipped a fix and are checking the remaining affected photos. No submitted photos are stuck in processing. Some images still need review because certain JPEG uploads cannot be read, several large TIFF uploads may exceed processing limits, and we are investigating the remaining failures. Your photos and submissions remain safe.

Identified - Oct 1, 12:36 UTC

Since 02:45 UTC, some photos, including new contest submissions, may not load. Your photos and submissions are safe and no data has been lost. We found the cause and are working on a fix.