We have continued to monitor LeMUR traffic and success rates have remained high with no further instances of elevated 5xx errors so this issue is being marked as resolved. Our team has identified a potential cause of these increased errors and we are working to confirm that as the root cause. Once we have conclusively determined the root cause we will follow up with more information in the form of a post-mortem update.
Posted Oct 31, 2023 - 23:17 UTC
Monitoring
We have seen a reduction in the number of 5xx errors for LeMUR. We will continue to work to investigate the root cause of the increased error rate but at this point, success rates for LeMUR have returned to the normal range and remained there for the last ~30 minutes.
We will continue to monitor traffic to ensure good performance and will follow up with more info once we have identified the root cause of the increase in errors.
Posted Oct 31, 2023 - 21:48 UTC
Investigating
We are currently investigating an increase in the number of 5xx errors from LeMUR. This is currently affecting ~10% of traffic, with the majority of jobs completing successfully. If you have retry logic in your process for handling failed requests those retries are likely to succeed. Our team is currently investigating the cause of the increased error rate and we will provide further updates as we learn more.