Increase in processing times for Async transcription
Incident Report for AssemblyAI
Postmortem

We wanted to reach back out to share more detailed information on the incidents that occurred on 8/7 and 8/8. These incidents were caused by separate issues See the information below for a description of each issue and the steps taken to remedy them.

8/7
Incident Cause
An inefficient database usage pattern change was submitted and deployed on 8/2. Although inefficient, due to the standard load prior to deployment, no regression was detected. We encountered a new peak load on 8/7 which along with this inefficiency led to a large increase in latency (turnaround times) which was the incident faced this day.

Resolution
We identified and reverted the database usage change committed on 8/2 that led to this slowdown.
We upgraded our database instance size.

8/8
Incident Cause
A full table query was run against our write replica database as a team worked to transfer data to BigQuery for business intelligence tooling. This led to database contention and slowed down our production service.

Resolution
We implemented more fine-grained controls and roles for database access along with an approval process to verify production database queries are run against the correct replica and will not impact customers.

If you have any questions about this information feel free to reach out to support@assemblyai.com.

Posted Aug 20, 2023 - 23:32 UTC

Resolved
Processing times have remained in the normal range and this incident is being transitioned to resolved.
Posted Aug 08, 2023 - 21:16 UTC
Monitoring
Processing times for our Async endpoint have returned to the normal range and remained there for the last ~15 minutes. We will be monitoring traffic to ensure continued good performance going forward.
Posted Aug 08, 2023 - 20:18 UTC
Investigating
We are currently seeing an increase in processing times for our Async API. Our Engineering team is actively investigating the issue and we will post additional updates as we learn more.
Posted Aug 08, 2023 - 19:56 UTC
This incident affected: APIs (Asynchronous API).