webapp and ingestion failures
Incident Report for Lightstep
Resolved
This incident has been resolved.
Posted Jul 21, 2021 - 13:14 PDT
Update
We are continuing to monitor for any further issues.
Posted Jul 21, 2021 - 13:14 PDT
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Jul 21, 2021 - 11:07 PDT
Update
We have identified that the remaining trace assembly issues are limited to microsatellite customers. We have implemented a fix and are monitoring.
Posted Jul 21, 2021 - 11:07 PDT
Identified
We have implemented a fix for the initial problem and are working to restore our system to its original state. Data loss will have occured for both span and metric ingestion for the period of the incident (9:55AM - 10:25AM PST).

There are some remaining issues with trace assembly that we are actively investigating.
Posted Jul 21, 2021 - 10:51 PDT
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Jul 21, 2021 - 10:25 PDT
Identified
The issue has been identified and a fix is being implemented.
Posted Jul 21, 2021 - 10:21 PDT
Investigating
We are currently investigating an issue affecting both webapp usage and span and metric ingestion.
Posted Jul 21, 2021 - 10:10 PDT
This incident affected: Data Ingestion and Aggregation (Trace Assembly, Statistics), Web UI (Explorer, Dashboards, Service Directory, Root Cause Analysis, Login), Alerts, API, and Third-Party Notification Methods (PagerDuty Events API, Slack Apps/Integrations, Slack Connections, Slack Messaging).