webapp and ingestion failures

Incident Report for ServiceNow Cloud Observability

Resolved

This incident has been resolved.

Posted Jul 21, 2021 - 13:14 PDT

Update

We are continuing to monitor for any further issues.

Posted Jul 21, 2021 - 13:14 PDT

Monitoring

A fix has been implemented and we are monitoring the results.

Posted Jul 21, 2021 - 11:07 PDT

Update

We have identified that the remaining trace assembly issues are limited to microsatellite customers. We have implemented a fix and are monitoring.

Posted Jul 21, 2021 - 11:07 PDT

Identified

We have implemented a fix for the initial problem and are working to restore our system to its original state. Data loss will have occured for both span and metric ingestion for the period of the incident (9:55AM - 10:25AM PST).

There are some remaining issues with trace assembly that we are actively investigating.

Posted Jul 21, 2021 - 10:51 PDT

Monitoring

A fix has been implemented and we are monitoring the results.

Posted Jul 21, 2021 - 10:25 PDT

Identified

The issue has been identified and a fix is being implemented.

Posted Jul 21, 2021 - 10:21 PDT

Investigating

We are currently investigating an issue affecting both webapp usage and span and metric ingestion.

Posted Jul 21, 2021 - 10:10 PDT

This incident affected: Observability - Data Ingestion (Trace Assembly, Trace Statistics), Observability - Third-Party Notification Methods (PagerDuty Events API, Slack Apps/Integrations, Slack Connections, Slack Messaging), Observability - Web UI (Service Directory, Dashboards, Change Intelligence, Explorer), Observability - Alerting (Streams Alerting), and Observability - API.