Previous incidents

SignalPrint log ingestion will be temporarily paused to facilitate reconfiguration of backend stream resources. The SignalPrint API will continue to function normally during the operation and data sent to the API will buffer until the streams are resumed. Dashboards will not display newly-ingested data until the maintenance is complete, but otherwise we anticipate no impact to service availability.

Following a standard application deployment, monitors for the Rules API and SignalPrint API backends notified Verosint engineering of significantly elevated error responses from those endpoints. Engineering staff determined that a faulty VCS repository configuration allowed the affected services configurations to be updated without deploying the application version for which said configuration was intended. Building and deploying an appropriate artifact restored both services, the associated ...

The API backend experienced a brief service interruption on 05 June 2023, beginning at 20:33 UTC and ending at 20:39 UTC. Root cause was determined to be a stale network configuration on a subset of backend cluster nodes which prevented new ingress service instances from initializing. Rolling replacement of the affected cluster nodes ensured the correct network configuration was applied and API service monitors recovered shortly after the first few nodes joined the cluster.

Upgrading the core network driver in our backend microservice K8S cluster. Minimal service disruption is anticipated during the rollout.