Previous incidents

Rule evaluation requests were temporarily failing after underlying library code changed. We have reverted the version of said library and redeployed the previous version of the Rules API, which resolved the incident.

Between 9:34 am CDT and 9:46 am CDT, deployment of a dependent back-end service coincided with exceptionally high overall platform load, exposing an issue which resulted in a large number of 400 responses from the Rules API. The service degradation automatically resolved once the deployment completed, but Verosint engineering is aware of the issue, working on a permanent fix, and will take additional safety precautions during subsequent deployments of the dependent back-end service until the ...

SignalPrint log ingestion will be temporarily paused to facilitate reconfiguration of backend stream resources. The SignalPrint API will continue to function normally during the operation and data sent to the API will buffer until the streams are resumed. Dashboards will not display newly-ingested data until the maintenance is complete, but otherwise we anticipate no impact to service availability.

Following a standard application deployment, monitors for the Rules API and SignalPrint API backends notified Verosint engineering of significantly elevated error responses from those endpoints. Engineering staff determined that a faulty VCS repository configuration allowed the affected services configurations to be updated without deploying the application version for which said configuration was intended. Building and deploying an appropriate artifact restored both services, the associated ...

The API backend experienced a brief service interruption on 05 June 2023, beginning at 20:33 UTC and ending at 20:39 UTC. Root cause was determined to be a stale network configuration on a subset of backend cluster nodes which prevented new ingress service instances from initializing. Rolling replacement of the affected cluster nodes ensured the correct network configuration was applied and API service monitors recovered shortly after the first few nodes joined the cluster.