Overview

Due to incorrect deployment of the web client's parameters, the URL to our authentication broker pointed to the wrong region. Our deployment tools that pre-populate the parameters to all components configured our system with the incorrect value. This resulted in timed-out authentication requests which prevented users from logging in to the platform and agent.

Logged-in users with a valid authentication token weren’t affected by this issue and were able to log in and work as usual.

‌

Detection

The Perimeter 81 Customer Success team received several reports from different users and escalated the issue to the R&D team as a critical issue.

‌

Root Cause Analysis

The team pushed some new configurations and environment variables to the Perimeter 81 repository for the upcoming releases. An automatic action triggered an old process to deploy our Front End version with the new variables. This resulted in an old deprecated module replacing a new module in production. The old module had an incorrect routing to the authentication broker which triggered the issue.

Since the deployment was not planned, we didn’t run the post-deployment regression testing and therefore the old module and wrong routing were not detected by the team.

‌

Resolution

The correct environment variables were pushed back to the production system and automatically deployed to all relevant components.