Starting at approximately 14:00 UTC on December 2, 2024, our team began receiving reports of severe performance degradation on customer devices. Upon investigation, we identified that the root cause stemmed from an inability for the Webroot agent to communicate with an essential backend service. This disconnect led to a substantial increase in traffic volume as agents repeatedly attempted to reach our cloud backend.
To mitigate the situation, we rapidly scaled our backend server capacity, which restored timely feedback loops between the agent and our systems. By doing so, we stabilized overall performance and brought the service back to normal operations.
In our post-incident analysis, we discovered that the excessive load correlated with the days immediately following holiday weekends. While minor spikes are not uncommon, the extended post-Thanksgiving weekend created a particularly large surge that exacerbated the performance degradation this time around.
Moving forward, our agent and cloud teams will proactively review backend call limits and resource allocations leading into holiday periods. Additionally, we will update our endpoint agent to be more resilient and responsive to changing backend conditions, ensuring smoother performance even during high-demand scenarios.
We remain committed to delivering top-tier endpoint protection and antimalware services. Thank you for your understanding and patience during this incident.
- The OpenText Webroot Product and Engineering team