On the September 13th Connection Issues
Yesterday Telegram experienced something extraordinary. At 17:45 UTC there was an immense spike in user activity on Telegram that exceeded our peak load by 5 times.
Telegram is different from most other internet services in that we use a distributed server infrastructure, so peak loads cannot make the whole of Telegram go down. However, if one of the Telegram clusters is severely affected, a part of our users can experience difficulties sending and receiving messages.
This is exactly what happened yesterday. Due to the spike, one of the Telegram server clusters went down in part and approximately 15% of the users who were online at the time experienced connection issues from 17:45 to 18:10. The issues were partially fixed at 18:10, but about 11% of online users could still face slow or no performance until 19:00 when the problem was permanently fixed.
The regions that were most affected are Germany, Iraq and the CIS – most notably, Uzbekistan, Russia, Ukraine, Kazakhstan and Belarus. We briefly commented on the situation via Twitter, but I’d like to tell more extensively about it here.
1. First of all, we are extremely sorry for the inconvenience this has caused. The cluster that went down hasn’t had any issues for a few years now, so I can imagine the shock of the users affected by yesterday’s downtime. We understand that you use Telegram constantly for work and leisure, and that we must be online 100% of the time. We take yesterday’s issues very seriously.
2. Secondly, to make sure this doesn’t happen again, we identified the cause of the problem and have eliminated the bottlenecks in our infrastructure that caused the downtime. We think that x5 peak load spikes won’t be a problem any longer. We've also set up a plan to be able to cope with x20 load spikes (however unlikely the x20 scenario may seem) before the end of 2017.
While unfortunately no one can completely rule out the chance of another downtime some time in the future (there’s always a slight probability of some unexpected Black Swan event), I can assure you that we at Telegram are working hard at taking this chance to the lowest in the industry.
We strive to be the winner in every aspect including availability. So far the Telegram uptime stats look good compared to other major communication services – also because Telegram can never go down entirely around the world. But we are not satisfied with just that, and our work continues.
Thanks for your attention, and again – apologizes to anyone who was affected.
Yesterday Telegram experienced something extraordinary. At 17:45 UTC there was an immense spike in user activity on Telegram that exceeded our peak load by 5 times.
Telegram is different from most other internet services in that we use a distributed server infrastructure, so peak loads cannot make the whole of Telegram go down. However, if one of the Telegram clusters is severely affected, a part of our users can experience difficulties sending and receiving messages.
This is exactly what happened yesterday. Due to the spike, one of the Telegram server clusters went down in part and approximately 15% of the users who were online at the time experienced connection issues from 17:45 to 18:10. The issues were partially fixed at 18:10, but about 11% of online users could still face slow or no performance until 19:00 when the problem was permanently fixed.
The regions that were most affected are Germany, Iraq and the CIS – most notably, Uzbekistan, Russia, Ukraine, Kazakhstan and Belarus. We briefly commented on the situation via Twitter, but I’d like to tell more extensively about it here.
1. First of all, we are extremely sorry for the inconvenience this has caused. The cluster that went down hasn’t had any issues for a few years now, so I can imagine the shock of the users affected by yesterday’s downtime. We understand that you use Telegram constantly for work and leisure, and that we must be online 100% of the time. We take yesterday’s issues very seriously.
2. Secondly, to make sure this doesn’t happen again, we identified the cause of the problem and have eliminated the bottlenecks in our infrastructure that caused the downtime. We think that x5 peak load spikes won’t be a problem any longer. We've also set up a plan to be able to cope with x20 load spikes (however unlikely the x20 scenario may seem) before the end of 2017.
While unfortunately no one can completely rule out the chance of another downtime some time in the future (there’s always a slight probability of some unexpected Black Swan event), I can assure you that we at Telegram are working hard at taking this chance to the lowest in the industry.
We strive to be the winner in every aspect including availability. So far the Telegram uptime stats look good compared to other major communication services – also because Telegram can never go down entirely around the world. But we are not satisfied with just that, and our work continues.
Thanks for your attention, and again – apologizes to anyone who was affected.