As COVID-19 continues to unfold, forcing staff to earn a living from home, the companies of ISPs, cloud suppliers and conferencing companies a.okay.a. unified communications as a service (UCaaS) suppliers are experiencing elevated visitors.
ThousandEyes is monitoring how these will increase have an effect on outages and the efficiency challenges these suppliers bear. It is going to present Community World a roundup of attention-grabbing occasions of the week within the supply of those companies, and Community World will present a abstract right here. Cease again subsequent week for one more replace, and see extra particulars right here.
Replace Oct. 12
Globally, the variety of outages noticed in all three classes elevated by 12% vs. the week earlier than, from 233 to 261. Within the US they elevated 15%, from 111 to 128.
ISP outages worldwide elevated 18%, from 168 to 199. Within the US they rose 15%, from 96 to 110.
The variety of outages in public cloud networks globally dropped from 28 to 23, a lower of 18%. Within the US they stayed regular at 4.
In whole there have been two collaboration app outages, each within the US.
A notable outage for the week began Oct. 5 about 6 a.m. PDT and affected the collaboration app Slack. ThousandEyes assessments returned 503 server errors, indicating the service was unavailable, in addition to timeouts, suggesting that the appliance was operating slower than regular. These issues had been intermittent. No community points had been noticed connecting to Slack’s edge servers, that are hosted inside AWS. Slack confirmed points inside their backend techniques and that they had been resolved at 10 p.m. PDT. Click on right here for an interactive view of the outage.
Replace Oct. 5
Globally, the entire variety of outages noticed throughout all three classes elevated by 21% from the week earlier than, from 193 to 233. The rise was mirrored within the US, the place outages rose from 84 to 111, a 32% enhance.
The variety of ISP outages worldwide elevated by 15%, rising from 146 to 168, accounting for 72% of all outages noticed. Within the US, the quantity rose from 72 to 96, a 33% enhance.
Globally, cloud supplier outages greater than doubled from 11 to 28, a 155% enhance. Within the US, the quantity rose from one to 4.
For the primary time in three weeks there have been no collaboration-app-network outages noticed globally. The week earlier than there have been two.
A notable outage occurred about Three a.m. EDT on Sept. 30 when Cogent, a US primarily based multinational transit service supplier, skilled a service disruption that affected customers world wide trying to entry Microsoft, Amazon, Fb, and Google companies. The outage lasted 41 minutes unfold over three hours and affected a number of elements of Cogent’s US community. The timing and sample of the outage point out traffic-engineering exercise because the trigger. The service was restored about 5:50 a.m. EDT. Click on right here for an interactive view of the outage.
Replace Sept. 28
The variety of outages noticed worldwide in all three classes decreased by 16% from the week prior, from 230 to 193. Within the U.S., the variety of outages elevated by 11, a 15% enhance.
International ISP outages dropped from 175 to 146, down 17%. However within the U.S., outages rose from 63 to 72, a rise of 14%.
Public cloud outages worldwide elevated from eight to 11, up 38%. U.S. public cloud outages remained steady at one.
Collaboration app community outages jumped 300% globally from one to 4. Most of that was because of the 300% enhance within the U.S. from one to 3.
Google suffered a notable disruption about 9 p.m. EDT Sept. 24 that prevented many customers world wide from accessing companies together with Gmail, YouTube, Google Calendarand Google Meet. Entrance-end servers remained reachable in the course of the outage, however requests to entry companies returned obtain errors. Google confirmed {that a} pool of servers that dealt with utility visitors on the backend had crashed. Service was restored about 9:30 p.m. EDT. Click on right here for an interactive view of the outage
Replace Sept. 21
The variety of outages reported globally in all three classes was 230 for the week Sept. 14-20, up 50% from 153 the week earlier than. Within the U.S., the rely was 73 for the most recent week, up two from the week earlier than.
ISP outages rose 62%, from 108 to 175 worldwide, and from 57 to 63 within the U.S., and increse of 11%.
Public-cloud supplier outages had been down a 3rd, from 12 to eight, with the rely within the U.S. dropping from 5 to only one.
Collaboration-app community suppliers suffered a single outage this week, with that one occurring within the U.S. The week earlier than, there have been none.
Instagram and Amazon suffered notable outages in the course of the week.
About 11:10 a.m. PDT on Sept. 17 Instagram skilled a service disruption that prevented many customers worldwide from utilizing the appliance. With no community or reachability points with its front-end servers, and customers receiving HTTP 502 error notifications, the trigger seemed to be anapplication back-end subject. Service started to return about 11:15 a.m. PDT, wth full service restored by 11:45 a.m. PDT. Click on right here for an interactive view of the outage.
About 2:45 p.m. EDT Sept. 14 Amazon suffered a 29-minute outage centered on nodes in Columbus, Ohio, and affecting Amazon cloud-compute situations at its Hilliard, Ohio, knowledge middle. The outage affected 99 interfaces and was contained to the one location. The impression was that some customers experiencing non-responsive or sluggish EC2 situations. The outage was cleared simply previous Three p.m. EDT.
Replace Sept. 14
Globally the variety of outages noticed between Sept. 7 and 13 in all three classes decreased by 40% from the week earlier than, from 256 to 153, the bottom determine noticed since early February. Within the U.S., the variety of outages dropped from 134 to 71, a 47% lower.
The variety of ISP outages worldwide dropped 50% from 216 to 108. Within the lower was 54% from 123 to 57, the bottom weekly quantity since early April.
Cloud-provider outages globaly decreased by 25% from 16 to 12. Within the U.S. they remained at 5 for the second consecutive week.
Globally and within the U.S. for the primary week since early August, no collaboration app community outages had been recorded.
Cogent Communications suffered three outages about 11:45 a.m. EDT on Sept. 11, lasting 36 minutes. They three lasted 13 minutes, Four minutes and 19 minutes, unfold throughout simply over an hour. All three centered on Cogent node in Newark, N.J., and affected clients throughout the U.S. and likewise within the U.Okay., Netherlands, Canada, Mexico and India. The shoppers had been utilizing the community to entry companies equivalent to Visa on-line companies, Microsoft workplace, and Shopify. The outages occurring throughout enterprise hours and their focus point out that some type of control-plane situation was the trigger. The issue was cleared about 12:50 p.m. EDT.
Replace Sept. 7
Globally the variety of outages noticed in all three classes decreased by 33% from the week prior, from 381 to 256. Within the U.S., the variety of outages dropped by 46, lowering from 180 to 134, a 26% lower from the week prior.
Worldwide, the variety of ISP outages decreased by 103, dropping from 319 to 216, a 32% lower and accounting for 84% of all outages noticed this week. Within the U.S., the variety of ISP outages decreased by 27, dropping from 150 to 123, an 18% lower.
Cloud supplier outages globally elevated by a 3rd from 12 to 16. Within the U.S., outages greater than doubled, rising from two to 5.
There was only one collaboration app supplier outage worldwide — not within the U.S. — down from two.
PCCW International suffered two outages beginning about 12:40 a.m. EDT Sept. 3, one lasting 20 minutes, and the opposite lasted six. The primary centered on PCCW nodes positioned in Atlanta, Ga., and affecting companies utilizing the Charlotte Colocation and Affiliated Laptop Providers networks. The second began about half an hour after the primary cleared and centered on PCCW nodes in Ashburn, Va., and affected entry to Oracle Cloud companies. All outages had been cleared by 1:45 a.m. EDT. The trigger was doubtless the results of a traffic-engineering train.
About 6 p.m. PDT, Comcast suffered a four-minute outage that affected customers within the western U.S. centered on Comcast core units in Sunnyvale, Calif. and primarily affecting companies throughout Comcast Xfinity networks (Comcast Cable Communications). The outage would doubtless have prompted web connectivity slowdowns and disruption for customers.
Replace Aug. 31
Globally the variety of outages noticed throughout all three classes elevated by 29% from the week prior, rising from 296 to 381. This was the most important variety of outages recorded in a single week this 12 months. Within the U.S. outages elevated 70% in comparison with the week prior from 106 to 180.
The overwhelming majority of outages had been as a result of ISP issues. Worldwide the quantity jumped from 214 to 319, with the rely within the U.S. rising from 80 to 150.
Public cloud outages declined worldwide fom 27 to 12 and from 4 to 2 within the U.S.
Collaboration apps networks stayed stead at two worldwide, with each of them occurring within the U.S. the place the rely rose from zero to 2.
CenturyLink suffered a significant outage simply after 6 a.m. EDT Aug. 30 that hit a broadrange of suppliers and companies together with Twitter, Microsoft (Xbox Reside), Discord, Reddit, Cloudflare, OpenDNS, and Hulu. Shortly after the outage started, suppliers began rerouting visitors from CenturyLink to alternate suppliers in an effort to alleviate the impression, nonetheless, given the dimensions and distribution of CenturyLink’s community, many companies had been nonetheless unreachable, ThousandEyes mentioned. At 8:13 a.m. EDT, CenturyLink introduced it was investigating points affecting some companies inside their Mississauga, Ontario, Canada knowledge middle. Having recognized the trigger as an incorrect flowspec announcement from the Mississauga knowledge middle, CenturyLink requested that its Tier 1 Web supplier companions de-peer and ignore any visitors coming from its community. (BGP movement specification (flowspec) is a characteristic that means that you can quickly deploy and propagate filter insurance policies amongst numerous BGP peer routers.) With a purpose to resolve the difficulty, CenturyLink reset all of the gear and begin with clear BGP routing tables, a course of that took nearly 5 hours to finish. Simply earlier than 3:00 p.m. EDT, CenturyLink introduced that the difficulty had been resolved and all companies had been restored.
Replace Aug. 24
Globally the entire variety of outages noticed throughout all three classes in the course of the week Aug. 17-23 elevated by 21% in comparison with the week prior, rising from 245 to 296. This enhance within the U.S. rose from 90 to 106 a rise of 18% from the week prior.
ISP outages worldwide rose from 166 to 214 and from 72 to 80 within the U.S.
Public cloud community outages dropped worldwide from 28 to 27, and stayed the identical within the U.S. at 4.
Collaboration app community outages rose from zero to 2 globally, however remained at zero within the U.S.
ThousandEyes flagged three notable outages in the course of the week.
Simply after Eight a.m. EDT on Aug. 18, Spotify suffered an outage that prevented customers from streaming songs from the service. The outage lasted simply over an hour and would play songs for a couple of seconds, then pause and return an error. The outage is believed to be assosicated with an expired TLS certificates. Click on right here for a proof on the impression of certificates expiration.
About 11:30 p.m. EDT on Aug. 17, Equinix suffered an influence outage to a colocation middle in Docklands, London. About 2 a.m. the failure of an output static change from a UPS system triggered a hearth alarm, leading to lack of energy for a number of clients. At 3:50 a.m. companies began to be restored and had been absolutely restored by 4:50 p.m. EDT. Affected clients included BT, Sky, Virgin Media, Giganet, Epsilon, SiPalto, EX Networks, Fast2Host, ICUK.web, and Evoke Telecom.
About 10:50 p.m. PDT on Aug. 19 Cogent Networks suffered a 36-minute outage affecting U.S. customers’ entry to Microsoft networks and related companies, in addition to CDN content material for companies equivalent to TikTok and ESPN. The outage affected nodes throughout the U.S. and apparently resulted from a configuration adjustment. A second outage two hours later at 11:26 p.m. PDT lasted 24 minutes and certain was related to the primary outage’s configuration adjustment. It affected customers within the U.S., Asia-Pacific and Europe, Mid-East and Africa. Click on right here for an interactive view of the outages.
Update Aug. 17
International outages throughout all three classes fell between the weeks of Aug. 3-9 and Aug. 10-16 from 294 to 245 (-17%) and within the U.S. from 123 to 90 (-27%).
ISP outages dropped worldwide from 227 to 166 and from 109 to 72 within the U.S.
Public cloud outages worldwide fell from 30 to 28 and from 5 to 4 within the U.S.
Collaboration app community outages worldwide remained at zero for the second week in a row.
Cogent Networks suffered a notable outage at about 10:30 p.m. EDT on Aug. 13 that lasted about 40 minutes and affected its Atlanta, Ga., community. It affected entry to Microsoft networks and related companies, equivalent to Sharepoint, Workplace, Azure companies and internet hosting, and seemed to be positioned within the Cogent knowledge middle in Atlanta. Based mostly on the affected interfaces and nodes it seems it was a results of configuration changes relatively than a control-plane subject.
Leave a Reply