GSL
Global Maintenance - Limited Impact
Update - Changes are now largely complete around the world with some limited changes still to be made. Customers in those specific sites will be contacted directly with more specific updates.

All up this project ran two weeks longer than expected.

We have adjusted more than 600,000 lines of config in aggregate across all routers worldwide.
We have removed an additional 170,000 lines of config considered technical debt caused by the last 5 years of growth.
We have added more than 350 100G ports to the network to support internal infrastructure and various services.
We have added 84 external 100G ports to the network over the last two months.

Most things should be returning to normal, albeit there may be some routing regressions or suspect paths may be active.

If routing looks sub-optimal, please either reach out to noc@globalsecurelayer.com or via your related support avenues if you're not a direct customer. We are actively reviewing global routing at this stage to make widespread improvements.
Sep 18, 2023 - 16:31 UTC
Update - Work is still progressing.

North America has been complete and stable since the 23rd.

At this stage it looks like we are going to run right up to our previous estimation/deadline and run 1-2 days over for global completion, pending any significant issues that arise.

We have begun the migration of cities in Australia today, we do expect there to be some scenic routing while some other dependent sites are migrated through the week.

We apologize in advance for the headaches this causes, we hope that the scenic routing is short lived through this week.

Please still reach out with traces to your contacts or to noc@globalsecurelayer.com if something is incorrect, it may be something we can repair immediately, or if not we can advise with more specifics on when it will be restored.
Sep 02, 2023 - 01:27 UTC
Update - At approximately 9:00AM UTC some automation systems were updated to account for changes within the new network.

This caused adverse affects to a number of locations, most notably Sydney, Frankfurt & Tokyo.

A large amount of traffic was re-routed for approximately 6 minutes with some others black holed for roughly 1 minute.

Apologies for this inconvenience, This event should not re-occur.
Aug 11, 2023 - 09:18 UTC
Update - During turnup of Pittsburgh some traffic entering the network over an IXP was incorrectly backhauled to a remote DDoS scrubber for a period of time today.

This has been rectified.
Aug 10, 2023 - 02:40 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Aug 07, 2023 - 03:00 UTC
Scheduled - Over the past six months, we have diligently worked on upgrading our global network to accommodate new products, additional features, enhanced customization for our technical customers, and to significantly improve the quality of life and reliability in regions prone to frequent cascading cable cuts and infrastructure failures.

With our preparations now complete, we are ready to implement the more noticeable changes throughout the month of August, with the project expected to conclude in the first week of September.

During this period, users may observe the following temporary changes:
- Occasional incorrect routing or re-routing to nearby locations.
- Reduction in peak DDoS mitigation capacity on certain days.
- Bypassing of some locations for network traffic.
- Slightly higher than expected latency between specific regions.

We want to reassure you that, apart from the latency caused by the use of different paths, there should be no significant underlying throughput or performance issues during this maintenance period.

If you notice an incorrect route persisting for more than 24 hours, we kindly request you to inform us by sending an email to noc@globalsecurelayer.com, regardless of whether you are a direct customer or not.

Your feedback is essential for maintaining the integrity and performance of our network. If you have any concerns, feel free to reach out to us.
IPV6 Issues - Amsterdam and Frankfurt
Investigating - In the wake of our ongoing global maintenance, we've identified some challenges related to IPv6 in Amsterdam and Frankfurt. Our dedicated team is fully aware of these issues and is diligently working to resolve them.

Currently, we do not have a precise estimate for the resolution timeline, but rest assured that we are actively addressing the situation. We are committed to providing you with an update as soon as we have more information.

If you are experiencing IPv6-related issues and would like our team to investigate your case specifically, please don't hesitate to submit a support ticket.
Please provide any relevant information in your ticket, such as traceroutes, source/destination IPV6 addresses etc.
Your concerns are important to us, and we're here to assist you.

Thank you for your patience as we work to enhance the reliability and performance of our network. Your feedback and support are greatly appreciated.
Sep 12, 2023 - 15:08 UTC
IP Transit Operational
(AU) Adelaide Operational
(AU) Brisbane Operational
(AU) Melbourne Operational
(AU) Sydney Operational
(AU) Perth Operational
(NZ) Auckland Operational
(EU) Amsterdam Operational
(EU) Marseille Operational
(EU) Frankfurt Operational
(UK) London Operational
(HK) Hong Kong Operational
(SG) Singapore Operational
(JP) Tokyo Operational
(USA) Ashburn Operational
(USA) Chicago Operational
(USA) Dallas Operational
(USA) Phoenix Operational
(USA) Los Angeles Operational
(USA) New York Operational
(USA) Atlanta Operational
(USA) Seattle Operational
(USA) Guam Operational
Backbone Network Operational
(AU) Adelaide Operational
(AU) Brisbane Operational
(AU) Melbourne Operational
(AU) Sydney Operational
(AU) Perth Operational
(NZ) Auckland Operational
(EU) Amsterdam Operational
(EU) Marseille Operational
(EU) Frankfurt Operational
(UK) London Operational
(HK) Hong Kong Operational
(SG) Singapore Operational
(JP) Tokyo Operational
(USA) Ashburn Operational
(USA) Chicago Operational
(USA) Dallas Operational
(USA) Phoenix Operational
(USA) Los Angeles Operational
(USA) New York Operational
(USA) Atlanta Operational
(USA) Seattle Operational
(USA) Guam Operational
Datacentres Operational
(AU) Sydney - Equinix SY5 Operational
(AU) Sydney - NextDC S1 Operational
(AU) Brisbane - NextDC B2 Operational
(AU) Melbourne - NextDC M1 Operational
(AU) Adelaide - Equinix AE1 Operational
(AU) Adelaide - ColoCity DC3 Operational
(AU) Perth - Equinix PE2 Operational
(AU) Perth - NextDC P2 Operational
(NZ) Auckland - DataCentre220 Operational
(SG) Singapore - Equinix SG3 Operational
(JPN) Tokyo - Equinix TY8 Operational
(HK) Hong Kong - Equinix HK1 Operational
(USA) Ashburn - Equinix DC10 Operational
(USA) New York - Equinix NY2 Operational
(USA) Chicago - Equinix CH1 Operational
(USA) Los Angeles - Coresite LA2 Operational
(USA) Dallas - Equinix DA11 Operational
(EU) Amsterdam - Equinix AM5 Operational
(EU) Amsterdam - Nikhef Operational
(EU) Frankfurt - Equinix FR5 Operational
(EU) Marseille - Interxion MRS2/3 Operational
(EU) Amsterdam - Iron Mountain AMS1 Operational
(USA) Phoenix - PhoenixNAP Operational
(UK) London - Equinix LD8 Operational
(UK) London - Telehouse North Operational
(UK) London - Telehouse West Operational
(USA) Los Angeles - Equinix LA1 Operational
(USA) Chicago - Equinix CH3 Operational
(USA) Atlanta - Digital Realty 56 Marietta Operational
(USA) Seattle - Digital Realty The Westin Building Exchange Operational
(USA) Guam - RTI Guam GNC Operational
(AU) Melbourne - Equinix ME1 Operational
(AU) Perth - NextDC P1 Operational
Customer Portals Under Maintenance
Support and Billing Operational
Bare Metal Management Operational
Protection Reporting Operational
Protection Managment (Creatia) Operational
Remote Protection Management (Mercury) Under Maintenance
Operational
Degraded Performance
Partial Outage
Major Outage
Maintenance
Past Incidents
Sep 25, 2023

No incidents reported today.

Sep 24, 2023

No incidents reported.

Sep 23, 2023
Frankfurt network - delayed
Completed - The scheduled maintenance has been completed.
Sep 23, 22:00 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Sep 23, 16:00 UTC
Scheduled - Due to unforeseen delays, the original maintenance event scheduled to be executed yesterday had to be rescheduled to today:

We have scheduled emergency maintenance in Frankfurt to resolve routing issues and replace a faulty line card. Transit customers without a redundant service and unprotected L2 Ethernet customers may encounter a maximum downtime of 10 minutes during this maintenance window. However, we anticipate that the actual downtime will be only 5 minutes.

During this maintenance window, customers should anticipate increased latency to and from Frankfurt. We anticipate that this higher latency will not persist for more than 24 hours.
Sep 23, 06:46 UTC
Scheduled Emergency Maintenance - Frankfurt Network
Completed - The scheduled maintenance has been completed.
Sep 23, 00:00 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Sep 22, 14:00 UTC
Scheduled - We have scheduled emergency maintenance in Frankfurt to resolve routing issues and replace a faulty line card. Transit customers without a redundant service and unprotected L2 Ethernet customers may encounter a maximum downtime of 10 minutes during this maintenance window. However, we anticipate that the actual downtime will be only 5 minutes.

During this maintenance window, customers should anticipate increased latency to and from Frankfurt. We anticipate that this higher latency will not persist for more than 24 hours.
Sep 22, 07:51 UTC
Sep 22, 2023
Scheduled Emergency Maintenance - Hong Kong Network
Completed - The scheduled maintenance has been completed.
Sep 22, 06:00 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Sep 22, 02:00 UTC
Scheduled - We have scheduled emergency maintenance in Hong Kong to resolve routing issues and remove a faulty device. Transit customers without a redundant service and unprotected L2 Ethernet customers may encounter a maximum downtime of 10 minutes during this maintenance window. However, we anticipate that the actual downtime will be only 5 minutes.
Sep 22, 00:40 UTC
Sep 21, 2023

No incidents reported.

Sep 20, 2023

No incidents reported.

Sep 19, 2023

No incidents reported.

Sep 18, 2023

Unresolved incident: Global Maintenance - Limited Impact.

Sep 17, 2023

No incidents reported.

Sep 16, 2023

No incidents reported.

Sep 15, 2023

No incidents reported.

Sep 14, 2023
Sydney - Hardware Upgrade
Completed - The scheduled maintenance has been completed.
Sep 14, 19:00 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Sep 14, 15:00 UTC
Scheduled - We will be installing new core routers in Equinix SY5. Customers who have been notified via email will be migrated to this new hardware.
Customers without a redundant cross connect will experience a small outage while their session is shifted to new equipment. Each of these outages will be less than five minutes.

All relevant customer sessions will be drained prior to maintenance to ensure there is no impact.
Sep 14, 01:18 UTC
Sep 13, 2023

No incidents reported.

Sep 12, 2023

Unresolved incident: IPV6 Issues - Amsterdam and Frankfurt.

Sep 11, 2023

No incidents reported.

← Incident History Powered by Atlassian Statuspage