Web Wiz Network Announcements

We will be carrying out Windows Updates to our Windows Shared Hosting Servers and to customers servers with Managed Cloud and Managed Dedicated Servers during the times set out below.

After testing the latest updates we roll out Windows updates to all Web Wiz Managed Windows Servers.

We will be carrying out emergency maintenance wok on our network overnight tonight.

All work is now completed and and the network has been running stable for the last 30 minutes.

This is the second part  of a network upgrade, moving from our Cisco 1 Gbps Layer 2 network to a more advanced HPE Aruba Networks 10 Gbps Layer 3 network. 

Start Date: 28th May 2019 Start Time: 04:00

Risk End Date: 28th May 2019 Risk End Time: 7:00

There maybe some packet loss not lasting more than a few minutes while the new network routes are activated and traffic reroutes across the new paths.

Reason: 

Keeping with offering eco-friendly services to customers, our new network will be decreasing the number of switches we have by 50%, which will reduce power requirements while providing improved performance and security. 

One of the 10Gbps Fibre Optic Transceivers has begun to fail intermittently. This has caused some package loss on our secondary route and MSTP flapping on our primary route. Engineers are working to fix the issue and have installed a replacement transceiver and are working on the MSTP issue.

Engineers have found the cause of the MSTP flapping and intermittently blocking forwarding on ports causing short outages lasting a few minutes each time was caused by a faulty Fibre Optic Cable which has been replaced. We have seen the network running stably for 3 hours. We will continue to monitor the network for any further issues.

We have seen periods of intermittent packet loss return lasting a few seconds each time. We have called in all engineers to work throughout the night to try and resolve the issue.

It has been a busy night, we have reconfigured the entire network in the hope of finding the cause of the intermittent packet loss, which is work we planned to do next month. This has added extra security and performance to the network, but did not stop the intermittent packet loss.

The intermittent packet loss was eventually traced to the Intel 10 Gbps Fibre Optic Transceivers in the gateway firewalls. Replacing these with 1 Gbps Fibre Optic Transceivers and rolling back others to standard 1Gbps Ethernet ports has fixed the packet loss issue and we have seen the network run stable with no packet loss for 2 hours.

We have placed an order for replacement Intel 10 Gbps Transceivers using a different model and hope to have these later in the week for testing within the lab.

We are aware of an issue with our Microsoft recursive DNS Servers are not able to perform DNS Lookups of hostnames for the Shared Hosting Services.

It would appear that both primary and secondary recursive DNS servers are missing their Root Hints.

Authoritative DNS Servers which do not run Microsoft DNS Service are not affected.

We are working to resolve the issue.

DNS Services have been running stable now a period of time and all related services are working.

We are continuing to investigate how both recursive DNS Servers lost their Root Hints at the same time as such an issue is unheard of so not something we had previously built a contingency plan for. 

For those who do not know what the Root Hints are, this is a file built into DNS Servers which tell the DNS Server where the Root DNS Servers for the Internet are located in order to perform DNS Lookups of Hostnames.

An investigation has found the issue was caused by a bug in Windows Server DNS Service that when Windows Server reboots it updates the Root Hints deleting all the IPv4 address of the Internets DNS Root Servers. We have raised a support ticket with Microsoft regarding this issue.

To avoid further problems relating to this issue we have purchased SimpleFailover. This software allows us to monitor DNS Responses in real-time and send alerts if the DNS servers fail to complete DNS Lookups.

The primary effect of the DNS outage on customers was that websites were not able to perform DNS Lookups for database connections on our 'websitelive.net' domain. 

To make sure that the issue does not impact on related services we have configured our recursive DNS Servers to be a Secondary DNS Zone for our 'websitelive.net' domain. By making this change, it will improve performance and as an external lookup is not required allow websites to still lookup hostnames on our 'websitelive.net' domain even if DNS Lookups are failing.

We have now learned from Microsoft the cause of the issue. Microsoft fixed a vulnerability in the DNS Service in Windows Server 2016 this then caused the bug. If IPv6 is enabled the DNS Service only loads in the IPv6 addresses of the Internet Root Servers and deletes IPv4 addresses.

We want to apologise to all customers affected by this outage.

We will be carrying out Windows Updates to our Windows Shared Hosting Servers and to customers servers with Managed Cloud and Managed Dedicated Servers during the times set out below.

After testing the latest updates we roll out Windows updates to all Web Wiz Managed Windows Servers.