Web Wiz - Solar Powered Eco Hosting


Web Wiz Network Announcements


Windows Server Updates - June 2019

12 June 2019, in Shared Hosting, Cloud Servers, Dedicated Servers

We will be carrying out Windows Updates to our Windows Shared Hosting Servers and to customers servers with Managed Cloud and Managed Dedicated Servers during the times set out below.

Start of work - At Risk period:
Start Date: 13th June 2019

End of work - At Risk period:
Risk End Date: 14th June 2019

Times work will be carried out:
Start Time: 05:00 GMT
End Time: 8:30 GMT

Service Impact:
Servers will need to be rebooted after updates are applied.

Reason:
Microsoft releases Windows updates and patches on the second Tuesday of each month. These updates fix security vulnerabilities, performance and other issues.

After testing the latest updates we roll out Windows updates to all Web Wiz Managed Windows Servers.



Emergency Network Maintenance - 30th May 2019 ** Completed **

30 May 2019, in Network

We will be carrying out emergency maintenance wok on our network overnight tonight.

Start of work - At Risk period:
Start Date: 30th May 2019 Start Time: 10:00

End of work - At Risk period:
Risk End Date: 31st May 2019 Risk End Time: 05:00

Service Impact:
There maybe some packet loss and a few very short outages that are expected to last no longer than a few minutes while new network equipment is installed and switches and routers are reconfigured.

Reason:  
We had intermittent connectivity issues on the 28th and 29th May. To fix these issues older equipment was used and a number of workaround configurations were implemented. This network maintenance is to replace the older equipment and remove the workarounds.  

Update 31st May 00:52
All work is now completed and and the network has been running stable for the last 30 minutes.



Network Upgrade to 10 Gbps Core - 28th May 2019

25 May 2019, in Network, Data Centre

We will be moving over to our new 10 Gbps Core Network on Tuesday 28th May 2019.

This is the second part  of a network upgrade, moving from our Cisco 1 Gbps Layer 2 network to a more advanced HPE Aruba Networks 10 Gbps Layer 3 network. 

Start of work - At Risk period:
Start Date: 28th May 2019 Start Time: 04:00

End of work - At Risk period:
Risk End Date: 28th May 2019 Risk End Time: 7:00

Service Impact:
There maybe some packet loss not lasting more than a few minutes while the new network routes are activated and traffic reroutes across the new paths.

Reason: 
Keeping with offering eco-friendly services to customers, our new network will be decreasing the number of switches we have by 50%, which will reduce power requirements while providing improved performance and security. 


Update 12:52
One of the 10Gbps Fibre Optic Transceivers has begun to fail intermittently. This has caused some package loss on our secondary route and MSTP flapping on our primary route. Engineers are working to fix the issue and have installed a replacement transceiver and are working on the MSTP issue.

Update 18:23
Engineers have found the cause of the MSTP flapping and intermittently blocking forwarding on ports causing short outages lasting a few minutes each time was caused by a faulty Fibre Optic Cable which has been replaced. We have seen the network running stably for 3 hours. We will continue to monitor the network for any further issues.

Update 19:28
We have seen periods of intermittent packet loss return lasting a few seconds each time. We have called in all engineers to work throughout the night to try and resolve the issue.

Update May 29th 03:09
It has been a busy night, we have reconfigured the entire network in the hope of finding the cause of the intermittent packet loss, which is work we planned to do next month. This has added extra security and performance to the network, but did not stop the intermittent packet loss.

The intermittent packet loss was eventually traced to the Intel 10 Gbps Fibre Optic Transceivers in the gateway firewalls. Replacing these with 1 Gbps Fibre Optic Transceivers and rolling back others to standard 1Gbps Ethernet ports has fixed the packet loss issue and we have seen the network run stable with no packet loss for 2 hours.

We have placed an order for replacement Intel 10 Gbps Transceivers using a different model and hope to have these later in the week for testing within the lab.



DNS Service Outage ** Resolved **

23 May 2019, in Shared Hosting

We are aware of an issue with our Microsoft recursive DNS Servers are not able to perform DNS Lookups of hostnames for the Shared Hosting Services.

It would appear that both primary and secondary recursive DNS servers are missing their Root Hints.

Authoritative DNS Servers which do not run Microsoft DNS Service are not affected.

We are working to resolve the issue.


Update 07:10
DNS Services have been running stable now a period of time and all related services are working.

We are continuing to investigate how both recursive DNS Servers lost their Root Hints at the same time as such an issue is unheard of so not something we had previously built a contingency plan for. 

What are Root Hints?
For those who do not know what the Root Hints are, this is a file built into DNS Servers which tell the DNS Server where the Root DNS Servers for the Internet are located in order to perform DNS Lookups of Hostnames.


Update 12:39 ** Resolved **
An investigation has found the issue was caused by a bug in Windows Server DNS Service that when Windows Server reboots it updates the Root Hints deleting all the IPv4 address of the Internets DNS Root Servers. We have raised a support ticket with Microsoft regarding this issue.

Measures to prevent recurrence
To avoid further problems relating to this issue we have purchased SimpleFailover. This software allows us to monitor DNS Responses in real-time and send alerts if the DNS servers fail to complete DNS Lookups.

The primary effect of the DNS outage on customers was that websites were not able to perform DNS Lookups for database connections on our 'websitelive.net' domain. 

To make sure that the issue does not impact on related services we have configured our recursive DNS Servers to be a Secondary DNS Zone for our 'websitelive.net' domain. By making this change, it will improve performance and as an external lookup is not required allow websites to still lookup hostnames on our 'websitelive.net' domain even if DNS Lookups are failing.


Update 14:49 - What caused the outage
We have now learned from Microsoft the cause of the issue. Microsoft fixed a vulnerability in the DNS Service in Windows Server 2016 this then caused the bug. If IPv6 is enabled the DNS Service only loads in the IPv6 addresses of the Internet Root Servers and deletes IPv4 addresses.

As a workaround to address the problem we have created a PowerShell Script that runs when the Servers are booted that adds the missing Root Hints to the DNS Services.


We want to apologise to all customers affected by this outage.



Windows Server Updates - May 2019

15 May 2019, in Shared Hosting, Cloud Servers, Dedicated Servers

We will be carrying out Windows Updates to our Windows Shared Hosting Servers and to customers servers with Managed Cloud and Managed Dedicated Servers during the times set out below.

Start of work - At Risk period:
Start Date: 16th May 2019

End of work - At Risk period:
Risk End Date: 17th May 2019

Times work will be carried out:
Start Time: 05:00 GMT
End Time: 8:30 GMT

Service Impact:
Servers will need to be rebooted after updates are applied.

Reason:
Microsoft releases Windows updates and patches on the second Tuesday of each month. These updates fix security vulnerabilities, performance and other issues.

After testing the latest updates we roll out Windows updates to all Web Wiz Managed Windows Servers.



Page  123...21>



Become a Fan on Facebook Follow us on Twitter Connect with us on LinkedIn Community Forums Web Wiz Blogs Web Wiz News
About Web Wiz | Contact Web Wiz | Terms & Conditions | Cookies | Privacy Policy

Web Wiz is the trading name of Web Wiz Ltd. Company registration No. 05977755. Registered in England and Wales.
Registered office: Web Wiz Ltd, Unit 18, The Glenmore Centre, Fancy Road, Poole, Dorset, BH12 4FB, UK.

Prices exclude VAT unless otherwise stated. VAT No. GB988999105 - $, € prices shown as a guideline only.

Copyright ©2001-2019 Web Wiz Ltd. All rights reserved.