Kernel Panic (Resolved)
  • Priority - Critical
  • Affecting System - ldn16-uk
  • ldn16-uk has encountered a kernel panic and is now restarting.

    Apologies for any inconvenience caused.

  • Date - 14/12/2019 20:24 - 14/12/2019 21:25
  • Last Updated - 14/12/2019 21:26
Network Unresponsive (Resolved)
  • Priority - Critical
  • Affecting System - nc7-us
  • nc7-us's network is currently not responding.

    The local NoC are currently investigating the issue.

    Update #1 - nc7-us encountered a kernel panic, it is now restarting.

  • Date - 06/11/2019 18:16 - 06/11/2019 19:00
  • Last Updated - 14/12/2019 21:02
Network Unresponsive (Resolved)
  • Priority - Critical
  • Affecting Other - ldn8-uk
  • ldn8-uk is unresponsive with the bonded link showing as down.

    Local NoC have been alerted and are going to investigate.

    Apologies for any inconvenience caused, once the issue is resolved an RFO will be published.

    Update 1 - Both Diverse NIC's are unresponsive at this time, new cables, NiC's and links are currently being tested.

  • Date - 28/08/2019 19:06 - 02/09/2019 00:55
  • Last Updated - 02/09/2019 08:58
Service Unavailable (Resolved)
  • Priority - Critical
  • Affecting System - manage.quadhost.net
  • Our Virtual Server Controller (manage.) is currently unavailable as it is hosted on ldn8-uk which is currently encountering issues.

    When the issue with ldn8-uk has been resolved the portal will be once again available.

    Apologies for any inconvenience caused.

  • Date - 28/08/2019 19:06 - 02/09/2019 06:30
  • Last Updated - 02/09/2019 08:57
E-mail Delivery (Resolved)
  • Priority - High
  • Affecting System - mandrillapp.com
  • We are seeing disruption in Mandrill's SMTP servers connectivity, outbound e-mails from the website and management portal are failing and will be resent once the service is online and operational.

    This doesn't affect Shared Hosting, VPS or Dedicated Server customers using our own relays or IP Space.

  • Date - 26/07/2019 13:00 - 26/07/2019 17:00
  • Last Updated - 28/08/2019 19:27
Hypervisor Restart (Resolved)
  • Priority - Critical
  • Affecting System - ldn16-uk
  • ldn16-uk has encountered a tainted kernel and triggered a panic and is restarting.

    Apologies for any inconvenience caused.

  • Date - 26/07/2019 19:14
  • Last Updated - 26/07/2019 21:22
Upstream Maintenance (Resolved)
  • Priority - Medium
  • Affecting System - ldn-uk
  • Scheduled maintenance is planned by our upstream in London at 00:00-05:00 on the 6th, 7th, 8th & 9th at 00:00-05:00.

    No loss of service is expected due to diverse routing but  services are potentially at risk at this time.

    Update: Disruption Observed to some routes in/out of London on 8/7/2019.

  • Date - 06/07/2019 00:00 - 09/07/2019 05:00
  • Last Updated - 24/07/2019 05:47
Hypervisor Unresponsive (Resolved)
  • Priority - Critical
  • Affecting System - nc7-us
  • nc7-us is not responding, local NoC are investigating.

    Local NoC have noted a loose power cable to the hypervisor as the cause of the loss of power, server and NAT network have been restarted.

  • Date - 22/07/2019 17:44 - 22/07/2019 19:47
  • Last Updated - 22/07/2019 19:47
Hypervisor Fault (Resolved)
  • Priority - Critical
  • Affecting System - nc8-us
  • nc8-us has reported a hardware anomaly, we are investigating.

    UPDATE: nc8-us now has an immediate scheduled hardware intervention scheduled.

    Please see the applicable e-mail concerning the actions taken on a per service/product basis for customers on this hypervisor.

    If you have any questions or queries then feel free to contact the support team.

  • Date - 07/07/2019 17:15 - 08/07/2019 00:12
  • Last Updated - 07/07/2019 22:12
NAT Connectivity to Singapore (Resolved)
  • Priority - Critical
  • Affecting System - tai-sgX
  • Connectivity and routing to tai-sgX is disrupted, packetloss on IPv4 is seen, with IPv6 dropped.

    Issue is open with upstream for investigation and resolution.

    Apologies for any inconvenience caused by the upstream disruption.

    UPDATE :
    1) Due to the delays in resolution, if another location is suitable then instances can be migrated to restore service to specific customers/instances.

    2) At this time we are evaluating other upstreams/vendors to migrate the NAT services to avoid any future issues of this nature.

    3) Should users wish to be migrated to another location during the issue resolution please open a ticket with the migrations department confirming the ideal location and if the data needs to be retained as part of the migration. 

    4) Upstream changes have been agreed and are now in progress for deployment.

    5) IPv4 connectivity has been restored, IPv6 connectivity will be restored shortly.

  • Date - 22/05/2019 06:00 - 22/06/2019 10:30
  • Last Updated - 01/07/2019 18:05
Hypervisor Unresponsive (Resolved)
  • Priority - Critical
  • Affecting System - nc7-us
  • nc7-us is not responding.

    Local NoC are investigating.

    Issue resolved.

  • Date - 28/06/2019 20:59 - 28/06/2019 23:27
  • Last Updated - 28/06/2019 21:44
Hypervisor Unresponsive (Resolved)
  • Priority - Critical
  • Affecting System - nc7-us
  • nc7-us is unresponsive.

    Issue lodged with local NoC/Upstream for investigation.

    Appears at this time to be a power disruption, hypervisor has restarted, containers and NAT network is now starting.

  • Date - 09/06/2019 09:40 - 09/06/2019 14:14
  • Last Updated - 09/06/2019 12:14
North Carolina - Upstream Connectivity Loss (Resolved)
  • Priority - Critical
  • Affecting System - ncX-us
  • Upstream connectivity in North Carolina via Cogent is disrupted.

    Local NOC are investigating.

  • Date - 03/06/2019 16:52 - 03/06/2019 17:10
  • Last Updated - 03/06/2019 17:11
Upstream Outage (Resolved)
  • Priority - Critical
  • Affecting System - taiX-sg
  • taiX-sg upstream route is no longer responding.

    Awaiting confirmation from upstream as to cause and resolution time.

  • Date - 14/05/2019 17:14 - 14/05/2019 17:20
  • Last Updated - 21/05/2019 07:15
Upstream Outage (Resolved)
  • Priority - Critical
  • Affecting System - nc-*
  • Upstream in North Carolina is unresponsive, awaiting local NoC investigation.

    Apologies for any inconvenience caused.

  • Date - 07/05/2019 10:55 - 07/05/2019 12:05
  • Last Updated - 12/05/2019 11:07
Kernel Panic (Resolved)
  • Priority - Critical
  • Affecting System - nc7-us
  • nc7-us has encountered a kernel panic and is now restarting.

    Apologies for any inconvenience caused.

  • Date - 05/05/2019 19:17 - 05/05/2019 19:37
  • Last Updated - 07/05/2019 11:10
ak1-nz Unresponsive (Resolved)
  • Priority - Critical
  • Affecting System - ak1-nz
  • ak1-nz is unresponsive, issue has been lodged with upstream provider for resolution.

  • Date - 25/04/2019 01:03 - 25/04/2019 07:09
  • Last Updated - 25/04/2019 07:09
IP Nulled due to Inbound DDoS (Resolved)
  • Priority - Critical
  • Affecting Other - NL NAT Shared IP
  • The NL NAT shared IPv4 address (185.169.235.5) has been nulled at this time due to an inbound DDoS attack towards it.

    Our apologies for any inconvenience caused by the nullroute, it has been done to prevent disruption to other customers and connectivity.

    IPv6 connectivity is unaffected.

  • Date - 25/03/2019 20:50 - 25/03/2019 21:38
  • Last Updated - 25/03/2019 21:45
Network Unresponsive (Resolved)
  • Priority - Critical
  • Affecting System - ldn10-uk
  • ldn10-uk network is unresponsive, issue escalated to upstream for investigation and resolution.

  • Date - 24/03/2019 23:01 - 25/03/2019 00:50
  • Last Updated - 25/03/2019 01:07
PoP Unresponsive (Resolved)
  • Priority - Critical
  • Affecting Other - ak1-nz
  • ak1-nz and upstream hops are unresponsive.

    Ticket/issue already lodged with upstream.

    Update - Upstream hops responsive, final hops are not.

  • Date - 20/03/2019 01:22 - 20/03/2019 05:53
  • Last Updated - 20/03/2019 07:54
Hypervisor Unresponsive (Resolved)
  • Priority - Critical
  • Affecting System - nc7-us
  • nc7-us is unresponsive, issue escalated to local remote hands for investigation.

    Update: nc7-us is now booted, NAT network is up and containers are starting.

    Apologies for any inconvience caused by the kernel panic.

  • Date - 14/03/2019 10:19 - 14/03/2019 11:19
  • Last Updated - 16/03/2019 14:41
Planned Migrations & I/O Performance (Resolved)
  • Priority - Medium
  • Affecting System - nc7-us
  • nc7-us I/O Performance is currently impacted due to a RAID rebuild, once complete existing level of performance will return.

    Apologies for any inconvience caused.

    ----

    Continual rebuilds of nc7's RAID array are keeping the service online however we believe now the best course of action is to perform a full hypervisor replacement.

    nc7-us is now planned for retirement and will be transferred to new equipment shortly, timeframes to be e-mailed to each customer based on the product type.

  • Date - 12/03/2017 12:33
  • Last Updated - 14/03/2019 10:38
PoP Unresponsive (Resolved)
  • Priority - Critical
  • Affecting System - ak1-nz
  • ak1-nz and upstream hops are unresponsive.

    When upstream portal is responsive ticket will be lodged; contact has already been attempted.

  • Date - 05/03/2019 19:14 - 05/03/2019 21:44
  • Last Updated - 06/03/2019 09:03
Kernel Panic (Resolved)
  • Priority - High
  • Affecting System - nc7-us
  • Kernel on nc7-us has encountered a kernel panic, hypervisor is now restarting.

  • Date - 25/02/2019 16:28 - 25/02/2019 17:28
  • Last Updated - 05/03/2019 19:16
Hypervisor Unresponsive (Resolved)
  • Priority - Critical
  • Affecting System - tai2-sg
  • tai2-sg has encountered a kernel panic, server is restarting.

  • Date - 20/02/2019 17:00 - 20/02/2019 17:15
  • Last Updated - 20/02/2019 17:49
PoP Unresponsive (Resolved)
  • Priority - Critical
  • Affecting System - tai-sg
  • tai-sg is unresponsive from multiple locations, issue escalated to local NoC for resolution.

  • Date - 11/02/2019 06:40 - 11/02/2019 06:55
  • Last Updated - 11/02/2019 06:55
Server Unresponsive (Resolved)
  • Priority - Critical
  • Affecting System - nc7-us
  • nc7-us is not responding to internal or external monitoring.

    Issue escalated to local NoC for investigation.

    Update - Server had a kernel panic, hypervisor now restarting, on completion of container(s) start NAT networking will be restarted.

  • Date - 08/02/2019 06:47 - 08/02/2019 07:07
  • Last Updated - 08/02/2019 10:16
IPv6 Connectivity (Resolved)
  • Priority - High
  • Affecting System - nc7
  • Loss in IPv6 connectivity observed from North Carolina.

    Issue escalated to upstram.

  • Date - 05/02/2019 23:30 - 06/02/2019 19:00
  • Last Updated - 07/02/2019 21:24
IP Assignments (Resolved)
  • Priority - Medium
  • Affecting System - Resource Bundle
  • IP Address assignment during resource bundle creation is not assigning correct IP's.

    Issue is currently being resolved with the management panel development team.

    UPDATE - Issue confined to OVH based hypervisors at present.

  • Date - 06/07/2018 14:21
  • Last Updated - 31/01/2019 21:36
Connectivity Issues (Resolved)
  • Priority - Critical
  • Affecting System - tai-sg
  • tai-sg upstream connectivity has dropped.

    Issue escalated to upstream for resolution.

  • Date - 29/11/2018 20:08 - 29/11/2018 20:18
  • Last Updated - 31/01/2019 21:35
Loss of Connectivity (Resolved)
  • Priority - High
  • Affecting System - var-bg
  • A small number of customers are reporting a loss of connectivity within certain routes/subnets.

    Engineers are working to resolve the issue.

  • Date - 14/11/2018 11:25 - 14/11/2018 12:00
  • Last Updated - 29/11/2018 20:13
Network Hardware Fault (Resolved)
  • Priority - Critical
  • Affecting System - ldn11-uk
  • ldn11-uk is unresponsive over bonded network.

    Local NoC are investigating.

    Update - Service restored.

  • Date - 13/11/2018 03:07 - 13/11/2018 15:19
  • Last Updated - 13/11/2018 16:30
Connectivity Issues (Resolved)
  • Priority - Medium
  • Affecting System - tai-sg
  • Network connectivity disruption has been reported within our SG location, local NoC is investigating.

    As information regarding the issue is available updates will be posted to this status page.

  • Date - 30/09/2018 11:30 - 30/09/2018 12:30
  • Last Updated - 13/11/2018 08:58
Hypervisor Unresponsive (Resolved)
  • Priority - Critical
  • Affecting System - ak1-nz
  • ak1-nz is not responsive.

    Local NoC have been informed and we are awaiting their update and intervention.

  • Date - 31/10/2018 06:14 - 31/10/2018 06:44
  • Last Updated - 13/11/2018 08:57
Hypervisor Unresponsive (Resolved)
  • Priority - Critical
  • Affecting System - ndk2-nl
  • ndk2-nl's upstream network is unresponsive.

    Local NoC informed and investigating.

    Update - Connectivity Restored.

  • Date - 24/09/2018 10:06 - 24/09/2018 11:39
  • Last Updated - 24/09/2018 11:47
Hypervisor Unresponsive (Resolved)
  • Priority - Critical
  • Affecting System - gra1-fr
  • gra1-fr is unresponsive, local NoC have been informed and are awaiting intervention outcome.

    Issue identified and resolved by local NOC.

  • Date - 15/09/2018 07:05 - 15/09/2018 08:42
  • Last Updated - 15/09/2018 08:54
Hypervisor Kernel Panic (Resolved)
  • Priority - Critical
  • Affecting System - ldn11-uk
  • ldn11-uk has encountered a kernel panic.

    Hypervisor restarting, manufacturer informed of recurring issue for resolution.

    Update - Hypervisor and NAT network restarted, VM's being booted now.

    Update #2 - Kernel panic encountered again, awaiting vendor response regarding mainboard solution. Node restarted, VM's coming up.


    Update #3 - Hypervisor is currently awaiting intervention from hardware vendor for system test.

    Update #4 - Vendor has performed hardware check(s), firmware updates made and hypervisor now restarting.

    Update #5 - NAT Network now back online.

    Update #6 - Kernel panic occured again, preparing for migrations to reduce extended downtime pending further hypervisor investigation.

    Update #7 - Additional hardware intervention performed alongside software and kernel change to 042stab133.2 for resolution.

    Update #8 - No further panic's encountered, hypervisor now considered stable once again after 12hrs without interuption
    (previous panic's encoutnered within 1hr).

  • Date - 29/08/2018 10:29 - 31/08/2018 12:43
  • Last Updated - 31/08/2018 12:43
Upstream Connectivity Loss (Resolved)
  • Priority - Critical
  • Affecting System - ak1-nz
  • Upstream connectivity has been lost in ak1-nz, upstream alerted and awaiting intervention.

  • Date - 29/08/2018 14:07 - 29/08/2018 14:37
  • Last Updated - 29/08/2018 14:20
Hypervisor Unresponsive (Resolved)
  • Priority - Critical
  • Affecting System - ldn11-uk
  • ldn11-uk has encountered a kernel panic.

    Hypervisor restart in progress.

  • Date - 26/08/2018 00:09 - 26/08/2018 01:14
  • Last Updated - 29/08/2018 11:21
Hypervisor Kernel Panic (Resolved)
  • Priority - Critical
  • Affecting System - ldn11-uk
  • ldn11-uk has encountered a kernel panic and is restarting.

    Once restarted the NAT network will be braught online.

    Apologies for any inconvience caused.

    Update - Restart complete, NAT network coming up now.

  • Date - 08/08/2018 17:53 - 08/08/2018 18:25
  • Last Updated - 08/08/2018 18:25
Hypervisor Unresponsive (Resolved)
  • Priority - High
  • Affecting System - ak1-nz
  • ak1-nz is unresponsive at present.

    Local NoC have been alerted to the issue and are working to resolve.

  • Date - 01/08/2018 05:01 - 01/08/2018 20:39
  • Last Updated - 01/08/2018 22:10
Network Unresponsive (Resolved)
  • Priority - Critical
  • Affecting System - var3-bg
  • var3-bg network is unresponsive, local NOC are investigating the cause.

    UPDATE - var3-bg upstream connectivity restored.

  • Date - 31/07/2018 15:21 - 31/07/2018 16:41
  • Last Updated - 31/07/2018 16:41
Hypervisor Unresponsive (Resolved)
  • Priority - Critical
  • Affecting System - ak1-nz
  • ak1-nz and upstream switches are unsresponsive at this time.

    Upstream vendor is investigating.

    Update - Hypervisor now responding.

  • Date - 23/07/2018 20:15 - 23/07/2018 21:00
  • Last Updated - 23/07/2018 21:13
Hypervisor Unresponsive (Resolved)
  • Priority - Critical
  • Affecting System - rbx1-fr
  • rbx1-fr is unresponsive.

    OVH NoC technicians have been made aware and are carrying out an intervention to see what the cause is.

    Update - Rack switch fault occured, unit replaced and system now online.

  • Date - 12/06/2018 06:28 - 12/06/2018 09:10
  • Last Updated - 12/06/2018 11:43
DDoS Attack (Resolved)
  • Priority - Critical
  • Affecting System - tai-sg
  • Singapore is currently under a large scale DDoS attack, nullroutes have been applied where possible to minimise impact.

  • Date - 11/06/2018 17:00 - 11/06/2018 17:39
  • Last Updated - 11/06/2018 18:39
IPv6 Connectivity (Resolved)
  • Priority - High
  • Affecting System - tai2-sg.quadhost.net
  • IPv6 connectivity in/out of Singapore is currently experiencing intermitant connectivity issues.

    Upstream is aware and working on prompt resolution.

    Update: Upstream pinpointing cause to determine resolution timescale.

    Update #2: Link up, performing tests.

    Update #3: Test has passed, routes now up.

  • Date - 27/05/2018 14:45 - 02/06/2018 14:00
  • Last Updated - 02/06/2018 14:27
Hypervisor Unresponsive (Resolved)
  • Priority - Critical
  • Affecting System - ldn16-uk
  • ldn16-uk's network is unresponsive.

    Investigation now in progress.

    UPDATE 1:
    System has kernel panicked, now restarting.

    UPDATE 2:
    Once system has restarted (DONE), VM's will be returned to state (IN PROGRESS) and NAT network will be online shortly thereafter.

    UPDATE 3:
    ldn16-uk is now online and operational again.

    Apologies for any inconvience caused.

  • Date - 20/05/2018 13:32 - 20/05/2018 14:46
  • Last Updated - 20/05/2018 14:46
IPv6 Connectivity (Resolved)
  • Priority - Critical
  • Affecting System - gra1-fr
  • IPv6 connectivity on gra1-fr.quadhost.net is currently non responsive.

    Local NoC have been informed and are investigating.

  • Date - 15/05/2018 11:40 - 15/05/2018 23:23
  • Last Updated - 20/05/2018 13:41
Network Outage (Resolved)
  • Priority - Critical
  • Affecting System - nc6-us,nc7-us,nc8-us,nc9-us
  • Network is unresponsive in North Carolina, US location.

    Local NoC are investigating at this time.

    Apologies for any inconvience caused.

  • Date - 01/05/2018 15:11 - 01/05/2018 15:33
  • Last Updated - 01/05/2018 18:35
Hypervisor Unresponsive (Resolved)
  • Priority - Critical
  • Affecting System - nc9-us
  • nc9-us is no longer responding.

    Local NoC have been alerted and are investigating the issue.

  • Date - 19/04/2018 20:45 - 19/04/2018 21:02
  • Last Updated - 01/05/2018 15:20
Hypervisor Unresponsive (Resolved)
  • Priority - Critical
  • Affecting System - ldn11-uk
  • ldn11-uk has encountered a kernel panic and has restarted.

    VM's are now booting and NAT based network stack will be back online shortly.

  • Date - 23/04/2018 09:41 - 23/04/2018 09:59
  • Last Updated - 23/04/2018 10:00
Hypervisor Restart (Resolved)
  • Priority - High
  • Affecting System - ndk2-nl
  • ndk2-nl appears to have kernel panicked and restarted.

    Reboot in progress.

    Update: Hypervisor back online.

    Upstream Notice:

    "We have had an outage in a small part of our datacenters (NLDW2).
    We have found the cause and resolved the failure.
    Our sincere apologies for the inconvenience and down-time.
    We will make sure the correct measures are taken to prevent this from happening in the future.

    Due to the failure your server might have been rebooted.
    Our team has checked every individual server in this part of our datacenter to see if it came back online.
    Please contact us if any assistance is needed. "

  • Date - 18/04/2018 07:30 - 18/04/2018 07:49
  • Last Updated - 18/04/2018 16:40
Hypervisor Unresponsive (Resolved)
  • Priority - Critical
  • Affecting System - ndk2-nl
  • ndk2-nl is unresponsive, local NoC are investigating.

    Update - Hypervisor Responding

  • Date - 05/04/2018 10:10 - 05/04/2018 19:48
  • Last Updated - 05/04/2018 19:49
Network Unresponsive (Resolved)
  • Priority - Critical
  • Affecting System - nc9/nc10
  • vlan2 Network segment in North Carolina unavailable, affecting nc9/nc10 hypervisors.

    Awaiting diagnosis and resolution from local NoC.

  • Date - 19/03/2018 12:30 - 19/03/2018 14:30
  • Last Updated - 19/03/2018 14:49
Hypervisor Unresponsive (Resolved)
  • Priority - Critical
  • Affecting System - ldn11-uk
  • ldn11-uk is not responding, IPMI shows kernel panic.

    Hypervisor restart in progress.

    Apologies for any inconvience caused by this outage.

  • Date - 18/03/2018 13:48 - 18/03/2018 14:32
  • Last Updated - 19/03/2018 12:45
Hypervisor Restart - Kernel Panic (Resolved)
  • Priority - High
  • Affecting System - ldn11-uk
  • ldn11-uk has encountered a kernel panic.

    Hypervisor is currently restarting.

    Once reboot cycle commenced NAT stack will be braught up.

    UPDATE - Hypervisor rebooted, Containers now starting and NAT stack enqueued.

  • Date - 11/03/2018 03:00 - 11/03/2018 04:03
  • Last Updated - 11/03/2018 04:23
Hypervisor Unresponsive (Resolved)
  • Priority - Critical
  • Affecting System - tai1-sg/tai2-sg
  • tai1-sg/tai2-sg are unresponsive.

    Local NoC alerted, awaiting action.

    Update - Connectivity Restored.

  • Date - 10/03/2018 11:03 - 10/03/2018 11:06
  • Last Updated - 10/03/2018 11:06
Hypervisor Unresponsive (Resolved)
  • Priority - Critical
  • Affecting System - ak1-nz
  • ak1-nz and upstream network is unresponsive.

    Issue lodged with upstream pending response.

  • Date - 23/02/2018 17:11 - 23/02/2018 17:20
  • Last Updated - 10/03/2018 11:05
Nullroute on i-83 US (Resolved)
  • Priority - Critical
  • Affecting System - US NAT Shared IPv4
  • Shared IPv4 address is currently nulled by our upstream in NC, USA; to protect our network.

    IPv6 connectivity is still operational.

    Apologies for any inconvenience caused during the nullroute.

    Once feasible the null will be lifted.

     

  • Date - 23/02/2018 00:36 - 23/02/2018 01:36
  • Last Updated - 23/02/2018 09:23
Hypervisor Unresponsive (Resolved)
  • Priority - Critical
  • Affecting System - tai2-sg
  • tai2-sg is unresponsive.

    Awaiting confirmation from local NoC regarding state & issue.

    Update - Upstream have resolved the network issue and network is responding.

  • Date - 18/01/2018 12:16 - 18/01/2018 13:53
  • Last Updated - 18/01/2018 13:57
Hypervisor Unresponsive (Resolved)
  • Priority - Critical
  • Affecting System - tai2-sg
  • tai2-sg is unresponsive.

    Awaiting confirmation from local NoC regarding state & issue.

  • Date - 13/01/2018 21:24 - 13/01/2018 21:25
  • Last Updated - 13/01/2018 21:25
Hypervisor Unresponsive (Resolved)
  • Priority - High
  • Affecting System - tai2-sg.quadhost.net
  • tai2-sg is unresponsive.

    Awaiting confirmation from local NoC regarding state & issue.

    Update #1 - Local NoC have still not responded regarding the outage, considering our other options in the region for redeployment.

    Update #2 - Local NoC have resolved issue, hypervisor is now being patched as per Spectre/Meltdown Notice and restarted.

  • Date - 03/01/2018 15:03 - 09/01/2018 14:53
  • Last Updated - 09/01/2018 14:54
CPU Vulnerability Patch (Resolved)
  • Priority - High
  • Affecting System - All Hypervisors (OpenVZ/KVM)
  • To address the recent Spectre & Meltdown CPU Vulnerabilities a maintenance window is scheduled from 18:00 07/01/2017 for affected hardware to be updated and restarted.

    This event will cause a small window of downtime as each hypervisor is restarted.

    We apologise for any inconvience caused during the brief window however this intervention is required due to the security risks associated to the vulnerability.

    Update #1 - Maintenance window confirmed to start at 18:00, each node will be restarted individually to minimise any issues as the hypervisors restart.

    Update #2 - Commencing, see below:

    Live Progress Updates:

    Complete:
    bhs1,
    nc6,nc7,nc8,nc9,nc10,
    rbx1,gra1,
    ak1,
    var2,var3,
    ndk2,
    ldn8, manage, ldn15, ldn10, ldn11, ldn16

    Postponed:
    tai1,tai2

  • Date - 07/01/2018 13:30 - 08/01/2018 01:34
  • Last Updated - 08/01/2018 01:34
CPU Vulnerability Patch (Resolved)
  • Priority - High
  • Affecting System - London Shared Hosting Clusters
  • To address the recent Spectre & Meltdown CPU Vulnerabilities that exist on the hardware that operates our main shared hosting cluster a maintenance window is scheduled from 23:00 07/01/2017 for affected hardware to be updated and restarted.

    Failover measures will be triggered during the main host restart, once back online all systems will fall back to the primary server.

    No DNS changes are requried for users managing their own DNS records.

  • Date - 07/01/2018 23:00 - 08/01/2018 00:17
  • Last Updated - 08/01/2018 00:17
Interface Offline (Resolved)
  • Priority - Medium
  • Affecting System - manage.
  • manage. is currently unavailable whilst its parent hypervisor is restarted as per the Spectre & Meltdown Patching.

    We apologise for any inconvience caused during the patching.

  • Date - 07/01/2018 23:20 - 07/01/2018 23:35
  • Last Updated - 07/01/2018 23:36
Hypervisor Unresponsive (Resolved)
  • Priority - Critical
  • Affecting System - tai2-sg.quadhost.net
  • tai2-sg is unresponsive.

    Awaiting confirmation from local NoC regarding state & issue.

    Update - Hypervisor now responding.

  • Date - 30/12/2017 00:57 - 31/12/2017 01:50
  • Last Updated - 31/12/2017 02:14
Packetloss (Resolved)
  • Priority - High
  • Affecting System - rbx1-fr
  • Packetloss noted to rbx1-fr.quadhost.net, pending response from OVH regarding losses.

    RBX links flapped, OVH is manually rebuilding routing configuration due to error on Cisco platform side.

    Issue can be seen on OVH tracker here.

  • Date - 06/12/2017 21:20 - 06/12/2017 21:52
  • Last Updated - 06/12/2017 21:30
Packetloss (Resolved)
  • Priority - High
  • Affecting System - gra1-fr
  • Packetloss noted to gra1-fr.quadhost.net, pending response from OVH regarding losses.

    Loss seen from all networks outside of Asia & London, appears to be potentially linked to optical flap however waiting for confirmation from OVH NoC.

    Update #1 - Loss level has reduced however routing is less than optimal, mitigation capabilities are also suspected to be affected at this time.

  • Date - 06/12/2017 19:40 - 06/12/2017 21:52
  • Last Updated - 06/12/2017 21:29
Hypervisor Packetloss (Resolved)
  • Priority - Critical
  • Affecting System - ndk2-nl
  • ndk2-nl is experiencing packetloss, appears to be the upstream network encountering issues.

    Local NoC have been made aware, awaiting resolution from them ASAP.

    Update #1 - Packetloss has subsided.


    RFO Update - one of the 100G modules on upstream side was broken, module replaced and connectivity restored.

  • Date - 03/12/2017 18:08 - 03/12/2017 18:33
  • Last Updated - 03/12/2017 19:02
Hypervisor Unresponsive (Resolved)
  • Priority - Critical
  • Affecting System - ldn10-uk
  • ldn10-uk is unresponsive.

    Awaiting local NoC update regarding issue.

    Update #1 -Local NoC have confirmed the hypervisor had locked and it is now restarting.

    Virtual Servers will be back online shortly, apologies for any inconvience caused.

  • Date - 02/12/2017 14:07 - 02/12/2017 18:05
  • Last Updated - 03/12/2017 18:26
Account Verification (Resolved)
  • Priority - Medium
  • Affecting System - SMS Delivery to US/Canada
  • SMS based Verification account verification alerts are currently being delayed or failing when sent to a number of North American based carriers.

    Our upstream vendor is working to resolve the issue as soon as possible.

  • Date - 31/10/2017 10:00 - 01/11/2017 10:13
  • Last Updated - 02/12/2017 14:51
Hypervisor Unresponsive (Resolved)
  • Priority - Critical
  • Affecting System - bhs1-ca
  • bhs1-ca is unresponsive and are any other IP's in the same segment.

    OVH monitoring confirms server is online, looks to be a connection/routing issue in BHS.

    Awaiting confirmation from OVH of issue.

    No confirmation from OVH on issue, hypervisor is now responding.

  • Date - 01/12/2017 11:18 - 01/12/2017 11:58
  • Last Updated - 01/12/2017 11:59
Hypervisor Unresponsive (Resolved)
  • Priority - Critical
  • Affecting System - ldn11-uk
  • ldn11-uk has become unresponsive over standard network and IPMI.

    Update - Appears a bug in the UEFI has caused machine to hang following a panic.

    Update #2 - Machine is now restarting.


    Apologies for any inconvience caused during this time.

  • Date - 28/11/2017 00:33 - 28/11/2017 04:52
  • Last Updated - 28/11/2017 09:52
Hypervisor Unresponsive (Resolved)
  • Priority - Critical
  • Affecting Other - ldn11-uk
  • ldn11-uk has become unresponsive over standard network and IPMI.

    Update - Appears a bug in the UEFI has caused machine to hang following a panic.

    Update #2 - Machine is now restarting.

    Update #3 - Hypervisor has restarted, NAT Networking is now coming back online.

    Apologies for any inconvience caused during this time.

  • Date - 26/11/2017 00:30 - 26/11/2017 09:44
  • Last Updated - 26/11/2017 09:44
Management Portal (Resolved)
  • Priority - Medium
  • Affecting System - ldn11-uk
  • ldn11-uk is currently encountering issues with the management portal.

    All instances located on ldn11-uk are currently online and operational however may show as offline within the management portal and interactions will fail at this time.

    Management portal developers are aware of the issue and working on a resolution as soon as possible.

    Update - Issue has been resolved.

  • Date - 18/11/2017 23:50 - 19/11/2017 09:37
  • Last Updated - 19/11/2017 09:38
Hypervisor Unresponsive (Resolved)
  • Priority - Critical
  • Affecting System - ldn11-uk
  • ldn11-uk has become unresponsive over standard network and IPMI.

    Update - Appears a bug in the UEFI has caused machine to hang following a panic.

    Update #2 - Machine is now restarting.

    Update #3 - Hypervisor has restarted, NAT Networking is now coming back online.

    Apologies for any inconvience caused during this time.

  • Date - 17/11/2017 22:58 - 18/11/2017 10:43
  • Last Updated - 18/11/2017 10:43
Network Loss (Resolved)
  • Priority - High
  • Affecting System - ak1-nz
  • ak1-nz's network is not responding, issue escalated to upstream vendor.

    Update - Upstream connectivity has been restoreed.

  • Date - 08/11/2017 08:14 - 08/11/2017 09:43
  • Last Updated - 08/11/2017 09:54
Hypervisor Unresponsive (Resolved)
  • Priority - Critical
  • Affecting System - ak1-nz
  • ak1-nz is currently unresponsive.

    Local NoC are investigating at this time.

    Apologies for any inconvience caused.

    Update: Service is now restored.

  • Date - 31/10/2017 08:56 - 31/10/2017 10:50
  • Last Updated - 31/10/2017 10:50
Shared IP Nulled (Resolved)
  • Priority - High
  • Affecting System - nc7-us
  • NAT Shared IP #1 in use on nc7-us has been nulled to protect other customers on the hypervisor.

    Update - Once attack level has subsided the null will be lifted by our upstream vendor.

    Update #2 - Awaiting further confirmation the attack has subsided from our upstream.

    Update #3 - Null has been lifted.

  • Date - 29/09/2017 19:13 - 01/10/2017 09:56
  • Last Updated - 02/10/2017 02:57
Hypervisor Unresponsive (Resolved)
  • Priority - Critical
  • Affecting System - nc7-us
  • nc7-us is unresponsive on public network and IPMI, local NoC are investigating.

    Updates to be provided as soon as possible.

  • Date - 01/10/2017 09:54 - 02/10/2017 01:14
  • Last Updated - 02/10/2017 02:57
Shared IPv4 Nulled (Resolved)
  • Priority - Critical
  • Affecting System - nc7-us (NAT)
  • NAT Shared IPv4 #2 is currently nulled by our upstream due to repeated attacks against the IP.

    Our upstream will lift the null once the attack has subsided.

    Apologies for any inconvience caused during this time.

  • Date - 15/09/2017 00:00 - 15/09/2017 19:27
  • Last Updated - 15/09/2017 19:38
Outage (Resolved)
  • Priority - Critical
  • Affecting System - ndk2-nl
  • ndk2-nl is not responsive, awaiting local NoC investigation.

    Apologies for any inconvience caused during this time.

  • Date - 12/09/2017 22:43 - 13/09/2017 08:50
  • Last Updated - 13/09/2017 13:34
Outage (Resolved)
  • Priority - Critical
  • Affecting System - ak1-nz
  • ak1-nz is not responding remotely, NoC are currently investigating.

    Update: Service has been restored, apologies for any inconvience caused during this outage.

  • Date - 01/09/2017 22:23 - 01/09/2017 22:43
  • Last Updated - 01/09/2017 22:43
NAT Shared IP Nulled (Resolved)
  • Priority - High
  • Affecting System - nc7-us
  • NAT Shared IPv4 has been nulled due to an inbound attack at the carrier level, null will be lifted once attack subsides.

  • Date - 31/08/2017 19:10 - 31/08/2017 20:03
  • Last Updated - 31/08/2017 22:05
Shared IPv4 NAT Address Nulled (Resolved)
  • Priority - High
  • Affecting System - nc7-us.quadhost.net
  • The NAT based solutions shared IPv4 address for nc7-is has been null routed to protect the upstreams network at this time due to an inbound attack.

    The null route will be lifted as soon as possible.

    Apologies for any inconvenience caused during this outage.

  • Date - 22/08/2017 19:00 - 23/08/2017 11:25
  • Last Updated - 23/08/2017 11:55
Outage (Resolved)
  • Priority - Critical
  • Affecting System - tai-sg
  • tai-sg is unresponsive over standard network and IPMI, awaiting confirmation from local NoC on issue.

    Update - Service restored.

  • Date - 21/08/2017 16:55 - 21/08/2017 17:11
  • Last Updated - 21/08/2017 17:17
IPv6 Outage (Resolved)
  • Priority - Critical
  • Affecting System - tai-sg
  • IPv6 routes to SG are currently being dropped by upstream, awaiting their response and confirmation of the issue.

    Partial IPv6 connectivity has been restored, pushing local upstreams for RFO and cause.

  • Date - 02/08/2017 14:00
  • Last Updated - 12/08/2017 16:56
Coinify Payments (Resolved)
  • Priority - Low
  • Affecting System - Payment Gateway
  • Customers who typically pay via Coinify are advised to switch to BitPay.

    Coinify have ceased all operations at the time being until the upcoming BTC fork.

    Please see https://www.coinify.com/status for further details.

    Coinify has now been re-activated.

  • Date - 01/08/2017 15:00 - 03/08/2017 09:50
  • Last Updated - 12/08/2017 16:54
Hypervisors Not Responding (Resolved)
  • Priority - Critical
  • Affecting System - nc6-us/nc8-us
  • nc6-us/nc8-us are not responding at this time, our technicans are currently investigating the issue within the datacentre.

  • Date - 14/07/2017 14:15 - 14/07/2017 16:12
  • Last Updated - 23/07/2017 14:12
IPv4 Nullroute (Resolved)
  • Priority - Critical
  • Affecting System - nc7-us (Shared IP)
  • NAT Shared IPv4 #2 for nc7-us is currently nulled to protect the hypervisor, we anticipate the null to be removed by our upstream within 24hrs.

  • Date - 05/07/2017 17:33 - 10/07/2017 17:58
  • Last Updated - 10/07/2017 17:58
Outage (Resolved)
  • Priority - Critical
  • Affecting System - ak1-nz
  • ak1-nz is not responding.

    Our technicians are working to resolve the issue, appologies for any inconvience caused during this time.

    Update #1: ak1-nz is now responding.

  • Date - 26/06/2017 05:25
  • Last Updated - 04/07/2017 12:04
DDoS Attack (Resolved)
  • Priority - Critical
  • Affecting System - pun1-in
  • DDoS attack in progress against the subnet gateway for pun1-in.

    Update: Still awaiting upstream confirmation of attack subsidance for the null to be lifted.

    Update #2: We are working with the upstream connectivity provider to re-access options available for re-connection.

    Update #3: Due to the continued attacks towards India our upstream has since terminated the agreement, services will be migrated to London to allow customer data collection and to offer additional protection against the prolonged attacks.

  • Date - 13/06/2017 13:59 - 03/07/2017 00:00
  • Last Updated - 04/07/2017 11:51
Kernel Panic (Resolved)
  • Priority - Critical
  • Affecting System - ldn16-uk
  • ldn16-uk has encountered a kernel panic, hypervisor now back online with NAT network coming back online shortly.

  • Date - 07/06/2017 23:00 - 07/06/2017 23:07
  • Last Updated - 07/06/2017 23:08
DDoS Attack (Resolved)
  • Priority - Critical
  • Affecting System - pun1-in
  • DDoS attack towards pun1-in in progress.

  • Date - 31/05/2017 11:41 - 01/06/2017 12:47
  • Last Updated - 31/05/2017 13:42
Outage (Resolved)
  • Priority - Critical
  • Affecting System - ndk2-nl
  • ndk2-nl is unresponsive, local NoC are investigating.

  • Date - 29/05/2017 12:40 - 29/05/2017 13:14
  • Last Updated - 29/05/2017 12:46
DDoS Attack (Resolved)
  • Priority - Critical
  • Affecting System - tai-sg
  • DDoS attack in progress against tai-sg core network services causing packetloss.

  • Date - 19/05/2017 23:52 - 26/05/2017 15:02
  • Last Updated - 20/05/2017 00:35
pun1-in (Resolved)
  • Priority - Critical
  • Affecting System - DDoS Attack
  • pun1-in is under a large scale DDoS attack causing packetloss.

  • Date - 18/05/2017 05:08 - 18/05/2017 06:19
  • Last Updated - 18/05/2017 05:09
DDoS Attack (Resolved)
  • Priority - Critical
  • Affecting System - tai-sg
  • Large scale DDoS attack is currently flooding ranges in SG, nullroutes being applied where required.

  • Date - 13/05/2017 21:20 - 14/05/2017 09:20
  • Last Updated - 14/05/2017 10:05
FLK-DE (Resolved)
  • Priority - Critical
  • Affecting Other - flk-de Systems Outage
  • Systems connection loss detected in FLK, DE.

    Currently investigating, issues being resolved with Hetzner.

    Update - Our technicians are still working with Hetzner to resolve the remaining issues.

    Update #2 -
    Once again apologies for any inconvience caused during this outage. 

    Data is secure at this time, once the issue with Hetzner is resolved SLA credits/extensions will be made where applicable.

    ----

    Update #3 - 

    Unfortunately after many discussions with Hetzner where possible, regarding the impact to our services and repeated issues originating from a small group of malicious signups it is clear that moving forward we will not be staying in Falkenstein.

    During the first talks with Hetzner we have since discussed the proposal to move to other upstreams and locations in DE, however at this stage the avenues explored are not a suitable fit, nor provide for some of the requirements as to why we utilised Hetzner.

    Whilst these discussions have caused delay whilst being explored any applicable services will now be re-instated in London, UK unless otherwise requested.

    1. Existing Users Requiring Data
    Data is still secure at this stage and has not been seized/destroyed throughout the abuse investigation.

    We have limited access to the drives and data will be restored to our London UK servers starting immediately where it can be recovered or your service can be migrated and continue.

    2. Existing Users Not Requiring Data
    Please contact the migrations team and request your instance be re-created in the form of a new server in a suitable location, if no location is specified our team will create a new instance in London, UK.

    Compensation/SLA
    For services with an SLA (Standard Plans/Packages) in place that have not already been migrated/credited accordingly before this announcement please ensure you have no open tickets pending resolution and open a ticket with the Billing Team to resolve.

    Services without any SLA (Typically i-83.net Services) will be issued a credit for the time period lost or a service extension for the product as a goodwill gesture.

    FAQ's

    Why London?
    It allows us increased flexibility in the hardware and network operated, and reduces the impact of an event like this reoccurring whilst being a short hop away from DE.
    - If London is not suitable for a destination then please open a ticket with the migrations team.

    What about other location
    At this time there are no plans for any other locations to be altered or adjusted, all other PoP's are operating typically.

    What have QuadHost Ltd. learnt from this specific outage?
    The impact of this outage to i-83 customers has been noted, with attention to the handling and communication from staff during this time. To improve and resolve any issue handling moving forward we have changed our focus and plans for 2017; with our support team retraining with a staff expansion and overall platform changes that are now on our roadmap.

    If you have any further questions or queries about our DE location and affected plans please open a ticket with the support team.

    Update #4 - Data Restoration
    Instances that are not already re-created are now being migrated to London and restored.

    Update #5 - Restoration Complete
    All Instances have now been migrated to London and are listed under the management panel. Restored services will have previous login credentials, re-created services will use the details in the welcome e-mail.

  • Date - 14/03/2017 09:10 - 08/05/2017 21:00
  • Last Updated - 08/05/2017 21:02
OVH Intervention (Resolved)
  • Priority - Critical
  • Affecting System - rbx1-fr
  • OVH Monitoring noticed a fault on the server and we have scheduled an intervention in order to fix this fault, updates to be relayed as soon as possible.

    Update #1
    Issue has been escalated to 2nd Level OVH Technicans now due to failed response from NoC over the bank holiday weekend, resolution anticipated shortly.

    Update #2
    Local NoC has confirmed machine is now rebooting back to our OS and configuration after hardware tests have now completed without error.

  • Date - 29/04/2017 09:47 - 04/05/2017 17:47
  • Last Updated - 04/05/2017 17:47
DDoS Attack (Resolved)
  • Priority - Critical
  • Affecting System - pun1-in
  • pun1-in is unresponsive, local NoC are investigating the cause.

    Another rolling DDoS Attack targeting the datacenter causing nullroutes to be placed via upstreams.

    Upstream is currently working to deploy mitigation where possible despite the location not being protected, following their deployment our own subnets are now unresponsive, awaiting for resolution on upstreams side.

    Update #1

    Datacenter is still under sustained DDoS attack, awaiting confirmation from mitigation provider on the new approach for resolution from local NoC.

  • Date - 27/04/2017 23:12 - 03/05/2017 10:46
  • Last Updated - 04/05/2017 17:46
Kernel Panic / fsck (Resolved)
  • Priority - Critical
  • Affecting System - ldn16-uk
  • fsck underway on ldn16-uk due to a kernel panic this morning.

  • Date - 29/04/2017 10:00 - 02/05/2017 17:30
  • Last Updated - 02/05/2017 17:40
DDoS Attack (Resolved)
  • Priority - Critical
  • Affecting System - pun-in,ldn-uk,ndk-nl
  • A large DDoS attack is currently impacting various IP blocks in India, London and NDK.

    At this time nullroutes are being placed for short periods on the main targets of the attacks however they are rolling through ranges causing disruption to customers within that space.

  • Date - 22/04/2017 17:00 - 27/04/2017 22:00
  • Last Updated - 28/04/2017 00:13
Shared IPv4 Null (Resolved)
  • Priority - Critical
  • Affecting System - nc7-us
  • Shared IPv4 Null in place on nc7-us due to rolling attack, null will be lifted at the earliest convenience.

  • Date - 03/04/2017 18:32 - 04/04/2017 16:36
  • Last Updated - 04/04/2017 18:46
Network Issues (Resolved)
  • Priority - Critical
  • Affecting System - pun1-in
  • pun1-in is unresponsive, awaiting local NoC confirmation.

  • Date - 04/04/2017 10:37 - 04/04/2017 14:02
  • Last Updated - 04/04/2017 14:02
Outage (Resolved)
  • Priority - Critical
  • Affecting System - ndk2-nl
  • ndk2-nl is no longer responding, confirming hardware status with local NoC.

    Update - Connectivity is now restored.

  • Date - 29/03/2017 10:05 - 29/03/2017 11:00
  • Last Updated - 29/03/2017 11:01
Shared IPv4 Nulled (Resolved)
  • Priority - High
  • Affecting System - nc7-us
  • Shared IP #2 has been nulled upstream due to abuse, working with upstream to resolve ASAP.

  • Date - 16/03/2017 15:53 - 16/03/2017 17:33
  • Last Updated - 17/03/2017 03:12
IPv6 DDoS Attack (Resolved)
  • Priority - Critical
  • Affecting System - London
  • Certain IPv6 blocks in London have been temporarily nulled due to a large scale DDoS attack against the blocks.

    Apologies for any inconvience caused by this null.

    Update - Null removed.

  • Date - 14/03/2017 21:37 - 15/03/2017 00:00
  • Last Updated - 15/03/2017 01:24
ak1-nz (Resolved)
  • Priority - Critical
  • Affecting System - Outage
  • ak1-nz is non responsive, issue has been isolated and service will be restored shortly.

    Update - Service restored.

  • Date - 08/03/2017 19:01 - 08/03/2017 19:39
  • Last Updated - 08/03/2017 19:39
Outage (Resolved)
  • Priority - Critical
  • Affecting System - ak1-nz
  • ak1-nz is currently showing network drops from multiple locations.

    Local NoC are aware and are investigating the issue - Additional details to follow as investigation underway.

    Update - Network back up, pending full RFO from network provider.

  • Date - 22/02/2017 19:43 - 22/02/2017 22:19
  • Last Updated - 22/02/2017 20:22
Switch Failure (Resolved)
  • Priority - Critical
  • Affecting System - LDN-UK-CORE2
  • Core switch 2 dropping packets, network failed over to core 1.

    NoC team investigating to bring core 1 online once issue isolated.

    Update - Network back online.

  • Date - 20/02/2017 12:06 - 20/02/2017 12:10
  • Last Updated - 20/02/2017 12:17
Licensing Issue (Resolved)
  • Priority - Critical
  • Due to licensing dispute issues at present certain modules are being re-issued/verified which may cause temporary licensing check errors.

  • Date - 16/02/2017 12:00 - 16/02/2017 15:39
  • Last Updated - 16/02/2017 14:08
Fibre Cut (Resolved)
  • Priority - Medium
  • Affecting System - ak1-nz
  • International traffic fibre links currently down/damaged causing routing losses for international traffic.

    Link repair already in progress and should be completed shortly.

    No local traffic affected.

  • Date - 13/02/2017 08:00 - 14/02/2017 11:03
  • Last Updated - 14/02/2017 20:37
Migration Intervention (Resolved)
  • Priority - High
  • Affecting System - ldn13-uk / ldn14-uk
  • Hardware event has caused ldn13-uk to go into protection mode, instances are being live switched to ldn14-uk, a small downtime may occur whilst storage catchup occurs prior to the instance rebooting.

    Apologies for any inconvience caused during this time. The average catchup period is ~3minutes at present.

    Protection control minimises downtime impact by actively montioring system status and migration any instances prior to circumstances further degrading causing a prolonged outage.

    Hardware event also flagged ldn14-uk shortly after causing a logic loop, system(s) are currently confirming chunk state and re-allocating.

    Protection control now disabled for the interim.

  • Date - 10/02/2017 12:02 - 14/02/2017 20:37
  • Last Updated - 11/02/2017 00:22
Outage (Resolved)
  • Priority - Critical
  • Affecting System - flk2-de
  • Service restored.

  • Date - 07/02/2017 15:21 - 07/02/2017 15:08
  • Last Updated - 07/02/2017 20:15
Outage (Resolved)
  • Priority - Critical
  • Affecting Other - pun1-in
  • pun1-in is no longer responding via ICMP/SSH nor is IPMI up, appears to be facility related.

    Awaiting response from local NoC.

  • Date - 31/01/2017 12:39 - 07/02/2017 15:32
  • Last Updated - 07/02/2017 15:32
RAID Rebuild (Resolved)
  • Priority - Medium
  • Affecting System - ndk1-nl
  • RAID rebuild commenced on ndk1-nl, disk/IO performance may be degraded during peak times during the rebuild process.

    No data has been lost nor system outages caused due to the disk dropping from the array.

    Update - Rebuild still ongoing.

    Update #2 - Failure detected; kernel panic output, machine now rebooting.

    Update #3 - 2nd Panic has occured, system intervention now underway.

    Update #4 - Rebuild completed.

  • Date - 14/01/2017 09:01 - 26/01/2017 09:35
  • Last Updated - 26/01/2017 17:05
RAID Rebuild (Resolved)
  • Priority - Medium
  • Affecting System - nc7-us
  • RAID rebuild commenced on nc7-us, disk/IO performance may be degraded during peak times during the rebuild process.

    No data has been lost nor system outages caused due to the disk dropping from the array.

  • Date - 15/01/2017 10:29 - 17/01/2017 19:46
  • Last Updated - 18/01/2017 15:54
RAID Rebuild (Resolved)
  • Priority - Medium
  • Affecting System - nc7-us
  • RAID rebuild commenced on nc7-us, disk/IO performance may be degraded during peak times during the rebuild process.

    No data has been lost nor system outages caused due to the disk dropping from the array.

    Update - Rebuild still ongoing.

  • Date - 08/01/2017 00:03 - 13/01/2017 17:26
  • Last Updated - 15/01/2017 13:27
Main Site Migration (Resolved)
  • Priority - High
  • Affecting System - Main Website
  • Main site to be migrated to new servers, some visitors may experience a drop in access whilst the transition takes place.

    Scheduled to commence at 22:00 on 11/01/2017, estimate completion time: 02:00 on 12/01/2017.

  • Date - 11/01/2017 22:00 - 12/01/2017 02:00
  • Last Updated - 12/01/2017 02:08
IPv6 Connectivity (Resolved)
  • Priority - Critical
  • Affecting System - rbx1-fr
  • IPv6 connectivity has dropped beyond the gateway on rbx1-fr.quadhost.net , issue currently with OVH for investigation on their side.

  • Date - 06/01/2017 16:30 - 06/01/2017 17:22
  • Last Updated - 09/01/2017 23:04
RAID Rebuild (Resolved)
  • Priority - Medium
  • Affecting System - var3-bg
  • RAID rebuild commenced on var3-bg, disk/IO performance may be degraded during peak times during the rebuild process.

    No data has been lost nor system outages caused due to the disk dropping from the array.

  • Date - 25/12/2016 04:13 - 28/12/2016 03:11
  • Last Updated - 06/01/2017 16:56
Outage (Resolved)
  • Priority - Critical
  • Affecting System - tai2-sg
  • tai2-sg is not responding over SSH nor via IPMI, local NoC now investigating, updates to follow.

    Update #1 - tai2-sg is now back online.

    Update #2 - both sides of the network offline again, local NoC investigating.

    Update #3 - network now coming back up.

  • Date - 04/01/2017 15:04 - 04/01/2017 16:32
  • Last Updated - 06/01/2017 16:56
Outage (Resolved)
  • Priority - Critical
  • Affecting System - var3-bg
  • var3 is unresponsive, passed to local NoC for hardware testing.

    var3 is back online.

  • Date - 01/01/2017 02:42 - 01/01/2017 09:47
  • Last Updated - 01/01/2017 09:47
Nulled IPv4 (Resolved)
  • Priority - High
  • Affecting System - flk2-de
  • flk2-de IPv4 has been nulled as a reactive measure by our upstream, the null will be lifted shortly.

    Null now lifted.

  • Date - 29/12/2016 06:44 - 29/12/2016 13:51
  • Last Updated - 29/12/2016 13:51
IPv6 Session Drop (Resolved)
  • Priority - Critical
  • Affecting System - pun1-in
  • IPv6 BGP Session has dropped on upstream side, awaiting their action to bring back up.

  • Date - 20/12/2016 12:28 - 21/12/2016 01:16
  • Last Updated - 21/12/2016 01:16
bhs1-ca (Resolved)
  • Priority - Critical
  • Affecting Other - Connectivity Issues
  • bhs1-ca is unresponsive, issue passed to local OVH NoC for resolution.

  • Date - 20/12/2016 23:20 - 20/12/2016 23:42
  • Last Updated - 20/12/2016 23:48
Management Interface (Resolved)
  • Priority - Medium
  • Affecting System - pun1-in
  • Management portal interface is currently offline for instances housed on pun1-in.

    Awaiting intervention from panel development team to resolve.

  • Date - 20/12/2016 20:00 - 20/12/2016 20:41
  • Last Updated - 20/12/2016 22:41
Nulled Shared IP (Resolved)
  • Priority - Critical
  • Affecting System - nc7-us
  • One of the shared IPv4 has been nulled upstream.

    Nullroute to be lifted shortly.

  • Date - 19/12/2016 12:45 - 20/12/2016 22:02
  • Last Updated - 20/12/2016 22:02
Outage (Resolved)
  • Priority - Critical
  • Affecting System - pun1-in
  • pun1-in is no longer responsive, local NoC are investigating.

  • Date - 16/12/2016 09:24 - 17/12/2016 18:37
  • Last Updated - 17/12/2016 18:37
Nullroute (Resolved)
  • Priority - Critical
  • Affecting System - nc7-us
  • One of the shared IP's on nc7-us has been temporarily nulled, once the issue is resolved the null will be lifted.

  • Date - 12/12/2016 19:54 - 16/12/2016 09:24
  • Last Updated - 12/12/2016 19:55
Outage (Resolved)
  • Priority - Critical
  • Affecting System - flk2-de
  • flk2-de is no longer responding, local NoC are investigationg.

    Update connectivity has been restored.

  • Date - 07/12/2016 09:09 - 07/12/2016 11:50
  • Last Updated - 07/12/2016 11:50
Outage (Resolved)
  • Priority - Critical
  • Affecting System - tai-sg
  • Connectivity to hypervisors in tai-sg has been impacted, investigation by local NoC has begun.

    Update - Connectivity Restored

  • Date - 06/12/2016 12:48 - 06/12/2016 14:23
  • Last Updated - 06/12/2016 14:23
Outage (Resolved)
  • Priority - Critical
  • Affecting System - flk2-de
  • flk2-de is no longer responding via network interfaces, investigating.

  • Date - 05/12/2016 13:41
  • Last Updated - 05/12/2016 17:35
rDNS Cluster Upgrade (Resolved)
  • Priority - Medium
  • Affecting System - rdns1 / rdns2 / rdns3
  • Commencing 10:00 GMT 3rd Dec 2016 we will be starting to migrate our reverse DNS cluster to our new version, commencing as follows: rdns2, rdns3 & rdns1.

    No downtime is expected due to the failover nature of the solution.

    Update - rDNS cluster now upgraded.

  • Date - 03/12/2016 10:00 - 03/12/2016 16:07
  • Last Updated - 03/12/2016 16:07
Packet Loss (Resolved)
  • Priority - High
  • Affecting Other - tai1-sg.quadhost.net
  • Packetloss seen to tai1-sg, currently investigating.

    Packetloss resolved.

  • Date - 26/11/2016 17:04 - 28/11/2016 13:29
  • Last Updated - 28/11/2016 13:30
System Stability Issue (Resolved)
  • Priority - Critical
  • Affecting System - ak1-nz
  • ak1-nz is not currently stable, having kernel panic'd repeatedly.

    New orders have been held and we are awaiting further responses from local upstreams regarding a timeframe for resolution.

    NAT containers held on oldak1-nz is being migrated to ak1-nz immediately to prevent future outages in Auckland location.

    Apologies for any incovience this may cause, we are taking the approach to migrate at the earliest convience to prevent any future unexpected downtime.

  • Date - 01/11/2016 00:00 - 25/11/2016 00:00
  • Last Updated - 26/11/2016 19:15
London IPv6 (Resolved)
  • Priority - Critical
  • Affecting Other - IPv6 Connectivity Issues
  • IPv6 Connectivity in London is currently degraded with connectivity issue.

    Investigation ongoing.

    Update - IPv6 connectivity is now restored.

  • Date - 22/11/2016 23:21 - 23/11/2016 01:30
  • Last Updated - 23/11/2016 01:33
NAT Network Changes (Resolved)
  • Priority - High
  • Affecting System - ndk1-nl
  • ndk1-nl is undergoing IP Space changes at this time to due issues with our upstreams IP Space.

    109.236.83.202 is no longer in use and has been swapped to 185.169.235.5.

    Individual IPv6 addresses have also been nulled and will be replaced with the new /64 block visible in your client area.

    If you have any issues feel free to open a ticket with our technical support team to resolve your issue.

  • Date - 21/11/2016 00:00 - 21/11/2016 23:59
  • Last Updated - 22/11/2016 23:34
RAID Rebuild (Resolved)
  • Priority - Medium
  • Affecting System - nc7-us.quadhost.net
  • One of the disks in the storage layer on nc7-us showed as degraded, disk has been replaced and RAID rebuild is now ongoing.

    I/O Performance will be degraded until rebuild is completed.

  • Date - 20/11/2016 13:45 - 21/11/2016 18:09
  • Last Updated - 22/11/2016 00:27
RAM Failure (Resolved)
  • Priority - Critical
  • Affecting System - ndk1-nl
  • ndk1-nl showing RAM failure after hardlock, local NoC to replace RAM, awaiting confirmation from them prior to powering the hypervisor back online.

  • Date - 21/11/2016 19:10 - 21/11/2016 20:11
  • Last Updated - 21/11/2016 21:35
Hypervisor not responding (Resolved)
  • Priority - Critical
  • Affecting System - ak1-nz
  • Hypervisor not responding, escalating to local upstream for resolution.

  • Date - 28/10/2016 22:07
  • Last Updated - 12/11/2016 16:21
Node Outage (Resolved)
  • Priority - Critical
  • Affecting System - rei1-fr
  • rei1-fr is down over both ethernet uplinks.

    Local NoC has been requested to check the machine, updates to follow.

    Update #1 - Local NoC has aknowledged an issue.

    Update #2 - Local NoC is failing to work with us to find a resolution, contegency plans are being rolled out

    Update #3 - All active users should now have a open a ticket with the support department to confirm how they wish to proceed.

    It goes without saying, this is not the level of service typically provided nor the level we wish to do so - changes will be put in place at the new location. Credit/Compensation will be provided to each customer once the initial issue is resolved, for further information please see open ticket(s).

  • Date - 30/09/2016 10:15 - 30/10/2016 00:00
  • Last Updated - 28/10/2016 22:22
Degraded I/O Performance (Resolved)
  • Priority - Critical
  • Affecting System - rbx1-fr
  • Degraded Disk I/O spotted on a number of instances, caused by a local RAID storage system rebuild.

  • Date - 10/10/2016 21:44
  • Last Updated - 28/10/2016 22:21
Packetloss (Resolved)
  • Priority - Critical
  • Affecting System - flk2-de
  • Packetloss seen to flk2-de, investigation by hetzner in progress.

    Currently fits the same traffic pattern as the attack seen to OVH.

    Traffic volume subsided, still pending response from Hetzner.

    Attack seen once again.

  • Date - 15/10/2016 21:47 - 17/10/2016 20:08
  • Last Updated - 28/10/2016 22:21
Hypervisor Issues (Resolved)
  • Priority - Critical
  • Affecting System - pun1-in
  • Issues with pun1-in have been detected, issue is being investigated.

  • Date - 25/10/2016 20:14 - 25/10/2016 20:34
  • Last Updated - 28/10/2016 22:21
Fault Detected (Resolved)
  • Priority - High
  • Affecting System - gra1-fr
  • Our internal monitoring and OVH Monitoring has detected a fault with connectivity to gra1-fr.

    OVH staff are to perform an intervention shortly, updates to follow.


    Large attack has bypassed VAC and is currently attacking gra1-fr, OVH working to resolve.

  • Date - 15/10/2016 21:41 - 15/10/2016 22:43
  • Last Updated - 15/10/2016 22:43
System Outage (Resolved)
  • Priority - Critical
  • Affecting System - flk2-de
  • flk2-de is non responsive again, issue esclated to Hetzner pending response.

    New hypervisor is pending delivery for planned migrations, awaiting diagnosis of existing server from local NoC.

    Update - Hetzner is carrying out a hardware intervention on the hypervisor.
    Update #2 - Intervention complete, instances booting up now and RAID rebuild commencing.
    Update #3 - Box network unresponsive again as of 9AM BST.
    Update #4 - Box stable.
    Update #5 - Box showing intermitted packetloss once again.

  • Date - 09/10/2016 22:18 - 10/10/2016 21:46
  • Last Updated - 11/10/2016 08:58
Packetloss (Resolved)
  • Priority - Critical
  • Affecting System - flk2-de
  • Packetloss seen to flk2-de once again.

    Esclated to Hetzner for investigation.

    Hetzner have confirmed hardware failure in addition to seeing the intermittent packetloss.

    Replacement hypervisor being deployed shortly in addition to a maintenance window being scheduled, for additional information please contact us.

  • Date - 06/10/2016 10:40 - 09/10/2016 22:44
  • Last Updated - 09/10/2016 22:42
Packetloss (Resolved)
  • Priority - High
  • Affecting Other - flk2-de
  • Packetloss seen to flk2-de.

    Under investigation by the local NoC.

    Update #1 - Connectivity appears to be restored.
    Update #2 - Packetloss returned
    Update #3 - Packetloss subsided
    Update #4 - Packetloss returned
    Update #5 - System restored by Hetzner, suspected powerloss/failure - awaiting their response.
    Update #6 - Packetloss returned, Hetzner investigating hardware issue.
    Update #7 - System hardlock reoccured
    Update #8 - Hetzner NoC doing RAM replacement and BIOS update, system now restarting.
    Update #9 - Node is online, network operational and VM's starting.

  • Date - 02/10/2016 17:05 - 04/10/2016 17:12
  • Last Updated - 04/10/2016 17:13
RAID Rebuild (Resolved)
  • Priority - High
  • Affecting System - flk2-de
  • flk2-de is reporting degrading of the primary storage array, disk I/O will be impacted whilst a RAID rebuild is carried out.

    Apologies for any inconvience caused.

    Update - Rebuild still in progress.

    Update #2 - RAID rebuild completed.

    Update #3 - On rebuild completion another disk is reporting a failure - full diagnostic commenced whilst arranging a drive swap.

    Update #4 - Swap will include hardware change, reboot of hypervisor inevitable. Array (Data) is still secure/safe - no cause for concern.

    Update #5 - Hardware swapped, Hypervisor rebooting and instances being powered up.

    Update #6 - Hypervisor booted, instances back online, rebuild re-commencing.

  • Date - 27/09/2016 22:05 - 03/10/2016 14:05
  • Last Updated - 04/10/2016 16:25
Connectivity Issues (Resolved)
  • Priority - Critical
  • Affecting Other - nc7-us
  • nc7-us has been nulled temporarily by our upstream due to a large attack.

    We are working with them to restore service as soon as possible.

    Update #1 - Inbound attacks over IPv6 still ongoing

    Update #2 - Connectivity Restored

  • Date - 01/10/2016 05:40 - 02/10/2016 17:15
  • Last Updated - 02/10/2016 17:15
Packetloss (Resolved)
  • Priority - High
  • Affecting System - nc7-us.quadhost.net
  • Packetloss seen to nc7-us, NoC investigating now.

    Normal network funcationality is now restored.

  • Date - 30/09/2016 18:45 - 30/09/2016 19:44
  • Last Updated - 30/09/2016 19:44
Outage (Resolved)
  • Priority - Critical
  • Affecting System - flk2-de
  • flk2-de network is no longer responsive, awaiting LARA application for diagnosis.

    Update - Lara/Lantronix shows no output from the machine, hardware diagnosis by local NoC now underway.

    Update #2 - Hypervisor now cleared hardware checks and is rebooting, containers and network to start slowly once machine is online.

    Update #3 - VM's are now back online.

  • Date - 29/09/2016 23:00 - 30/09/2016 01:13
  • Last Updated - 30/09/2016 01:13
SSL Certificate Renewal (Resolved)
  • Priority - High
  • Affecting System - i-83.net Microsite
  • i-83.net may be breifly unavailable or prompt certificate errors at approx 14:15 BST on 29/09/2016 whilst we revoke the existing SSL and migrate to a new certificate.

    Update - Cert now authorised and in place, please do a hard refresh if you still encounter errors.

  • Date - 29/09/2016 14:15 - 29/09/2016 14:30
  • Last Updated - 29/09/2016 14:31
DDoS Attack (Resolved)
  • Priority - Critical
  • Affecting System - nc7/flk2
  • A number of our NAT hypervisors are under a co-oirdinated DDoS attack.

    Awaiting mitigation where possible else IP's will be nulled as a short term measure.

  • Date - 23/09/2016 14:30 - 23/09/2016 18:50
  • Last Updated - 23/09/2016 18:51
High Loads/System Degradation (Resolved)
  • Priority - Critical
  • Affecting System - rbx1-fr
  • rbx1 is under a large sustained DDoS attack causing increased load and network connectivity issues.

    Working with our upstream to mitigate the impact and resolve the instability as soon as possible.

  • Date - 23/09/2016 01:51 - 23/09/2016 04:06
  • Last Updated - 23/09/2016 04:06
Service Degraded (Resolved)
  • Priority - High
  • Affecting System - ldn6-uk
  • Our automatic system monitoring has detected a loss of performance on ldn6-uk.quadhost.net, in addition a hardware anomaly has been noted.

    Basic investigation has begun, updates to follow.

    Update: Due to the nature of the fault instances will be migrated to ldn11-uk, for additional details see the e-mails being sent to your registered system mail.

    Update #2: Migrations are now completed.

    Update #3: The Virtualizor development team are investigating a few cases of mis-assigned IP Addresses.

    Update #4: All configurations should now be re-aligned, please contact us if you are still encountering issues.

  • Date - 10/09/2016 20:08 - 12/09/2016 23:00
  • Last Updated - 12/09/2016 23:00
SBL Listings (Resolved)
  • Priority - Critical
  • Affecting Other - Spamhaus
  • We are aware Spamhaus have listed a number of our IPv4 blocks over a number of locations.

    This is an "escalation" from them for a specific set IPv6 blocks that are used by a customer.

    The blocks in question are no longer visible on our network and we have stopped announcing the space.

    We are awaiting updates from Spamhaus regarding moving forward and having our own and shared IP space delisted.

    We apologise to all affected clients for any inconvience caused by the recent actions of Spamhaus.

    https://www.spamhaus.org/sbl/listings/quadhost.net

  • Date - 11/09/2016 21:51 - 12/09/2016 17:28
  • Last Updated - 12/09/2016 17:52
System Fault Detected (Resolved)
  • Priority - Critical
  • Affecting System - bhs1-ca
  • bhs1-ca has reported a hardware fault.

    Local technicians are diagnosing the issue, updates to follow.

    Hypervisor had locked, local NoC have now powercycled, machine now booting.

    Hypervisor back online, all instances booting and NAT redirection reloaded.

  • Date - 05/09/2016 13:00 - 05/09/2016 13:47
  • Last Updated - 05/09/2016 13:47
Node Outage (Resolved)
  • Priority - Critical
  • Affecting System - flk2-de
  • Upstream connectivity on flk2-de has gone away, pending response as to why from local NoC.

    Connectivity has been restored.

  • Date - 31/08/2016 12:21 - 31/08/2016 13:45
  • Last Updated - 31/08/2016 14:03
Hypervisor Outage (Resolved)
  • Priority - Critical
  • Affecting System - nc7-us
  • nc7-us is no longer responding over ICMP nor via monitoring channels.

    Investigation has begun, updates to follow.

    Network connectivity from our upstream has been limited due to a outbound attack.

    Connectivity is now restored.

  • Date - 16/08/2016 18:40 - 16/08/2016 19:29
  • Last Updated - 16/08/2016 19:29
IPv6 Outage (Resolved)
  • Priority - Critical
  • Affecting System - pun1-in
  • IPv6 is no longer responding, investigating.

    Upstream connectivity has now been restored.

  • Date - 24/07/2016 21:30
  • Last Updated - 24/07/2016 22:02
Outage (Resolved)
  • Priority - Critical
  • Affecting System - var3-bg.quadhost.net
  • Hypervisor is no longer responding, local NoC is investigating, updates to follow.

    Service is restored.

  • Date - 21/07/2016 15:50 - 21/07/2016 16:16
  • Last Updated - 21/07/2016 16:39
Outage (Resolved)
  • Priority - Critical
  • Affecting System - tai1-sg.quadhost.net
  • System has gone offline, our local NoC is currently investigating.

    Service has been restored.

  • Date - 19/07/2016 17:11 - 19/07/2016 20:30
  • Last Updated - 21/07/2016 15:55
Planned Maintenance (Resolved)
  • Priority - Critical
  • Affecting System - tai1-sg
  • tai1-sg is scheduled for hardware maintenance at 9:30 BST (17:30 SGT).

    During the maintenance the hypervisor will be powercycled requiring all VM's to be gracefully shutdown and restarted once the system is back online.

    We apologise for any inconvience this may cause.

  • Date - 19/07/2016 10:30 - 19/07/2016 14:40
  • Last Updated - 19/07/2016 14:46
Hardware Fault Detected (Resolved)
  • Priority - Critical
  • Affecting System - rbx1-fr
  • Our monitoring system has just detected a fault on rbx1-fr.

    Our technicians are currently awaiting response from OVH regarding the issue.

    UPDATE - Manual file system check cleared, system booted and online, NAT networking is coming back up as VM's power up.

  • Date - 01/07/2016 16:59 - 01/07/2016 17:14
  • Last Updated - 02/07/2016 01:48
Ongoing Issues in India (IPv4 Connectivity) (Resolved)
  • Priority - Critical
  • Affecting System - del1-in
  • IPv4 connectivity for del1-in is currently impacted due to an issue upstream with AS55470 (all support channels have been contacted - awaiting their diagnosis and response).

    UPDATE - Main hypervisor access and NAT forwarding have been restored. BGP is still suspended.

    2nd UPDATE - We are withdrawing from our current provider in New Delhi, India due to the continued issues faced with the unstable network - please see e-mail sent to all active customers.

  • Date - 09/06/2016 00:31
  • Last Updated - 21/06/2016 03:03
DDoS Against Hypervisor (Resolved)
  • Priority - High
  • Affecting System - tai1-sg.quadhost.net
  • The shared/main hypervisor IP for tai1-sg has been nulled once again by our upstream due to a large inbound DDoS.

    Our upstream has confirmed that if these attacks continue they will be ceasing service - we are now looking at other options including moving the hypervisor to a DDoS Protected provider to prevent future issues.

    Updates to follow based on responses from upstream and options explored.

    UPDATE - Our upstream and ourselves are looking at DDoS Protection options moving forward.

  • Date - 02/06/2016 15:45 - 13/06/2016 23:30
  • Last Updated - 13/06/2016 23:31
DDoS Against Hypervisor (Resolved)
  • Priority - High
  • Affecting System - tai1-sg.quadhost.net
  • The shared IPv4 for SG based NAT instances is currently nulled by our upstream due to a large DDoS attack.

    The null will be lifted when the attack traffic has reduced.

    Update - Null has now been lifted.

  • Date - 29/05/2016 07:05 - 30/05/2016 14:23
  • Last Updated - 30/05/2016 14:41
DDoS Against Hypervisor (Resolved)
  • Priority - Critical
  • Affecting System - del1-in.quadhost.net
  • A DDoS attack has caused the upstream to null the connection to del1-in.

    Update - We are working with our upstream to re-connect the hypervisor shortly.
    2nd Update - Most of the null's are now lifted, remainders will stay in place until the continued attack drops.

  • Date - 24/05/2016 20:12 - 27/05/2016 07:37
  • Last Updated - 29/05/2016 10:23
Connectivity Issues (Resolved)
  • Priority - High
  • Affecting System - IPv6 Connectivity in London
  • Connectivity issues have been reported on IPv6 in London, UK.

    Our technicians are currently investigating, updates to follow.

    UPDATE

    16:40 - Upstream 1 (CoreIX) is having issues routing to certain subnets, network dropped from blend temporarily. Will be re-added to blend at a later date.

  • Date - 08/05/2016 16:08 - 08/05/2016 16:43
  • Last Updated - 08/05/2016 16:43
Connectivity Issues (Resolved)
  • Priority - High
  • Affecting System - del1-in
  • Connectivity to del1-in has degraded, investigation in progress.

    Updates to follow shortly.

    Resolved.

  • Date - 05/05/2016 04:37 - 05/05/2016 04:59
  • Last Updated - 05/05/2016 05:00
Server Nulled (DDoS) (Resolved)
  • Priority - Critical
  • Affecting System - tai1-sg
  • Server tai1-sg has been nulled by the local NoC team following a large DDoS attack.

    Once the attack subsides connectivity will be restored.

  • Date - 29/04/2016 08:48 - 29/04/2016 09:17
  • Last Updated - 29/04/2016 09:54
Control Panel Functionality (Resolved)
  • Priority - Medium
  • Affecting System - NAT/Budget Varna VPS Control Panel
  • Virtual Server Instances based on var1's legacy integrated control panel has been taken offline pending a platform migration.

    For any administrative or support assistance please open a ticket.

    Apologies for any inconvience.

    Update - New hypervisors have been added to our main platform and client instances are being recreated for migration.

  • Date - 04/01/2016 20:44 - 24/04/2016 00:00
  • Last Updated - 24/04/2016 01:56
Nullroute on flk2 (Resolved)
  • Priority - Critical
  • Affecting System - flk2-de.quadhost.net
  • Our upstream has locked our shared IP on flk2-de.quadhost.net due to a large number of outbound attacks from NAT instances that reside on the hypervisor.

    Updates:

    15:22 4th April 2016 - Our counter report has been filed and we are awaiting the nullroute to be lifted.
    15:31 4th April 2016 - Upstream has lifted nullroute.

  • Date - 04/04/2016 14:15 - 04/04/2016 15:31
  • Last Updated - 04/04/2016 15:31
Control Panel Functionality (Resolved)
  • Priority - Medium
  • Affecting System - ldn1-uk
  • We have temporarily disabled the control panel interface for instances hosted on ldn1-uk during urgent system updates.

    If you require any actions to be carried out on your instance in the interim feel free to open a support ticket, a member of the team will be happy to assist you.

  • Date - 28/03/2016 20:19 - 01/04/2016 18:03
  • Last Updated - 28/03/2016 20:21
LDN6-UK Outage (Resolved)
  • Priority - Critical
  • Affecting Other - London NAT Range
  • ldn6-uk appears to have been nulled by upstream - suspected down pending mitigation for a DDoS attack based on symptoms and network flow before the outage started.

    Investigation ongoing, updates to follow.

    Update - Connectivity is now restored, full RFO to be posted shortly following investigation and feedback from GTT.

  • Date - 16/03/2016 23:38 - 17/03/2016 02:29
  • Last Updated - 17/03/2016 02:44
rbx1-fr Outage (Resolved)
  • Priority - Critical
  • Affecting System - rbx1-fr
  • rbx1-fr.quadhost.net was put into restricted mode by the OVH Anti-Hack due to a outbound flood from a number of instances.

    The hypervisor was put into read-only mode by OVH, we have since resolved the issue with support and had the machines netboot removed, this did not go to plan due to an issue in the OVH panel.

    Support was then contacted and advised the machine would need to be powered down to load from our disks and boot.

    System Booted, all VM's will start to power on again over the next few minutes.

  • Date - 28/01/2016 18:30 - 28/01/2016 18:50
  • Last Updated - 28/01/2016 19:11
Hardware Relocation (Resolved)
  • Priority - High
  • Affecting System - ns1/rdns1 Migration
  • Our ns1 and rdns1 hardware is being relocated to our new facility in Miami.

    No actions are required on your part, ns2 & rdns2 will take over requests during this period.

    Update - rdns1 is back online and handling requests, we are bringing ns1 online next.

    Update #2 - both systems back online.

  • Date - 22/08/2015 20:55 - 23/08/2015 14:00
  • Last Updated - 23/08/2015 20:18
Planned Network Maintenance (Resolved)
  • Priority - High
  • Affecting Other - Network Maintenance in North Carolina
  • During the maintenance window customers may experience a brief period of instability as routes fail over to alternate providers and re-converge throughout the internet.

    UPDATE - Maintenance Completed.

  • Date - 11/08/2015 12:00 - 11/08/2015 12:16
  • Last Updated - 11/08/2015 17:56
System Upgrade (Resolved)
  • Priority - High
  • Affecting System - VPS Management Portal
  • Our VPS management portal will switch to limited access mode whilst a disk upgrade is carried out.

    UPDATE - Disk Upgrade Complete and normal access re-enabled.

  • Date - 19/07/2015 14:53 - 19/07/2015 14:56
  • Last Updated - 19/07/2015 20:08
Planned Portal Maintenance (Resolved)
  • Priority - High
  • Affecting System - Account Area
  • Our main website and account portal will be undergoing planned maintenance commencing 03:00:00 GMT, we anticipate this to take upto 1 hour.

    During this period quadhost.net and the customer portal may experience brief connectivity issues.

    No other sites or servers will be affected during this time.

    UPDATE - Maintenance completed within 39minutes and 32 seconds.

  • Date - 17/07/2015 03:04 - 17/07/2015 03:44
  • Last Updated - 17/07/2015 03:07
NC,US - Network Maintenance (Resolved)
  • Priority - High
  • Affecting System - Network
  • Network maintenance work is planned during this time frame in our North Carolina Facility in America.

    We do not expect loss of connectivity but ping time and routes will change during this time as our infastructure undergoes maintenance.

  • Date - 10/01/2015 23:59 - 11/01/2015 06:00
  • Last Updated - 17/07/2015 03:07
OpenVZ Vulnerability Patch (Resolved)
  • Priority - High
  • Affecting System - All OpenVZ Based Servers
  • All OpenVZ based Virtualization servers have been patched in light of CVE-2014-9322.

    Nodes have been restarted to utilise the latest kernel and VM's will be booting over the course of a few minutes upon reboot; if you server is still encountering issues please raise a ticket.

  • Date - 17/12/2014 21:00 - 17/12/2014 21:15
  • Last Updated - 23/12/2014 14:08
CPU Upgrade on nc1.us (Resolved)
  • Priority - High
  • nc1.us will be offline temporarily today whilst carrying out a CPU upgrade.

    We anticipate the machine to be offline for around 15minutes at approx 17:30 GMT on 10/11/2014.

    All active accounts have been informed.

  • Date - 10/11/2014 17:01 - 10/11/2014 17:33
  • Last Updated - 10/11/2014 19:22

Powered by WHMCompleteSolution