NJ Server Access (Resolved)
  • Priority - Critical
  • Affecting Server - New Jersey
  • FYI, our provider in New Jersey has sent us a last minute mandatory change of IP addresses for our NJ Server.  Users will not be able to access server until we make changes to accomodate these new IP addresses.  We hope to have this resolved today.

    Thanks

    New Jersey server should be up and running.. We'll be continuig testing.. if any protocols are not working, please email support and we'll work on iit..
    thanks for your patience!

  • Date - 07/01/2019 05:45
  • Last Updated - 08/12/2019 13:35
OpenVPN Shared UDP 443 Unable to Connect (Resolved)
  • Priority - High
  • Affecting System - OpenVPN
  • Some users are reporting that OpenVPN is not connecting when using Shared UDP topology. We are working on a fix for the problem. In the meantime, we recommend switching the port from Automatic to 53 and the connection will work for you. If you have any questions please open a support request.

  • Date - 02/04/2019 13:31 - 06/17/2019 11:34
  • Last Updated - 02/04/2019 13:34
Authentication Issues (Resolved)
  • Priority - Critical
  • Some users are experiencing Authentication issues. We are investigating this.


    update - The issue should be solved for everyone. If you attempted to change your username and password and received an error then you will want to use the original username/password or change them again.

  • Date - 01/05/2019 11:17 - 02/04/2019 13:31
  • Last Updated - 01/05/2019 12:42
SG Down for Maintenance (Resolved)
  • Priority - High
  • Affecting Server - Singapore
  • Currently Singapore is offline for hardware maintenance. It should be back online soon.

  • Date - 12/11/2018 14:11 - 02/04/2019 13:31
  • Last Updated - 12/11/2018 14:13
FL Offline (Resolved)
  • Priority - Critical
  • Affecting Server - Miami
  • Florida is offline. We are investigating the problem.

  • Date - 11/19/2018 22:04 - 02/04/2019 13:31
  • Last Updated - 11/19/2018 22:05
UK and VA Connection Problems (Resolved)
  • Priority - Critical
  • Affecting System - OpenVPN
  • Users connecting to UK and VA have been experiencing authentication errors intermittently depending on the Protocol and Topology being connected to. We have resolved the connection issue and are in the process of fixing the underlying cause.

  • Date - 09/26/2018 12:41 - 11/19/2018 22:04
  • Last Updated - 10/01/2018 12:45
Streaming Issues (Resolved)
  • Priority - Critical
  • Affecting System - USA and UK Content Streaming
  • We are experiencing problems with our USA and UK content streaming. We are working to get it back online ASAP. We expect to have it completed this week.

  • Date - 08/18/2018 23:46 - 10/01/2018 12:41
  • Last Updated - 08/20/2018 23:49
GA/TX OpenVPN Shared UDP (Automatic / Port 44 (Resolved)
  • Priority - High
  • Affecting System - GA/TX OpenVPN
  • Connections to GA or TX via OpenVPN Shared UDP (Automatic / Port 443) sometimes does not finish connecting. The problem has to do with how the datacenter's DDoS filters are implemented. We are working with the data center to come up with a solution. Please stay tuned.


    On Aug 4th we will decommission these locations because the datacenter was unable to prevent legitimate traffic from being filtered during DDoS events. There is an announcement here https://my.liquidvpn.com/announcements.php?id=362

  • Date - 07/02/2018 17:13 - 10/01/2018 12:41
  • Last Updated - 07/29/2018 22:23
Nami and Ahri Coming Offline (Resolved)
  • Priority - Critical
  • Affecting Other - Nami and Ahri
  • Nami and Ahri are being replaced in June. Please disconnect from them before 6/2/2018

  • Date - 06/02/2018 13:04 - 07/29/2018 22:22
  • Last Updated - 05/15/2018 13:22
Authentication Errors (Resolved)
  • Priority - Critical
  • Affecting System - Authentication for VPN Servers
  • Some users that were signing in and out of servers this morning received authentication errors after signing in and out more than their accounts simultanious sessions on OpenVPN. Our authentication system was not clearing stale sessions properly. We have solved the problem and everything should be back to normal. If you have any issues please let us know.

  • Date - 04/24/2018 05:26 - 04/24/2018 10:26
  • Last Updated - 04/24/2018 09:35
IL, NJ and CA Streaming on L2TP/SSTP/OpenConn (Resolved)
  • Priority - High
  • Affecting System - IL, NJ and CA L2TP/SSTP/OpenConnect
  • IL, NJ and CA are not able to stream Netflix and iPlayer via L2TP, SSTP or OpenConnect at this time. We are working on a solution. We will have more information about this soon.

  • Date - 04/01/2018 16:46 - 04/24/2018 09:35
  • Last Updated - 04/01/2018 16:49
Server Reboots Due to Emergency Patching (Resolved)
  • Priority - Critical
  • Affecting Other - CVE-2017-5753 / CVE-2017-5715 / CVE-2017-5754
  • We have patched our servers for CVE-2017-5753 / CVE-2017-5715 / CVE-2017-5754 the patches require reboots which will take place over the next hour or so. Downtime for each server should be under 15 minutes.

  • Date - 01/04/2018 12:57 - 04/01/2018 16:46
  • Last Updated - 01/04/2018 13:01
UK3 10Gbps Upgrade (Resolved)
  • Priority - High
  • Affecting System - UK3
  • We are waiting on 10Gbps Nics to arrive in the UK so we can start this upgrade. We expect to have this server upgraded before December 9th

  • Date - 11/23/2017 15:35
  • Last Updated - 01/04/2018 12:57
Romania Cannot Stream Content (Resolved)
  • Priority - High
  • Affecting Server - Romania
  • Content streaming is currently not working on Romania. We expect it to be fixed tomorrow.

  • Date - 12/08/2017 21:52 - 04/01/2018 16:46
  • Last Updated - 12/08/2017 21:53
Two Content Streaming Proxies Crashed (Resolved)
  • Priority - Critical
  • Affecting System - Content Streaming Crash
  • We run four different proxies to stream censored content and reach blocked websites. Two of these proxies crashed without triggering a warning or alert. Because of this, some users have been receiving Backend Not Available errors on lots of popular content streaming and/or blocked websites for several days. We run 2 proxies in each region and normally our internal routers route traffic to both proxies to distribute load, if a proxy fails it is supposed to route all of the traffic to the 2nd proxy but if both fail users end up seeing the Backend Not Available error. Normally, this would have been caught right away but because of how the proxies failed our internal routers and the internal alerting system detected them as being online so a portion of our traffic to those sites was routed to both of downed proxies. Support was unable to replicate the problem and we failed to find the issue until earlier this morning. There is no excuse for missing this and we are very sorry for any problems this caused users. We are working on a more robust alerting system for our proxies. We are undertaking a massive infrastructure project to upgrade all of our VPN servers to 10Gbps and the recent setbacks (losing 4 USA locations within a week) has us scrambling to get replacements in place instead of working on some of the other aspects of our network. Our support personnel is fielding an abnormal number of tickets because of all the suddenly downed servers. We clearly botched this one. 


    LiquidVPN Staff



    Update - Content is streaming again.

  • Date - 12/07/2017 13:04 - 12/07/2017 19:34
  • Last Updated - 12/07/2017 19:34
CA, TX and FL Servers Offline Because of Abus (Resolved)
  • Priority - High
  • Affecting Other - CA, TX and FL
  • Earlier today these locations went dark because of DMCA notices. The DMCA notices were sent in a shady method making 1 DMCA notice look like 20 by sending 1 notice per track downloaded despite all 20 tracks were part of a single file. We will keep you updated as we hear more.

  • Date - 11/27/2017 18:37 - 01/04/2018 12:56
  • Last Updated - 12/04/2017 23:53
VA Public IPs Are Working (Resolved)
  • Priority - High
  • Affecting Other - VA Network
  • We had a routing problem in VA that kept the public IPs from working properly when using our internal DNS servers. This has been solved.

  • Date - 12/04/2017 23:47 - 12/05/2017 00:46
  • Last Updated - 12/04/2017 23:48
GA Terminated Due to DMCA Notices (Resolved)
  • Priority - High
  • Affecting Other - GA Servers
  • GA, FL and CA were all hosted with the same host. All three locations have been terminated. We have a replacement CA server on the way. We are looking for FL and GA alternatives. Until we have them online any connections heading towards CA, GA or the Tampa FL location will be routed to the nearest 10Gbps location which is VA for now.

  • Date - 11/29/2017 12:09 - 01/04/2018 12:57
  • Last Updated - 12/04/2017 23:33
VA1 10Gbps Upgrade (Resolved)
  • Priority - High
  • Affecting System - VA1
  • Ashburn VA Installation is complete. 10Gbps upgrade will not happen until the hardware is onsite.


    12/1/2017
    Server has been upgraded to 10Gbps

  • Date - 11/18/2017 15:09 - 12/03/2017 19:15
  • Last Updated - 12/03/2017 19:15
GA1 10Gbps Upgrade (Resolved)
  • Priority - High
  • Affecting System - GA1
  • Georgia is online as of 11/18 but is running at 1Gbps until the new 10Gbps hardware arrives on site.


    Terminated due to DMCA notices

  • Date - 11/18/2017 15:12 - 11/29/2017 12:44
  • Last Updated - 11/29/2017 12:43
CA3 Upgrade to 10Gbps (Resolved)
  • Priority - High
  • Affecting System - CA3
  • CA will be upgraded to 10Gbps as soon as the hardware is delivered.

    11/22/2017
    The 10Gbps NICs arrived at LAX today. We are scheduling the Installation for 11/25/2017 between 3 AM and 6 AM. Please use another server at that time.


    Terminated due to DMCA notices

  • Date - 11/18/2017 15:07 - 11/29/2017 12:43
  • Last Updated - 11/29/2017 12:43
JAX and LAX Phased Out (Resolved)
  • Priority - Medium
  • Affecting Other - Network
  • The Jax and LAX hosts network no longer meet our requirements. New locations in CA and FL are going online before JAX and LAX go offline.

  • Date - 11/03/2017 13:46 - 11/19/2017 13:49
  • Last Updated - 11/19/2017 13:49
Upgrading NL1 and NL1-OVPN to 10Gbps (Resolved)
  • Priority - Critical
  • Affecting System - NL1
  • We are upgrading NL1 and NL1-OVPN to 10Gbps. Currently NL1 (L2TP/SSTP) has been complete but NL1-OVPN will be down until tomorrow.

    Completed

  • Date - 11/13/2017 20:01 - 11/18/2017 15:07
  • Last Updated - 11/18/2017 15:07
Upgrading LAX2 and JAX1 to 10Gbps DDoS Protec (Resolved)
  • Priority - High
  • Affecting System - LAX2 and JAX1
  • We are upgrading LAX2 and JAX1 to 10Gbps servers. We expect there to be 24 to 48 hours of downtime at both of these locations while we have the new equipment installed.

  • Date - 11/08/2017 20:20 - 11/13/2017 20:01
  • Last Updated - 11/08/2017 20:22
Canada Servers Offline (Resolved)
  • Priority - Critical
  • Affecting Server - Canada
  • We are investigating the outage.

  • Date - 10/23/2017 16:41 - 10/24/2017 13:29
  • Last Updated - 10/23/2017 16:42
Public IPs Not Working on UK2-OVPN (Resolved)
  • Priority - High
  • Affecting Server - UK2-OVPN
  • Public IP topology is not currently working on UK2-OVPN we are waiting to hear back from the ISP

  • Date - 09/20/2017 21:12 - 10/24/2017 13:29
  • Last Updated - 09/20/2017 21:13
Chicago Network Maintenance (Resolved)
  • Priority - Critical
  • Affecting Server - Chicago
  • Event Details

    Ticket Number:

    [SW-1-80917]

    Start Date/Time:

    8th of September 2017  04:00 CDT

    End Date/Time:

    8th of September 2017  09:00 CDT

    Direct Impact:

    All customers using services in Chicago location might experience small network connectivity drops during first hour of maintenance window, i.e. from 04:00 CDT to 05:00 CDT. However, we would like to reserve the remaining time (until 09:00 CDT) for any unexpected situations, and we would like you to report any abnormalities (e.g. loss of connectivity, packet loss), so that our networking team can react and adjust the routing configuration accordingly.

    Reason:

    Due to recently observed drop in networking performance that affected up/download speeds for some of the services we are going to perform a series of troubleshooting tests and implement precautionary measures on our uplink level.

  • Date - 09/08/2017 04:00 - 09/20/2017 21:14
  • Last Updated - 09/06/2017 19:00
LAX2 SSD Replacement (Resolved)
  • Priority - High
  • Affecting System - LAX2
  • LAX2 is going down for some SSD replacements

  • Date - 08/25/2017 17:30 - 08/25/2017 21:12
  • Last Updated - 08/25/2017 17:40
LAX2 SSD Failure (Resolved)
  • Priority - Critical
  • Affecting Other - LAX2
  • There was an SSD failure during routine server rebuild. We are waiting for a replacement SSD to be installed.

  • Date - 08/20/2017 00:31 - 08/20/2017 20:14
  • Last Updated - 08/20/2017 13:33
Netflix, HBO and other Sites Not Found at LAX (Resolved)
  • Priority - High
  • Affecting System - Smart Content Streaming
  • A typo in an IP address has stopped Netflix, HBO and a few others from streaming on our new infrastructure. Our staff is working to solve the problem. We will make an announcement once it has been rolled out. If you have questions please let us know.

  • Date - 08/12/2017 14:09 - 08/12/2017 21:15
  • Last Updated - 08/12/2017 14:19
California Site Upgrade (Resolved)
  • Priority - High
  • Affecting System - All California sites
  • We are taking down two California servers at a time and migrating the data this week.

    Update 8/8/2017
    We have 2 last migrations to finish up tomorrow.

  • Date - 08/03/2017 12:05 - 08/09/2017 23:20
  • Last Updated - 08/08/2017 15:17
Beta Servers - Shared UDP Outage (Resolved)
  • Priority - Low
  • Affecting Other - UDP on Shared IP Topology
  • We pushed some tweaks to NAT rules on our beta servers today that had an unexpected impact on the Shared IP UDP Topology. It will connect, resolve DNS addresses but it cannot reach the Internet. We are researching the problem and will roll out a fix as soon as possible. Our Beta and Alpha servers are on the bleeding edge of what is possible in DevOps so please bear with us while we work out the kinks of the new LiquidVPN infrastructure.

  • Date - 07/28/2017 20:21 - 08/03/2017 12:17
  • Last Updated - 07/28/2017 20:39
Replace NAMI (Resolved)
  • Priority - Critical
  • Affecting Server - Canada
  • The new OpenVPN server is finally up and running. Let us know if you have any problems using it.

  • Date - 07/06/2017 10:25 - 07/19/2017 20:25
  • Last Updated - 07/08/2017 00:26
Germany is Offline (Resolved)
  • Priority - Critical
  • Affecting System - Titus
  • The Germany server Titus is completely unresponsive. the data center is investigating the issue.

  • Date - 05/17/2017 19:10 - 05/18/2017 15:52
  • Last Updated - 05/17/2017 22:11
Web server Migration Monday, May 22nd at 12:3 (Resolved)
  • Priority - Critical
  • Affecting Server - host
  • On Monday, May 22nd at 12:30 am ET the billing area will be offline for about an hour while we migrate everything over to a new server. During that time the VPN network will be online but getting updates or downloading a new server list will not work.

  • Date - 05/22/2017 00:30 - 05/22/2017 15:02
  • Last Updated - 05/17/2017 16:32
Ashe and Jax offline (Resolved)
  • Priority - High
  • Affecting System - Ashe and Jax
  • We are investigating the outages in Virginia.

  • Date - 03/22/2017 00:00 - 04/19/2017 00:26
  • Last Updated - 03/22/2017 00:00
CZ Servers Most Likely Going Offline for Good (Resolved)
  • Priority - Medium
  • Affecting System - CZ Servers
  • Our CZ location has been having a lot of issues recently with poor hardware. We have decided to close it down and look for a better location.

  • Date - 03/14/2017 19:54 - 03/22/2017 00:00
  • Last Updated - 03/14/2017 19:57
Infrastructure Upgrade (Resolved)
  • Priority - Critical
  • Affecting System - OpenVPN Servers
  • We have rolled out upgrades to our OpenVPN servers. You will need to reconnect.

  • Date - 03/14/2017 16:59 - 03/21/2017 23:58
  • Last Updated - 03/14/2017 17:01
CZ Outage (Resolved)
  • Priority - Critical
  • Affecting System - Bard Czech Republic (CZ)
  • Update 3/11/2017 11:59 PM the data center is still not giving us any updates as to why Bard is offline still. We may have to cancel both of our servers there and find a replacement.

  • Date - 03/10/2017 14:52 - 03/14/2017 19:53
  • Last Updated - 03/11/2017 23:01
UK Outage (Resolved)
  • Priority - Critical
  • Affecting Server - UK2-OVPN
  • There is an outage in the UK and 1 of our OpenVPN servers is offline there. We are investigating.

  • Date - 03/07/2017 04:38 - 03/11/2017 22:52
  • Last Updated - 03/07/2017 08:39
California Maintaince Allistar and Brand (Resolved)
  • Priority - Critical
  • Affecting System - Allistar and Brand being replaced
  • These servers are being replaced. Please use our other California location.

  • Date - 02/13/2017 15:02 - 03/07/2017 08:39
  • Last Updated - 02/14/2017 17:41
Incoming DDoS Attacks on Echo (California) (Resolved)
  • Priority - High
  • Affecting System - Echo (OpenVPN California)
  • There was an incoming DDoS attack on a couple of our IP addresses today in California. The Echo server was affected. We are researching the attacks still but would like to remind users that we offer DDoS protection on all LiquidVPN servers located in Virginia, New Jersey, Miami, Germany, Amsterdam and Singapore.

  • Date - 02/02/2017 13:03 - 02/15/2017 14:48
  • Last Updated - 02/02/2017 15:25
Chicago is Down (Resolved)
  • Priority - Critical
  • Affecting System - All Chicago Servers
  • There is an outage in Chicago, it started about 14 hours ago. We are waiting for more information

  • Date - 01/10/2017 02:10 - 02/15/2017 14:48
  • Last Updated - 01/10/2017 02:14
NL Alblasserdam Network Migration Bizarro and (Resolved)
  • Priority - High
  • Affecting System - Bizarro and Inferno Migration
  • Bizarro and Inferno will come offline shortly because they will be part of a network migration happening at the host. Please let us know if you have any questions.

  • Date - 11/28/2016 18:15 - 05/16/2017 13:25
  • Last Updated - 11/28/2016 18:20
OpenVPN NL Server Blacksun Being Patched (Resolved)
  • Priority - High
  • Affecting System - Blacksun NL
  • The server is being patched and upgraded, a reboot will be required. Expect some downtime.

  • Date - 11/05/2016 18:40 - 11/28/2016 18:15
  • Last Updated - 11/05/2016 19:04
Switzerland offline (Resolved)
  • Priority - High
  • Affecting Server - Switzerland
  • We are investigating the issue. Their support has been unresponsive for several hours.

  • Date - 10/24/2016 09:17 - 10/26/2016 00:44
  • Last Updated - 10/24/2016 09:18
Romania Downtime Tomorrow (Resolved)
  • Priority - Critical
  • Affecting Server - Romania
  • Downtime scheduled for 01:00 - 05:00, maximum 2 interruptions, upto 15 minutes each. Downtime is to replace a broken line card.

  • Date - 10/21/2016 01:00 - 10/26/2016 00:43
  • Last Updated - 10/20/2016 16:16
Kansas City Outage (Resolved)
  • Priority - High
  • Affecting Server - Kansas City
  • All three of our Kansas City servers went offline a few minutes ago. We are investigating the issue right now.

  • Date - 10/20/2016 14:28 - 10/20/2016 16:11
  • Last Updated - 10/20/2016 14:30
NJ DDoS (Resolved)
  • Priority - High
  • Affecting Server - New Jersey
  • A large portion of our IP space has been targeted by an attack exceeding 500 Gbps in size that caused several major upstream links that we have to be completely saturated. At this time, there have been several immediate changes made to our network infrastructure that will prevent the attack from causing complete unavailability. While our current infrastructure can handle the attacks as a whole, the attack traffic did not come in through balanced routes causing specific uplinks to be completely saturated. We have immediately started the process of upgrading uplinks to all upstream providers to multiple 100 Gbps links, which will allow us to completely absorb the attack without saturation in the near future. This permanent solution is expected to be in place soon (date TBA as we are still coordinating with upstream providers) and is considered to be our highest priority project to ensure long term availability and quality of service to our customers. 

    Our number one goal is to provide the best possible service to customers. We appreciate your patience and understanding in this matter.

  • Date - 09/21/2016 17:07 - 10/20/2016 16:11
  • Last Updated - 09/21/2016 17:07
UK Outage (Resolved)
  • Priority - High
  • Affecting Other - UK Network
  • Currently the UK network is offline. We are looking into this issue right now.


    Update 9/13/2016 @ 8:47 
    There was emergency maintenance on the network to replace their core router. The servers should be back online shortly.

    Update 9/14/2016 @ 12:35
    The datacenter still has not brought our servers back online. They are claiming that the servers that have been running the same network configuration for up to 2 years are offline because of a broadcast storm. We are looking at new UK datacenters.

  • Date - 09/13/2016 19:38 - 09/20/2016 15:27
  • Last Updated - 09/14/2016 11:41
UK Datacenter Offline (Resolved)
  • Priority - Critical
  • Affecting System - UK
  • We are investigating the outage in the UK

  • Date - 09/09/2016 18:35 - 09/13/2016 19:47
  • Last Updated - 09/09/2016 18:35
DDoS Attacks Aimed at California Server(s) (Resolved)
  • Priority - Medium
  • Affecting System - OpenVPN
  • Over the last few days, we have been hit with a few DDoS attacks on an OpenVPN server in California. As of right now we are not quite sure if this is an attack aimed at us or aimed at one of our users. We ask all of our users that are at risk of being hit with a DDoS to use one of our DDoS protected locations https://www.liquidvpn.com/vpn-server-locations/

  • Date - 09/07/2016 00:59 - 10/20/2016 16:12
  • Last Updated - 09/08/2016 22:17
Emergency Maintenance Notice For Titan & (Resolved)
  • Priority - Critical
  • Affecting System - Rotterdam Netherlands
  • Due to a problem with core infrastructure the datacenter is switching over to redundant systems, all sessions will be restored after the switch but higher latency and some packet loss is expected.

  • Date - 09/06/2016 12:31 - 09/08/2016 22:17
  • Last Updated - 09/06/2016 13:32
Germany going down for maintenance (Resolved)
  • Priority - Critical
  • Affecting Server - Germany
  • We are rebooting the Germany server and updating the kernel.

  • Date - 08/09/2016 00:00 - 08/09/2016 11:31
  • Last Updated - 08/09/2016 00:01
Chicago Network Maintenance (Resolved)
  • Priority - High
  • Affecting Server - Chicago
  • Start Date/Time: 8th August 2016 10:00 AM CEST
    End Date/Time: 8th August 2016 11:00 AM CEST

    Hardware upgrade will be performed by the datacenter.

  • Date - 08/08/2016 10:00 - 09/06/2016 13:22
  • Last Updated - 08/04/2016 14:46
Switzerland Packetloss (Resolved)
  • Priority - Medium
  • Affecting Server - Switzerland
  • Switzerland is reporting major packet loss. We are investigating
    8993.440ms, lost 60%
    Date/Time: 2016-07-26 17:09:01

  • Date - 07/26/2016 16:12 - 07/31/2016 22:31
  • Last Updated - 07/31/2016 22:30
Network Outage NJ (Resolved)
  • Priority - High
  • Affecting Server - New Jersey
  • The network event was traced to a 3rd party public peer announcing erroneous routing information. The peer has been disabled and filters have been configured to accept trusted prefixes only. Core engineers will continue to monitor the environment closely for 24 hours.

  • Date - 07/01/2016 08:15 - 07/21/2016 12:00
  • Last Updated - 07/01/2016 14:33
Chicago Maintenance 2016-06-17 01:00 AM CDT (Resolved)
  • Priority - High
  • Affecting Server - Chicago
  • Chicago Network upgrade users will experience periods of downtime during this maintenance window. Some latency and/or sub-optimal routing may be experienced during this maintenance window.

  • Date - 06/17/2016 01:00 - 09/06/2016 13:22
  • Last Updated - 06/09/2016 23:58
Emergency Sweden Router Update (Resolved)
  • Priority - Critical
  • Affecting System - RouterOS Update
  • We are updating the RouterOS firmware in Sweden. Expect up to 20 minutes of downtime

  • Date - 05/07/2016 17:33 - 05/27/2016 13:26
  • Last Updated - 05/07/2016 17:35
Sweden OpenVPN Server Reboot (Resolved)
  • Priority - Critical
  • Affecting Server - phantomworx
  • We are rebooting the OpenVPN server in Sweden right now

  • Date - 05/06/2016 20:30 - 05/07/2016 17:33
  • Last Updated - 05/06/2016 20:30
Sweden Packet Loss (Resolved)
  • Priority - Medium
  • Affecting System - The whole Sweden network
  • We are investigating a problem in Sweden that is causing packet loss on all of our servers there.

  • Date - 05/06/2016 15:12 - 05/06/2016 20:30
  • Last Updated - 05/06/2016 15:14
Worldstream Outage (Resolved)
  • Priority - High
  • Affecting Server - Netherlands
  • The worldstream network is offline completely right now and our servers there are offline as well.

  • Date - 05/02/2016 00:00 - 05/06/2016 15:12
  • Last Updated - 05/02/2016 00:46
PPTP/L2TP/SSTP Routing Issue Found and Fixed (Resolved)
  • Priority - Critical
  • Affecting System - Several Locations
  • Earlier today during routine testing we discovered a route from the PPTP/L2TP/SSTP servers to the DNS servers did not cover the full DHCP scope. The DHCP scope was expanded two months ago because a bug found in the Mikrotik firmware was rendering some IPs inaccessible after a user terminated their session. We found 9 servers with different levels of PPTP/L2TP/SSTP routing issues. Two of the servers were only affected when they ran out of public IP addresses to hand out to users and the rest were missing half of the new DHCP scope. This means after the servers reached certain capacity users would be able to connect but would not be able to use the private DNS servers and/or SMART DNS. If you have any questions please contact support.

  • Date - 05/01/2016 15:10 - 05/01/2016 21:11
  • Last Updated - 05/01/2016 20:20
California Reboot (Resolved)
  • Priority - Medium
  • Affecting Server - LAX1
  • One of the California server clusters were rebooted at 6:15 EST

  • Date - 04/27/2016 18:17 - 05/06/2016 15:12
  • Last Updated - 04/27/2016 18:18
UK IP's Got Null Routed (Resolved)
  • Priority - Medium
  • Affecting Server - UK2-OVPN
  • There was a UDP Flood on one some of our UK IPs and they were null routed without the datacenter telling us about it. We are working to get the IPs back on.

  • Date - 04/26/2016 16:09 - 04/27/2016 18:17
  • Last Updated - 04/27/2016 00:12
Switzerland Outage (Resolved)
  • Priority - Critical
  • Affecting Server - Switzerland
  • The datacenter in Switzerland does not appear to be on the internet anymore. It is most likely a DDoS. The servers there should be back online soon.

  • Date - 04/21/2016 20:35 - 04/27/2016 00:09
  • Last Updated - 04/21/2016 20:37
California drive issues (Resolved)
  • Priority - Critical
  • Affecting Server - LAX1
  • We are having some drive issues in California and may need to reboot a host there. If it happens some services will go offline for a little while. This affects Brand and Alistar

  • Date - 04/13/2016 11:41 - 04/21/2016 20:35
  • Last Updated - 04/13/2016 11:44
Emergency Reboot California Node (Resolved)
  • Priority - Critical
  • Affecting Server - LAX1
  • We are rebooting some of California servers right now. Expect about 15 to 20 minutes of downtime.


    The node is not coming back online like it should there appears to be a drive failure. We are waiting for staff to check to see which disks in the array are bad.

  • Date - 04/12/2016 21:26 - 04/13/2016 11:41
  • Last Updated - 04/12/2016 23:42
4/3/2016 Server Maintenance - Reboots Requir (Resolved)
  • Priority - High
  • Affecting System - The LiquidVPN Network
  • On 4/3/2016 at 11:00 PM EST we will finish OS upgrades they require a reboot after they are applied. This reboot affects about 25% of our servers. We expect each server to be down for no more than 10 minutes. If you have questions or concerns please contact us.


  • Date - 04/03/2016 23:00 - 09/06/2016 13:22
  • Last Updated - 03/31/2016 00:11
Kansas City Routing Issues (Resolved)
  • Priority - Critical
  • Affecting Other - Whole Kansas City Network
  • The datacenter in Kansas city appears to have messed up our static routes again. It has bypassed our detection methods because everything reports online and while people can connect their data is not getting past the datacenters gateway. Please do not attempt to use Kansas City until this matter is resolved. We do not know how stable it will be until we manually check everything and confirm they have fixed. Please stay tuned for updates.

    The data center has replaced the missing routes.

  • Date - 03/10/2016 20:28 - 03/23/2016 19:40
  • Last Updated - 03/23/2016 19:40
NY Server Suspended Due To Duplicate DMCA Not (Resolved)
  • Priority - Critical
  • Affecting Server - New Jersey
  • Hello, Please be advised that the server ny.liquidvpn.com (108.61.6.82) has been suspended due to a large quantity of abuse complaints received within the last 30 minutes. There have been over 50 abuse complaints have been processed, and more are being received retroactively. Kindly terminate the offending user and notify us and we will unsuspend the server. Thank you, ReliableSite.Net LLC

  • Date - 03/18/2016 18:08 - 03/21/2016 20:28
  • Last Updated - 03/18/2016 18:09
Network Trouble in Rotterdam (Resolved)
  • Priority - Medium
  • Affecting System - Hex and Titan Connectivity Issues
  • There appears to be some network troubles in Rotterdam people are having trouble connecting and/or staying connected to both of our servers there.

  • Date - 03/12/2016 18:21 - 03/18/2016 18:09
  • Last Updated - 03/12/2016 18:24
User Control Panel Reboot (Resolved)
  • Priority - High
  • Affecting System - Webserver
  • Our webserver will be rebooted on Friday, March 11th, 2016 between 00:01 and 01:01 CST (06:01 - 07:01 GMT) for software updates. We expect downtime to be 15 minutes or less.

  • Date - 03/11/2016 00:00 - 03/12/2016 18:24
  • Last Updated - 03/08/2016 17:46
Emergency Update in Germany for Drown Patchin (Resolved)
  • Priority - Critical
  • Affecting Server - Germany
  • Our Germany servers were found to have a compromised version of OpenSSL that was vulnerable to the recently discovered Drown vulnerability https://drownattack.com/ we just finished patching and rebooting the server.

  • Date - 03/04/2016 21:48 - 03/04/2016 22:50
  • Last Updated - 03/04/2016 21:52
Network Wide SmartDNS Outage (Resolved)
  • Priority - Critical
  • Affecting System - USA SmartDNS
  • The DC that provides our SmartDNS functions is experiencing a major network outage. They have not been very forthcoming with information. We are making plans to move these features to another datacenter. In the mean time hope the current DC will get its act together and get their network in order soon.


    Update 3/3/2016
    We have installed a temporary fix for Netflix and many other content providers. However Hulu is not working yet. We are working on getting Hulu back up and running ASAP



    Update 3/4/2016
    New SmartDNS servers are online and working

  • Date - 03/01/2016 03:23 - 03/04/2016 11:30
  • Last Updated - 03/04/2016 11:30
PPTP/L2TP/SSTP Connection Issues (UPDATE) (Resolved)
  • Priority - Critical
  • Affecting System - Whole Network
  • Applying the glibc emergency patch the other day has ended up causing some problems with L2TP/PPTP/SSTP we are rolling out another fix right now but each PPTP/L2TP/SSTP server will require a reboot.


    We are now waiting on a Mikrotik engineer to get in touch with us about a few of our routers that are still displaying some issues after emergency patching.



    Update 3/3/2016

    After hours of research with the Mikrotik engineers we have found a solution to the problems we have had with the routing. We are pushing out the fix now.

  • Date - 02/25/2016 21:37 - 03/04/2016 21:58
  • Last Updated - 03/03/2016 00:00
Manchester Server Experiencing Problems (Resolved)
  • Priority - High
  • Affecting System - UK Dedicated/Shared IP OpenVPN Server
  • There appears to be network trouble at M247. One of our Manchester servers and Romania servers are having trouble and both of them are hosted through M247. We are waiting for input from the ISP.

  • Date - 02/27/2016 15:23 - 02/27/2016 22:15
  • Last Updated - 02/27/2016 15:25
PPTP/L2TP/SSTP Connection Problems (Resolved)
  • Priority - Critical
  • Affecting System - Network Wide
  • We are investigating the problems some people are having connecting to L2TP, SSTP and PPTP servers. We are implementing what we hope to be a fix but will not know for sure until we can observe the network for a few hours.

  • Date - 02/25/2016 14:29 - 02/25/2016 21:37
  • Last Updated - 02/25/2016 14:32
KC Network Troubles (Resolved)
  • Priority - Critical
  • Affecting System - KC Router
  • Kansas City has been up and down all day. It is the server that hosts our router meaning all of KC resources are offline. We believe there is an intermittent hardware failure and are waiting for a comprehensive hardware scan to finish. It could take 12 hours or more.

  • Date - 02/20/2016 22:49 - 02/24/2016 23:48
  • Last Updated - 02/20/2016 23:12
Canada Echo DDoS Attack (Resolved)
  • Priority - Medium
  • Affecting Server - Canada
  • One of our Canada servers "Echo" is experiencing a DDoS attack. We expect service to return shortly.

  • Date - 02/18/2016 00:54 - 02/20/2016 23:01
  • Last Updated - 02/18/2016 00:55
Emergency Patching Tonight Requires Restarts (Resolved)
  • Priority - Critical
  • Affecting System - The LiquidVPN Network
  • Most of our servers are vulnerable to a bug discovered today that would allow an attacker to gain access to servers running Linux. To protect against this bug we have been applying patches to our infrastructure for the last 4 hours. Later tonight after hours we will begin rebooting our servers. We expect each server will see 10 to 15 minutes of downtime.

  • Date - 02/17/2016 17:10 - 03/12/2016 18:24
  • Last Updated - 02/17/2016 17:10
M247 Network Issues (Romania and UK Affected) (Resolved)
  • Priority - Medium
  • Affecting System - M247 Network Issues
  • Currently, there is intermittent packet loss to both of our network segments hosted by M247. If you are on any of our UK or RO servers you may experience some connection problems.

  • Date - 02/11/2016 16:46 - 02/13/2016 20:51
  • Last Updated - 02/11/2016 16:49
NL OpenVPN Server Bizarro NIC failure (Resolved)
  • Priority - Critical
  • Affecting Server - Netherlands
  • The NIC failure on our NL OpenVPN server Bizarro had a NIC fail. We have replaced it and confirmed things are up and running again.

  • Date - 01/27/2016 05:33 - 01/27/2016 11:34
  • Last Updated - 01/27/2016 10:36
Chicago Network Upgrade January 28th 2016 (Resolved)
  • Priority - High
  • Affecting Server - Chicago
  • Please be aware that engineers will be performing a scheduled upgrade to the Chicago network on the 28th of January 2016 from 4:30 AM Central Standard Time (CST) to 5:00 AM  Central Standard Time (CST). During this window, you may experience some packet loss and/or a short disruption in connectivity. If you have any questions please contact technical support here https://my.liquidvpn.com/submitticket.php



    T
    he LiquidVPN Team

     

    scheduled for 28th January 2016

    Time: 4:30 AM Central Standard Time (CST) to 5:00 AM  Central Standard Time (CST)

  • Date - 01/23/2016 21:26 - 02/17/2016 17:10
  • Last Updated - 01/23/2016 21:33
California Shared IP Topology DDoS (Resolved)
  • Priority - Medium
  • Affecting Server - LAX1
  • One of our IPs (shared IP Topology) in California is experiencing a DDoS attack. This will affect users across several servers.

  • Date - 01/19/2016 18:38 - 01/23/2016 21:26
  • Last Updated - 01/19/2016 18:41
Valhalla and Khan Missouri Servers Being Move (Resolved)
  • Priority - High
  • Affecting Server - Kansas City
  • We will begin the server moves in approximately 30 minutes. We do not anticipate more than 30 to 45 minutes of downtime per server. We will begin the server move process in approximately 30 minutes.

  • Date - 01/17/2016 21:05 - 01/18/2016 22:03
  • Last Updated - 01/17/2016 21:09
Null Routed California IP Due to DDoS attack (Resolved)
  • Priority - Medium
  • Affecting Server - LAX1
  • An Incoming DDoS attack on one of the shared California IP addresses has triggered an automatic Null Route. If you feel you are the target of the DDoS attack, we highly suggest using one of our DDoS protected locations (NJ and Flordia both have DDoS protection)

  • Date - 01/10/2016 18:10 - 01/17/2016 21:14
  • Last Updated - 01/10/2016 18:13
Null Routed California IP Due to DDoS attack (Resolved)
  • Priority - Medium
  • Affecting Server - LAX1
  • There is an incoming DDoS attack on Brand's shared exit IP in California. Please use another server or switch to a different topology. The null route will be lifted as soon as the attack is over.


    Update
    We have rerouted traffic to a new IP address. If anyone is still experiencing problems let us know.

  • Date - 01/09/2016 17:06 - 01/10/2016 18:04
  • Last Updated - 01/09/2016 17:25
IP in LAX Null Routed due to DDoS (Resolved)
  • Priority - Medium
  • Affecting Server - LAX1
  • We have null routed an IP in LA due to an incoming DDoS attack aimed at our server.

  • Date - 12/24/2015 01:25 - 01/09/2016 12:51
  • Last Updated - 12/24/2015 01:26
DDoS Resulting in Null Route NL Blacksun (Resolved)
  • Priority - Medium
  • Affecting System - Blacksun
  • One of our NL servers is the current target of a DDoS attack. We have null routed the affected IP. Please use another NL server for now.

  • Date - 12/19/2015 15:22 - 12/20/2015 13:27
  • Last Updated - 12/19/2015 15:26
Chicago Network Trouble (Resolved)
  • Priority - Critical
  • Affecting System - Chicago Network
  • Chicago is experiencing network issues right now

  • Date - 12/10/2015 12:58 - 12/19/2015 15:26
  • Last Updated - 12/10/2015 12:59
Romania DDoS (Resolved)
  • Priority - High
  • Affecting System - Romania Servers
  • Currently Romania is very unstable due to a DDoS on the network. Please use another gateway until it calms down.

  • Date - 12/06/2015 21:42 - 12/10/2015 12:59
  • Last Updated - 12/06/2015 21:44
SSTP Not Working Properly on OS X (Resolved)
  • Priority - High
  • Affecting System - LiquidVPN OS X Client
  • When trying to connect to SSTP via OS X an error appears stating the service is not started. We are investigating this right now and will update this ticket when we find out more infortmation.

  • Date - 11/27/2015 11:38 - 12/05/2015 23:51
  • Last Updated - 11/27/2015 12:09
Chicago Performance Issues (Resolved)
  • Priority - High
  • Affecting Server - Chicago
  • We are aware of the terrible performance coming out of Chicago right now. The reason is because this is the first time stacking 2 1Gbps ports in an 802.3ad bonded interface. The server is likely to go on and off as we test configurations. Yesterday when we tested this in a limited environment we did not run into this issue. Please stay tuned for more information.


    We have found the cause and believe we have fixed it but we wont know until we really begin to load people on it.

  • Date - 11/19/2015 17:07 - 11/20/2015 14:51
  • Last Updated - 11/20/2015 14:51
Switzerland DDoS (Resolved)
  • Priority - High
  • Affecting Server - Switzerland
  • The upstream provider in Switzerland is experiencing a DDoS and is causing some packet loss on the node. Please use a different location if possible.

  • Date - 11/15/2015 09:54 - 11/15/2015 15:39
  • Last Updated - 11/15/2015 09:55
NJ Outage (Resolved)
  • Priority - High
  • Affecting Server - New Jersey
  • There is an outage in New Jersey. We will update once we have more information.

    The OS is no longer loading. We may have to rebuild the node. Stay tuned to this ticket for more information.


    Update 11/13/2015 1:11 AM
    Our NJ node is still offline. We have had a major hardware failure and are waiting for new hardware to be installed before we can rebuild it

  • Date - 11/12/2015 14:56 - 11/15/2015 09:54
  • Last Updated - 11/13/2015 00:11
Switzerland Offline (Resolved)
  • Priority - Critical
  • Affecting Server - Switzerland
  • Switzerland is completely offline. The datacenters website is down as well. We will update this ticket as soon as we get more info.



    The DDoS on the Switzerland network is over and everything is working like normal again.


    THe network in Switzerland keeps going up and down. Please be cautious if you decide to use this server. We will update this ticket again when everything is back to normal

  • Date - 11/02/2015 13:32 - 11/12/2015 14:58
  • Last Updated - 11/02/2015 16:03
Packet Loss in Switzerland (Resolved)
  • Priority - Medium
  • Affecting Server - Switzerland
  • We are experiencing major packet loss on both of our Switzerland servers. An Alternative route is advised.

  • Date - 10/23/2015 17:30 - 10/29/2015 16:11
  • Last Updated - 10/23/2015 17:39
New Jersey Network Offline For Good on 11/15/ (Resolved)
  • Priority - Low
  • Affecting Server - New Jersey
  • We are removing our servers from the New Jersey Datacenter on 11/15/2015 as soon as we have more information about the replacements we will post it.

  • Date - 10/21/2015 19:32 - 11/24/2015 00:18
  • Last Updated - 10/21/2015 18:33
Some New Jersey IPs are Null Routed by ISP (Resolved)
  • Priority - Medium
  • Affecting Server - New Jersey
  • We have a handful of NJ IPs that have been null routed by the datacenter because of excessive abuse. We are closing this server down at the end of the contract.

  • Date - 10/21/2015 13:41 - 10/23/2015 17:39
  • Last Updated - 10/21/2015 13:42
New Jersey Limited Due to Abuse (Resolved)
  • Priority - Medium
  • Affecting Server - New Jersey
  • The New Jersey datacenter has added our shared IP to their UDP filter because of the DMCA notices being generated right now.

  • Date - 10/05/2015 21:20 - 10/19/2015 22:31
  • Last Updated - 10/05/2015 21:26
New Jersey Offline Completely (Resolved)
  • Priority - Critical
  • Affecting Server - New Jersey
  • We are not sure why the datacenter deleted the route to our main subnet rendering every server in the cluster offline but they did. We are waiting for them to fix it once again. We understand everyone is upset because they cannot access the DDoS protected servers in NJ, we are upset as well. After 2 years at this datacenter it may be time to move on. We will keep you posted. In the meantime if you need DDoS protection use the Florida location.

  • Date - 10/05/2015 13:25 - 10/05/2015 21:20
  • Last Updated - 10/05/2015 13:31
Public IP's in New Jersey (Zeta) not wor (Resolved)
  • Priority - High
  • Affecting Server - New Jersey
  • The datacenter appears to have removed the static routes that allowed us to hand out public IPs in New Jersey again. This means that anyone trying to use the modulating or public topologies are not going to be able to connect to the internet.

  • Date - 10/04/2015 11:28 - 10/05/2015 21:20
  • Last Updated - 10/04/2015 11:30
New Jersey Null Routes Due to Abuse (Resolved)
  • Priority - Medium
  • Affecting Server - New Jersey
  • Due to excessive abuse the datacenter in New Jersey null routed several of our IP addresses.


    The datacenter accidently removed some of our static routes so while New Jersey is back online only OpenVPN Shared can browse the internet.

  • Date - 09/29/2015 22:41 - 10/05/2015 21:20
  • Last Updated - 09/30/2015 23:15
Romania OpenVPN Offline (Resolved)
  • Priority - High
  • Affecting Server - Romania
  • Our Romania OpenVPN server is not responsive. We are investigating now.

  • Date - 09/30/2015 04:55 - 09/30/2015 16:26
  • Last Updated - 09/30/2015 04:56
Confirm Password Error During Registration (Resolved)
  • Priority - High
  • Affecting System - New User Registration
  • New user signups are being rejected due to an error about their desired password not being correct. 


    Update 12:55 PM

    The new version of this file has been uploaded and the issue has been resolved.



  • Date - 09/28/2015 00:00 - 09/28/2015 12:52
  • Last Updated - 09/28/2015 11:56
Netherlands Kernel Upgrades (Resolved)
  • Priority - Critical
  • Affecting System - Titan and Hex
  • We have upgraded the Kernel on Titan and Hex (netherlands) and are rebooting both servers now. Please allow up to 10 minutes for the server to come back online

  • Date - 09/23/2015 22:49 - 09/27/2015 18:03
  • Last Updated - 09/23/2015 22:52
Singapore Offline For Emergency Rebuild (Resolved)
  • Priority - Critical
  • Affecting System - Singapore
  • Singapore is having a motherboard failure and we are waiting on new hardware. The server should be back up and running within the next 24 hours.



    9/18/2015 10:21PM

    Singapore is once again fully operational.

  • Date - 09/17/2015 14:51 - 09/18/2015 21:22
  • Last Updated - 09/18/2015 21:22
Emergency Router Maintenance in UK (Resolved)
  • Priority - Critical
  • Affecting System - Manchester 2
  • A memory leak was causing our routers load in manchester to spike causing very poor connectivity and lots of latency. We upgraded the firmware and everything appears to have been solved. We will monitor the situation and update if required.

  • Date - 09/12/2015 06:03 - 09/16/2015 20:15
  • Last Updated - 09/12/2015 06:06
Sessions Hanging Causing Connection Problems (Resolved)
  • Priority - Critical
  • Affecting Server - masterradius
  • Our main authentication gateway has a bad RAID controller causing some data integrety problems. This was causing sessions to hang and some users to have problems connecting to the VPN. We have taken the gateway offline and are now functioning with the alternative gateways only until we rebuild the main gateway. We do not expect any problems with load running on just the secondary gateway but if anyone has trouble please open a ticket. Once the main authentication gateway is back online this ticket will be updated.

    Update 7:15 AM 9/8/2015
    Last night we tried to enable the primary gateway and ran into some trouble with sessions starting to hang again. Today we will replace the RAID controller and begin rebuilding the authentication server.


    Update 5:38 PM 9/8/2015
    The primary server has been completely rebuilt and everything should be back to normal.

  • Date - 09/06/2015 09:31 - 09/08/2015 16:40
  • Last Updated - 09/08/2015 16:40
my.liquidvpn.com Maintenance (Resolved)
  • Priority - High
  • Affecting System - Web Server and Software Maintenance


  • We will be upgrading our billing portal software and implementing a more userfriendly checkout system on 8/24/2015 we expect about an hour of downtime. During the downtime the control panel will not be active. Users VPN accounts will not be affected.

  • Date - 08/24/2015 00:00 - 08/25/2015 10:05
  • Last Updated - 08/21/2015 13:37
NL Maintenance August 25 to August 26 (Resolved)
  • Priority - Critical
  • Affecting System - Bizzaro and Hex
  • We are sending you this update, following up on our last mailing from August 11th 2015 regarding the unplanned downtime.
    On the night of August 25th to 26th, we will be performing a scheduled maintenance to upgrade our Core Routing Engine:

    Begin: Tuesday, August 25th 2015 - 11:59 PM
    End: Wednesday, August 26th 2015 - 01:00 AM

    Planned downtime: 5-15 minutes

  • Date - 08/25/2015 23:59 - 09/02/2015 10:54
  • Last Updated - 08/17/2015 14:38
Germany DDOS (Resolved)
  • Priority - High
  • Affecting Server - Germany
  • Leaseweb our host in Germany is experiencing a DDoS on a core router. Expect frequent disconnects until it is over with.

  • Date - 08/16/2015 00:00 - 08/17/2015 14:35
  • Last Updated - 08/16/2015 01:24
Netflix and Hulu Not Working at some location (Resolved)
  • Priority - High
  • Affecting Server - phantomworx
  • There was a problem with our reverse proxy servers that were causing hulu and netflix problems for some non USA servers. This has been fixed.

  • Date - 08/10/2015 15:20 - 08/15/2015 16:20
  • Last Updated - 08/15/2015 15:22
New Jersey Bones PPTP/L2TP IP Null Routed (Resolved)
  • Priority - Medium
  • Affecting Server - New Jersey
  • One or possibly more of the IP addresses handed out to clients when they connect to bones has been null routed. We are waiting for the datacenter to fix this issue.

    The ISP is still trying to track down where their routing error is.

  • Date - 08/09/2015 22:17 - 08/15/2015 15:20
  • Last Updated - 08/10/2015 16:52
Router Upgrades (Resolved)
  • Priority - Medium
  • Affecting System - All Locations That Support IP Modulation
  • All this month we will be upgrading our routers firmware. This applies only to locations that support IP Modulation. This should be a seamless change but it is possible there could be up to a maximum of 5 minutes downtime for each router.

  • Date - 08/01/2015 22:15 - 10/19/2015 22:31
  • Last Updated - 08/01/2015 22:18
Emergency DNS Server Maintenance (Resolved)
  • Priority - Critical
  • Affecting System - Bind
  • We have finished patching our DNS servers for vulnerability CVE-2015-5477 https://kb.isc.org/article/AA-01272 however this patch was rolled out rather quickly on all of our DNS servers. We will continue to test our network to make sure the patch has no unintended side effects. If you run into any problems with name resolution (Websites dont open but things like skype work) please open a ticket and let us know what server you are having the problem on.

  • Date - 07/30/2015 13:31 - 08/01/2015 22:15
  • Last Updated - 07/30/2015 13:35
Web Server Reboot (Resolved)
  • Priority - Critical
  • Affecting Server - phantomworx
  • The webserver that hosts our control panel will be rebooted on July 24th, 2015, between 00:01 and 06:00 CDT (05:01 - 11:00GMT). There should be no more than 20 to 30 minutes of downtime to complete this upgrade and reboot the hardware node. We will select a time when there are as few people on the server as possible and then do the reboot.

  • Date - 07/24/2015 00:01 - 08/01/2015 22:15
  • Last Updated - 07/23/2015 13:38
Romania Network DDoS (Resolved)
  • Priority - Critical
  • Affecting Server - Romania
  • Our Romania datacenter is currently being DDoSed. Our connections are being affected. Please use caution if connected to this location.

  • Date - 07/10/2015 21:08 - 07/11/2015 23:34
  • Last Updated - 07/10/2015 21:10
Switzerland Network Troubles (Resolved)
  • Priority - Critical
  • Affecting Server - Switzerland
  • The privatelayer network appears to be completely offline right now. Their website is down as well. Please use other locations for now. We do not expect the downtime to last too much longer.

  • Date - 06/28/2015 13:52 - 06/30/2015 15:09
  • Last Updated - 06/28/2015 13:54
Kansas City Power Outage (Resolved)
  • Priority - Critical
  • Affecting Server - Kansas City
  • We are experiencing a power outage due to an issue with the local city power. Kansas City is offline currently. We will make an announcement once it comes back online.

  • Date - 06/27/2015 14:06 - 06/27/2015 15:59
  • Last Updated - 06/27/2015 14:07
Web server Reboot Sunday, July 21st 05:01 - 0 (Resolved)
  • Priority - High
  • Affecting System - my.liquidvpn.com
  • On Sunday, July 21st, 2015 between 00:01 and 0:30 CDT (05:01 - 05:30 GMT) we will be rebooting the server that hosts our billing area. Our front end server will not go offline during this time only our billing area is being updated.

  • Date - 07/21/2015 00:01 - 07/23/2015 13:36
  • Last Updated - 06/19/2015 23:44
California Router Crashed (Resolved)
  • Priority - Critical
  • Affecting Server - LAX1
  • We are investigating one of our routers in California that crashed. We will update the ticket once we have more information.

  • Date - 06/17/2015 13:38 - 06/22/2015 23:22
  • Last Updated - 06/17/2015 13:39
UK OpenVPN Static IP UDP (Resolved)
  • Priority - Critical
  • Affecting Server - UK2-OVPN
  • The Static IP Daemon was experiencing connection problems. It has been restarted and is now working again.

  • Date - 06/14/2015 11:48 - 06/14/2015 12:49
  • Last Updated - 06/14/2015 11:50
Kansas City Maintenance (Resolved)
  • Priority - Critical
  • Affecting Server - Kansas City
  • Window Start Time:  Tuesday, June 16, 1:00am CDT

    Window End Time:  Tuesday, June 16, 2:00am CDT

    Expected Impact:   ~ 15 minute

    Reason: Hardware upgrade to improve capacity and security.

  • Date - 06/16/2015 01:00 - 06/22/2015 23:23
  • Last Updated - 06/09/2015 12:49
June Maintaince (Resolved)
  • Priority - Medium
  • Affecting System - VPN Firewall Rules
  • This month we will be deploying new firewall rules throughout the network. The downtime should be minimal but anyone experiencing anomalies should get in contact with us.

  • Date - 06/09/2015 12:19 - 07/10/2015 21:08
  • Last Updated - 06/09/2015 12:47
UK Network Outage (Resolved)
  • Priority - Critical
  • Affecting Server - UK2-OVPN
  • There is an outage in the UK. We are investigating.

  • Date - 06/08/2015 03:18 - 06/09/2015 12:19
  • Last Updated - 06/08/2015 03:18
Kansas City Outage (Resolved)
  • Priority - Critical
  • Affecting Server - Kansas City
  • Kansas City is experiencing connectivity problems. We are investigating

  • Date - 05/27/2015 00:17 - 06/03/2015 21:50
  • Last Updated - 05/27/2015 00:19
Smart Streaming Error - HAProxy (Resolved)
  • Priority - Critical
  • Affecting Server - HaProxy
  • It appears some of our SMART streaming IPs got blocked by providers like Hulu. We have rerouted traffic to another IP and the issue has been fixed. Thank you for your patience.

  • Date - 05/18/2015 13:29 - 05/27/2015 00:17
  • Last Updated - 05/20/2015 13:31
Los Angeles Department of Water and Power wil (Resolved)
  • Priority - Critical
  • Affecting Server - LAX1
  • The maintenance will take place on Tuesday May 19th and Wednesday May 20th at 0500 hours.

    Technicians will be working on the transformers from 4-8 hours each day. All critical load will be transferred to Generator power during this window. We do not expect any direct downtime as a result of the maintaince however in the event of a power failure we caution users against leaving their connection unattended during these hours. 

  • Date - 05/19/2015 05:00 - 05/27/2015 00:17
  • Last Updated - 05/17/2015 12:25
Romania Kernel Panic (Resolved)
  • Priority - Critical
  • Affecting Server - Romania
  • Our OpenVPN servers kernel paniced at 12:10 AM EST on 5/15/2015 it is back online on the failover cluster depending on the cause expect intermittent problems for OpenVPN connections here.

  • Date - 05/14/2015 23:26 - 05/16/2015 15:38
  • Last Updated - 05/14/2015 23:29
May LiquidVPN Maintenance (Resolved)
  • Priority - High
  • Affecting System - LiquidVPN Network
  • 5/2/2015
    California Emergency Patching for both servers.


  • Date - 05/01/2015 00:00 - 06/03/2015 21:51
  • Last Updated - 05/03/2015 00:39
Weekly Scheduled Maintenance Information (Resolved)
  • Priority - Medium
  • Affecting System - VPN Network
  • Starting April 17 2015 we will begin weekly scheduled maintaince on our servers to keep users connection performing at an optimum level. This scheduled maintaince will run from now on every week. This will be the only announcement for weekly maintaince unless any type of downtime will be required. 


    Friday - Canada, UK, NL
    Saturday - USA
    Sunday - The rest of the network

    4/17/2015 @ 1:00 PM EST
    UK 2 - Kernel Updates on OpenVPN and PPP servers will require both servers to be rebooted. There will be about 30 minutes of downtime on both of the UK 2 servers.

    5/2/2015
    California Emergency Patching.

  • Date - 04/14/2015 15:23 - 05/03/2015 00:36
  • Last Updated - 05/03/2015 00:36
Toronto Dynamic UDP Daemon Quit Functioning (Resolved)
  • Priority - High
  • Affecting Server - Canada
  • Earlier today toronto dynamic udp stopped allowing connections. We have corrected the problem and it should now work correctly.

  • Date - 04/27/2015 00:29 - 04/27/2015 01:30
  • Last Updated - 04/27/2015 00:30
Switzerland Network Issues (Resolved)
  • Priority - Medium
  • Affecting Server - Switzerland
  • The datacenter in Switzerland is reporting issues with their main upstream provider. Expect some connection problems while they work through the problems. This ticket will be updated once more news becomes available.

    We have rebooted our router in Switzerland. Connectivity will be inturrupted for a time while the reboot happens.

  • Date - 04/25/2015 18:13 - 04/27/2015 00:31
  • Last Updated - 04/25/2015 20:23
Virginia Connectivity issues (Resolved)
  • Priority - Critical
  • Affecting Server - Virginia
  • Virgina has stopped responding. We are investigating.


    Update 3:22 PM EST

    Apparently Leaseweb has routed our IPs to another server again. We are working to get this corrected.

    Update 6:27 PM EST

    Leaseweb has confirmed that one of their users has stolen our IP(s) but their support has been very slow and unhelpful so we are waiting for a higher level engineer to fix this.

    Update 4/22/2015 8:17 PM EST
    Virginia gateway has been rebuilt and is now online.

  • Date - 04/21/2015 12:29 - 04/22/2015 19:18
  • Last Updated - 04/22/2015 19:18
Romania Network Issues (Resolved)
  • Priority - Critical
  • Affecting Server - Romania
  • Romania appears to be having network issues. We are seeing packet loss at two routers there.

  • Date - 04/19/2015 21:50 - 04/21/2015 12:30
  • Last Updated - 04/19/2015 21:52
Virginia OpenVPN issues resolved (Resolved)
  • Priority - High
  • Affecting Server - Virginia
  • For several hours this morning virginia openvpn was having problems staying online. We have remedied this and it should now be available for everyone once again.

  • Date - 04/13/2015 15:06 - 04/13/2015 16:07
  • Last Updated - 04/13/2015 15:08
Adding More California Servers (Resolved)
  • Priority - Medium
  • Affecting Server - LAX1
  • We have ordered 2 new servers in California and will begin installing them once the datacenter has gotten the 2 RAID cards needed for the servers.

    4/6/2015
    The servers are online and are being configured.

    4/11/2015
    Servers are online configurations are published and Liquid Viscosity connections have been updated.

  • Date - 03/30/2015 13:12 - 04/11/2015 18:48
  • Last Updated - 04/11/2015 18:48
New Website Going Live Tomorrow (Resolved)
  • Priority - High
  • Affecting System - www.liquidvpn.com
  • After months of development the new website is ready to go live. Tomorrow on 4/6/2015 at 9:00 PM EST the new webserver and website will come online. We expect minimal actual downtime (15 to 30 minutes) the new website is ran on two servers instead of 1 so the control panel address will change to my.liquidvpn.com. We will make an announcement about an hour before the process begins and then make another one once it is over.

  • Date - 04/06/2015 21:00 - 04/07/2015 19:16
  • Last Updated - 04/05/2015 23:45
Emergency Maintenance in Romania (Resolved)
  • Priority - Critical
  • Affecting Server - phantomworx
  • Romania will undergo some emergency maintaince today. We have a failing SSD drive that needs to be replaced.

  • Date - 03/25/2015 13:15 - 03/28/2015 11:20
  • Last Updated - 03/25/2015 13:17
Amsterdam Replacement Again (Resolved)
  • Priority - Critical
  • Affecting Server - Netherlands
  • Amsterdam appears to have some issues with the routing tables. At this point we are waiting for more information. It is recommended you use the other Netherlands server until we announce that it is back online.

    Leaseweb is replacing our Amsterdam server currently it is unusable. We will push out new configs once we have the server setup.

    10:16 PM EST
    The new server is online. Everything should be back to normal

    3/20/2015 9:00 AM EST
    This morning Amsterdam began displaying the same unusual signs of daemons returning errors and packet loss. A Leaseweb engineer is testing the IPs. We should have more information soon.

  • Date - 03/18/2015 04:21 - 03/19/2015 22:16
  • Last Updated - 03/22/2015 14:35
Amsterdam OpenVPN Shared TCP Flapping (Resolved)
  • Priority - Critical
  • Affecting Server - Netherlands
  • Amsterdam OpenVPN Shared TCP has been flapping (working then erroring out until our network restarts it and the cycle repeats) we have solved the problem and it should be okay to use once again.

  • Date - 03/18/2015 02:07 - 03/18/2015 10:07
  • Last Updated - 03/18/2015 09:10
Invalid License Files (Resolved)
  • Priority - High
  • Affecting System - Liquid Viscosity Licensing
  • Users that signed up before today and have attempted to re-download the Liquid Viscosity zip file from the Client Area will recieve a corrupted License.lic file. To manually fix this you will need to regenerate Liquid Viscosity. There is a quick step by step to show you how here

    https://www.liquidvpn.com/billing/knowledgebase/407/Solve-Liquid-Viscosity-License-Errors.html  

    We are working on a final solution right now. Once it is in place this will not be required.


    This has been fixed earlier this morning. You should no longer need to use the work around.

  • Date - 03/17/2015 00:17 - 03/17/2015 12:34
  • Last Updated - 03/17/2015 12:34
Upgrade Amsterdam on 11-03-2015 at 17:00 CET (Resolved)
  • Priority - Critical
  • Affecting Server - Netherlands
  • Amsterdam will be coming offline on 11-03-2015 at 17:00 CET. We will be replacing it with a new server and will publish new confgurations once its online. We expect no more then 8 hours downtime for this location. We recommend users use one of the other Netherlands servers.

    Completed Upgrade. New Configuration Files have been uploaded.

  • Date - 03/10/2015 20:28 - 03/15/2015 23:49
  • Last Updated - 03/11/2015 15:56
Romania Emergency Maintaince (Resolved)
  • Priority - Critical
  • Affecting Server - Romania
  • Our Romania Colo is currently upgrading their network. The server should be back online in about an hour.

  • Date - 02/24/2015 19:24 - 03/05/2015 23:13
  • Last Updated - 02/24/2015 19:25
Liquid Viscosity Switzerland and UK Modulatin (Resolved)
  • Priority - Critical
  • Affecting Other - Liquid Viscosity for Windows
  • A Flaw was found in the way Liquid Viscosity imports our Switzerland and UK modulating configurations. Rendering the automatic DNS Leak protection turned off. The developers have been notified and we are waiting for a fix to be rolled out. We highly suggest you turn on DNS leak protection if you connect to Switzerland or UK Modulating on a modulating IP (Only the modulating IPs are affected)


    To turn on DNS leak protection follow these steps.

    1. Open the Preferences window
    2. Highlight the connection you want to modify
    3. Click the Gear button. It is next to the Edit button at the bottom of the preferences window
    4. Duplicate the connection
    5. Click Edit
    6. Open the Networking Tab
    7. In the DNS Section change the mode from Automatic to Full DNS
    8. Click Save and Connect

    As soon as a fix is rolled out we will make an announcement.

    An update has been pushed out. Your Liquid Viscosity will update your configs over the course of the next 24 hours.

  • Date - 02/14/2015 15:47 - 02/24/2015 19:25
  • Last Updated - 02/19/2015 17:43
Romania Routing Problem (Resolved)
  • Priority - Critical
  • Affecting Server - Romania
  • During optimizations a problem was discovered in Romania. Expect intermittent connection issues while we troubleshoot.

  • Date - 02/19/2015 17:40 - 02/24/2015 19:25
  • Last Updated - 02/19/2015 17:41
Amsterdam Login errors and Pending Server Upg (Resolved)
  • Priority - High
  • Affecting Server - Netherlands
  • Amsterdam was not allowing new connections for 443 UDP. It has been fixed and we are updating the OS. You may notice your connection restart during the update.

  • Date - 02/14/2015 20:02 - 02/14/2015 23:33
  • Last Updated - 02/14/2015 21:24
UK 1 Reboot (Resolved)
  • Priority - Critical
  • Affecting Server - phantomworx
  • UK 1 had to be rebooted due to a kernel panic. We are investigating.

  • Date - 02/12/2015 14:08 - 02/14/2015 21:24
  • Last Updated - 02/12/2015 14:09
Rolling firmware upgrades and network fixes (Resolved)
  • Priority - Critical
  • Affecting Other - Throughout the network
  • We are applying firmware upgrades to all of our Mikrotik routers. This will impact each server for up to 5 minutes.

    Kansas City SSTP has been fixed.
    Switzerland SSTP has been added

    Update 9:27 PM EST

    The rolling firmware update is completed.

  • Date - 02/09/2015 18:54 - 02/09/2015 20:27
  • Last Updated - 02/09/2015 20:27
Kansas City DDoS (Resolved)
  • Priority - High
  • Affecting Server - Kansas City
  • An MCI router is being DDoSed a hop or two in front of our KC router. KC is completely offline until this is over.

  • Date - 02/05/2015 16:51 - 02/08/2015 23:27
  • Last Updated - 02/05/2015 16:59
UK 1 Major Failure (Resolved)
  • Priority - Critical
  • Affecting System - UK 1
  • there was a major outage reported at 3:18 AM EST. We are investigating it now.


    Update 3:33 PM

    The server is back online we are still investigating the cause.

  • Date - 02/01/2015 03:28 - 02/02/2015 11:50
  • Last Updated - 02/01/2015 14:34
Upgrading throughput at 5 locations (Resolved)
  • Priority - Critical
  • Affecting Server - phantomworx
  • 5 locations will be getting a second NIC setup to double the bandwidth. Reboots are required. These will be happening throughout the day.

    California
    UK
    Switzerland
    Netherlands
    Romania

    Reboots have been completed. Please report any OpenVPN connection problems to support.

  • Date - 01/26/2015 16:24 - 01/27/2015 13:02
  • Last Updated - 01/26/2015 16:37
NJ Firewall Modifications (Resolved)
  • Priority - High
  • Affecting Server - New Jersey
  • We are working on new firewall rules in New Jersey to help stop the major increase in DMCAs there. Expect intermittent connectivity issues until the new rules are in place.

  • Date - 01/25/2015 12:09 - 02/05/2015 14:02
  • Last Updated - 01/25/2015 12:11
Romania Outage (Resolved)
  • Priority - Critical
  • Affecting Server - Romania
  • Romania has become un-responsive. We are investigating the problem right now.

  • Date - 01/20/2015 01:32 - 01/20/2015 01:42
  • Last Updated - 01/21/2015 11:44
Switzerland OpenVPN (Resolved)
  • Priority - Critical
  • Affecting Server - Switzerland
  • We are investigating an outage in Switzerland OpenVPN.

    The datacenter has addressed the issue that was causing packet loss to one of the OpenVPN IPs in Switzerland.

  • Date - 01/19/2015 11:37 - 01/19/2015 18:44
  • Last Updated - 01/19/2015 18:44
Canada and New Jersey Router Reboot (Resolved)
  • Priority - Critical
  • Affecting System - Skynet
  • Our routers in Canada and New Jersey were running outdated firmware. We updated and rebooted both routers. There was a 60 second gap in connectivity because of this reboot.

  • Date - 01/18/2015 12:25 - 01/18/2015 12:30
  • Last Updated - 01/19/2015 18:42
Kansas City Outage (Resolved)
  • Priority - Critical
  • Affecting Server - Kansas City
  • We are investigating an outage in Kansas City


    One of the sticks of RAM failed causing the server to crash.

  • Date - 01/14/2015 11:00 - 01/14/2015 16:33
  • Last Updated - 01/14/2015 16:33
UK Network Connectivity (Resolved)
  • Priority - High
  • Affecting System - UK Network
  • Our UK network appears to be flapping. We are investigating.

  • Date - 12/25/2014 00:14 - 12/26/2014 19:44
  • Last Updated - 12/25/2014 00:15
Kansas City Flapping (Resolved)
  • Priority - Medium
  • Affecting Server - phantomworx
  • Our Kansas City router is flapping We are working to resolve the issue.


    This turned out to be a DDOS aimed at a user on a public IP address. This server is not DDoS protected so the IP was simply null routed for awhile.

  • Date - 12/22/2014 01:32 - 12/22/2014 14:36
  • Last Updated - 12/22/2014 14:36
Patching Servers (Resolved)
  • Priority - Critical
  • Affecting System - Linux Servers
  • The bug CVE-2014-9322 reported yesterday required emergency testing and patching today. Some users may have noticed their sessions get disconnected this was due to our patching. We will be closely monitoring the network to make sure the emegency patching did not have any unexpected effects. 


  • Date - 12/18/2014 06:45 - 12/19/2014 09:16
  • Last Updated - 12/18/2014 15:58
California Network Flapping (Resolved)
  • Priority - Critical
  • Affecting Server - LAX1
  • California is flapping.

  • Date - 12/17/2014 19:59 - 12/17/2014 23:12
  • Last Updated - 12/17/2014 20:00
New Jersey Network Outage (Resolved)
  • Priority - Critical
  • Affecting Server - New Jersey
  • Some of our New Jersey resources have gone offline. We are investigating

    There was a DDOS it has been mitigated

  • Date - 11/29/2014 20:33 - 11/29/2014 20:41
  • Last Updated - 11/29/2014 21:35
KC Outage (Resolved)
  • Priority - Critical
  • Affecting Server - Kansas City
  • The whole KC network has gone offline. We are investigating.

    There is some issues with the internal routing tables in the KC data center. Please do not use KC until we post another update.

  • Date - 11/26/2014 21:02 - 11/27/2014 00:58
  • Last Updated - 11/26/2014 22:26
Romania PPTP/L2TP DNS Resolver (Resolved)
  • Priority - High
  • Affecting Server - Romania
  • The same ARP issue in Canada appears to be an issue in Romania as well. Users on one of the PPTP/L2TP subnets quit resolving IP addresses after the routers ARP table was refreshed.

    We have fixed the issue affecting L2TP/PPTP in Romania. Now the first 5 connections will resolve properly.

  • Date - 11/25/2014 17:40 - 11/25/2014 19:36
  • Last Updated - 11/25/2014 18:36
Canada PPTP/L2TP DNS Not Resolving (Resolved)
  • Priority - Critical
  • Affecting Server - Canada
  • Canada's PPTP/L2TP has been reported as not resolving DNS names via LiquidDNS. There was a typo in the route that went undetected due to to ARP. It has been corrected and L2TP/PPTP is now able to resolve LiquidDNS addresses.

  • Date - 11/25/2014 11:22 - 11/25/2014 14:23
  • Last Updated - 11/25/2014 13:26
Outage in Kansas City (Resolved)
  • Priority - Critical
  • Affecting Server - Kansas City
  • An outage in Kansas City has been reported

    It appears to be because of a large scale attack in the general area as reported by http://map.ipviking.com/

    Location back online

  • Date - 11/22/2014 08:10 - 11/22/2014 11:07
  • Last Updated - 11/22/2014 10:07
Abnormal Traffic Patterns in DE (Resolved)
  • Priority - High
  • Affecting Server - Germany
  • We have been experiencing some abnormal traffic patterns in DE and have shut down some of the servers there in an attempt to isolate the exact cause. If anyones legit traffic was terminated we apologize in advance.

    Update 11/19/2014

    There appears to be a misconfigured windows server on the Leaseweb network in Germany and it is routing hundreds of thousands of netbios packets to our servers. Please use Germany with care as we cannot guarantee the quality of service coming out of there until this issue is fixed.

    Update 11/20/2014 7:46 AM

    LeaseWeb finally terminated the account generating all of the netbios traffic. Germany is back online

  • Date - 11/19/2014 02:15 - 11/20/2014 06:47
  • Last Updated - 11/20/2014 06:47
IP reporting on control panel (Resolved)
  • Priority - High
  • Affecting System - website homepage label
  • We ran some updates that appears to have broken the IP display on the main control panel screen. We are investigating.

  • Date - 11/17/2014 08:25 - 11/17/2014 13:56
  • Last Updated - 11/17/2014 08:35
California Router Update (Resolved)
  • Priority - Critical
  • Affecting System - California
  • An update to the router in California has been pushed out.

  • Date - 11/13/2014 01:45 - 11/20/2014 06:47
  • Last Updated - 11/13/2014 00:47
UK Network Issue (Resolved)
  • Priority - Critical
  • Connectivity in UK is intermittent.

  • Date - 11/12/2014 14:30 - 11/12/2014 16:48
  • Last Updated - 11/12/2014 14:30
website upgrade (Resolved)
  • Priority - Critical
  • Affecting System - liquidvpn.com
  • There is going to be some intermittent issues with liquidvpn.com while we finish setting up all of the new infrastructure. Thank you for your patience.

  • Date - 10/23/2014 16:13 - 10/23/2014 23:24
  • Last Updated - 10/23/2014 16:15
New Jersey DDoS Attack (Resolved)
  • Priority - Critical
  • Affecting Other - New Jersey DDOS
  • DDoS Attack targetting the Network Core
    Incident Start: Oct 21, 2014 2:35PM Eastern
    Incident End: Oct 21, 2014 2:50PM Eastern

    At approximately 2:35PM Eastern on October 21st, 2014, our monitoring systems alerted us of issues pertaining to our New Jersey network. After researching, it was determined that a denial of service attack had targetted our network core. All malicious traffic has been mitigated and service was returned to a normal working state minutes after the notification. We are making network changes to prevent this from becoming a recurring occurence. Any subsequent maintenance activities will be scheduled and you'll receive a separate notice.


  • Date - 10/21/2014 14:35 - 10/22/2014 23:17
  • Last Updated - 10/21/2014 17:34
Romania Emergency Maintenance Sunday Oct 12th (Resolved)
  • Priority - Critical
  • Affecting Server - Romania
  • We will like to inform you that on Sunday, 12th of Octomber at 5AM (GMT + 2) we will start emergency maintenace on a core router in Romania. Most users will still be able to connect but all traffic will be impacted. We HIGHLY recommend using another location during this time. 

    Link of converted times
    http://www.worldtimebuddy.com/?qm=1&lid=122,5,2643743,8&h=122&date=2014-10-12&sln=5-6


    This ticket will be updated once it is complete.
    T
    here is a routing problem in Romania and we are looking into it. Currently only shared connections are working correctly.


    Update 10/13/2014

    The routing problem has been fixed. Romania is back online and working like it should.

  • Date - 10/12/2014 11:00 - 10/13/2014 15:22
  • Last Updated - 10/13/2014 15:22
Backup VPN Authentication Server Corrupted (Resolved)
  • Priority - Critical
  • Affecting Server - phantomworx
  • A RAID Controller failed on our backup VPN authentication server today. Services should not be affected. Unless something happens to the primary before we are able to re-create the backup on a new server. We are in the process of adding a new failover authentication server as we speak. This ticket will get updated once it is completed.

    A new backup authentication solution is now in place. This is a High Availability failover cluster.


    The network has been updated all nodes have been integrated with the new authentication system.



  • Date - 10/08/2014 15:00 - 10/09/2014 01:22
  • Last Updated - 10/09/2014 01:22
Server Updates (Resolved)
  • Priority - High
  • Affecting System - VPN Network
  • LiquidVPN has begun rolling out new server security updates. Some users may experience OpenVPN connectivity loss for a few seconds if a process restart is required on the server your connected to.

    The server updates are complete. Thank you for your patience.

  • Date - 10/05/2014 19:00 - 10/06/2014 20:13
  • Last Updated - 10/06/2014 20:13
California Infrastructure Upgrade (Resolved)
  • Priority - Medium
  • Affecting System - California
  • On 10/11/2014 California will go offline for two - four hours while our infrastructure is upgraded. We recommend using an alternative location between 9:00 PM and 1:00 AM Eastern Standard Time. To convert this to your local time please visit this site
    http://www.worldtimebuddy.com/?qm=1&lid=100,5,2643743,8&h=100&date=2014-10-11&sln=1-6

  • Date - 10/11/2014 21:00 - 10/11/2014 23:39
  • Last Updated - 10/02/2014 16:03
Smart Streaming Service Outage (Resolved)
  • Priority - Critical
  • Affecting System - Advanced Media Streaming Capabilities
  • Earlier today we had to add resources to our SMART Streaming system and upon reboot the kernel panicked. We are in the process of troubleshooting the issue. Please note that until this system comes back online your streaming capabilities may be impacted.

  • Date - 10/01/2014 12:24 - 10/01/2014 16:48
  • Last Updated - 10/01/2014 13:31
New Jersey DDoS (Resolved)
  • Priority - High
  • Affecting System - NJ OpenVPN Incoming DDoS
  • The New Jersey server has been the target of a 2002 Mbps attack at approximately 9/28/2014 4:59:16 PM (GMT -04:00:00) We have null routed the IP. This attack is aimed at the OpenVPN gateway IP so some OpenVPN connectivity may be inturrupted until the route is put back in place.

  • Date - 09/28/2014 16:04 - 09/28/2014 18:27
  • Last Updated - 09/28/2014 16:07
ShellShock and its Aftermath (Resolved)
  • Priority - Critical
  • Affecting System - All Systems Using Bash
  • On 9/24/2014 a major new bug was discovered called Shellshock CVE-2014-6271. Incomplete patches were released later that day. We patched all of our non debian servers with the incomplete patch. Earlier today we became aware of CVE-2014-7169. The patch for the new bug CVE-2014-7169 was released a few hours ago. We immediately pushed out updates to all of our infrastructure. We are now in the process of evaluating our servers for any breaches that occured before the bug became publicly known. If anything is found this ticket and an announcement will be made.

  • Date - 09/24/2014 13:20 - 10/01/2014 13:24
  • Last Updated - 09/26/2014 13:40
Rotterdam TCP Connectivity (Resolved)
  • Priority - High
  • Affecting System - Rotterdam
  • Rotterdam TCP connectivity is cut in half. Users cannot connect on any TCP ports from the alternate enterence IP.


    Problem has been resolved.

  • Date - 09/25/2014 09:33 - 09/25/2014 12:27
  • Last Updated - 09/25/2014 12:26
Romania OpenVPN Upgrade (Resolved)
  • Priority - High
  • Affecting Server - Romania
  • We are upgrading Romanias hardware to better handle the load due to the DDoS

  • Date - 09/18/2014 17:55 - 09/18/2014 19:53
  • Last Updated - 09/18/2014 17:56
Romania OpenVPN Ongoing DDoS (Resolved)
  • Priority - Medium
  • Affecting Server - phantomworx
  • Romania is suffering from an ongoing DDoS. Really multiple DDoS's to be exact. We are working on a solution right now but until it is in place the OpenVPN connections may not connect. We should have something in place tonight to get everything working like normal again.


    Update 9/18/2014

    The DDoS is still taking place off and on. This has been going on for almost 2 weeks now. If anyone has had problems connecting to Romania during our IPS and firewall rule modifications we apologize.

  • Date - 09/15/2014 18:54 - 09/20/2014 19:05
  • Last Updated - 09/18/2014 17:22
PPTP/L2TP For NL 1 Not Working After Update (Resolved)
  • Priority - High
  • Affecting Server - Netherlands
  • We are investigating an issue with NL 1 PPTP/L2TP. After updating it stopped allowing connections.

  • Date - 09/18/2014 13:54 - 09/18/2014 17:19
  • Last Updated - 09/18/2014 13:55
Romania Router is being DDoSed (Resolved)
  • Priority - Critical
  • Affecting Server - Romania
  • We are currently being DDoSed at Romania. Please use other servers until it is over.


    Update 9/8/2014

    The Romania DDOS appears to just be aimed at waisting bandwidth and interrupting/slowing down service. We are adding some QoS and L7 filters that will only allow new VPN connections up until a point to ensure users do not see any major performance issues. If you cannot connect to Romania this is most likely the cause. We will update this ticket again as soon as the DDoS is over.

  • Date - 09/07/2014 22:10 - 09/08/2014 22:23
  • Last Updated - 09/08/2014 14:53
UK UPGRADE (Resolved)
  • Priority - High
  • Affecting Server - UK2-OVPN
  • We are upgrading the UK network right now. Connections may be inturrupted while we modify the network.


  • Date - 09/03/2014 18:22 - 09/03/2014 22:29
  • Last Updated - 09/03/2014 22:29
Canada Router Maintaince (Resolved)
  • Priority - Critical
  • Affecting Server - Canada
  • Canada Router has undergone emergency maintaince this morning. It has just been rebooted.

  • Date - 09/03/2014 12:58 - 09/03/2014 18:48
  • Last Updated - 09/03/2014 13:01
Canada Cogent Uplink Performance (Resolved)
  • Priority - Medium
  • Affecting System - Brighthouse, TWC, Comcast and Road Runner Cogent Uplink to our Canada Network
  • Some LiquidVPN subscribers are getting poor connectivity to Canada. We have investigated this issue and it appears to be a problem with our colo's upstream provider to the broadband networks Bright House, TWC, Comcast and Road Runner. The upstream provider in question is Cogent. Cogent is widely known as being a budget carrier that oversells their connectivity. This issue is intermittent and it appears to only affect midwest and east coast users.

    We are looking for new resources in Canada that can provide a better connection to the ISPs in question and give us the IP address space we need to provide IP modulation and dynamic connections. Finding a better connection is simple, finding one willing to give us the kind of IPv4 space we need is a bit more difficult. We have also begun working with our colo in Canada to try and have them bypass the cogent uplink for these networks. Most likely this will not result in anything beneficial happening but it is worth mentioning. The connectivity issues appear to be heavily based on network congestion. If you have any questions or recommendations for Canada hosts please do not hesitate to get in touch with us.

    Update-

    Amanah is going to re-route our traffic through a different end point in an attempt to bypass cogent all together. We are waiting for them to let us know when it has been completed.


    Update - Amanah has bypassed cogent and the speed is vastly improved in Canada.

  • Date - 08/27/2014 06:17 - 08/28/2014 13:02
  • Last Updated - 08/28/2014 13:01
Zion OpenVPN Issues (California) (Resolved)
  • Priority - Critical
  • Affecting Server - LAX1
  • One of the California servers went offline. We are investigating.

    It appears an upstream provider went offline for a few seconds and the path to our NOC was disruppted. None of the hardware actually went offline some people may have lost their connection for a few minutes.

  • Date - 08/19/2014 01:04 - 08/19/2014 02:12
  • Last Updated - 08/19/2014 01:12
.zip files are not downloading properly (Resolved)
  • Priority - High
  • Affecting System - Website billing area
  • We updated the backend code the other day to make it easier to download the latest Liquid Viscosity files and it broke downloading of normal .zip and .dmg files. We are working on a fix for this right now. Anyone needing OpenVPN configs for the time being can get them from the following urls


    An update is being worked on as we speak.

    The update is complete and .zip downloads are working again normally.

  • Date - 08/17/2014 22:38 - 08/18/2014 16:19
  • Last Updated - 08/18/2014 15:20
California emergency reboot (Resolved)
  • Priority - Critical
  • Affecting Server - LAX1
  • Some of our California resources are being restarted due to a system failure.

  • Date - 08/17/2014 11:46 - 08/17/2014 13:54
  • Last Updated - 08/17/2014 11:47
Router Down in UK (Resolved)
  • Priority - Critical
  • Affecting Server - UK2-OVPN
  • There is a router down in UK currently. Stay tuned for updates.

    Backup router has been put in place and connectivity has been restored.

  • Date - 08/15/2014 17:00 - 08/15/2014 19:29
  • Last Updated - 08/15/2014 18:30
LiquidVPN Network Maintaince (Resolved)
  • Priority - Critical
  • Affecting System - VPN Network
  • Dear Customer,

     

    As you may be aware from recent news coverage the size of the IPv4 routing table exceeded 512,000 routes this week. This caused widespread disruption to many users, businesses and other ISP's.

     

    In preparation for the continued growth of the global routing table we are performing some emergency maintenance on our network. This involves us performing some upgrades and changes which may cause some disruption to services but is important to ensure services continue to operate normally.

     

    For those customers with diverse connectivity to our network there will be no extended downtime, however you may see sub-optimal routing for a few minutes as our network reconverges. For those customers with a single connection into our network services will be disrupted whilst this work is ongoing.

     

    We will be performing upgrades across our network within the maintenance window below. We expect no router to be unavailable for any more than 30 minutes, however in most cases we would expect services to return quicker than this.

     

    Start: Tuesday 26th August 2014 22:00 BST

    Finish: Wednesday 27th August 2014 04:00 BST

     

    Should you have any questions then please do not hesitate to raise a ticket with us at support@liquidvpn.com

  • Date - 08/26/2014 22:00 - 08/28/2014 06:16
  • Last Updated - 08/15/2014 12:18
Romania is Down (Resolved)
  • Priority - Critical
  • Affecting Server - Romania
  • Romania has just went offline. We are investigating it now.


    The uplink between road runner (comcast/brighthouse etc) and M247 our RO host started flapping. It has been fixed.

  • Date - 08/07/2014 14:30 - 08/08/2014 20:33
  • Last Updated - 08/07/2014 14:37
NL 2 DDOS (Resolved)
  • Priority - High
  • Affecting Server - NL-2
  • Our NL 2 network is being DDoSed right now. Connectivity is sketchy at this time.

  • Date - 07/31/2014 19:58 - 08/01/2014 00:20
  • Last Updated - 07/31/2014 19:59
Canada Crashed (Resolved)
  • Priority - Critical
  • Affecting Server - Canada
  • Canada crashed this morning. It is back online now and we are investigating the root cause right.

  • Date - 07/29/2014 10:56 - 07/29/2014 11:57
  • Last Updated - 07/29/2014 11:08
UK OpenVPN Server Update (Resolved)
  • Priority - Critical
  • Affecting Server - UK2-OVPN
  • Our UK Servers are not on our private DNS system yet. The 3rd party zero log DNS servers that were being used black listed some IPs. As such some people were unable to browse while connected to this location.

  • Date - 07/28/2014 01:12 - 07/28/2014 02:12
  • Last Updated - 07/28/2014 01:14
UK1 (Static IP Users Only) DDOS (Resolved)
  • Priority - Critical
  • Affecting Server - phantomworx
  • UK 1 is being DDoSed. Only users with static UK IPs are affected by this.

  • Date - 07/23/2014 23:58 - 07/25/2014 12:02
  • Last Updated - 07/24/2014 00:00
Romania Network Problems (Resolved)
  • Priority - Critical
  • Affecting Server - phantomworx
  • Romania Network is experiencing network problems

  • Date - 07/19/2014 08:16 - 07/19/2014 15:37
  • Last Updated - 07/19/2014 08:17
Server Status Page (Resolved)
  • Priority - Medium
  • Affecting Other - VPN Status Page
  • The VPN Status page is now much more accurate. You can view it by going to the client area and looking in the sidebar under VPN Management > Server Status

  • Date - 07/14/2014 20:29 - 07/14/2014 21:30
  • Last Updated - 07/14/2014 20:31
Canada cluster is flapping (Resolved)
  • Priority - Critical
  • Affecting Server - phantomworx
  • Canada cluster is going up and down (flapping) we are investigating it.

    Network is being hit with a 50Gbps DDOS. There will be some connectivity issues there until its over. 

  • Date - 07/04/2014 23:27 - 07/05/2014 08:19
  • Last Updated - 07/05/2014 01:00
New LiquidVPN Server & Security Upgrades (Resolved)
  • Priority - High
  • Affecting System - All VPN Clusters
  • This ticket will serve as the road map to the VPN Server and Security Updates outlined in this blog post. All VPN clusters will also recieve their own private and encrypted DNS servers instead of using a 3rd party DNS service.

    The server update is scheduled to go resume on June 08, 2014* After that date some connections may not work without running an update of the software or upgrading OpenVPN configuration Files. An email will be sent out prior to the update reminding everyone to upgrade their connections.

    *Due to the DDOS attacks on the webserver the server upgrade has had to be pushed back from June 05 2014 to June 08 2014 while we implement some new technology on the web server. 

    5/15/2014 - Las Vegas
    The first official node with the updated server build, improved PFS, 4096 bit security, new dedicated Certificate Authority with Air Gap and encrypted DNS is online. 

    The rest of the network will flip over to the new build in the coming weeks. Keep an eye on this ticket for the most up to date information.

    Canada - Completed on 5/18/2014

    Viscosity build w/ new connections

    https://www.liquidvpn.com/billing/dl.php?type=d&id=36 

    Viscosity Without Automatic Connection Updates
    https://www.liquidvpn.com/billing/dl.php?type=d&id=37 

    Raw OpenVPN Configs
    https://www.liquidvpn.com/billing/downloads/10/OpenVPN

    Stay tuned to this thread for more updates on our progress.

    Kansas City, New Jersey and Romania now has access to their own private Logless encrypted DNS 5/23/2014
    These 3 locations have been upgraded but the changes are still pending. We will not push the changes out until all of the servers are ready to make the switch and the users have been given warning along with links to download the new configs. If you would like to use the private DNS servers in any of these locations simply point the DNS of your TAP adapter to 10.10.10.10 (can do it from the viscosity settings or network settings)

    New Jersey has been completed and is defaulting to the private DNS and has access to the private speed test. 6/7/2014
    You will need to update your configs to connect to New Jersey OpenVPN. Currently all of New Jersey is on the private DNS by default except Dynamic TCP. We are working on getting Dynamic TCP fixed but it could take 24 hours. New Jersey private speedtest server is online. To use it point browser to http://test.liquid.vpn

    6/14/2014 Kansas City Migration has started. Once an OpenVPN server has 0 connections on it we will bring it down and then back up with the latest configurations. 

  • Date - 05/15/2014 19:18 - 07/09/2014 15:17
  • Last Updated - 07/04/2014 18:52
New Jersey OpenVPN Maintaince (Resolved)
  • Priority - High
  • Affecting Server - phantomworx
  • New Jersey is having more bandwidth added. Downtime will last 15 to 45 minutes.

    New Jersey is completely back online. 

  • Date - 07/04/2014 16:13 - 07/04/2014 18:51
  • Last Updated - 07/04/2014 18:51
Network Maintenance Event 6/27/2014 (Resolved)
  • Priority - High
  • Affecting Server - Kansas City
  • Start Time: 15:00 CST, 06/27/2014

    Duration: Approximately 30 minutes

    New distribution routers are being rolled out. 

  • Date - 06/24/2014 22:32 - 07/04/2014 18:51
  • Last Updated - 06/24/2014 22:34
UK 1 Failure (Static IP's) (Resolved)
  • Priority - Critical
  • OpenVPN Services on UK 1 are back online. There was a problem and the server rebooted on its own last night. The services restarted theirselves but did not do so correctly and never came fully online.

  • Date - 06/23/2014 09:20 - 06/23/2014 10:40
  • Last Updated - 06/23/2014 09:42
New Accounts Not Being Created Automatically (Resolved)
  • Priority - Critical
  • Affecting Server - masterradius
  • Currently new accounts are not being created automatically. It is because of the new CDN and some security we have recently added. Any new signups must be created by hand. Please stay tuned to this ticket for updates.


    Issue has been resolved.

  • Date - 06/21/2014 20:42 - 06/21/2014 23:45
  • Last Updated - 06/21/2014 23:45
Mail Routing Problems (Resolved)
  • Priority - Critical
  • Affecting System - Email
  • Currently some incoming email is falsely being marked as spam and bounced. We are investigating it now.

  • Date - 06/17/2014 15:32 - 06/18/2014 16:00
  • Last Updated - 06/17/2014 15:35
TOR Exit Nodes Blocked (Resolved)
  • Priority - Medium
  • Affecting System - Website
  • The new WAF was blocking Tor traffic. The rule has been modified and Tor users can once again browse the site.

  • Date - 06/10/2014 23:01 - 06/10/2014 23:01
  • Last Updated - 06/10/2014 22:03
Webserver DDOS (Resolved)
  • Priority - High
  • Affecting Server - host
  • Our webserver was the target of two seperate 1500Mbps DDOS attacks yesterday. We have enabled DDOS protection and are monitoring the situation. Because parts of the webserver were inaccessible during the DDOS attacks we have begun switching some core systems over to an isolated server on the VLAN. it wont help much for this attack but would offer some safeguards for any future attacks.

    6/04/2014
    There was a 3rd attack this morning. We have changed the IP addresses of our server so that the DDOS protection can mitigate any further DDOS attempts.

    6/06/2014
    Everything appears to be back to normal right now but a lot of internal changes have been made to the web servers. Please report any odd findings or out of place posts to us immediately. Any user finding a glitch and reporting it will be credited 3 months or more of free service. This ticket will remain open until we are reasonably sure the attacks have stopped.


    6/10/2014
    Closing the ticket

  • Date - 06/01/2014 13:59 - 06/10/2014 13:48
  • Last Updated - 06/10/2014 12:51
Vulnerability Patching Scheduled Tonight (Resolved)
  • Priority - High
  • Affecting System - Webserver and VPN nodes
  • There has been a criticle vulnerability found in OpenSSL again today http://www.theguardian.com/technology/2014/jun/06/heartbleed-openssl-bug-security-vulnerabilities We must patch and reboot all of our OpenVPN nodes and the web server. We will try and take this window to switch over more of our VPN nodes to our new network stack if time permits. A new Liquid Viscosity Client has already been released. Within 24 hours you will be prompted to update the client.

    We apologize in advance for the short notice and any inconvenience that this Hardware Node maintenance may cause. This critical bug fix within the kernel requires us to put this update in place as soon as possible. If you have any questions, please feel free to open a ticket. Please note that the nature of this bug renders it virtually ineffective against our VPN connections but we are still moving forward with the maintaince. 


    Security Advisory https://www.openssl.org/news/secadv_20140605.txt

  • Date - 06/06/2014 22:00 - 06/07/2014 17:52
  • Last Updated - 06/06/2014 18:37
Las Vegas OpenVPN (Resolved)
  • Priority - Critical
  • Affecting Server - phantomworx
  • OpenVPN cluster went offline. 

  • Date - 05/31/2014 16:43 - 05/31/2014 19:47
  • Last Updated - 05/31/2014 16:44
UK 2 Outage (Resolved)
  • Priority - Critical
  • Affecting Server - phantomworx
  • We are investigating

  • Date - 05/29/2014 18:08 - 05/29/2014 20:38
  • Last Updated - 05/29/2014 19:38
Kansas City Outage (Resolved)
  • Priority - Critical
  • Affecting Server - phantomworx
  • Investigating the outage

    There is a major hardware failure in Kansas City. Outage could last several hours.

  • Date - 05/24/2014 16:17 - 05/24/2014 22:54
  • Last Updated - 05/24/2014 22:54
Canada Offline (Resolved)
  • Priority - Critical
  • Affecting Server - phantomworx
  • Canada is offline. We are investigating.

    This appears to be the same thing that happened 5 days ago. We have just upgraded the Canada firmware. A reboot will happen in 15 minutes.

  • Date - 05/23/2014 16:46 - 05/23/2014 18:29
  • Last Updated - 05/23/2014 18:29
Romania Network Problems (Resolved)
  • Priority - High
  • Affecting Server - phantomworx
  • Romaina is flapping (up and down) there may be intermittent connectivity problems until we find out what is going on with the network.

    Update 8:04 PM EST
    A Cisco tech advised an IOS upgrade. It did not fix the problem and now the core router must be replaced. All network traffic to and from Romania is now being routed through a single overloaded router. Connections to Romaina will be sketchy until this is resolved.

  • Date - 05/21/2014 11:45 - 05/23/2014 16:47
  • Last Updated - 05/21/2014 19:07
New Jersey Outage (Resolved)
  • Priority - Critical
  • Affecting Server - phantomworx
  • Our core router crashed in New Jersey this morning. We are investigating the overall cause. It is back online.

  • Date - 05/20/2014 10:02 - 05/21/2014 00:35
  • Last Updated - 05/20/2014 10:03
Canada Emergency Maintaince (Resolved)
Canada Updates Require a Reboot at 4:30 EST (Resolved)
  • Priority - Medium
  • Affecting Server - phantomworx
  • We will be rebooting all 3 servers in Canada at 4:30 PM EST today to apply patches. The expected downtime is less then 10 minutes.

  • Date - 05/15/2014 16:36 - 06/06/2014 16:37
  • Last Updated - 05/15/2014 12:38
Kansas City Outage (Resolved)
  • Priority - Critical
  • Affecting Server - Kansas City
  • We are investigating an outage in Kansas City.

    Our router went offline. It is back online but we are investigating the root cause. Will update as more information becomes available. 

  • Date - 05/11/2014 13:08 - 05/15/2014 12:38
  • Last Updated - 05/11/2014 13:24
Canada Network Down (Resolved)
  • Priority - Critical
  • Affecting Server - Canada
  • Canada's network was completely offline this morning. They had a core router and switch fail. It is back online now.

  • Date - 05/10/2014 07:54 - 05/11/2014 01:11
  • Last Updated - 05/10/2014 12:55
Frankfurt Outage (Resolved)
  • Priority - Critical
  • Affecting Server - Germany
  • Major outage in Leasewebs germany datacenter. All Germany locations are offline.

    Network is back online

  • Date - 04/26/2014 22:22 - 04/27/2014 02:01
  • Last Updated - 04/27/2014 02:01
LiquidIPS Detection Mode (Resolved)
  • Priority - High
  • Affecting System - IPS
  • Our new LiquidIPS has been put back into learning mode while we work on eliminating any false positives that was causing some legitimate traffic to be rate limited or in a couple cases dropped all together. The more users using LiquidIPS the faster it will go into full IPS mode again. If you would like to help us get LiquidIPS up to speed open a ticket and we will enable it on your account for 1 month free of charge.

  • Date - 04/21/2014 17:14 - 05/15/2014 19:18
  • Last Updated - 04/21/2014 17:22
Canada Connects but Can not browse (Resolved)
  • Priority - Critical
  • Affecting Server - Canada
  • Our Canada host www.Amanah.com null routed (AKA Blocked) 125 of our IP addresses because we recieved 3 DMCA notices from them this morning. We are waiting to find out what is going to happen here.

  • Date - 04/10/2014 19:04 - 04/21/2014 15:46
  • Last Updated - 04/21/2014 15:46
Service Upgrade (TLS Authentication) (Resolved)
  • Priority - Medium
  • Affecting System - All Servers
  • In an effort to provide users with the highest security we will begin updating our servers configurations to provide a standardized configuration across all of our servers. This means the oldest servers that are using an older version of OpenVPN and using a non standard encryption will be updated to our new standards. We will also begin providing the extra laywer of security (TLS-Auth) currently in place in Kansas City across the whole network. We will implement TLS Authentication on all connections except IP modulation where there is a massive slowdown when TLS-Auth is enabled. This will require the use of an extra key (ta.key) the configuration files will need to be updated to connect. Because of this we will do the rollout gradually. Starting today with Canada and Kansas City. Here is our Schedule and the notes from the installations. This may change depending on support requests related to the changes.


    3/29/2014 Changes
    Canada - Add TLS-Auth and Ta.Key
    Kansas City Remove TLS-Auth on Modulating Connections

    Completed


    4/14/2014
    Due to the heartbleed bug this minor upgrade has instead become a complete overhaul when it comes to security. The overhaul will require clients to download all new configurations and certificates. It will inturrupt service and because of this our plan is get everything ready and then switch everything over at 1 time. Expect more news in the following weeks as the update gets closer to its golive date. We are expecting it to happen sometime in May.


    4/3/2014 Changes

    Netherlands - Upgrade Ciphers, add TLS-Auth

    4/8/2014 Changes
    Germany - Upgrade Ciphers, add TLS-Auth

    4/13/2014 Changes
    New Jersey - Add TLS-Auth

    4/18/2014 Changes
    Romania - Add TLS-Auth

    4/23/2014 Changes
    UK - Add TLS-Auth


  • Date - 03/29/2014 14:20 - 05/02/2014 22:59
  • Last Updated - 04/14/2014 00:02
VPN Script Library back online (Resolved)
  • Priority - High
  • Affecting System - https://www.liquidvpn.com/script-library-categries/
  • VPN Script Library was giving out 404 errors. It has been fixed and is now back online.

  • Date - 04/12/2014 17:51 - 04/12/2014 18:52
  • Last Updated - 04/12/2014 17:53
PPTP not working on heartbleed updated server (Resolved)
  • Priority - Critical
  • Affecting Server - phantomworx
  • Apparently some of the updates ran to patch the heartbleed vulnerability has stopped PPTP from working. We are investigating this problem right now and will post an update as soon as we find a fix.



    PPTP is working again on all the servers except Canada which is still null routed.

  • Date - 04/11/2014 01:31 - 04/11/2014 14:57
  • Last Updated - 04/11/2014 13:58
Heartbleed SSL Exploit Update (Resolved)
  • Priority - Critical
  • Affecting System - All Servers
  • The hearbleed SSL exploit was announced and we have been updating all of our servers regenerating SSL certificates and testing our server build all day. All of the sessions have been restarted and over the next 24 hours we will be restarting servers one by one.


    Update 4/8/2014 4:00 EDT

    Some of our OpenVPN servers used 1.0.1f which was a vulnerable OpenSSL version. On 4/7/2014 at 3:00 EDT we began patching all of our affected VPN servers. At 9:00 EDT the last OpenVPN server was patched. If you were on a vulnerable OpenVPN server you are not in jeopardy of having your OpenVPN data channel compromised because LiquidVPN uses ephemeral session keys and perfect forward secrecy. There is a chance that VPN username and passwords could be compromised but this would only allow a 3rd party to use one of your VPN sessions. We do highly recommend users change their username and password from the client area.

    Our webserver used a vulnerable OpenSSL version and was upgraded at 9:00 EDT. This poses minimal risk to anyone unless they are manually forcing their browsers to not support Perfect Forward Secrecy or are using Yandexbot 3, IE 6, IE 8.


    Update 4/9/2014 18:23

    Our new SSL certificate has been issued for LiquidVPN.com and is now in place. We are still doing some detailed testing and once we are satisified that everything including the Liquid Viscosity patch is rolled out expect a more detailed post on our blog. The Liquid Viscosity update is in beta right now.

  • Date - 04/07/2014 16:56 - 04/09/2014 10:47
  • Last Updated - 04/09/2014 18:30
Canada OpenVPN Filesystem Error (Resolved)
  • Priority - Critical
  • Affecting Server - Canada
  • The OpenSSL bug that was announced on 4/7/2014 required us to do some upgrades and we discovered a problem with the server that runs OpenVPN in Canada. We are investigating it now.

  • Date - 04/08/2014 21:24 - 04/09/2014 10:47
  • Last Updated - 04/08/2014 21:27
Server Monitor Accuracy (Resolved)
  • Priority - Medium
  • Affecting System - PUPKC5
  • VPN Server monitor only polls 1 authorization server instead of all 3.

  • Date - 02/24/2014 20:17 - 05/15/2014 12:39
  • Last Updated - 04/05/2014 18:06
Canada PPTP/L2TP Offline (Resolved)
  • Priority - Critical
  • Affecting Server - Canada
  • PPTP/L2TP was not configured properly. It was only allowing a limited number of users to connect to it before refusing to grant access. It is now resolved and is working properly.

  • Date - 03/31/2014 00:00 - 04/01/2014 17:08
  • Last Updated - 03/31/2014 19:46
Emergency Server Upgrade (Resolved)
  • Priority - Critical
  • Affecting System - LiquidVPN Website and Billing Portal
  • Our server has been recieving a lot more traffic lately and it has made our website respond slowly from time to time. We have tried just about everything to mitigate the problem without taking the server down to upgrade it. We are now at the point where it must be upgrades. The process has been started. There will be about 30 minutes of downtime at the end of the upgrade. Please be patient everything will be back online soon.


    Thanks,

    Dave Cox
    www.liquidvpn.com

  • Date - 03/31/2014 18:38 - 04/01/2014 17:08
  • Last Updated - 03/31/2014 18:41
New UK Server - Maintenance (Resolved)
  • Priority - Critical
  • Affecting Server - UK2-OVPN
  • The new UK server will be coming down for maintaince shortly. We need to fix a small issue with the routing and network adapters. Hope to have it back online in under 30 minutes.

  • Date - 03/28/2014 17:47 - 03/28/2014 22:00
  • Last Updated - 03/29/2014 17:54
Network Bandwidth Bottleneck (Resolved)
  • Priority - Critical
  • The UK node has become very popular and has showed down considerably. Adding another server to deal with the current load. ETA to finalize installation is 3/26/2014

  • Date - 03/25/2014 11:39 - 03/28/2014 17:41
  • Last Updated - 03/28/2014 17:41
Server locked up (Resolved)
  • Priority - Critical
  • Affecting Server - Romania
  • One of our romania servers locked up. It was rebooted. It is working again. We are investigating the cause.

  • Date - 03/27/2014 02:02 - 03/27/2014 02:12
  • Last Updated - 03/27/2014 01:33
Add Dynamic - Shared - Modulating Canada Node (Resolved)
  • Priority - Medium
  • Affecting System - VPN Network
  • We just signed up for new servers in Canada. Installation and testing will happen over the weekend. We expect the servers to be online by Tuesday. This will add over 200 new IP addresses to our service.

  • Date - 03/12/2014 17:06 - 03/18/2014 00:00
  • Last Updated - 03/22/2014 11:54
Romania OpenVPN NIC failed (Resolved)
  • Priority - Critical
  • Affecting Server - Romania
  • Romania router suffered a hardware failure this morning. The Nic used for our routing failed. Meaning any openVPN connections stopped working at about 10:45 AM this morning. Everything is back online now 

  • Date - 03/22/2014 09:37 - 03/22/2014 12:48
  • Last Updated - 03/22/2014 11:54
Adding Server resources to Canada Cluster (Resolved)
  • Priority - Critical
  • Affecting Server - masterradius
  • We are aware of the popularity of the new Canada network and as such we are adding 2 more network cards to the servers there. Expect intermittent downtime for the next hour or so.

  • Date - 03/21/2014 22:09 - 03/22/2014 02:23
  • Last Updated - 03/22/2014 01:23
[New Jersey] DDoS Attacks - 108.61.1xx.xxx (Resolved)
  • Priority - High
  • Affecting Server - New Jersey
  • The IP 108.61.1xx.xxx has been the target of a 9565 Mbps attack at approximately 3/13/2014 8:11:16 PM (GMT -04:00:00). The attack has been logged and we have null routed the IP in question.

    The following IP addresses have been logged and reported.

    221.7.252.40
    151.100.4.54
    192.198.84.17
    54.219.164.106
    151.100.4.93
    212.126.101.131
    217.148.93.70
    188.130.33.170
    222.81.175.158
    89.45.194.230
    121.18.213.231
    93.113.171.24
    218.205.155.149
    103.13.229.224
    220.180.207.101
    54.84.218.203
    202.29.80.23
    217.146.74.17
    210.34.14.31
    221.11.21.37
    221.7.151.221
    187.17.71.66
    210.6.112.58
    212.78.81.18
    200.27.180.203
    202.143.147.83
    65.111.175.90
    60.191.40.42
    163.27.140.1
    62.75.216.58
    163.27.197.1
    163.23.96.1
    222.27.186.144
    222.16.165.66
    211.25.206.34
    175.135.203.122
    202.207.16.252
    218.90.220.226
    212.46.48.28
    61.178.145.195
    116.249.124.67
    113.135.195.55
    163.27.206.196
    87.230.88.162
    193.231.3.84
    222.220.144.219
    202.143.190.246
    192.192.82.104
    210.216.163.2
    212.72.182.175
    81.29.176.54
    58.216.224.60
    61.184.81.130
    163.27.177.2
    222.77.182.220
    27.17.53.25
    216.145.6.2
    221.139.104.88
    185.15.244.224
    64.62.145.42
    117.27.146.30
    174.121.117.93
    59.11.85.244
    63.147.125.206
    222.174.12.251
    222.240.184.178
    183.178.12.246
    61.136.169.194
    213.247.44.39
    203.157.10.19
    115.179.101.52
    221.206.36.230
    60.172.158.122
    195.130.72.10
    113.10.201.250
    211.9.40.73
    123.157.211.242
    59.53.216.133
    46.218.77.59
    221.143.42.12
    222.187.207.171
    163.13.129.26
    218.65.221.112
    163.27.235.58
    84.22.33.10
    61.139.33.29
    123.233.250.158
    220.163.43.114
    203.90.188.60
    113.4.136.227
    192.249.65.173
    64.237.44.81
    211.63.1.144
    211.115.107.202
    87.116.5.110
    183.63.137.5
    222.132.78.2
    221.131.2.165
    192.161.63.24
    221.209.110.10
    218.106.182.78
    222.184.251.130
    140.123.176.242
    61.187.98.31
    192.198.80.122
    54.200.223.250
    89.19.23.122
    212.75.225.67
    218.29.15.44
    210.0.198.236
    212.112.241.96
    103.242.233.22
    121.8.210.27
    54.213.196.33
    132.205.57.226
    122.228.198.3
    213.134.35.9
    54.72.5.158
    114.35.113.38
    210.51.47.201
    159.226.94.123
    184.22.42.36
    66.226.30.100
    84.14.58.3
    125.39.8.3
    118.244.203.153
    221.193.200.102
    123.131.134.18
    114.32.102.20
    80.92.240.53
    202.101.184.35
    222.24.19.56
    69.69.204.78
    116.255.163.49
    140.113.75.110
    222.24.19.122
    123.138.18.230
    87.229.108.184

  • Date - 03/13/2014 20:11 - 03/14/2014 00:16
  • Last Updated - 03/13/2014 19:39
VPN Configuration Repository Emergency Mainte (Resolved)
  • Priority - High
  • Affecting Other - https://www.liquidvpn.com/billing/vpndownloads.php
  • Currently our VPN Configuration File Repo is undergoing emergency maintenance. Please use the downloads from 
    https://www.liquidvpn.com/billing/downloads.php

    Update:
    Everything is now located in our downloads section use the link provided above.

  • Date - 03/04/2014 22:05 - 03/09/2014 20:23
  • Last Updated - 03/09/2014 19:24
Romainia DDOS (Resolved)
  • Priority - Low
  • Affecting Server - Romania
  • Someone attempted to bog down Romaina with a DNS based DDOS attempt today

  • Date - 02/28/2014 14:34 - 02/28/2014 15:37
  • Last Updated - 02/28/2014 14:45
Unable to reset password via forgot password (Resolved)
  • Priority - High
  • Affecting System - www.liquidvpn.com/billing/
  • All users that forget their passwords to our billing area will need to open a ticket to have their passwords reset until we can find out what is keeping the new passwords from being sent out.

  • Date - 02/27/2014 20:19 - 03/01/2014 15:23
  • Last Updated - 02/27/2014 20:21
Kansas City - Outage (Resolved)
  • Priority - Critical
  • Affecting Server - Kansas City
  • We are investigating the router problem we are having inside the Kansas City node.

  • Date - 08/31/2013 14:51 - 08/31/2013 18:51
  • Last Updated - 02/24/2014 20:42
USA West OpenVPN & Modulating IPs (Resolved)
  • Priority - Critical
  • Affecting System - USA West
  • USA West router rebooted without proper shutdown causing both instances of OpenVPN (Regular and Modulating IP) to quit receiving connections. It has been fixed. We are investigating root cause.

  • Date - 09/12/2013 09:33 - 09/12/2013 13:34
  • Last Updated - 02/24/2014 20:42
Ziln and KC upgrades (Resolved)
  • Priority - Critical
  • Affecting Other - Ziln and Kansas City
  • Kansas City and Ziln is being upgraded. Will need to restart each of them for changes to take effect.

  • Date - 09/24/2013 18:05 - 09/25/2013 01:42
  • Last Updated - 02/24/2014 20:42
US South Maintenance (Resolved)
  • Priority - Critical
  • Affecting System - USA South
  • US South network is being investigated for problems.

  • Date - 01/18/2014 21:09 - 01/18/2014 20:14
  • Last Updated - 02/24/2014 20:41
Kansas City Dynamic is rolling (Resolved)
  • Priority - Critical
  • Affecting Server - Kansas City
  • We had to proactively block some things on Midwest because of the abuser we have on the network. We accidently added Dynamic IPs to Modulating and as such everyone connected to Dynamic UDP is modulating. We are correcting this and the speed will return to that port soon.

  • Date - 02/13/2014 11:36 - 02/13/2014 12:10
  • Last Updated - 02/24/2014 20:40
Kansas City Network (Resolved)
  • Priority - Critical
  • Affecting Server - Kansas City
  • There is a major outage at our Kansas City datacenter. All of our servers are unreachable at this time. We will update this ticket when there is more information to give.


    It appears that the network is back online and that it was a problem with Hurricane Electric. (Shocking)

  • Date - 02/24/2014 18:20 - 02/24/2014 18:25
  • Last Updated - 02/24/2014 20:40
Logging enabled on USA South (Resolved)
  • Priority - High
  • Affecting System - USA South
  • Logging will be enabled on USA South starting at 2:00 AM 2/13/2014 EST


    This will effect all of our USA South network. This is in response to an attack made against Bank of America from our USA south network. 


    2/12/2014 4:58 PM Update

    The data center has terminated all 8 of our servers located there. We wanted to give everyone 24 hours before we enabled logging so that people would have a chance to see the mailer we sent out and/or this but Bank of america sent 2 more abuse reports from yesterday and today plus there were countless more SPAM and DMCA reports. The network will be much fuller until we can replace those servers. Please bare with us.

  • Date - 02/11/2014 23:32 - 02/20/2014 20:40
  • Last Updated - 02/12/2014 19:11
Main Auth Server for Personal VPN Plan (Resolved)
  • Priority - Critical
  • Affecting Server - masterradius
  • Our main auth server for the personal unlimited plan has been restored but some of the accounts seem to be corrupted. If you have an affected account please contact support so that we can fix it for you.

  • Date - 02/10/2014 20:19 - 02/12/2014 19:11
  • Last Updated - 02/11/2014 20:27
OpenVPN KC-USWest-Ziln TCP Connectivity (Resolved)
  • Priority - Critical
  • Affecting System - OpenVPN on Modulating Nodes
  • We have implemented a new network configuration to help releive the processing load on our Modulating servers. It is live on the Dallas node only. After we finish testing it we will push it out to the other modulating servers. Until then you may experience connectivity issues with Ziln, Kansas City and USA West.


    Kansas City has been completely upgraded and is now working fully.

  • Date - 09/26/2013 20:06 - 10/08/2013 12:49
  • Last Updated - 10/05/2013 10:51
Dedicated IP Servers are down while we add ne (Resolved)
  • Priority - High
  • Affecting Other - Netherlands and Midwest
  • Working on the new Netherlands and Midwest Nodes that contain the dedicated IPs. There has been some routing problems that are out of our control. Will update again soon.

  • Date - 09/05/2013 15:53 - 09/06/2013 14:14
  • Last Updated - 09/05/2013 19:47
Ziln Adding IPs (Resolved)
  • Priority - Critical
  • Affecting System - EU ZILN
  • Adding IPs to ZILN.

  • Date - 09/04/2013 22:48 - 09/04/2013 23:46
  • Last Updated - 09/05/2013 03:48