Rails Machine Status

Host Server : Imperial scheduled Maintenance

Time: 9:00 EDT on Sunday April 20, 2014.
Window duration: 1 hour
Anticipated impact: 30 minutes

Description: Server Maintenance

Impacted services: A small number of customers hosted in our GNAX facility.

We will be conducting a scheduled maintenance procedure on Sunday April 20th, 2014 at our GNAX datacenter that will affect only a small number of our customers. These customers will be notified individually of their expected impact. We expect the maintenance to last no longer than 1 hour, and will begin at 9am EST. Check back here during the maintenance for any pertinent updates.

If you have any other questions, please file a support ticket. Thanks!

railsmachine Posted Thursday, Apr 17, 2014 by Rails Machine
 

[Resolved] Intermittent connectivity to our GNAX data center

We are seeing intermittent connectivity to our GNAX facility.

We've been working with our GNAX network team for status updates and we will be following up with more details about the cause.

Update: March 31, 2014 3:25pm EST

At this time all network connectivity to GNAX has been restored. The data center reported a failure in their core networking gear and a bad failover which caused the outage. As we get more updated information we'll post it here.

Update: April 1, 2014 1:40pm EST

Here is the RCA provided by our GNAX Data Center:

On Monday, March 31 at 2:17 PM EST, GNAX routers were unable to effectively route traffic to the internet. The issue stemmed from a large peer at the TIE peering fabric flooding the peer routers with unproductive routes, which crippled our route tables on the adjacent routers and then propagated and affected our core routers as BGP neighbors. The immediate fix re-converged routes at 3:12 PM EST.

To prevent this type of incident occurring again in the future, our network team has applied more stringent access lists in those peers. Also, our stricter configuration will terminate a BGP peer if they show a sudden, unexpected increase in routes, further protecting our customers from this type of occurrence in the future.

The immediate fix was determined and implementation was started in less than 30 minutes, as our network team launched in to action. However, due to the scale and variety of our network infrastructure, it took a few hours to fully diagnose and confirm the issue from the logs, design a more permanent resolution and carefully test it.

If you have any other questions, please file a support ticket. Thanks!

railsmachine Posted Monday, Mar 31, 2014 by Rails Machine
 

Presiden't Day Holiday Support Hours

The Rails Machine office will be closed Monday 2/17 for the President's Day holiday.

We'll be back online and resume normal business hours on Tuesday 2/18.

In case something goes awry for you, we’re always reachable and available for Urgent support requests which can be filed at https://support.railsmachine.com. All other non-urgent support requests will be addressed during the next business day: Tuesday, 2/18/2014

railsmachine Posted Saturday, Feb 15, 2014 by Rails Machine
 

Martin Luther King Jr. Holiday Support Hours

The Rails Machine office will be closed Monday 1/20 for the Martin Luther King Jr holiday.

We'll be back online and resume normal business hours on Tuesday 1/21.

In case something goes awry for you, we’re always reachable and available for Urgent support requests which can be filed at https://support.railsmachine.com. All other non-urgent support requests will be addressed during the next business day: Tuesday, 1/21/2014

railsmachine Posted Friday, Jan 17, 2014 by Rails Machine
 

New Years Holiday Support Hours

Here at Rails Machine we're wrapping up another awesome year and will be spending tomorrow celebrating and reflecting with our friends and family and hope you'll be doing the same.

As many of you will be in observance of, Wednesday (January 1st, 2014) is a Rails Machine holiday, and is a perfect chance for our team to spend a litte more quality time away from the internet. For your planning purposes, please also see our full listing of Rails Machine Holidays, meaning that all non-urgent support requests submitted after 8pm on Tuesday (December 31st) will be answered on Thursday (January 2nd).

In case something goes awry for you, we're always reachable and available for Urgent support requests which can be filed at https://support.railsmachine.com.

Again, a big thanks to all of our first class customers for an impressive collaborative year and have a thankful and happy holiday season! We'll catch you next year. Have a happy and safe holiday.

railsmachine Posted Tuesday, Dec 31, 2013 by Rails Machine
 

Christmas Holiday Support Hours

Here at Rails Machine, we're pretty much a 24/7/365 shop. However, every now and then we like to untether ourselves from our laptops and spend time with our friends, family, and loved ones.

As many of you will be in observance of, Tuesday and Wednesday (December 24th and 25th, 2013) are Rails Machine holidays, and are a perfect chance for our team to open presents, drink some eggnog, and spend some quality time away from the internet. For your planning purposes, please also see our full listing of Rails Machine Holidays, meaning that all non-urgent support requests submitted after 8pm on Monday (December 23rd) will be answered on Thursday (December 26th).

In case something goes awry for you, we're always reachable and available for Urgent support requests which can be filed at https://support.railsmachine.com.

Thanks to all of our first class customers for an impressive collaborative year and have a thankful and happy holiday season!

railsmachine Posted Monday, Dec 23, 2013 by Rails Machine
 

Veteran's Day Support

The Rails Machine office will be closed on Monday, November 11th, 2013, for Veteran's Day.

We'll be back online and resume normal business hours on Tuesday, November 12th, 2013.

In case something goes awry for you, we're always reachable and available for Urgent support requests which can be filed at https://support.railsmachine.com. All other non-urgent support request will be addressed during the next business day: Tuesday, November 12th, 2013.

railsmachine Posted Friday, Nov 8, 2013 by Rails Machine
 

Loss of connectivity to our GNAX data center

To start, we would like to let everyone know connectivity to our GNAX data center in Atlanta has been restored and all sites are back online. Please keep reading for an explanation of the events that lead up to the outage and what we've done to restore connectivity.

At 9:30am EST we started receiving alerts from our monitoring systems that one of our customer's servers was no longer accessible via SSH. There were only a few alerts to start so we began blocking the source IP at the server level and investigating each event to correlate the cause.

At 9:45am EST the frequency of the alerts increased and a trend developed to point to a single IP range that was the target of the connection flooding. Initially we were blocking the IP address at the server level since it was isolated to only a few target servers.

At 9:50am EST, when the alerts starting increasing rapidly, we confirmed the target was a single IP range and not a specific customer site. As we initially blocked the incoming traffic the failed SSH connection counts started increasing as the attack progressed through the IPs in the target range. This hadn't affected the availability of applications hosted on the target server, only their access via SSH.

At 9:53am EST the inbound SSH connections flooded our edge networking gear and other team members were pulled in to assist. We contacted our data center team to assist in dropping all inbound traffic so we could apply a fix to our edge networking gear.

By 10:00am EST we stopped all incoming traffic from the source on our edge networking gear. After verifying with our data center we had measures in place all traffic was then allowed to pass and our monitoring systems started reporting all alerts were recovering.

By 10:10am EST all alerts had cleared and our manual verification of sites that were affected had been completed.

One of the counter measures most data centers employ to mitigate incoming DoS and DDoS attacks is to null route all inbound traffic to the target to prevent saturating the upstream connectivity and causing a larger scale outage across all data center customers using any particular incoming uplink.

This a slow process to detect at the data center level due to the amount of traffic needed to flood their upstream connections but is more easily identified at the data center customer level (us) where the targeted attack is taking place. Our monitoring systems, by default, monitor for SSH connectivity as well as HTTP/HTTPS connectivity and response times among other things. This helps to more quickly identify the type of attack and its source and allows us to put stop gap measures in place until a final solution is determined and applied.

At this time, all connectivity has been restored and steps to more quickly mitigate these types of attacks have been put in place to help prevent future outages.

Please file a support request if you're still having issues. Thanks!

railsmachine Posted Monday, Oct 14, 2013 by Rails Machine
 

Columbus Day Support Schedule

The Rails Machine office will be close for Columbus Day on Monday, October 14th, 2013.

We'll be open normal business hours on Tuesday, October 15th.

In case something goes awry for you, we're always reachable and available for Urgent support requests which can be filed at https://support.railsmachine.com. All other non-urgent support request will be addressed during the next business day.

railsmachine Posted Saturday, Oct 12, 2013 by Rails Machine
 

Labor Day Support Schedule

The Rails Machine office will be close for Labor Day on Monday, September 2nd, 2013.

We'll be open normal business hours on Tuesday, September 3rd.

In case something goes awry for you, we're always reachable and available for Urgent support requests which can be filed at https://support.railsmachine.com. All other non-urgent support request will be addressed during the next business day.

railsmachine Posted Friday, Aug 30, 2013 by Rails Machine