Rails Machine Status

As part of our ongoing efforts to continually improve host stability and performance, we are scheduling maintenance on the host server 'penn' on Sunday 1/29 at 9AM EST. During this maintenance, we expect 15-30 minutes of downtime for customers on this host.

We'll be updating this post as the maintenance progresses.

Here at Rails Machine, we’re pretty much a 24/7/365 shop. However, every now and then we like to untether ourselves from our laptops and spend time with our friends, family, and loved ones.

Monday, January 16th, is the birthday of Martin Luther King, Jr., and a federal holiday. It’s also a Rails Machine Holiday, meaning that all non-urgent support requests submitted Monday will be answered on Tuesday, January 17th. 

In case something goes awry for you, we’re always reachable and available for Urgent support requests which can be filed at https://support.railsmachine.com/help.

Thanks, and have a great holiday!

We’re currently experiencing some network problems at GNAX (AtlantaNAP) causing a number of sites to be unavailable. We’re investigating this now and will provide updates as we have them.

6:30AM EST We're continuing to experience network problems at our GNAX datacenter. We're currently working with out datacenter team to identify and resolve the underlying issue.

6:50AM EST We are seeing connectivity to AtlanaNAP (GNAX) returning. We called the data center to find out the status of our network connectivity and they were aware of the problem and were working to resolve it. At that time the data center engineers couldn't give specific details to the cause of the outage, but we will be following up to find out the cause.

We've confirmed recovery across aboard now, so if you see any outsanding issues following this, please file a new Support Ticket.

The host server Odouls is being rebooted for emergency maintenance. Customers with servers on this host will not be able to connect during the reboot. This issue should be resolved within 15-30 minutes. We will update at that time.

9:50am EST Maintenance is now complete, file system checks have finished and all servers are back online. Please submit a support ticket if you need any further assistance.

At approximately 7:30pm EST on 12/27 we had drop in network connectivity for less than 5 minutes that caused loss of availability for servers in our AtlantaNAP data center (GNAX).

After a thorough investigation into the cause we discovered that our uptime caused the downtime.

We were able to identify a bug in our core switching gear's OS that caused the brief network connectivity loss. In short, time stamps stored to monitor process uptime caused an integer apocalypse that sent a kill signal to the redundant routing modules, forcing them to reboot at the same time instead of a graceful rolling reboot as expected.

We have discussed this with our hardware vendor and have a path to implement a fix to prevent this from happening again in the future. We will roll out the patch during a planned maintenance window once we have full details on how it will effect the switch uptime.

Details on the maintenance, any service interruptions, and time estimates will be posted here once we have a solid plan implemented for the patch process.

Here at Rails Machine, we’re pretty much a 24/7/365 shop However, every now and then we like to untether ourselves from our laptops and spend time with our friends, family, and loved ones.

As many of you will be in observance of, Monday December 26th and Monday January 2nd are Rails Machine holidays, and are a perfect chance for our team to open presents, drink some eggnog, and spend some quality time away from the internet. For your planning purposes, please also see our full listing of Rails Machine Holidays, meaning that all non-urgent support requests submitted after 8pm on Friday (December 23rd) will be answered on Tuesday (December 27th) and the same scenario plays out for after 8pm on Friday December 30th, in that we’ll address all non-urgent support requests come bright and early Tuesday morning on January 3rd. And of course we would be most appreciative and as a favor to us, for you please to NOT break anything major during the Rails Machine Holidays. But rest assured, we’ll be around to answer your urgent support requests if you do.

Thanks to all of our first class customers for an impressive collaborative year and have a thankful and happy holiday season!

The host server Hops is being rebooted for emergency maintenance. Customers with servers on this host will not be able to connect during the reboot. This issue should be resolved within 15-30 minutes. We will update at that time.

11:15pm EST The host server is back online and file system checks are completed, servers are coming online now.

11:22pm EST All servers are back online, if you need further assistance, please open a support ticket.

The host server Hops is being rebooted for emergency maintenance. Customers with servers on this host will not be able to connect during the reboot. This issue should be resolved within 15-30 minutes. We will update at that time.

11:48pm EST The server is back online and file system checks are running now.

12:01pm EST File system checks are completing and servers are coming back online now.

12:06pm EST All servers are online now. Please submit a support ticket if you need any further assistance.

The host server Rogue is being rebooted for emergency maintenance. Customers with servers on this host will not be able to connect during the reboot. This issue should be resolved within 15-30 minutes. We will update at that time.

10:33 pm EST The host server is back online and file system checks are running now.

11:00 pm EST All file system checks are finished and servers are coming back online now.

11:10 pm EST All servers are now back online. If you need any assistance, please open a support ticket.

The host server Wolf is being rebooted for emergency maintenance. Customers with servers on this host will not be able to connect during the reboot. This issue should be resolved within 15-30 minutes. We will update at that time.

4:07pm EST Maintenance is still on going at this time.

4:40pm EST Maintenance is complete and file system checks are runnning now. Guests are coming online now.

5:28pm EST All but 3 guests are back online and we are working those now.

5:55pm EST All servers are back online now. We are in contact with any customers who's servers haven't fully booted yet.

If you are seeing any problems, please submit a support ticket.