Owner Real Deal Posted April 2, 2011 Owner Report Share Posted April 2, 2011 Over the past couple of weeks we've had some performance issues with our servers mostly during the over-night period due to a couple of factors. I want to detail why these issues have happened, what we're doing to resolve them, as well as going over some scheduled maintenance for this evening. We always do our best to be as transparent and honest as possible and as such we're not going to keep any details from you. As you may already know, we run CloudLinux on our servers to help keep things stable and to prevent the over-usage of some accounts affecting everybody else. The system is generally very solid and does a wonderful job of doing what it's supposed to do. Recently we, at the advice of CloudLinux, upgraded to a newer version of the system that was supposed to offer increased performance and reliability when in fact it turns out that we ran into a couple of serious issues with the new version of the software that were exasperated by our R1Soft backup system which can be very intensive all on it's own. During normal use the new version of CloudLinux we were running on the servers performed well and did indeed give some performance gains especially when it came to hard drive access speeds on the server. The issues arose when our R1Soft backup system was run to back up your data and protect you from hardware failure and other data loss causes. This backup process tends to be very intensive as it needs to scan every bit on the disk to look for and record any changes to the data made to keep an up-to-date backup of your data. When this backup process ran, without any warning the servers locked up forcing us to not only reboot the systems but to also abort the backup process that had already been running for an hour or more. We have downgraded our CloudLinux installations across our servers back to a version we operated with for a very long time without issues and we don't anticipate having any major issues with the backup systems from this point forward. We have always kept backups of our servers and we tend to use those backups fairly regularly to fix issues for customers such as when somebody accidentally deletes a file or drops the wrong database or database table. This process due to being very intensive does cause a few minutes of extremely slow performance every night when it runs however this usually clears up in 2 to 3 minutes. We realize that this can be annoying at best however it's the cost of keeping up to date off-server copies of all data and databases. Due to all of the issues we've faced while performing backups over the last week we feel that we cannot rely upon the quality of the current backups stored of the systems and as such we need to perform fresh backups of all servers to ensure that if the need does arise to use the backups to restore data, that the data restored is accurate and problem free. We evaluated our options as far as when to run these fresh backups to cause the least impact in service performance for our customers and we determined that Sunday nights are the best times to do such intensive processes. The next decision was whether we should wait a week or two to perform this fresh backup or to go ahead and do the backup as soon as possible. We have chosen to perform the seed backups tonight as we feel a week without reliable backups is far too long as hardware failure is not something that can be expected or planned for. We do run redundant disk arrays in our servers to help protect against drive failure (up to 2 drives can fail per server without data loss) however running redundant arrays is not a substitute for reliable daily backups of the data. This process we expect to take between 6 and 8 hours per server and once we've performed this fresh backup the backup system should not cause any major performance degradation in the future beyond the expected two to three minutes per night that it takes for the backup system to spin up per server and get started. You may be frustrated with the issues we've faced on a couple of our servers over the last two weeks and we're right there with you on that frustration. We have posted a thread on the forum with the information contained from this email as well as some additional details about the backup process for this evening. If you you have any direct questions you would like to ask you are welcome to respond to this email or to visit the forums and publicly post your thoughts / questions / comments / suggestions as well. Link to comment Share on other sites More sharing options...
kingfish Posted April 2, 2011 Report Share Posted April 2, 2011 But its Saturday night. What am I gonna do without OTR? <_< Link to comment Share on other sites More sharing options...
moeroadkill Posted April 2, 2011 Report Share Posted April 2, 2011 ugh now im forced to do homework Link to comment Share on other sites More sharing options...
Make It Naaaashty Posted April 2, 2011 Report Share Posted April 2, 2011 OTR = my life Thanks for ruining my Saturday night B... Link to comment Share on other sites More sharing options...
Warren2ThaG Posted April 2, 2011 Report Share Posted April 2, 2011 (edited) http://www.coldwellbanker.com/property?action=list&freeTextAddress=Detroit,+MI&countryId=1&numberOfBeds=2&numberOfBaths=1&maxPrice=15000 check those nice prices on homes in Michigan, you can invest too. lets all move there? Edited April 2, 2011 by Warren2ThaG Link to comment Share on other sites More sharing options...
Flash Posted April 2, 2011 Report Share Posted April 2, 2011 Are you saying I have to go out tonight? Wtf man!! Link to comment Share on other sites More sharing options...
*Swish* Posted April 2, 2011 Report Share Posted April 2, 2011 Thanks god I'm going to my friend's house Link to comment Share on other sites More sharing options...
Owner Real Deal Posted April 3, 2011 Author Owner Report Share Posted April 3, 2011 Was there any downtime? I didn't seem to notice, lol. Link to comment Share on other sites More sharing options...
moeroadkill Posted April 3, 2011 Report Share Posted April 3, 2011 i got a error page once about and hour ago. dunno how long it was out for though Link to comment Share on other sites More sharing options...
Sun Tzu Posted April 3, 2011 Report Share Posted April 3, 2011 Was there any downtime? I didn't seem to notice, lol. None for me as well. Link to comment Share on other sites More sharing options...
*Swish* Posted April 3, 2011 Report Share Posted April 3, 2011 Was there any downtime? I didn't seem to notice, lol.None but there was a moment when it took one minute to load a page Link to comment Share on other sites More sharing options...
Cavamatic Posted April 3, 2011 Report Share Posted April 3, 2011 I had slow loading pages all night. Link to comment Share on other sites More sharing options...
Warren2ThaG Posted April 3, 2011 Report Share Posted April 3, 2011 (edited) Was there any downtime? I didn't seem to notice, lol. Minor at times like loadups, but i was buzzed so it was ok. I wont bite you. Edited April 3, 2011 by Warren2ThaG Link to comment Share on other sites More sharing options...
Owner Real Deal Posted April 4, 2011 Author Owner Report Share Posted April 4, 2011 Maintenance could be tonight instead, not sure. I didn't see any downtime last night, so I'm assuming all of it will be checked tonight. Who knows at this point. Hang in there, fellas. Link to comment Share on other sites More sharing options...
Cavamatic Posted April 4, 2011 Report Share Posted April 4, 2011 Running kinda slow right now.... Link to comment Share on other sites More sharing options...
ChosenOne Posted April 4, 2011 Report Share Posted April 4, 2011 Running super slow for me right now. Link to comment Share on other sites More sharing options...
Owner Real Deal Posted April 4, 2011 Author Owner Report Share Posted April 4, 2011 Gathering speed! Link to comment Share on other sites More sharing options...
Owner Real Deal Posted April 4, 2011 Author Owner Report Share Posted April 4, 2011 The maintenance IS tonight. They are performing backups, which is very important (we know this more than most, right?), and they want to be sure they have everything backed up ASAP. We should be good tomorrow, if not late tonight (things are running a bit faster right now, anyway). Link to comment Share on other sites More sharing options...
Recommended Posts