This is one of those awkward embarrassing days where Testifor.com almost stopped working for 2 days… Morning Saturday November 16, 2013 to Evening, November 17, 2013… Some by mistake of our server and some our own petty mistakes… So lets go on with the story of what went wrong and we would never want to repeat it… (we want to look at this blog in the future and laugh about it)
The beauty of it was that it failed just the next day after we had exhibited ourselves at the most prestigious TECHCRUNCH INDIA Event held in Bangalore on Friday, November 15th 2013.
On November 16, 2013, Morning over by at about 10:00 AM we started getting Exception Notifications on our email (We get an email whenever any thing fails on the server) and initially we neglected it as when we tested on the server the website was working fine… Suddenly over at about 12:00 noon… the errors increased and we too panicked. We ran more number of tests and suddenly out of nowhere the server failed to respond and was not even letting us to login
We had no clue on what to do , but we were very determined to solve the problem and get out of the problem… So what we did
1. We did a Soft Reboot on the server and it did not work… rather it locked us out completely (The server guys calls it SSH port 22 was closed).. We tried that once again it we were complete lost.
2. We did a Hard Reboot on the server…but this time it gave us hope… at least the SSH opened and we could enter into the server
We took actions and got the server to work… however we tried to evaluate the problem we couldn’t. It was not a Hacker attack or a compromised port…. We didn’t even find any RootKits… So we brought it to notice of our Hosting providers.
They wanted to run some tests and had to bring the server down for about 1 hrs time, so we decided to do it at midnight i.e. between 2:00 AM to 5:00 AM which is our peak closure time.
On November 17, 2013, our hosting providers ran tests on the server and they say they have rectified errors, however they had to restart the server… Once again as with any restart the server, we were supposed to restart some of the services which makes the product work perfectly… which we missed. This was our mistake… which we rectified only by afternoon / evening…This caused about 1500 requests which were not served with necessary data in these 2 days.
So ideally for all our customers their widgets and plugins might not have worked between 10:00AM on Saturday 16, November 2013 to until 03:00 PM on the same day and once again between 06:00 AM on Sunday 17, November 2013 to until 04:00 PM on the same day.
We have been working on making the servers safer and better. We might be changing our servers or move to new hosting providers if that is required.
P.S: We regret if this has caused any inconvenience to our customers and we shall take steps to make things better and keep the servers safe…
For any clarifications please send me an email to email@example.com
- Satish N Kota (Creator – TestimonialsFor)