Wednesday, April 9, 2008
We're Back...And Here's Our Tale of Woe
Posted by Jason Dunn in "Thoughts Media Status Updates" @ 08:51 AM
After what is, I believe, the longest down-time in the history of these sites, we're back and live. Such a long window of down-time deserves an explanation more detailed than a haiku (did anyone notice the first version of the haiku wasn't actually a proper haiku? And this is from the guy who just got back from Japan!), but this explanation has to reach back a few years to make complete sense.
We've been running on the same main machine for web and database serving for several years - a dual Xeon box from ev1 (we also had a second box for email and image serving). Long story short(er), it was largely unmanaged - I got help for specific problems when I needed it, but by and large it was completely left alone for almost four years. Yeah, that's very bad news from a security standpoint. I'm NOT a server guy, so I think I buried my head in the sand and hoped for the best. Not a good plan, right? Part of the problem was that the OS was old and what we really needed was to move to a new box with a new OS - but I didn't want to do that until all the sites were on vBulletin and I knew how much hardware I needed (phpBB is much more resource intensive than vBulletin is). Getting from phpBB to vBulletin took, literally, almost two years of development work - we had to develop a new CMS from scratch, something flexible and powerful enough to be used for years to come.
Somehow, miraculously, over the years we never had major problems on our main server (likely because it had no control panel software to hack) - until December of last year. A few days after my dear friend Crystal passed away, and during the weekend when I was preparing for her funeral (I was doing some video/photos stuff for it), our secondary server (email/images) got hacked via Cpanel. So while grieving her loss I was also dealing with having huge email problems and worrying about losing data. Over the next couple of days, Jorj was moving that server to a virtual machine on our main server. Incredibly, as if the universe itself was punishing me, the morning of the funeral day our other server got hacked and our install of phpBB was breached several times. No data was lost, but we had spammers and virus/spyware distributors trying to hijack our forums. We fought them off, and closed the holes as best we could. I can't explain how grateful I am to Jorj and Janak for their help - Jorj in particular spent a great deal of time working on the server in between celebrating Christmas with his family (and this is all volunteered time). His selflessness was/is awe-inspiring.
The problem with being hacked is that once it happens, you're never entirely sure what they left behind to hack you later with - the best recourse of action is to "burn the box", meaning to abandon the OS and start fresh. This is why I was never able to explain our holiday 2007 down-time to all of you - you never want to say "Oh, we were hacked" unless you're sure you're ready to withstand another attack - and we weren't. We needed a new server anyway, so we did an emergency move from our old server (which now contained two servers) to our new server purchased from The Planet. Because it happened so fast, and because we were still planning on migrating Pocket PC Thoughts to vBulletin, we didn't want to set up phpBB on the new server - so we ran our two previously hacked servers each inside a virtual machine (VM) on the new server.
Things seemed to be OK for a while, and we successfully migrated Pocket PC Thoughts to vBulletin - all still on the same unstable and wobbly VM. We migrated our email/image server to a fresh VM, and were making preparations to do the same with our main web/database server now that we were 100% on vBulletin. I asked the team to work on it while I was in Japan, and we'd move soon when I got back. I returned home on Friday the 4th, and the morning of Saturday the 5th I upgraded vBulletin to the latest version that had just come out that day or perhaps a day earlier. Things were going fine - or so I thought.
The next day, Sunday night the 6th of April, I was driving home with my wife from a family dinner, and Jon Westfall called my mobile phone: "We've been hacked - bad." So I rushed home and saw that our vBulletin forums had been hacked. It turns out that our install of vBulletin had been hacked prior to moving Pocket PC Thoughts over - and we didn't know it because the hacker chose to wait. A scramble ensued, the almost-haiku went up, and the team of Janak, Jorj, Darius, Fabrizio and myself struggled to figure out what happened and how to fix it. The hack was deep and wide - and specifically targeted at our vBulletin database. We initiated an emergency move over to a fresh VM on the new server Monday morning, but the concerns over what happened to our vBulletin database remained. For a litany of reasons we didn't have a very recent database backup, so we had to take the slow and painful steps of seeking out every possible place the hacker could have gotten into. With some great help from David from SEOvB.com, last night we managed to get things fixed up - though we lost our Pocket PC Thoughts forum template (style) and the developer I paid to create it didn't have a copy of his work....
So here we are. We're on two fresh VM installs, managed by Jorj, and one some fast hardware: a Quad-Core 2.13 GHz Kentsfield Xeon 3210 CPU, 8 GB of RAM, and some decently fast 500 GB hard drives (though we might need to upgrade these later). Now that we've jettisoned phpBB, and our old VM, things around here should be faster - though Jorj did want me to point out that because we're essentially starting over with this new VM, performance tuning will need to be done before we're really optimized. But it sure seems faster to me already!
There's a lot of work left to be done, but that's the story as to why we've been down since Sunday night. I'm much more confident now in the server we have, and the people managing it (instead of yours truly), so I believe we're going to have great uptime from now on. The Pocket PC Thoughts migration happened in a rush, so there are things that still need fixing there, including launching some subscriber features. Stay tuned, and thank you for your support - it's been a rough three months, but I believe the worst is behind us now and it's going to be a great remainder of the year for the Thoughts Media communities...
Sincerely,
Jason Dunn
Thoughts Media Inc.
We've been running on the same main machine for web and database serving for several years - a dual Xeon box from ev1 (we also had a second box for email and image serving). Long story short(er), it was largely unmanaged - I got help for specific problems when I needed it, but by and large it was completely left alone for almost four years. Yeah, that's very bad news from a security standpoint. I'm NOT a server guy, so I think I buried my head in the sand and hoped for the best. Not a good plan, right? Part of the problem was that the OS was old and what we really needed was to move to a new box with a new OS - but I didn't want to do that until all the sites were on vBulletin and I knew how much hardware I needed (phpBB is much more resource intensive than vBulletin is). Getting from phpBB to vBulletin took, literally, almost two years of development work - we had to develop a new CMS from scratch, something flexible and powerful enough to be used for years to come.
Somehow, miraculously, over the years we never had major problems on our main server (likely because it had no control panel software to hack) - until December of last year. A few days after my dear friend Crystal passed away, and during the weekend when I was preparing for her funeral (I was doing some video/photos stuff for it), our secondary server (email/images) got hacked via Cpanel. So while grieving her loss I was also dealing with having huge email problems and worrying about losing data. Over the next couple of days, Jorj was moving that server to a virtual machine on our main server. Incredibly, as if the universe itself was punishing me, the morning of the funeral day our other server got hacked and our install of phpBB was breached several times. No data was lost, but we had spammers and virus/spyware distributors trying to hijack our forums. We fought them off, and closed the holes as best we could. I can't explain how grateful I am to Jorj and Janak for their help - Jorj in particular spent a great deal of time working on the server in between celebrating Christmas with his family (and this is all volunteered time). His selflessness was/is awe-inspiring.
The problem with being hacked is that once it happens, you're never entirely sure what they left behind to hack you later with - the best recourse of action is to "burn the box", meaning to abandon the OS and start fresh. This is why I was never able to explain our holiday 2007 down-time to all of you - you never want to say "Oh, we were hacked" unless you're sure you're ready to withstand another attack - and we weren't. We needed a new server anyway, so we did an emergency move from our old server (which now contained two servers) to our new server purchased from The Planet. Because it happened so fast, and because we were still planning on migrating Pocket PC Thoughts to vBulletin, we didn't want to set up phpBB on the new server - so we ran our two previously hacked servers each inside a virtual machine (VM) on the new server.
Things seemed to be OK for a while, and we successfully migrated Pocket PC Thoughts to vBulletin - all still on the same unstable and wobbly VM. We migrated our email/image server to a fresh VM, and were making preparations to do the same with our main web/database server now that we were 100% on vBulletin. I asked the team to work on it while I was in Japan, and we'd move soon when I got back. I returned home on Friday the 4th, and the morning of Saturday the 5th I upgraded vBulletin to the latest version that had just come out that day or perhaps a day earlier. Things were going fine - or so I thought.
The next day, Sunday night the 6th of April, I was driving home with my wife from a family dinner, and Jon Westfall called my mobile phone: "We've been hacked - bad." So I rushed home and saw that our vBulletin forums had been hacked. It turns out that our install of vBulletin had been hacked prior to moving Pocket PC Thoughts over - and we didn't know it because the hacker chose to wait. A scramble ensued, the almost-haiku went up, and the team of Janak, Jorj, Darius, Fabrizio and myself struggled to figure out what happened and how to fix it. The hack was deep and wide - and specifically targeted at our vBulletin database. We initiated an emergency move over to a fresh VM on the new server Monday morning, but the concerns over what happened to our vBulletin database remained. For a litany of reasons we didn't have a very recent database backup, so we had to take the slow and painful steps of seeking out every possible place the hacker could have gotten into. With some great help from David from SEOvB.com, last night we managed to get things fixed up - though we lost our Pocket PC Thoughts forum template (style) and the developer I paid to create it didn't have a copy of his work...
So here we are. We're on two fresh VM installs, managed by Jorj, and one some fast hardware: a Quad-Core 2.13 GHz Kentsfield Xeon 3210 CPU, 8 GB of RAM, and some decently fast 500 GB hard drives (though we might need to upgrade these later). Now that we've jettisoned phpBB, and our old VM, things around here should be faster - though Jorj did want me to point out that because we're essentially starting over with this new VM, performance tuning will need to be done before we're really optimized. But it sure seems faster to me already!
There's a lot of work left to be done, but that's the story as to why we've been down since Sunday night. I'm much more confident now in the server we have, and the people managing it (instead of yours truly), so I believe we're going to have great uptime from now on. The Pocket PC Thoughts migration happened in a rush, so there are things that still need fixing there, including launching some subscriber features. Stay tuned, and thank you for your support - it's been a rough three months, but I believe the worst is behind us now and it's going to be a great remainder of the year for the Thoughts Media communities...
Sincerely,
Jason Dunn
Thoughts Media Inc.
