Hi everyone,
It’s been two days since the situation came back to normal and we wanted to provide a detailed explanation of the issue. At the beginning of the week -as every night- our server checks if there’s updates avaliable -updates from OS (in our case CentOS 7), RPMs to cPanel versions-, there was a new cPanel version avaliable and some others scripts updates so cPanel automatically started to update everything, issue starts here; since memory usage (RAM) was high the updates increased it and made it surpass its limits turning the server down, in normal situations this wouldn’t affect anything, server would just have a small downtime but as there was an update going on cPanel and OS failed, most of the sites were still browsable but some of them weren’t and cPanel wasn’t working at all.
What we wanted to do first -and the easiest thing to do it- was buy another server and use the transfer tool that cPanel provides to move accounts over, that would only take ~2 hours if not less, unfortunately due to the failure update “transfer tool” wasn’t working, what could we do? Reinstall the server and restore all the sites from our backup server using the script we use. Here comes the second problem, the first problem of this second problem was that It was impossible for us to reissue the license from script’s client area so it was impossible for us to access the script from WHM, after they reissued it for us we noticed that old backups didn’t show up in WHM, reaching the script developers they told us that the script database was hosted on “the old server” not the storage one making it impossible to restore the accounts right away, we got really pissed off and mad at them since before purchasing the script we asked everything about it and they did not tell us that. How can a script backup every single file of a site and its databases but not backup its own database? who was the genius that thought that? – We told them – What could we do next? Hire one of their programmers to manually rebuild the database so we could restore all sites, we could also restore site to site the backups but that would take much longer since the backups are not full backups but incremental backups. Things didn’t work out as we expected, rebuilding the database didn’t work either but we had another solution that could at least speed things up as the other solution was manually restore everything.
We want to also point out that the issue could have been also solved faster if we had only restored monthly backups and not daily incremental ones, but we understand how much effort do you put in your site(s) and we had to restore all files. We made a mistake not making an announcement earlier, that won’t happen again, but we did reply everyone who emailed or dmed us with all information we had at the moment and we kept them up to date with the process until we started to update vía Twitter, also we made clear on the first reply that there wouldn’t be any data loss.
Now that everything have been solved, we have created our client area, you need to registrer through my.fan-village.net, on the support board you can ask for help, log in to cPanel, change cPanel password and a few other features. Client area is hosted on another server so in case the server goes down you will still be able to access to it! Also, we will be able to send emails to all hostess with the mass mail tool the client area has. We want to let you know that we’ll also be installing a forum soon and a new responsive theme will be installed along a tutorials center for newbies.
We have also replaced our old backup system to JetBackup -which as proven- is better than the old one we used, so in case this happened again -it won’t but just in case- the issue would be fixed in minutes. We have also increased the RAM on the server.
click to zoom
We want to apologize again for the issue, sadly, this things happens and the only thing we can do by our end is fix it and improve. We love Fan Village as much as you love your sites, well.. we even have our own sites hosted here and as you may know, we don’t want to loose anything. We want to remember that we have been online for 10 months and all prior months we have had 99.9% uptime and we want to keep it that way.
Regards,
FAN VILLAGE.
For those who still want to move to another hosts, we’ll be making a new post about how to migrate your sites, if we own the domain you can also migrate your site, we can push the domain to your new hosting or to you.
