What happend with server

All, some time on the 12th/13th March, there was apparently some kind of power surge that affected my hosted server. The first sign of that was that the database service notified me of corrupted tables then stopped running. After that I connected to the machine and discovered that most commands I typed didn’t work. After some frantic emails and calls I managed to get the machine restarted but it didn’t come back up… After the service engineer had a look at it and found nothing physically wrong with the machine (I rhought there might be a with a previous model), the support staff ran a file-system check and told me how the system was broken. Basically the directory that stored all the important config files was blank! This was Wednesday. After that the delay was getting another machine installed as a replacement with the current one running from a ‘rescue disk’ so that data could be transferred. Luckily, it looked like /etc was the only (!) directory really affected and the only mail lost was in fact my Inbox of about 6000 messages. So, from Friday evening to Sunday night, there was me recreating the setup on the new machine (up to a Xeon 2.8 from a celeron 2.4 and twice as much disk space). Handily, I managed to rescue the important apparently deleted files from the recovered files directory (created from the file-system check) so recreating all the user accounts wasn’t a problem (so I didn’t need to find out your passwords again).
What you can see from the server_roadmap page on wiki.gadg3ts.com is that I had been planning to convert the email files from flat-file mbox format (each mail ‘folder’ is one physical file containing all messages) to ‘Maildir’ which has a number of named directories containing an index of each message that relates to a single file, thereby making delivery faster and more resilient. Having this somewhat unfortunate opportunity to make this happen, this is what I’ve done. What this means to you is indexing or moving mail will be faster. What it means for me is that mail is less likely to be lost should anything untoward occur. Also, instead of having a large list of system users, all usernames, passwords and email aliases are stored in an ‘ldap directory’ which basically means adding/amending user details can be done through a web interface and the data can be automatically duplicated to another machine for safely. From a security point of view, it means I can further lock down access to only those users who have hosted sites and myself. What ldap also enables me to do is unify the user accounts (email) with those from the gongbong site, as they’ll all be stored in one place, but that’s still a while away (too many tasks not enough hours!). Technically, the email system now consists of postfix (mail reception and sending), cyrus (manages mailboxes), amavisdnew+clamav+spamassassin+dspam for mail filtering all (ok, mostly) running with their preferences in the ldap directory service. The cyrus sub-system handles pop & imap access, the physical mailboxes and the sieve filtering system (more on that soon). What this means for the users is that you can now do the following: *) send mail to username+folder@yourmaildomain for mail to be delivered to that folder *) Have server-side mail filters via the smartsieve interface (http://webmail.gongbong.com/smartsieve) and soon integrated to the webmail system. If any of you have used Microsoft Outlook with Exchange Server you’ll know what it’s all about. I’ve also been considering (and am likely to implement now that it’s possible) having email spam directed directly to a ‘Junk’ folder so as not to clutter your inboxes, however this may require some method of keeping them clean (delete after 14 days?) so any thoughts or feedback would be welcome. With regards to your previous mail – up until the 12th – none of that was actually lost (as far as I can tell, let me know if there was), unless you count my Inbox! :o ( I’m not sure if anything from Saturday (the period between creating the user mailboxes and migrating the old mail) disappeared, as the conversion of the previous mail ‘may’ have cleared out any mails received that day (due to the rebuilding of indexes of migrated mail), but that could have just been for me… I’m trying to determine that and can reference the received message numbers from the mail logs to what’s in the physical mailstore. Once I’d got the server running Thursday night I’d set postfix (what recieves mail from other machines) to ‘soft bounce’ before I’d created the mailboxes so that the rest of the world would keep trying until their local timeout, which is 5 days for standard settings. Ideally, I/we would have had a complete server backup to restore from and before all this happened I was thinking about sending down to the hosting centre an external drive for daily/weekly backups, which would probably be a good idea! For the services that allow replication of data between machines (mysql, ldap), I shall be actioning this hopefully this week so there’s a copy of the data on my storage machine at home. Anyway the plan is to have your (and my) data safely in 2 places, as as is said in technical circles, it’s not a matter of if something will go wrong, it’s when… This is all something that I don’t get paid for and so in a sense it is ‘just for fun’ or at least for keeping my technical skills current, yet it is an email service for almost 100 (!) people now as well as a few web hostings and I would prefer it was as reliable and professional as time and (my) availability allow. After all it is my primary email too! I had even wanted to take Friday off work (very stressed about having it all (not) working, but as it was only my 5th week there and they were depending on me for quite a lot if stuff I thought that wasn’t such a good idea… Trying the webmail from where I’m on a training course at the moment (IBM in Southwark) it all felt rather quick (as fast as a local application) so that’s one good thing to come out of all of this. There are likely to be other improvements/enhancements soon also, but I’ll let you know about them when they’re running! Thanks for listening! Sean

Leave a Reply

You must be logged in to post a comment.