Top wizardry

OK, you're probably way too familiar with the Linux top command. But do you know what all of these mean?

• top -b -n 1
• top -u www-data
• top -p 4365

If not, head over to this valuable article on using top more efficiently.

Backup all mysql databases to separate dump files

It's easy to just simple do a

mysqldump --alldatabases > mysqlbackup.sql

to backup everything into a large SQL dump, so that you have the inserts there if you have a booboo happening. But what a nightmare you face when you just have to get those lines that affect your recently-crashed site! Well, there's a bit more sophisticated, but much better solution: have each of your database backed up in a separate SQL dump file with the following shell script:

for i in /var/lib/mysql/*/; do
dbname=`basename $i`
/usr/bin/mysqldump $dbname > /home/db_backups
done

Of course you might have to add login parameters to the mysql line above, but that shouldn't be a problem. (Hint: -u myusername --password=mysecretpassword)

Serve all - and any - subdomains from one central script

This is probably the most common wildcard example, still, I needed some googling to find the perfect way to do it. The idea is that the setup should accept any kind of subdomains (we don't know in the beginning, what subdomains might come), and serve them from one central scritp. Of course all this without the user seeing what's going on behind the curtains. Let me make it clear with an example:

http://anything.example.com/somesubdir
should be served by
http://example.com/centralscript.cgi/anything/somesubdir

http://anotherthing.example.com/
should be served by
http://example.com/centralscript.cgi/anotherthing

etc.

So, as you see, we have a centralscript.cgi, which gets the subdomain name and the directories that were called as a parameter. This of course can be a PHP script too, whatever you prefer. Now, one more thing to the specification: www.example.com and the plain example.com should work as they are now, no centralscript.cgi magic here.

The solution was found at the wonderful webmasterworld forums - as so often. This topic discusses the exact same problem. I'll go into more details here and include the necessary nameserver configuration too, but I don't want to take credit for anything here, I would still be struggling without the fine guys at the webmasterworld.

So, first, let's set up our nameserver (bind9 is my choice, but whatever you use, this should work. I hope at least.), edit the zone file for your example.com domain. First of all, never forget to update the timestamp of the zone, so start with that, and then insert the following line to the end of your subdomain list:

* A 1.2.3.4

Where of course 1.2.3.4 is the IP of your domain name. Restart the nameserver (/etc/init.d/bind9 restart, or whatever you use), and check your setup with pinging anything.example.com. Provided that the test box uses the same nameserver too, you should get pretty fast response times. If not, maybe you should wait a bit until the domain zone file spreads to your test box - but be prepared for a bit more time than a cup of tea. Anyway, continue, if you can ping any subdomain under example.com - the nameserver part is done.

Next, Apache config and the mysterious rewrite engine. It's a topic worth to master, but honestly, I didn't have the time yet. The explanation of the techniques below can be found at the webmasterworld forums linked above.

So, open up your httpd.conf file, and look for the virtualhost already set up for your main example.com domain. (I won't explain it here how to do that, it's pretty basic.) Side-note: I'm very much against puting this kind of settings to .htaccess files. Only use those, if you absolutely don't have access to the central configuration file - just imagine, the httpd.conf files are loaded once you start the server, but the .htaccess files are loaded at every request. No need to overload your server without a reason.

First what you have to do is make sure that apache finds the appropriate virtualhost for all your subdomains. Add this line to the virtualhost settings:

ServerAlias *.example.com

To test if it works, restart apache, and go for http://whatever.example.com. You should see your original www.example.com main page, since no special rules are defined yet for this subdomain. If all goes well, open the httpd.conf file again, and paste this code below at the end of the virtualhost config of www.example.com:

# Rewrite <subdomain>.example.com/<path> to
#   example.com/centralscript.cgi/<subdomain>/<path> 
RewriteEngine On

# Skip rewrite if no hostname or if subdomain is www 
RewriteCond %{HTTP_HOST} . 
RewriteCond %{HTTP_HOST} !^www\. [NC] 

# Extract (required) subdomain (%1), and 
#   first path element (%3),
#   discard port number if present (%2) (in one line!)
RewriteCond %{HTTP_HOST}<>%{REQUEST_URI} 
    ^([^.]+)\.example\.com(:80)?<>/([^/]*) [NC] 

# Rewrite only when subdomain not equal to 
# first path element (prevents mod_rewrite recursion) 
RewriteCond %1<>%3 !^(.*)<>\1$ [NC] 

# Rewrite to /subdomain/path 
RewriteRule ^(.*) /centralscript.cgi/%1$1 [L]

# Do some logging
RewriteLog "/var/log/apache/rewrite.log"
RewriteLogLevel 9

Of course you'll have to replace example.com to your main domain, and centralscript.cgi for your script's name. Restart apache, and check if it works. If not, study the rewrite log set up above, if yes, comment out the two last lines - again, no need to overload the server with some useless logging junk.

I provide you a simple centralscript.cgi to test if the setup works, and also to see how the parameters are passed:

#!/usr/bin/perl
print "Content-type: text/plain\n\n";
print "CGI parameter (PATH_INFO): $ENV{'PATH_INFO'}";

Of course the usual blahblah applies: script should have the x flag, and ExecCGI should be allowed for the directory where it's in.

Common templates for different blogs in MovableType

Say you have more than one (we have eight, and growing...) number of MT blogs that have templates that are the same for every one of them. (For example the RSS feed, individual archive, whatever.) You'd love to have them at a central place, so when it's time to finetune, you don't have to modify all the blogs' templates for the same changes. I hate doing repetitive stuff, so instead I found out that how I can avoid it.

MovableType supports a feature that lets you link a template to a file. Yes, that's going to be the solution: we link the same temaplate (in this example the RSS feed index.xml one) to the same file at every blog. The only problem is that the "link template to a file" input box's rootdir is the root of your blog, which is of course different for the different blogs.

For example you have this setup:

/home/blogs/blog1/ - www.blog1.com
/home/blogs/blog2/ - www.blog2.com

Now if you try to link to "/common_index.xml.tmpl" at blog1, it will create the file /home/blogs/blog1/common_index.xml.tmp. Which is of course hard to see from blog2. What you can do is set up a directory for all the common templates (for example: /home/blogs/common_templates) and link to it like this: ../common_templates/index.xml.tml

I know it's a dirty hack, it might even be a bug in the MovableType engine (but so far it's working at MT 3.2), but it works like a charm, you just have to set it up once at every blog, then a modification at any blog will affect all the other blogs too. You might need to rebuild all of your blogs to see it working in case you're not modifying a dynamic template.

To master this technique, you can separate common parts from individual ones with referring to template modules. If you refer to a module with <$MTInclude module="head"$> then every blog will look for the template module "head" in its own templates, so you can achieve individual heads, while having common index templates. Way to go!

Simple, automated server link check

It's been ages since I've been looking for a solution to be able to check dead links on my sites. Just it never had a high priority - until now.

After some googling, installing, playing around, removing, googling, installing, etc... I've found the perfect script for me: CheckBot. The documentation is pretty much straightforward, but for a reference, I use this setup:

checkbot --verbose --cookies --file index.html --mailto my@addre.ss --dontwarn "(301|302|903|904|400|403)" --ignore "(feeds\.archive|jigsaw|https)" --sleep 0.2 http://www.site1.com http://www.site2.com

--verbose: this should be turned off for automatic running

--file index.html: that's because the report's in its own directory

--mailto: my@addre.ss: so that I see what was going on

--dontwarn "(301|302|903|904|400|403)": I pretty much only need the serious problems

--ignore "(feeds\.archive|jigsaw|https)": URLs with these strings I don't want to check, validators in the beginning, linked with another URL from every page, and I don't need https checking neither (I know the regexps could be better, but that was fine for me)

--sleep 0.2: Without this I experienced a sudden 1.2 load on the webserver, that's not nice. 0.2 seconds of delay between requests it OK for me - local files are still fast, and for remote files the network delay will be much longer anyway

And then the URLs at the end. Can be put to a crontab to run every week, and that's it.

Mysterious connection problem and resolution

This was weird. 2 days ago my connection to a company IMAP server simply stopped working. To cut a long story short, after lots of trying and head-scratching, I could at least identify the problem at its roots: whenever I telneted to a working port of the IMAP server (it has webmail too, etc) I was faced with an empty screen, and after the first keypress, the connection dropped.

Now I use a Windows XP laptop through a wifi router for work, I don't want to get into details why, it's just given. I tried to locate the source of the problem: trying from a remote server works OK. Trying from connecting the laptop to the router instead of wifi: NOT OK. Trying from the home linux server connected to the router: OK. Trying from Knoppix on my laptop: OK.

Shit, the problem's in the networking part of Windows XP. Hell, all I need now is a reinstall... like I'd have time for that. But I had an idea: if the stuff works from my server, why don't I try to tunel the IMAP traffic through it? Why not - besides the fact that I've never done anything like that before.

But surprisingly it worked like a charm, thanks to this guide. You might want to turn on compression and ask putty not to open a remote shell, since you're not using the ssh connection itself, just for the tunel. When set up, I started the tunel, and asked my IMAP client to connect to localhost:143 - and voilá, after all the struggling, I have a faster and a bit more secure connection now. Phew.

Creating and using SFV checkfiles on debian

This is quite straightforward, just I need it so rarely that I'm keep on forgeting, so here's the note. The package is cksfv, to create an sfv checkfile for sample.tgz use cksfv sample.tgz > sample.sfv, to check it just use cksfv sample.svf.

MovableType MTDateHeaders problem solved!

I've updated the original article in case someone finds it through google. Phew.