Simple, automated server link check

It's been ages since I've been looking for a solution to be able to check dead links on my sites. Just it never had a high priority - until now.

After some googling, installing, playing around, removing, googling, installing, etc... I've found the perfect script for me: CheckBot. The documentation is pretty much straightforward, but for a reference, I use this setup:

checkbot --verbose --cookies --file index.html --mailto my@addre.ss --dontwarn "(301|302|903|904|400|403)" --ignore "(feeds\.archive|jigsaw|https)" --sleep 0.2 http://www.site1.com http://www.site2.com

--verbose: this should be turned off for automatic running

--file index.html: that's because the report's in its own directory

--mailto: my@addre.ss: so that I see what was going on

--dontwarn "(301|302|903|904|400|403)": I pretty much only need the serious problems

--ignore "(feeds\.archive|jigsaw|https)": URLs with these strings I don't want to check, validators in the beginning, linked with another URL from every page, and I don't need https checking neither (I know the regexps could be better, but that was fine for me)

--sleep 0.2: Without this I experienced a sudden 1.2 load on the webserver, that's not nice. 0.2 seconds of delay between requests it OK for me - local files are still fast, and for remote files the network delay will be much longer anyway

And then the URLs at the end. Can be put to a crontab to run every week, and that's it.

Mysterious connection problem and resolution

This was weird. 2 days ago my connection to a company IMAP server simply stopped working. To cut a long story short, after lots of trying and head-scratching, I could at least identify the problem at its roots: whenever I telneted to a working port of the IMAP server (it has webmail too, etc) I was faced with an empty screen, and after the first keypress, the connection dropped.

Now I use a Windows XP laptop through a wifi router for work, I don't want to get into details why, it's just given. I tried to locate the source of the problem: trying from a remote server works OK. Trying from connecting the laptop to the router instead of wifi: NOT OK. Trying from the home linux server connected to the router: OK. Trying from Knoppix on my laptop: OK.

Shit, the problem's in the networking part of Windows XP. Hell, all I need now is a reinstall... like I'd have time for that. But I had an idea: if the stuff works from my server, why don't I try to tunel the IMAP traffic through it? Why not - besides the fact that I've never done anything like that before.

But surprisingly it worked like a charm, thanks to this guide. You might want to turn on compression and ask putty not to open a remote shell, since you're not using the ssh connection itself, just for the tunel. When set up, I started the tunel, and asked my IMAP client to connect to localhost:143 - and voilá, after all the struggling, I have a faster and a bit more secure connection now. Phew.