Category: <span>Hacks and Mash-Ups</span>

I had written back in October about OnlyMyEmail.com, and how great it was. Well I’d been meaning to write a follow-up for quite a while now, and here it is:

OnlyMyEmail.com sucks. No really, they really suck. Really.

Here’s the problem: Because I have so many email addresses, some of them going back more than 10 years, I get loads and loads of spam. Something like 2,000-3,000 messages a day. That’s on the order of 100-200 times as many legitimate emails that I’d like to actually have a chance to read.

OnlyMyEmail was great while it lasted, which was all of two days. I quickly exceeded their quota, and despite repeated attempts to contact someone there to see if it was possible to pay more money to remove the limitations, they dumped me. I couldn’t reach anyone there. Apparently this isn’t a problem that’s unique to me.

Bottom line: If you don’t care at all about customer service, don’t mind outages (possibly permanent), and don’t take your email communication too seriously, then OnlyMyEmail.com may be for you. If you get a lot of spam and really, really need your email to work, forget it.

So I searched around for a few days at the end of October, and found Runbox.com. They’re not free, but quite reasonable at $40/year. They support POP, IMAP, have web-mail, and even have a site for small-screen clients like PDAs, and give you 10GB of email storage — more than most people will ever use. They’re based in Norway, so I have some hope that they take privacy pretty seriously. Their servers are pretty responsive most of the time, despite the location. And best of all: they filter spam. Not as well, mind you, as OnlyMyEmail.com did (for the two days that I was actually able to use it), but pretty well.

After a few days though, a new problem became evident. I seemed to have two choices, depending on how I configured the Runbox spam filters: Either I would get a bunch of false-positives every day (meaning I still had to scan all the spam anyway), or if I turned off the learning filter I’d still get a few hundred spams every day. Sometimes more, depending on how well the Runbox people were keeping up with the spammers.

For a couple of months, I opted to scan the Spam mailbox once or twice a day. Sorting the mailbox by subject, I found it was relatively easy to scan for real email, since spam messages often use the same, or very similar subjects. Most even start with Re: which made it even easier.

But it still was far from perfect. If I missed something that was mistakenly labeled as spam, it might take a full day or more before I saw it and was able to respond. And I was still spending far too much of my time looking at spam — that’s what I wanted to avoid in the first place. At least it wasn’t taking up disk space on my machine.

Now though, just over four months later, I think I’ve finally got a working solution. It involves no fewer than four sets of filters, spread out over three different mail services, and my own computer. Here’s how it works:

How I seem to have finally solved my spam problem:

My main inbox is on a UserLand server. With a little help, I was able to set up some filters using a .procmailrc script for my account, that splits my incoming mail into three groups: definitely spam, low-risk, and high-risk, handling each group differently.

1. According to my script, a message is definitely spam when it has a keyword in the Subject, To or From fields which I know I don’t want to ever see: viagra, fuck, penis, virgin, etc. These all go immediately to /dev/null before ending up in any mailbox anywhere. Poof — gone. This accounts for about 60% of the spam I was receiving, off the top. But it wasn’t good enough. I didn’t want to have to spend a hour every day maintaining my .procmailrc either — what a pain.

2. Next, I identified low risk mail and let it through — partly. Low risk mail is any mail which matches my username (or one of them) in the To or CC fields: jake, jsavin, etc. These messages are forwarded to my Runbox account, where spamassassin and other filters catch almost all remaining spam that’s addressed to these users. Apple Mail then downloads my mail from Runbox (via IMAP), and filters it yet again, using its own trainable spam filter. Also, since the server at UserLand is forwarding directly to Runbox (instead of having Runbox log into my UserLand POP account), I get my email almost immediately, and don’t have to worry about Runbox being able to connect to UserLand’s POP server.

3. But what about all the other mail — the accounts I have on other domain names, and the catch-all throw-away addresses I use when signing up for stuff. This qualifies as high risk email. High risk mail is anything that’s left over after 1 & 2 above. High-risk mail is sent via my .procmailrc straight to my Gmail account, where Google’s gigantic distributed network of servers filters out even more spam. What’s still considered legitimate by Google is then downloaded by Apple Mail (via POP), and filtered yet again on my own computer.

What’s the net result, you ask? Well, in the last few days, I’ve gotten three or four false-positives (out of about 250 legitimate messages), and filtered out about 5,500 spam messages. Even better, I’m 99% sure that the spam being caught by Gmail will contain very few false-positives, so I won’t have to check there very often. What’s more, I’ve only seen about 50 spam messages caught at Runbox on any given day. That’s a whole lot less email to scan for missed messages than 2,000+ a day.

Oh happy, happy, joy, joy!!!

Is it true? Can I finally be nearly spam-free, if only fleetingly?
One can only hope.

Hacks and Mash-Ups

Tonight, while checking to see if there were any new updates to TiVo Tool, I stumbled across this thread on the DealDatabase Forums (where all serious TiVo hacking seems to get discussed), about the inclusion of a tystream demuxing module with VLC 0.8.2. Also linked on the thread is a module for streaming from TiVo to VLC.

Geek speak aside, what this means is that I can now stream shows from my TiVo straight to VLC player on my Mac. Coolio!

The trick for me was to figure out what invocation to use under the File -> Open Network command. After following the instructions on the DealDatabase thread, choose Open Network from the File menu, and enter tivo://192.168.0.10/plist into the first field in the dialog. (You have to replace the IP address with the IP or host name of your TiVo unit.)

Then open VLC’s playlist, and double-click the show you want to watch. Easy as that.

If you have trouble downloading the tystream and vserver modules for VLC, post a comment, and I’ll hook you up.

Happy streaming!

Hacks and Mash-Ups

Last weekend, I finally pulled the A drive from my Series 1 Philips TiVo, and set it up for shell and FTP access over my LAN. More interesting to me, and maybe to you, I also now have a way to transfer video over the network, to MPEG video files on my Mac OS X machine.

I had to dig through a whole bunch of Google searches, message board posts, install instructions, and irrelevant Windows information to make this happen, but in the end I overcame the odds and got it working.

Here’s my mini how-to for transferring TiVo shows to your Mac OS X machine. They should work with any Series 1 TiVo running version 3.x TiVo software, and decent-speed Mac running Panther (OS X 10.3) or better:

  1. Get a network card and install it. I’ve had a TurboNet card installed for about two years. It was cheap and setup was a breeze.
  2. Get shell access working on your TiVo. The best instructions I found were these from Steve Jenkins. Requirements for getting this working:

  • A PC with an IDE bus, which can boot from a CD-R.
  • A way to burn an ISO CD image to a CD-R disc.
  • Willingness and ability to remove a drive from your TiVo and install it on the correct bus, with the correct jumper settings in your PC. (Not so hard for some, but daunting for most newbies — get help from a geek friend if you need it.)
  • Willingness and ability to read instructions and follow them to the letter, or at least in sequential order. 😉
  1. Install TyStudio on your Mac. Normally the download would be here, but at the moment there’s no Mac OS X version available. At one time, someone on the DealDatabase forum built an OS X version, but I’ve lost the thread now. Instead, here’s a download of the version I installed on my PowerBook G4. It’s a little over 5MB.

  2. Install tyserver on your TiVo. Basically you just want to upload tyserver.ppc.sh to your TiVo, run it to install tyserver, then add commands to your startup script so it will run all the time. Note: I strongly recommend not letting the installer add its startup commands to your TiVo’s built-in rc.sysinit, since this can really mess things up something goes unexpectedly wrong. Instead add them to your own custom startup script — you created one when following the instructions in step 2, didn’t you? 😉

If the tyserver download link above doesn’t work anymore, you can use the version that I used on my Series 1.

Here’s what I put in my startup script:

  1. Restart your TiVo. I’m not sure if you need to do Standby and then power-cycle, or just do Standby and then wake your TiVo again. (If you’re command-line savvy, you can launch tyserver and tyindex in your telnet client instead of restarting.)

5a. Stop here, and wait for a couple of hours. The tyindex software running on your TiVo needs some time to collect data about the shows you’ve already recorded. In my case, I stopped here, and went to bed, then woke up on Sunday, and did the rest. The amount of time it takes for tyindex to complete will depend on the number of shows in your TiVo’s Now Showing list.

5b. Did you read 5a? I’m serious — wait for a bit. If you don’t, you’ll just be frustrated and think this won’t work. Trust me — it’ll work. (Or not. It worked for me, that’s all I’m saying.)

  1. Launch X11 on your Mac — it’s in your Applications/Utilities folder. (You should be able to do this in Jaguar as well, provided that you first install X11.)
  • Choose Customize from the Applications menu in X11. Then add a menu link for tyeditor from the TyStudio install you did in step 3. I’d installed my copy in /Applications/Video/tystudio/, so the path I used was /Applications/Video/tystudio/tyeditor. Now you should have a tyeditor command in your Applications menu, for launching TyStudio whenever you need it.

  • Choose your spiffy new tyeditor command from the Applications menu in X11. This brings up the TyStudio GUI. (Now we’re cooking with gas!)

  • Click the ‘Click here to load from Now Showing’ button. You should now see a list of shows. Double-click one whose value for ‘Indexed’ is ‘Yes’. (Ref: step 5.) I recommend you start with a short one — 1/2 hour or less.

  • Choose ‘Generic MPEG2’ from the Video to pop-up menu, and ‘No Transcoding’ from the Audio to pop-up menu. This will give you basically the same audio/video that your TiVo would play on your TV. Then click the Process button, choose a place to save the file, and you should be off to the races!

  • Just a few notes:

    • These files are really big. We’re talking about 1GB or more per hour, depending on your TiVo’s quality settings. Make sure you have enough drive space.
    • If you’re going to do a lot of this, I recommend that you use Tools -> Options in tyeditor to choose the video and audio options you’re going to use most frequently, so you don’t have to change them every time in the tyeditor main window.
    • I haven’t done much messing with the transcoding options, since I’m mostly using TyStudio to make digital archives of stuff that I want to keep around, that my TiVo has recorded. If you use transcoding, you’re going to lose quality. Series 1 TiVo’s record PCM audio at 32KHz. (PCM is roughly equivalent to AIFF or WAV.) Video is recorded at varying rates depending on the quality setting the recording was made with, but is always much larger than the audio portion of a recording.
    • If your network is fast enough (as 802.11b on my mac, with 10-base-T on the TiVo and a 10-base-T hub seems to be me), you should be able to start downloading a show on your Mac, and then start playing it a few seconds or minutes later, to in essence stream the show to your Mac’s video screen. Unfortunately, you can’t do this using QuickTime Player, since it checks the size of the file when you first open it, and will stop at that point despite the fact that you’re still downloading. VLC on the other hand, will keep playing a file until it either gets an error, or it reaches the actual end of the file. If you’re going to watch shows as they’re downloading, I recommend that you use VLC for playback instead of QuickTime Player. (MPlayer may work as well, but I like VLC better.)
    • While you’re hacking, you might as well install TivoWebPlus — this is a great app that no TiVo should be without.
    • TyStudio works on Windows and Linux as well. A quick search should make it very clear that they’re much easier to find than the Mac version was.

    Feel free to post a comment if you have a question about any of the above. If I have an answer, I’ll let you know. If not, I’ll ignore you. If you’re rude I’ll delete your comment, and if you’re rude again, I’ll block you, so be nice.

    Also, I haven’t found a way to stream MPEG data directly from the TiVo to a movie playing application on the Mac, without also downloading it to a disk on the Mac side. If you figure this out, please let me know. My filesystem will thank you. 😉

    Happy TiVo hacking!

    Hacks and Mash-Ups