Category: Hacks and Mash-Ups

I’ve been using an email and file-storage service called Fastmail.fm for about the last three years. I like it because it’s cheap (about $50/year), has lots of storage (10GB), and other perks like photo albums and web hosting on custom domains. They also have great spam filters, and they’re almost never down.

I prefer it over Hotmail (a.k.a. Live Mail) because its web UI is lean and fast, and it has IMAP access so I can access my mail easily on my PC or mobile device (iPhone at the moment). I also prefer it over Gmail since I don’t really like the way Gmail handles folders/categories, plus at the time I set it up, Gmail didn’t offer IMAP access, though they do now. Of course since Google released Google Buzz, there are new reasons not to like Gmail.

I still have a Gmail account, but I use Fastmail to both read and send email for my Gmail address. It’s easy to set this up in Fastmail and Gmail — all you need is a Fastmail personality, and to forward all your Gmail to your Fastmail address. Here’s how to set this up:

Configuring a Gmail personality in Fastmail:

  1. Click on Options at the top of the page
  2. Click Personalities (near the bottom of the first column)
  3. Click the Create New Personality button
  4. Enter your Gmail address under email address, and whatever you want to under display name and name
  5. Click the Save Personality button

Now you can send email using Fastmail, but “from” your gmail address.

Note: This does not save outgoing mail back to Gmail when sent from Fastmail. If you want that, then BCC your Gmail account when sending from Fastmail. (You can do that automatically if you want, by adding your Gmail address to the BCC addresses field on your gmail personality in Fastmail.)

Forwarding Gmail to Fastmail:

The other part is to set up forwarding incoming mail from Gmail to your Fastmail account:

  1. Log into Gmail
  2. Click Options at the top of the page
  3. Click the Forwarding and POP/IMAP tab
  4. Click the radio button in the top section to “Forward a copy of incoming mail to” and enter your Fastmail email address. You probably want to “keep Gmail’s copy in the inbox”
  5. Scroll to the bottom and click “Save Changes”

Now you can read all of your Gmail on the Fastmail site. Gmail’s spam filters are still active for mail coming from Gmail — they run before forwarding. Plus you get a second layer of spam filtering from Fastmail too.

PS: Can we please someday get IMAP access to Hotmail? Thanks!

Hacks and Mash-Ups iPhone

I recently purchased a new big disk (500GB), and a NAS enclosure. My goal here was to simplify my home network storage setup, and turn off some of my always-on computers. These were using about 450 watts of power together, which costs a couple hundred dollars a year, and is generally bad for my carbon footprint. The NAS disk, on the other hand, uses about 15 watts of power — even less than that when in standby, which is about 75% of the time.

While in the process of going through all my media, mostly music and video, and cleaning it up to move to the NAS disk, I also started thinking about backup strategies. The other reason I bought the NAS disk, besides cleaning up my media and saving power, was to have a central place that all my computers can send their critical stuff to be backed up.

That got me thinking again about data that lives out “in the cloud” on the services I use, on my blog host, etc. Which brings me to Netflix.

A number of people have mentioned the desire to back up their Netflix ratings, Queue, etc. I decided I wanted to do this too. But it turns out that it’s actually pretty difficult. I tried a few different Perl / Python solutions, but wasn’t able to get any of them to work for whatever reasons.

Then I stumbled on this:
http://badsegue.org/archives/2006/03/15/netflix-ratings-grabber

It’s totally low-tech, but get this — it totally works. I really don’t care about the output format, or that it runs as a JaveScript bookmarklet in my browser. (Latest version of Safari ok.) I got my data, and copied and pasted it into a text file, and for now I’m happy with that.

If I ever have to put it back on Netflix that will be a different story, but for now I’m satisfied that my data is safe, and in a format that I’m sure I could write my own code to do something interesting with.

In case you’re curious, here they are: {manilaSuite.gems.includeGem (11)}

Hacks and Mash-Ups

I think this is just about as cool as it gets these days, even though AudioFaucet is already well over a year old:

AudioFaucetOnTivo.png:

To give you an idea of what’s going on here, ponder this for a moment first:

  • iTunes is running on my Mac in a different room from my TV. Some big drives attached to the Mac are where all my digital media lives.
  • My TiVo HD recently got an update from TiVo (in fact it just informed me today) that enables its Home Networking apps (a.k.a. HME).
  • iTunes is a ‘scriptable’ application, meaning programs can talk to it using its scripting API (implemented using Apple’s standard AppleEvents, which are built into the system).
  • The screenshot above is drawn onto my TV screen by a Java application, running on my Mac in the other room, and is output to the TV by the TiVo.
  • Audio from the Mac is streaming over the network to the TiVo, and comes out of my HiFi speakers via an optical fiber connected to the TiVo.
  • When I hit the Thumbs Up button on my TiVo remote, the rating of the song on my iTunes library on my Mac is updated instantly.

So there are lots of interfaces at work here that are machine to machine, or software to software, and that have nothing to do with ‘human’ interfaces: USB (for the disks that hold my media files), AppleEvents (for AudioFaucet to talk to iTunes), TOS (digital audio over fiber optic), TCP/IP over the home network including a transparent coaxial link from the living room to the office in the basement, IR remote (proprietary TiVo), HDMI (digital video from the TiVo to the TV), and if the screenshot were of a podcast playing, then you can throw in RSS (a format, but also an interface).

I’ve been messing around with technology for about 30 years, with varying degrees of seriousness. This is by far the most fun and most versatile set of toys I’ve ever played with.

And I’m sure I’m only barely able to imagine how cool this stuff will be in just a few years. Every few years. For probably the rest of my life.

Ps. This works on Windows too, though I’m pretty sure the the interface is Windows Events, instead of AppleEvents.

Hacks and Mash-Ups TiVo

Alright, it had to happen eventually. Netflix changed something and my IMDB/Netflix movie mash-up stopped working the way it was supposed to.

It turns out that their URL scheme changed, but only on what were formerly displayMovie pages, and only on the main site (not the mobile site).

The upshot was that my script couldn’t tell any longer how to make an Add To Queue link from a movie’s URL on Netflix, nor could it convert a URL on the main site into a URL on the Netflix mobile site, when viewing using a mobile browser (or RSS reader).

Converting a displayMovie URL into an Add To Queue link used to be a matter of replacing displayMovie with addToQueue. That doesn’t work any more — now I have to actually parse the URL for the movieid, and make my own link. Same kind of thing for converting URLs on the main site into ones that work on the mobile site. Oh well… So much for low-tech. :-/

Update 8/27: Looks like the change at the beginning of the month fixed the problem. Coolio.

Hacks and Mash-Ups

I spent a little time last night on a small upgrade to my Good Flicks Netflix+IMDB+RSS mash-up.

Now each movie goes up as its own post, instead of all the week’s new releases going into a single post. This has the dual advantage that the RSS feed has an entry for each movie, which renders much better in the aggregators I’m using at the moment (and probably most others), and thet you can now leave comments next to each movie, in case you want to tell me it sucked, and by the way so does my app.

The other thing I did was to change the UI slightly, to accommodate the title/link/description model, so now below each movie title, you’ll see links to add to your queue, to visit the Netflix page for the movie, and also to visit the corresponding IMDB page.

Lastly, a couple of tips:

You can see a less crufty version of the page by using the print-friendly version (a feature of Manila).

And if you’re using the RSS feed on a mobile device, and your aggregator allows you to specify the User-Agent, use the IE Mobile User-Agent to get a version of the feed that links to the Netflix mobile site, instead of the big, heavy full site.

That User-Agent value is:

Mozilla/4.0 (compatible; MSIE 4.01; Windows CE; PPC; 240x320)

Though anything that contains Windows CE should work.

Hacks and Mash-Ups

I did a short programming project over the long weekend, building on a script I wrote last summer — it’s an IMDB+Netflix mash-up. Once a week, typically on Tuesdays, my computer at home goes through all the new DVD releases for the week, looks up the user ratings on IMDB, and then builds a post here on JakeSavin.com with links to all the movies in IMDB and Netflix, including links to add to my Netflix queue. Why, you ask? Read on…

Netflix and RSS

I love Netflix, and while the Netflix recommendation system is really cool, it doesn’t help me to find new releases in a timely manner. Now, Netflix does have some RSS feeds, in particular one that lists all the new DVD releases each week, which is a cool feature, but it doesn’t tie into the recommendation system, so its value is minimal as far as my Netflix experience is concerned. After all, I can get a list of new DVD releases in lots of places online and off, including the wall behind the counter at my local Blockbuster or Hollywood Video store.

The trouble with user ratings

Now, while I love Netflix recommendations, their user ratings have the same problems for me that Amazon’s and Yahoo’s do — most movie watchers have less discriminating tastes than I do, and a 5-star rating system doesn’t capture enough granularity to make the rating for a single movie work for me. In aggregate, it works great for the Netflix recommendation engine, but for a single movie it’s hit-or-miss at best. It may also be that there just aren’t enough users rating movies on these systems for the rating to carry much meaning, especially for movies that have just been released. This is where IMDB fits in.

IMDB has really high-quality user ratings

If you’re not familiar with IMDB, it’s a long lived, public database of movies and TV shows, with user ratings, and tons of other data about just about any movie you can think of. Aside from its vast capability for settling movie trivia arguments at parties, the thing I like the most about IMDB is the user ratings.

For whatever reason, IMDB users’ ratings match mine and Cindy’s tastes much more closely than do the ratings on Netflix or Amazon or Yahoo. This is probably at least in part because there may be more users providing ratings on IMDB, but it’s more likely that ratings have been collecting on IMDB for a longer period of time at the point that new DVDs are released, since they enter the IMDB system as soon as their existence is known, rather than near the DVD release date.

My guess is that the IMDB user ratings will also be more accurate than Netflix for other discriminating movie watchers as well.

The IMDB + Google + Netflix + RSS mash-up

So a while back, armed with this knowledge and some pretty good scripting skills, I hacked together a mash-up that loads the Netflix new release RSS feed, looks up all the titles using IMDB’s search engine, scrapes the user ratings from the HTML in the search result, and builds an outline (in OPML) on my computer with links to movies with ratings over 6.0. It worked pretty well for a while, and found lots of great movies I would otherwise have passed by. But it had some problems:

  • Many times, IMDB would return multiple results, and not necessarily for the movie I wanted
  • Sometimes the IMDB search would return the wrong movie — their search isn’t as good as Google’s
  • The resulting document was only useful for me: Cindy didn’t want to get on my computer to look at it, and so when I get busy with work or whatever, the new movies don’t make it into our queue — not to mention that nobody else could see them at all

So over the weekend I updated my hack. Here’s what I changed:

  • Now it uses Google to find the movies on IMDB, using a search like "movie title" 2006 inurl:title site:imdb.com — this usually gets to the correct page on IMDB
  • Once all the results have been found and IMDB user ratings compiled, it builds the HTML for a post on my blog, and sends it up to my New DVD Releases category — and the HTML now includes links straight to the Netflix AddToQueue page for the movie — this category is linked to under Good Flicks in my site navigation
  • As a bonus, my site is now checking the user-agent, so when serving to IE on Windows Mobile, all the Netflix links go to the appropriate page on the Netflix Mobile site, to make for happy mobile Netflix’ing

The end result: A page where anyone who has a Netflix account can go to find great new movies, sorted by IMDB user ratings. Just log into Netflix, go to my Good Flix page, and click the title of any movie to add it to your Netflix queue.

There’s still the occasional wrong movie returned by Google — for example right now it’s showing IMDB information for Cars instead of a new release called The Route — but far fewer than before. Also, movies that have been rated by only a few people, say under 400 or so, may not actually be any good since the first raters seem to be biased towards higher ratings, for whatever reason. For example, Tweek City is probably not worthy of an 8.2 out of 10, but that’s what the first 15 raters seem to think — I wonder how many of them are affiliated with the making of the movie. But — movies with high IMDB user ratings by 500 or more people are almost always worth watching. For example, Beer League might actually be funny with a 6.1 for 652 votes. At least the movies my mash-up finds are likely to be worth watching by me, and if you like them too, then all the better.

This would all work much better if IMDB had a reliable API for finding movies and getting their user rating data, but as far as I know they don’t, so for now it’s a hack. Maybe they’ll open up their database more in the future to stay competitive. After all, IMDB is owned by Amazon, and they’re definitely hip to web APIs in general. We’ll see…

So have fun, and let me know what you think of this hack?

Hacks and Mash-Ups

I had written back in October about OnlyMyEmail.com, and how great it was. Well I’d been meaning to write a follow-up for quite a while now, and here it is:

OnlyMyEmail.com sucks. No really, they really suck. Really.

Here’s the problem: Because I have so many email addresses, some of them going back more than 10 years, I get loads and loads of spam. Something like 2,000-3,000 messages a day. That’s on the order of 100-200 times as many legitimate emails that I’d like to actually have a chance to read.

OnlyMyEmail was great while it lasted, which was all of two days. I quickly exceeded their quota, and despite repeated attempts to contact someone there to see if it was possible to pay more money to remove the limitations, they dumped me. I couldn’t reach anyone there. Apparently this isn’t a problem that’s unique to me.

Bottom line: If you don’t care at all about customer service, don’t mind outages (possibly permanent), and don’t take your email communication too seriously, then OnlyMyEmail.com may be for you. If you get a lot of spam and really, really need your email to work, forget it.

So I searched around for a few days at the end of October, and found Runbox.com. They’re not free, but quite reasonable at $40/year. They support POP, IMAP, have web-mail, and even have a site for small-screen clients like PDAs, and give you 10GB of email storage — more than most people will ever use. They’re based in Norway, so I have some hope that they take privacy pretty seriously. Their servers are pretty responsive most of the time, despite the location. And best of all: they filter spam. Not as well, mind you, as OnlyMyEmail.com did (for the two days that I was actually able to use it), but pretty well.

After a few days though, a new problem became evident. I seemed to have two choices, depending on how I configured the Runbox spam filters: Either I would get a bunch of false-positives every day (meaning I still had to scan all the spam anyway), or if I turned off the learning filter I’d still get a few hundred spams every day. Sometimes more, depending on how well the Runbox people were keeping up with the spammers.

For a couple of months, I opted to scan the Spam mailbox once or twice a day. Sorting the mailbox by subject, I found it was relatively easy to scan for real email, since spam messages often use the same, or very similar subjects. Most even start with Re: which made it even easier.

But it still was far from perfect. If I missed something that was mistakenly labeled as spam, it might take a full day or more before I saw it and was able to respond. And I was still spending far too much of my time looking at spam — that’s what I wanted to avoid in the first place. At least it wasn’t taking up disk space on my machine.

Now though, just over four months later, I think I’ve finally got a working solution. It involves no fewer than four sets of filters, spread out over three different mail services, and my own computer. Here’s how it works:

How I seem to have finally solved my spam problem:

My main inbox is on a UserLand server. With a little help, I was able to set up some filters using a .procmailrc script for my account, that splits my incoming mail into three groups: definitely spam, low-risk, and high-risk, handling each group differently.

1. According to my script, a message is definitely spam when it has a keyword in the Subject, To or From fields which I know I don’t want to ever see: viagra, fuck, penis, virgin, etc. These all go immediately to /dev/null before ending up in any mailbox anywhere. Poof — gone. This accounts for about 60% of the spam I was receiving, off the top. But it wasn’t good enough. I didn’t want to have to spend a hour every day maintaining my .procmailrc either — what a pain.

2. Next, I identified low risk mail and let it through — partly. Low risk mail is any mail which matches my username (or one of them) in the To or CC fields: jake, jsavin, etc. These messages are forwarded to my Runbox account, where spamassassin and other filters catch almost all remaining spam that’s addressed to these users. Apple Mail then downloads my mail from Runbox (via IMAP), and filters it yet again, using its own trainable spam filter. Also, since the server at UserLand is forwarding directly to Runbox (instead of having Runbox log into my UserLand POP account), I get my email almost immediately, and don’t have to worry about Runbox being able to connect to UserLand’s POP server.

3. But what about all the other mail — the accounts I have on other domain names, and the catch-all throw-away addresses I use when signing up for stuff. This qualifies as high risk email. High risk mail is anything that’s left over after 1 & 2 above. High-risk mail is sent via my .procmailrc straight to my Gmail account, where Google’s gigantic distributed network of servers filters out even more spam. What’s still considered legitimate by Google is then downloaded by Apple Mail (via POP), and filtered yet again on my own computer.

What’s the net result, you ask? Well, in the last few days, I’ve gotten three or four false-positives (out of about 250 legitimate messages), and filtered out about 5,500 spam messages. Even better, I’m 99% sure that the spam being caught by Gmail will contain very few false-positives, so I won’t have to check there very often. What’s more, I’ve only seen about 50 spam messages caught at Runbox on any given day. That’s a whole lot less email to scan for missed messages than 2,000+ a day.

Oh happy, happy, joy, joy!!!

Is it true? Can I finally be nearly spam-free, if only fleetingly?
One can only hope.

Hacks and Mash-Ups

Tonight, while checking to see if there were any new updates to TiVo Tool, I stumbled across this thread on the DealDatabase Forums (where all serious TiVo hacking seems to get discussed), about the inclusion of a tystream demuxing module with VLC 0.8.2. Also linked on the thread is a module for streaming from TiVo to VLC.

Geek speak aside, what this means is that I can now stream shows from my TiVo straight to VLC player on my Mac. Coolio!

The trick for me was to figure out what invocation to use under the File -> Open Network command. After following the instructions on the DealDatabase thread, choose Open Network from the File menu, and enter tivo://192.168.0.10/plist into the first field in the dialog. (You have to replace the IP address with the IP or host name of your TiVo unit.)

Then open VLC’s playlist, and double-click the show you want to watch. Easy as that.

If you have trouble downloading the tystream and vserver modules for VLC, post a comment, and I’ll hook you up.

Happy streaming!

Hacks and Mash-Ups

Last weekend, I finally pulled the A drive from my Series 1 Philips TiVo, and set it up for shell and FTP access over my LAN. More interesting to me, and maybe to you, I also now have a way to transfer video over the network, to MPEG video files on my Mac OS X machine.

I had to dig through a whole bunch of Google searches, message board posts, install instructions, and irrelevant Windows information to make this happen, but in the end I overcame the odds and got it working.

Here’s my mini how-to for transferring TiVo shows to your Mac OS X machine. They should work with any Series 1 TiVo running version 3.x TiVo software, and decent-speed Mac running Panther (OS X 10.3) or better:

  1. Get a network card and install it. I’ve had a TurboNet card installed for about two years. It was cheap and setup was a breeze.
  2. Get shell access working on your TiVo. The best instructions I found were these from Steve Jenkins. Requirements for getting this working:

  • A PC with an IDE bus, which can boot from a CD-R.
  • A way to burn an ISO CD image to a CD-R disc.
  • Willingness and ability to remove a drive from your TiVo and install it on the correct bus, with the correct jumper settings in your PC. (Not so hard for some, but daunting for most newbies — get help from a geek friend if you need it.)
  • Willingness and ability to read instructions and follow them to the letter, or at least in sequential order. 😉
  1. Install TyStudio on your Mac. Normally the download would be here, but at the moment there’s no Mac OS X version available. At one time, someone on the DealDatabase forum built an OS X version, but I’ve lost the thread now. Instead, here’s a download of the version I installed on my PowerBook G4. It’s a little over 5MB.

  2. Install tyserver on your TiVo. Basically you just want to upload tyserver.ppc.sh to your TiVo, run it to install tyserver, then add commands to your startup script so it will run all the time. Note: I strongly recommend not letting the installer add its startup commands to your TiVo’s built-in rc.sysinit, since this can really mess things up something goes unexpectedly wrong. Instead add them to your own custom startup script — you created one when following the instructions in step 2, didn’t you? 😉

If the tyserver download link above doesn’t work anymore, you can use the version that I used on my Series 1.

Here’s what I put in my startup script:

  1. Restart your TiVo. I’m not sure if you need to do Standby and then power-cycle, or just do Standby and then wake your TiVo again. (If you’re command-line savvy, you can launch tyserver and tyindex in your telnet client instead of restarting.)

5a. Stop here, and wait for a couple of hours. The tyindex software running on your TiVo needs some time to collect data about the shows you’ve already recorded. In my case, I stopped here, and went to bed, then woke up on Sunday, and did the rest. The amount of time it takes for tyindex to complete will depend on the number of shows in your TiVo’s Now Showing list.

5b. Did you read 5a? I’m serious — wait for a bit. If you don’t, you’ll just be frustrated and think this won’t work. Trust me — it’ll work. (Or not. It worked for me, that’s all I’m saying.)

  1. Launch X11 on your Mac — it’s in your Applications/Utilities folder. (You should be able to do this in Jaguar as well, provided that you first install X11.)
  • Choose Customize from the Applications menu in X11. Then add a menu link for tyeditor from the TyStudio install you did in step 3. I’d installed my copy in /Applications/Video/tystudio/, so the path I used was /Applications/Video/tystudio/tyeditor. Now you should have a tyeditor command in your Applications menu, for launching TyStudio whenever you need it.

  • Choose your spiffy new tyeditor command from the Applications menu in X11. This brings up the TyStudio GUI. (Now we’re cooking with gas!)

  • Click the ‘Click here to load from Now Showing’ button. You should now see a list of shows. Double-click one whose value for ‘Indexed’ is ‘Yes’. (Ref: step 5.) I recommend you start with a short one — 1/2 hour or less.

  • Choose ‘Generic MPEG2’ from the Video to pop-up menu, and ‘No Transcoding’ from the Audio to pop-up menu. This will give you basically the same audio/video that your TiVo would play on your TV. Then click the Process button, choose a place to save the file, and you should be off to the races!

  • Just a few notes:

    • These files are really big. We’re talking about 1GB or more per hour, depending on your TiVo’s quality settings. Make sure you have enough drive space.
    • If you’re going to do a lot of this, I recommend that you use Tools -> Options in tyeditor to choose the video and audio options you’re going to use most frequently, so you don’t have to change them every time in the tyeditor main window.
    • I haven’t done much messing with the transcoding options, since I’m mostly using TyStudio to make digital archives of stuff that I want to keep around, that my TiVo has recorded. If you use transcoding, you’re going to lose quality. Series 1 TiVo’s record PCM audio at 32KHz. (PCM is roughly equivalent to AIFF or WAV.) Video is recorded at varying rates depending on the quality setting the recording was made with, but is always much larger than the audio portion of a recording.
    • If your network is fast enough (as 802.11b on my mac, with 10-base-T on the TiVo and a 10-base-T hub seems to be me), you should be able to start downloading a show on your Mac, and then start playing it a few seconds or minutes later, to in essence stream the show to your Mac’s video screen. Unfortunately, you can’t do this using QuickTime Player, since it checks the size of the file when you first open it, and will stop at that point despite the fact that you’re still downloading. VLC on the other hand, will keep playing a file until it either gets an error, or it reaches the actual end of the file. If you’re going to watch shows as they’re downloading, I recommend that you use VLC for playback instead of QuickTime Player. (MPlayer may work as well, but I like VLC better.)
    • While you’re hacking, you might as well install TivoWebPlus — this is a great app that no TiVo should be without.
    • TyStudio works on Windows and Linux as well. A quick search should make it very clear that they’re much easier to find than the Mac version was.

    Feel free to post a comment if you have a question about any of the above. If I have an answer, I’ll let you know. If not, I’ll ignore you. If you’re rude I’ll delete your comment, and if you’re rude again, I’ll block you, so be nice.

    Also, I haven’t found a way to stream MPEG data directly from the TiVo to a movie playing application on the Mac, without also downloading it to a disk on the Mac side. If you figure this out, please let me know. My filesystem will thank you. 😉

    Happy TiVo hacking!

    Hacks and Mash-Ups