Tweeted Links: An introduction, and a review

August 11, 2010

If you follow me on Twitter you have most likely seen my rants about my new service, Tweeted Links. The method behind the madness of it is basically this: I post a lot of links to Twitter, and often want to revisit them at a later time. I usually don’t Tweet anything meaningful with them that will help searching (e.g. KK) and usually shorten the URLs. Tweeted Links solves that problem! It keeps the links in a single location (a la and fetches the title of the page. It also solves the problem of shortened URLs by attempting to expand those too.

Another big point that I rather liked about it, was that from design, through development to deployment it took about 6 hours to get the first release out! I fixed a few bugs a couple of days later, so there has altogether been about 8 hours work put into it. Not too shabby, for something useful! I decided to use something new to build it, so used the Monk ‘glue’ framework which by default combines Sinatra, Ohm and Redis. It is a pretty nice system, however like most Ruby software the default skeletons (and at least the ones mentioned on the site) are rather opinionated. I myself prefer Bundler however none of them had this for dependency management. I wouldn’t have thought it would be terribly difficult to have a nice system that lets you add or remove different gems from the skeleton so you can customise it to your liking, that would be a nice feature for a future revision of the website. I then rounded it up with running on Ruby 1.9.2 and Unicorn.

In terms of actually using the tools, I have used Sinatra before and played around with Redis and Ohm a bit, however for those I mainly relearnt everything I (thought I) knew. This was the first time I had properly used a key-value storage engine, so something things were a bit strange. As an example I wanted to ‘expire’ old users after 6 hours so that their statuses would be refetched. In SQL this is dead simple, however how to replicate that in Redis? In the end I ended up fetching all the records and comparing the created_at field in code, I have doubts this is the optimum solution however….

I also had a few teething problems that went unnoticed (for about 15 days >_<) which I recently resolved. I’m not sure what happened however the expiry daemon just stopped working, even restarting it didn’t help. In the end I restarted Redis which sorted it, so I’m not sure what was to blame there. JNo was the last user to be added to the system, so I blame him. Everything seems to be working again now, at least for a while….

Another major problem (which I still haven’t resolved) is an issue with the Twitter search. Certain users just return no search results. The first user I found with this was the BBC News Twitter account, it had over 70,000 tweets so I thought maybe the search just didn’t index large accounts. However I then found a few more accounts that only had a couple of hundred tweets. I still haven’t got this resolved, mainly because I have yet to ask for help from the gods of Twitter 😛

So that about sums up Tweeted Links. I’m quite happy with the results so far, and am looking to expand it a bit further when I get time (I’m currently focussing on some Android work, which I’m sure there will be a few posts on over the next couple of weeks). I expect I will probably rewrite the system from scratch though, as to be honest, I’m not really too happy about how this key-value stuff has worked out so far. The rewrites will also involve some big data restructuring, so I don’t think it is a bad time to do so. Back to MySQL I say, yep I just said no to NoSQL!