Another big point that I rather liked about it, was that from design, through development to deployment it took about 6 hours to get the first release out! I fixed a few bugs a couple of days later, so there has altogether been about 8 hours work put into it. Not too shabby, for something useful! I decided to use something new to build it, so used the Monk ‘glue’ framework which by default combines Sinatra, Ohm and Redis. It is a pretty nice system, however like most Ruby software the default skeletons (and at least the ones mentioned on the site) are rather opinionated. I myself prefer Bundler however none of them had this for dependency management. I wouldn’t have thought it would be terribly difficult to have a nice system that lets you add or remove different gems from the skeleton so you can customise it to your liking, that would be a nice feature for a future revision of the website. I then rounded it up with running on Ruby 1.9.2 and Unicorn.

In terms of actually using the tools, I have used Sinatra before and played around with Redis and Ohm a bit, however for those I mainly relearnt everything I (thought I) knew. This was the first time I had properly used a key-value storage engine, so something things were a bit strange. As an example I wanted to ‘expire’ old users after 6 hours so that their statuses would be refetched. In SQL this is dead simple, however how to replicate that in Redis? In the end I ended up fetching all the records and comparing the created_at field in code, I have doubts this is the optimum solution however….

I also had a few teething problems that went unnoticed (for about 15 days >_<) which I recently resolved. I’m not sure what happened however the expiry daemon just stopped working, even restarting it didn’t help. In the end I restarted Redis which sorted it, so I’m not sure what was to blame there. JNo was the last user to be added to the system, so I blame him. Everything seems to be working again now, at least for a while….

Another major problem (which I still haven’t resolved) is an issue with the Twitter search. Certain users just return no search results. The first user I found with this was the BBC News Twitter account, it had over 70,000 tweets so I thought maybe the search just didn’t index large accounts. However I then found a few more accounts that only had a couple of hundred tweets. I still haven’t got this resolved, mainly because I have yet to ask for help from the gods of Twitter

So that about sums up Tweeted Links. I’m quite happy with the results so far, and am looking to expand it a bit further when I get time (I’m currently focussing on some Android work, which I’m sure there will be a few posts on over the next couple of weeks). I expect I will probably rewrite the system from scratch though, as to be honest, I’m not really too happy about how this key-value stuff has worked out so far. The rewrites will also involve some big data restructuring, so I don’t think it is a bad time to do so. Back to MySQL I say, yep I just said no to NoSQL!

It is also available as a PDF version.

(On a side note I have been using Google Docs for everything recently. The number of features has grown enormously since I first tried it out. No more Open Office. Yay!)

BFS and CFS are both different types of task schedulers used by the Linux kernel. CFS (Completely Fair Scheduler) is the default scheduler in the majority of distributions, however it isn’t though to be great. I won’t go into the details, however being relatively old it has built up quite a lot of bulk and the algorithms used are rather complicated.

BFS (Brain Fuck Scheduler) is the new kid on the block. It was written in 2007 after the author became annoyed with the random stalls experienced while using a Linux-based desktop machine. The scheduler is designed to offer low latency when used interactively, for example on a desktop machine, or a phone!

As stated latency is usually reduced, and random stalls should be reduced. However BFS also has another trick up its sleeve. In benchmarks it performed 80% better when encoding a video in x264 format!

BFS however isn’t (yet) going to make your Linux based systems super fast. Benchmark results are rather mixed, and discussions around it are rather heated. It is not currently included in the mainline Linux tree, and doesn’t look likely to be included anytime soon.

Either way, if you want to try it on your Android device, it is a quick flash away. Cyanogen Mod includes it by default, and there are plenty of different kernels out there you can try. Go ahead and try it, YMMV!

Further reading

• BFS – The Brain Fuck Scheduler by Con Kolivas – An introduction by the author
• FAQS about BFS – More details
• Open source collaboration done right – Thoughts of a codec developer on BFS

Intro
I have a 1TB drive which I use as a media drive. I currently have it partitioned as follows:

# fdisk /dev/sdb -l

Disk /dev/sdb: 1000.2 GB, 1000204886016 bytes
255 heads, 63 sectors/track, 121601 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0xffffffff

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1   *           1         608     4883728+  83  Linux
/dev/sdb2             609      121601   971876272+  8e  Linux LVM

Yeah nothing fancy, a small 5GB partition at the start and the rest setup in LVM. The small partition is to serve as a simple recovery partition if I want to access the contents of the drive on another OS; I can do so just by booting that up in a virtual machine. 5GB is probably a bit big, however it isn’t as if space is scarce.

In terms of LVM I just have a volume taking up the full size of the disk:

# lvdisplay
  --- Logical volume ---
  LV Name                /dev/vg/media
  VG Name                vg
  LV UUID                2OJIN6-sAx8-KRIz-SNvl-RU0I-2Xgl-nzUH4V
  LV Write Access        read/write
  LV Status              available
  # open                 0
  LV Size                926.85 GiB
  Current LE             237274
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           254:0

I have previously setup different partitions for certain things, however in the end that just got a pain when one partition got too small. This setup kind of defeats the point of LVM, however if I want another partition (e.g. to dual boot) or to add another disk I can easily do so.

ext4
Using the default options, on the face of it, ext4 doesn’t provide any significant advantages over ext3, or even ext2 when used for a media drive. However there are some nice features behind the scenes:

• Larger maximum filesystem size (ext3 is limited to 16TB)
• Extents (a set of contiguous blocks; improves performance and leads to decreased fragmentation)
• Multiblock allocation (as opposed to single block allocation; improves write performance for large files)
• Delayed allocation (doesn’t write the file to disk immediately, waiting until more of the file is in the cache, and then allocating a larger number of blocks; leads to decreased fragmentation)
• Faster fsck (Try running fsck on a 1TB ext2 volume ^_^)
• Persistant preallocation (Reserves a large number of blocks, which aren’t going to be used yet but will be at some point; on par with what modern P2P applications do by filling a file with zeros, however implemented into the filesystem so much more efficient)

A good introduction to ext4 (which these were taken from) is the ext4 article on Kernel Newbies.

Creating the filesystem

The options for mkfs.ext4 are basically unchanged from ext3, in order to enable the high performance options most of the work will be done in tune2fs and mount options. Even so for now

The b option sets the block size (this will most likely default to 4096 bytes), the M options set the percentage of space reserved for the super user, and the L option sets the filesystem label. This command shouldn’t take more than a couple of minutes to run.

Optimising

Next up is to pass some options to tune2fs to enable the advanced features. These options could have been passed to the previous command, however to easily explain they have been listed separated. You can also run this command to upgrade an existing ext3 file system. A lot of these are enabled by default, however they are listed here for completeness sake.

• has_journal – Enables the ext4 journal
• extent – Enables extents (see above)
• huge_file – Enables files over 2GB in size
• flex_bg – Allows inode metadata to be placed anywhere on the partition (as opposed to traditionally at the start)
• uninit_bg – Enables new blocks to not be initialised and enables block checksumming. Significantly speeds up creation and fsck
• dir_nlink – Enables unlimited sub directories
• dir_index – Enables a hashed B-tree for linking subdirectories
• extra_isize – Enables nanoseconds to be stored in timestamps

# tune2fs -O has_journal,extent,huge_file,flex_bg,uninit_bg,dir_nlink,dir_index,extra_isize /dev/vg/media
tune2fs 1.41.12 (17-May-2010)

As this just enables various options on the file system it will execute rather quickly, especially if the file system is empty. Note that if you are migrating an existing partition, most of these options will only apply to newly created files.

Mount options

The real killer performance can be obtained by setting certain mount options. Note that these seriously increase the likelihood of corruption if a system crash occurs, however in the case of a media drive, where there are a limited number of writes, I don’t feel that this is an issue.

• barrier=0 – Disables a protection option that ensures everything is written to the journal before being committed, however decreases performance by ~30%; this is the default on the majority of systems
• commit=60 – Only commits the journal to disk every 60 seconds
• noatime – Disables the access time attribute which causes the metadata to be updated every time a file is accessed
• data=writeback – Enables journaling of metadata only; can cause files to become corrupt if a system crash occurs
• journal_async_commit – Write the journal to disk asynchronously

These can be set in your /etc/fstab, e.g.:

/dev/vg/media	/media	ext4	defaults,user,barrier=0,commit=60,noatime,data=writeback,journal_async_commit	0	0

So thats it! If you find any other options that you deem useful, feel free to post them in the comments!

At the moment it isn’t really that impressive, however it looks like they are going to be doing some cool stuff with XAuth, a service that shares what social connections you have with a site you are visiting. So instead of a trillion ‘Share on ‘ buttons, you will just have one to share on everything. Nice!

Checkout the video below for an introduction to XAuth by the Meebo guys:

$ irb
ruby-1.8.6-p383 > a = Time.now
=> Wed Apr 14 15:36:21 +0100 2010
ruby-1.8.6-p383 > a.utc
=> Wed Apr 14 14:36:21 UTC 2010
ruby-1.8.6-p383 > a
=> Wed Apr 14 14:36:21 UTC 2010

This is actually in the Ruby docs, however who reads those? This is kind of unexpected behaviour, as in most cases in Ruby (however this is more of a Rails thing), if there are two methods the more dangerous of the two is suffixed by an exclamation mark. A good example is String#strip and String#strip!. The version without an exclamation mark doesn’t modify the original object, however the version with does:

$ irb
ruby-1.8.6-p383 > a = " hello "
=> " hello "
ruby-1.8.6-p383 > a.strip
=> "hello"
ruby-1.8.6-p383 > a
=> " hello "
ruby-1.8.6-p383 > a.strip!
=> "hello"
ruby-1.8.6-p383 > a
=> "hello"

So if you are going to use #utc, be careful and make sure it you aren’t reliant on the time offset later. You should really be using #dup or #getutc which won’t modify the receiver:

$ irb
ruby-1.8.6-p383 > b = Time.now
=> Wed Apr 14 15:36:59 +0100 2010
ruby-1.8.6-p383 > b.getutc
=> Wed Apr 14 14:36:59 UTC 2010
ruby-1.8.6-p383 > b
=> Wed Apr 14 15:36:59 +0100 2010

This is present in all current versions of Ruby, so upgrading won’t help either!

Back to the topic, the idea I had was to investigate how (ridiculously) easy it would be to scrape email addresses from Twitter. Having already created the Rainbow Bot, I knew that searching for particular terms on Twitter using a certain gem was a piece of cake. So I started searching for ‘@gmail’.

The results were pretty much as expected, rather a lot of users Tweeted their email address. Most of them used some sort of simple filtering (a space between the user and domain; non word characters before the domain), and it was rather easy to tell an address which was invalid (three consecutive dots is a sure fire no-no). Also interesting were the number of users who replaced the user of the address with ‘username’ or ‘twitter’, for example ‘Send CVs to my twitter @gmail.com’.

I then decided to expand this for a couple of other free email services (Hotmail and Yahoo), and as expected, the number of addresses went up. A simple search gave me a list of free email domains (NB: I have doubts that is a list of ‘every possible email domains’ ), which would allow me to expand this further. I didn’t even bother investigating handling of pagination, and without special permissions the Twitter search API only returns a limited subset of the total number of Tweets.

In total I managed to scrape 8000 addresses over a period of approximately twelve hours. So in conclusion, social networks appear to be a spammers paradise.

Here in Blighty we call it British Summer Time (BST). This year (2009) BST started on Sunday, March 29th at 01:00 GMT, and ended on Sunday, October 25th at 01:00 GMT (02:00 BST).

If you system is dealing with dates that are in GMT everything is fine and dandy. Not quite so if you are given dates in BST, however still fairly easy to handle if you have a good time library. The issues pop up when you get given times in local time, and have no idea whether they are BST or GMT. When the times are changed back in the autumn there is a period when 01:33 could be BST or GMT.

The solution for this is quite simple though: NEVER EVER SEND TIMES IN LOCAL TIME, ESPECIALLY NOT WITHOUT A TIMEZONE.

Original by

travelmatt
.

I am rather excited about the job though. My first proper job with a proper company! The guys at the interview seemed really cool, so it should be good working with them. The actual role is going to involve mainly doing backend Ruby development, so expect to see lots of Ruby posts in the future! They run a full scrum environment, so it should be good to see how that works in practice. I don’t think it was even mentioned during my degree, let alone put into practice… The location of the offices are in a great location as well, right next to Waterloo station. A quick hunt after the interview, there is a Sainsbury’s, a Starbucks, and lots of pubs, what more do I need?

Regarding the accomodation I have had a quick look online, however I figured it would be better to wait until a) I have a phone signal and b) I can go to places in person. As such tomorrow is going to be the prime house hunt day. As recommended by the recruitment company, I am looking in the South West, around Clapham Junction area. This apparently has lots of young professionals, and isn’t too pricey. I can’t afford anything in the North (yet) and the East is a bit far out from everything.

Prices aren’t too bad either, for a shared house you are looking at about £125, and depending on the area, about double that for a small flat / bedsit. I am looking on sites such as Gumtree as most of the letting agencies seem to have the more expensive properties. I am not really sure of the best way to go about this house hunt, so that will be another post when I have a better clue.