Skip to Content

Amazon S3 Filesystem for Linux: s3simple

Introduction

Over the last semester I created a rudimentary filesystem for Linux to access files on Amazon S3. The original goal was to create a system that was transparent to the end user and provide a form of semantic caching that would be optimized for applications like browsing photos or music and for network connected devices like a mobile phone or netbook. The motivation being that your files could reside in the cloud where they are safe, synchronized, and have unlimited room for growth. Due to my difficulty with writing the initial filesystem I did not have time to work on caching.

Development

The original plan was to develop this completely in Linux Kernel space so that it would be simple and compact. However, to simplify development and use libcurl for HTTP requests, I had to split the project into Kernal space and userspace components. This follows a similar structure to projects like FUSE and Coda:

s3simple architecture

In this structure, requests have to pass through a virtual character device defined by the system, which is not ideal (you have to poll the device) but seems to work fairly well. The filesystem itself leverages standard Linux system calls.

Here are a few resources I found very helpful while developing this. They serve as a great jumping off points for developing Linux filesystems or (virtual) device drivers.

Current Status and Future

Since this is a very rough implementation there are still a number of limitations that need to be addressed before this module is really usable:

  • Currently the system only works for files less than a block in size. This means that small text files are fine, but anything larger than that doesn't work.
  • The system currently can only be used as superuser. I attempted addressing this with udev rules, but still run into permission denied errors when looking at files.

For the future, I still think it would be nice to move the whole module into Kernel space and avoid having to run an extra program to make requests. This would keep the usage of the module much simpler. I also think it would be exciting to add support for some sort of semantic caching and an API to allow programs to define the type of prefetching to use.

Code

At this point I am hosting the code and build instructions on my GitHub account as s3simple. I am not going to have as much time to develop on this project over the next little while, but it's likely that I will be pushing small changes whenever I have some time and am interested.

Pager keys for Drupal and Views

About a half hour ago I found a really great script on Github for quickly browsing posts on a page. If you are on the homepage of my site, try typing the following hotkeys:

Key Action
h Next page
j Next post
k Previous post
l Previous page

If you have used the hotkeys in GMail or Google Reader, you will have likely used these shortcuts before. This usability pattern is showing up in quite a few places and I really like it. Take a look at how effective it is for browsing images on The Big Picture (make sure you're on a post), on Flickriver, and on FFFFOUND!.

Drupal is a great candidate for this pattern since quickly browsing through a list of nodes is quite common. Also, since Drupal comes with jQuery, adding the script wasn't too difficult. Here are some small adjustments to make it work.

This guide assumes that you are using Views to generate a list of nodes but it should be easy to adapt to other situations.

  1. Snag the jQuery version provided on Git and save it to your theme's directory.
  2. Add the page_keys.js to Drupal by adding the line scripts[] = paging_keys.js to your theme's .info.
  3. We have to make a few small adjustments to the script to make it play nice with Drupal. Go take a look at the settings array found on line 84 of paging_keys.js. Let's change it to the following:

    var config = {
      nodeSelector: '#main .node h2.node-title a',
      prevPageSelector: '.pager-previous a',
      nextPageSelector: '.pager-next a',
      pagingNavId: 'paging-nav',
      keyNext: 'j',
      keyPrev: 'k',
      keyNextPage: 'h',
      keyPrevPage: 'l',
      keyRefresh: 'r',
      additionalBodyClass: 'paging-keys',
      bottomAnchor: 'bottom'
    };

    This assumes that the list of nodes will be under the div '#main', but you will probably want to change this to be more specific. For my own site I've also used the view name in the selector to ensure that there is not any weird behaviour. The only other changes to the config array were for the Drupal pager.

  4. We want to add the anchor "bottom" to the last node of the View, allowing the script to move nicely between pages. To do this, we can add a small snippet to the node title link in node.tpl.php. Here is what my revised node title code looks like:

    <h2 class="title node-title">
      <a href="<?php echo $node_url; ?>"<?php if($view) { end($view->result)->nid == $node->nid ? print 'name="bottom"' : NULL; } ?>><?php echo $title; ?></a>
    </h2>

    This will check if we are displaying the last node for this view and add 'name="bottom"' to it's title link.

And you are done! Hopefully everything is working great. If you encounter problems, make sure that caching is turned off while you try to integrate this.

Update and content elsewhere

I have been pretty busy so far this summer and have been neglecting updates to this blog. I have a couple half written blog posts that I hope to get up soon: a summary of my user centred design course from last spring, a summary (and maybe discussion) of Jaron Lanier's essay Digital Maoism, a discussion of some of the text editors I've been using, and maybe an outline of my current ideas for a personal “toolbox” for business development. No promises on any of these, but if I hear requests I'm sure they will come up sooner.

I have been creating content elsewhere and here are a couple of posts on the Affinity Bridge blog that I'm proud to share:

I haven't really been taking photos with my good camera either but I will leave you with some snaps of circles from my iPhone.

photo.jpg photo.jpg

photo.jpg photo.jpg

photo.jpg photo.jpg

photo.jpg photo.jpg

Hello World in C, C++, and Java in Eclipse

A friend emailed a few months ago wondering how to compile a C program. Since this is an important first step for developers and I found that there were no simple and clear resources available related to Eclipse, I thought it might be good idea to go over it. So... here's how to setup and compile a "Hello world!" program in C, C++, and Java using the Eclipse IDE.

I've broken this up into five parts:

  1. Installing and setting up Eclipse
  2. Hello world in C with Eclipse
  3. Hello world in C++ with Eclipse
  4. Hello world in Java with Eclipse
  5. Compiling from the command line

1. Installing and setting up Eclipse

Eclipse is a free, cross-platform, open-source integrated development environment (IDE) that makes development easier. It features plugins for many languages, including PHP and Google Android, but it is also great for C, C++, and Java.

To download Eclipse, visit their download page (http://www.eclipse.org/downloads/) and choose either "Eclipse IDE for Java Developers" or "Eclipse IDE for C/C++ Developers". Install as is standard for your operating system. If you are going to be developing in different languages (I've been using all three at school right now) just download either one and install the other's packages later. To install additional packages, open up Eclipse and click Help > Software Updates, then click on the Available Software tab, expand "Ganymede Update Site", choose which plugins you want (probably either C and C++ Development or Java Development), and click Install on the right.

Eclipse keeps your panels organized using what's called a "Perspective", if you have both Java and C/C++ development installed you will have two perspectives which you can switch between:

Eclipse perspective buttons for C/C++ and Java

To compile programs in C/C++ or Java you may need to install the appropriate compiler. So, go ahead and install Java or GCC. On Mac OS, installing Apple Developer Tools (http://developer.apple.com/tools/xcode/) should do the trick if it's not already installed.

Now create a new project by clicking File > New > Project.. and follow the wizard. It will allow you to choose which language you want to use and setup all the necessary files for it. You'll have to give it a name, but can probably ignore the rest of the options in the wizard.

Now that we have Eclipse setup, lets compile a program.

2. Hello world in C with Eclipse

You should now have a new project setup and need to create your first source file. To do this, click File > New > Source File. I named mine HelloWorld.c.

Lets add some code to this file:
#include <stdio.h>
main() {
  printf("Hello world!\n");
}

Now click on the little hammer build icon and then the big green play button. This will compile and then run the program, showing "Hello world!" printed in your Console window.

3. Hello world in C++ with Eclipse

With a new C++ project setup, create a new file called HelloWorld.cpp. Then enter the following code:
#include <iostream.h>
main() {
  cout << "Hello World!\n";
}

Now click on the little hammer build icon and then the big green play button. This should compile and then run the program showing "Hello world!" printed in your Console window.

4. Hello world in Java with Eclipse

Now that you have a new project setup, you have to create your first class. To do this, click File > New > Class. Give it a name, I chose to use "HelloWorld". For now, you can probably ignore all of the other options. It will bring up a new file called HelloWorld.java which contains the following code:
public class HelloWorld {
 
}

Let's add our Hello World code by changing this to:
public class HelloWorld {
  public static void main( String[] args ) {
    System.out.println("Hello world!");
  }
}

Now click on the big green "play" button to compile and run the program - you will be prompted to choose how you want to run this, choose "Java Application". Then magically in the Console window you will see "Hello world!" printed!

5. Compiling from the command line

Since I think getting familiar with the command line (Terminal.app in OS X) is important I will also add instructions for compiling each of these files from the command line.

C:

  1. Navigate to the directory containing your files.
  2. To compile, run: gcc HelloWorld.c -o hello
  3. Then to run the program: ./hello

C++:

  1. Navigate to the directory containing your files.
  2. To compile, run: g++ HelloWorld.cpp -o hello
  3. This will probably give you a bunch of warnings but it will be fine. Then to run the program: ./hello

Java:

  1. Navigate to the directory containing your files.
  2. To compile, run: javac HelloWorld.java
  3. Then to run the program: java HelloWorld

Art in Technology, Art in the Web

At Northern Voice this year Darren Barefoot led a discussion surrounding online art — mostly just asking the question of where are the great works of online art? The most common theme was that of the collection, where curation and organization are what makes online art. Some examples are mashup music and the Big Picture photo blog.

I wanted to expand on another idea that was briefly brought up. Most people would agree that art is personal and one of my favourite quotes is that “art is for making people feel less alone.” Following that, what if art was very personal? Some services that can provide this for you are Last.fm or Apple's Genius. For Last.fm, it builds a database of your preferences based on the kinds of music you most commonly listen to and the music that you ‘favourite’. It then compares your preferences to that of the wider community to help bring you custom playlists with new music. Here, the art is in building a playlist that can mean something to you.

Last.fm, to me, is somewhat hit or miss, but here is what Last.fm just did for me (don't watch the videos, just listen, hopefully it's raining where you are).

It is raining outside and I turn on Last.fm. As I'm getting my umbrella out, walking down the steps, crossing the street, and walking through the mushy park up to the bus stop, this is playing:

If you can see this, then you might need a Flash Player upgrade or you need to install Flash Player if it's missing. Get Flash Player from Adobe.

(Fila Brazillia – Place de la Concorde)

Then as I hop on the bus, which is mostly silent and wet, this is playing...

(Blockhead – A Better Place)

It takes me to my stop where I get out and wait with another dripping woman for the elevator.

As soon as I enter my apartment, the following starts playing. It's warm, and fun but still sounds like rain:

If you can see this, then you might need a Flash Player upgrade or you need to install Flash Player if it's missing. Get Flash Player from Adobe.

(People Under the Stairs – Time To Rock Our Shit)

Sometimes technology can bring a little nuance.

Syndicate content