Skip to Content

engineering

Amazon S3 Filesystem for Linux: s3simple

Introduction

Over the last semester I created a rudimentary filesystem for Linux to access files on Amazon S3. The original goal was to create a system that was transparent to the end user and provide a form of semantic caching that would be optimized for applications like browsing photos or music and for network connected devices like a mobile phone or netbook. The motivation being that your files could reside in the cloud where they are safe, synchronized, and have unlimited room for growth. Due to my difficulty with writing the initial filesystem I did not have time to work on caching.

Development

The original plan was to develop this completely in Linux Kernel space so that it would be simple and compact. However, to simplify development and use libcurl for HTTP requests, I had to split the project into Kernal space and userspace components. This follows a similar structure to projects like FUSE and Coda:

s3simple architecture

In this structure, requests have to pass through a virtual character device defined by the system, which is not ideal (you have to poll the device) but seems to work fairly well. The filesystem itself leverages standard Linux system calls.

Here are a few resources I found very helpful while developing this. They serve as a great jumping off points for developing Linux filesystems or (virtual) device drivers.

Current Status and Future

Since this is a very rough implementation there are still a number of limitations that need to be addressed before this module is really usable:

  • Currently the system only works for files less than a block in size. This means that small text files are fine, but anything larger than that doesn't work.
  • The system currently can only be used as superuser. I attempted addressing this with udev rules, but still run into permission denied errors when looking at files.

For the future, I still think it would be nice to move the whole module into Kernel space and avoid having to run an extra program to make requests. This would keep the usage of the module much simpler. I also think it would be exciting to add support for some sort of semantic caching and an API to allow programs to define the type of prefetching to use.

Code

At this point I am hosting the code and build instructions on my GitHub account as s3simple. I am not going to have as much time to develop on this project over the next little while, but it's likely that I will be pushing small changes whenever I have some time and am interested.

Hello World in C, C++, and Java in Eclipse

A friend emailed a few months ago wondering how to compile a C program. Since this is an important first step for developers and I found that there were no simple and clear resources available related to Eclipse, I thought it might be good idea to go over it. So... here's how to setup and compile a "Hello world!" program in C, C++, and Java using the Eclipse IDE.

I've broken this up into five parts:

  1. Installing and setting up Eclipse
  2. Hello world in C with Eclipse
  3. Hello world in C++ with Eclipse
  4. Hello world in Java with Eclipse
  5. Compiling from the command line

1. Installing and setting up Eclipse

Eclipse is a free, cross-platform, open-source integrated development environment (IDE) that makes development easier. It features plugins for many languages, including PHP and Google Android, but it is also great for C, C++, and Java.

To download Eclipse, visit their download page (http://www.eclipse.org/downloads/) and choose either "Eclipse IDE for Java Developers" or "Eclipse IDE for C/C++ Developers". Install as is standard for your operating system. If you are going to be developing in different languages (I've been using all three at school right now) just download either one and install the other's packages later. To install additional packages, open up Eclipse and click Help > Software Updates, then click on the Available Software tab, expand "Ganymede Update Site", choose which plugins you want (probably either C and C++ Development or Java Development), and click Install on the right.

Eclipse keeps your panels organized using what's called a "Perspective", if you have both Java and C/C++ development installed you will have two perspectives which you can switch between:

Eclipse perspective buttons for C/C++ and Java

To compile programs in C/C++ or Java you may need to install the appropriate compiler. So, go ahead and install Java or GCC. On Mac OS, installing Apple Developer Tools (http://developer.apple.com/tools/xcode/) should do the trick if it's not already installed.

Now create a new project by clicking File > New > Project.. and follow the wizard. It will allow you to choose which language you want to use and setup all the necessary files for it. You'll have to give it a name, but can probably ignore the rest of the options in the wizard.

Now that we have Eclipse setup, lets compile a program.

2. Hello world in C with Eclipse

You should now have a new project setup and need to create your first source file. To do this, click File > New > Source File. I named mine HelloWorld.c.

Lets add some code to this file:
#include <stdio.h>
main() {
  printf("Hello world!\n");
}

Now click on the little hammer build icon and then the big green play button. This will compile and then run the program, showing "Hello world!" printed in your Console window.

3. Hello world in C++ with Eclipse

With a new C++ project setup, create a new file called HelloWorld.cpp. Then enter the following code:
#include <iostream.h>
main() {
  cout << "Hello World!\n";
}

Now click on the little hammer build icon and then the big green play button. This should compile and then run the program showing "Hello world!" printed in your Console window.

4. Hello world in Java with Eclipse

Now that you have a new project setup, you have to create your first class. To do this, click File > New > Class. Give it a name, I chose to use "HelloWorld". For now, you can probably ignore all of the other options. It will bring up a new file called HelloWorld.java which contains the following code:
public class HelloWorld {
 
}

Let's add our Hello World code by changing this to:
public class HelloWorld {
  public static void main( String[] args ) {
    System.out.println("Hello world!");
  }
}

Now click on the big green "play" button to compile and run the program - you will be prompted to choose how you want to run this, choose "Java Application". Then magically in the Console window you will see "Hello world!" printed!

5. Compiling from the command line

Since I think getting familiar with the command line (Terminal.app in OS X) is important I will also add instructions for compiling each of these files from the command line.

C:

  1. Navigate to the directory containing your files.
  2. To compile, run: gcc HelloWorld.c -o hello
  3. Then to run the program: ./hello

C++:

  1. Navigate to the directory containing your files.
  2. To compile, run: g++ HelloWorld.cpp -o hello
  3. This will probably give you a bunch of warnings but it will be fine. Then to run the program: ./hello

Java:

  1. Navigate to the directory containing your files.
  2. To compile, run: javac HelloWorld.java
  3. Then to run the program: java HelloWorld

Mind Control Device by Emotiv Systems

Here is a video demonstration of a thought control device created by Emotiv Systems of Australia. Even though this lecture is kind of long I think it is extremely important to watch — this kind of technology is going to surround us quite soon.

What is so remarkable is that the device will be retailing for only $299 USD and is completely non-invasive. There are so many applications for this it is boggling. I would love to see it in action and some of my first questions were:

  1. Can I record data of myself typing and can this be mapped to a point where I don't need a keyboard anymore?
  2. As per the example in the video, if I map a rotate in different axes can the software learn when I am trying to combine actions? Could I then dynamically rotate in many axes at once?
  3. Will this be available in an open way? Will we be able to apply the SDK on Linux for example?

Also, OCZ has a device called the Neural Impulse Actuator which will be available for $159 USD. I haven't seen a demonstration yet but I imagine its capabilities are somewhat similar.

I am excited about this as there is so much opportunity for business and innovation. It will be remarkable when we start seeing this kind of technology in our everyday lives and the potential is hard to imagine.

Next steps: They have an SDK available which I intend to poke around in and is there anyone in Vancouver already exploring this technology?

EECE 375/474 Project

For the past three and a half months I have been working with a group of my engineering peers at UBC to build a complete system for managing a library. We did our final demo today and everything went smoothly. The system uses RFID tags to keep track of books and communicates wirelessly using a ZigBee module. There is a built in charging circuit which helps keep the five NiMH powered up and an AtMega88 microcontroller to handle all the logic. Everything else is taken care of by a Drupal website with a few custom modules. Here is a quick demo of the unit:

The system supports adding new users (who have library cards with an embedded RFID tag), inserting new books, checking books in and out, passively finding lost books, and providing directions to find books in the library. Here are a few photos of the final construction process:

DSC_7316 DSC_7288 DSC_7286

DSC_7299  DSC_7290

DSC_7283  DSC_6684

I would say the project was a great success; we finished our integration testing early on and even had time to make library cards and build a nice metal case. Lots of credit goes out to the other members of my team: Aaron, Henry, Roland, and Steve. Very glad to be done!

Syndicate content