Distributed computing and Ruby

At work, I've been spending a lot of time working on porting our log management and analysis system over from a dying mysql implementation to a new system build upon an open source distributed database (Hypertable) using distributed Map/Reduce jobs for a variety of summarization and analysis tasks.

I've been amazed and gratified to be able to do the vast majority of this work within the comfort of Ruby, my favorite language. While Hypertable is written in C++, it uses Thrift to provide ruby and other language access, and since the primary developers of Hypertable are at Zvents (a rails shop), they've created a Rails plugin called HyperRecord that allows us to access Hypertable almost identically to how you would access mysql with ActiveRecord.

This has resulted in the ability to make the front end application for our stats & logging infrastructure a standard Rails app. Access restrictions are somewhat different in Hypertable than in a full relational database (its easiest to think of as an ordered hash... lookups by key or for a range of keys are supported, but conditions are expensive), but for most of our developers its just another rails app to work with.

The second place where I've been amazed by how much I've been able to stick to ruby is in designing and running our batched Map/Reduce jobs. We're using a framework called Cascading for designing our scalable batch workflows, built in Java and sitting on top of Hadoop. For those who aren't familiar, Hadoop is an open source implementation of Map/Reduce, written in Java, and Cascading allows for a higher-level conceptual model for parsing, analyzing, and modifying your data using Hadoop.

Cascading provides a number of built in filtering, text processing, and arithmetic map/reduce operations built in, and thanks to the wonder that is jruby, we're able to arrange our workflows entirely within ruby using Cascading.jruby. Only when we need a special operation that can't be constructed from the built-ins do we have to dip into Java.

So if you've been itching to dip your toes in the open source distributed computing revolution, but have been reluctant due to the Java heavy nature of Hadoop, take a look at HyperTable and Cascading!

Event Handler chains in Shoes

One of the beautiful things about Shoes is how simple and easy it is to set up event handlers on objects. As I talked about in a previous post creating a handler on a button is as simple as

b = button "Click Me"
b.click { alert "You clicked the button" }

This is great for simple things, but I hadn't been hacking on Shoes for very long before I wanted to do more than this. In particular, since one of my projects is creating an application to preview Shoes applications, I wanted to be able to add handlers that call back into my main application without worrying about their effect on the code being previewed.

To do this, you need to be able to have multiple handlers on an element at once, so that you can add new event handlers without disturbing the existing ones. I spent a while trying to understand the c-based implementation of event handlers, and figuring out all of the things I'd have to do to change it to a chain based implementation, but then realized that given the ruby nature of Shoes, this was a prime opportunity for monkeypatching.

Monkeypatching may be a little less performant, but this is for a debugger-style thing anyway (I'm thinking I want to get to something a lot like Firebug is for web development...), so who cares if its slow?

So in I dove. I'm calling the module ShoeShine, because its cleaning up my Shoes without changing the inside at all, and it turned out to be pretty simple to implement. First, I define a few methods to add, remove, and call handlers:

  def add_handler(type, &block)
    @_handlers ||= {}
    @_handlers[type] ||= []
    @_handlers[type].push block
    # set up callbacks from old handlers invocation into
    # new handler chain
    self.send "#{type}_without_chains".to_sym do |*args|
      self.call_handlers(type, *args)
    end
  end

  def clear_handlers(type)
    if @_handlers
      if @_handlers.delete(type)
        _handler_deleted(type)
      end
    end
  end

  def call_handlers(type, *args)
    return unless @_handlers[type
    @_handlers[type].each do |block| 
      block.call(*args)
    end
  end

Then I do some metaprogramming and monkeypatching to plug this in between handler registration and handler callback:

  HANDLERS.each do |handler|
    eval %(
      def #{handler}_with_chains(&block)
        # need to clear handlers to maintain old semantics
        clear_handlers(:#{handler})
        add_handler(:#{handler}, &block)
      end
    )
  end

  def self.included(klass)
    klass.instance_eval do
      ShoeShine::HANDLERS.each do |handler|
        if klass.instance_methods.include? handler.to_s
          # Replace old handlers with new handlers
          alias_method "#{handler}_without_chains".to_sym, handler       
          alias_method handler, "#{handler}_with_chains".to_sym
        end
      end
    end
  end

Finally, I include this module into all interesting Shoes classes:

Shoes.constants.each do |c|
  k = Shoes.const_get(c)
  if k.is_a? Class
    k.send :include, ShoeShine
  end
end

Voila! Suddenly, as well as having the traditional Shoes interface to handlers, you can also add additional handlers onto existing objects without disturbing the originals. This will open the door to all sorts of interesting things. I've already mentioned my intent to use this for debugging, but I'm also exploring using this as a way to get around the current Shoes limitations on click, hover, leave, etc handlers of only working on slots, allowing them to trickle down to paragraphs, widgets, etc.

I've got some early versions of this working, but its definitely not ready for prime time yet. For any who are interested, you can find both the ShoeShine code and my early trickle down work at github in the shoes-preview tree.

Can I just say how much I love method_missing?

I wanted to follow up on my previous explorations with ShoeQuery to add some helper methods to access pieces of shoes functionality... thinking I'd have to add pieces one by one. However, it turns out that due to the magic of method_missing, I was able to add access to basically all of them with a single 22-line method. It looks like:

  def method_missing(method, *args, &block)
    if args.empty? && !block_given?
      # if you have no arguments and no block, think of it as a query
      # and return the answer for the first element.
      if first.respond_to? method
        return first.send(method)
      else
        return elem.style(method)
      end
    elsif block_given?
      each {|elem| elem.send(method, *args, &block)}
    else
      # Otherwise, think of it as a setter
      setter = "#{method}=".to_sym
      if first.respond_to? setter
        each {|elem| elem.send(setter, *args, &block)}
      else
        each {|elem| elem.style(method => args[0], &block)}
      end
    end
    self
  end

And now, thanks to this I can write lines in shoes as follows:

# find all paragraphs, change their text to 'all ps', and color 
# them red
shoe_query('para').text("all ps").stroke(red)

# replace the click handlers on all buttons
shoe_query('button').click do
  para "All your button are belong to us now
end

Introducing ShoeQuery

ShoeQuery (better name needed) was born out of a desire to be able to find and manipulate elements in Shoes without having stashed away their locations ahead of time. It was inspired by jQuery's mechanisms for finding and manipulating DOM objects in Javascript.

The fundamental object (a ShoeQuery object) is pretty much an array of Shoes Elements, augmented with some extra functions. ShoeQuery objects are created via a helper (unsurprisingly named shoe_query) that lives in the Shoes::App object, and so should be accessible anywhere you want it. You can pass into that helper a selector, and it will return a ShoeQuery object containing the elements that match. You can also pass in an array of elements you already have handy, or an existing element, and it will also return a ShoeQuery object containing those elements. This form should be familiar for those who've used jQuery.

Right now, the things it lets you do are pretty limited, though a lot of the functionality jQuery specially provides already exists in Ruby. The one big feature is the ability to find elements by type, including css-like hierarchy. You can do this either by invoking shoe_query directly, or calling find on an existing shoe_query object (to search within its children). For example, given the demo app:

load 'shoe_query.rb'
Shoes.app do
  app = Shoes.app do
  para "foo"
  f = flow do
    stack do
      para "bar"
      para 'blah'
    end
  end
  b = button "click me to change all paragraphs"
  b.click do
    shoe_query('para').each do |p|
      p.text = "all ps"
    end
  end
  b2 = button "click me to just change the paragraphs inside the flow"
  b2.click do
    shoe_query('flow').find('para').each do |p|
      p.text = "flow ps"
    end
  end
  b3 = button "click me to change the button inside the stack"
  b3.click do
    shoe_query('stack para').each do |p|
      p.text = "stack p"
    end
  end

end

In this case, the first button modifies all three paragraph elements, while the latter two use two different mechanisms to scope their changes to within the flow and within the stack respectively

As I play with this more, I'll be extending it to include more functionality. Let me know if there's anything you want!

You can get ShoeQuery at github at http://github.com/kball/shoequery/tree/master

Shoes Class Hierarchy Redux: Introspection

After writing up the Shoes class hierarchy, someone pointed out that this is already included in the Shoes manual (and available online here. I took a look, and found some disagreements between that class hierarchy and the one I had come to. In particular, large numbers of classes are listed a being subclasses of Shoes::Basic, including some like EditBox and EditLine that are very definitely subclasses of Shoes::Native.

My first thought was that the manual must be just out of date, but upon looking into the source for the manual (located in the shoes source at lib/shoes/help.rb, as well as static/manual.txt), I discovered that the class hierarchy was actually being generated dynamically by introspecting the Shoes classes in the following method:

  def class_tree
    tree = {}
    Shoes.constants.each do |c|
      k = Shoes.const_get(c)
      next unless k.respond_to? :superclass

      c = "Shoes::#{c}"
      if k.superclass == Object
        tree[c] ||= []
      else
        k.ancestors[1..-1].each do |sk|
          break if [Object, Kernel].include? sk
          (tree[sk.name] ||= []) << c
          c = sk.name
        end
      end
    end
    tree
  end

This code takes the constants defined in Shoes, looks for those that have superclasses, and pulls out their ancestors up until Object. Why, then, if the manual is being generated dynamically, does it miss the fact that Shoes::EditLine, Shoes::EditBox, and a host of others inherit from Shoes::Native?

To debug this, I took a look at the ancestors tree of Shoes::EditBox using the shoes preview app I wrote about here, entering

Shoes.debug Shoes::EditBox.ancestors.inspect

and clicking 'Run Without App'. In the error message text box, the class hierarchy of Shoes::EditBox appeared:

[Shoes::EditBox, Shoes::Basic, Shoes::Native, Object, FileUtils, FileUtils::StreamUtils_, Kernel]

Shoes::Basic is definitely the first ancestor... but in the declaration of the EditBox class, Shoes::Native was clearly the ancestor:

(From shoes/ruby.c line 4965)

cEditBox  = rb_define_class_under(cShoes, "EditBox", cNative)

What's going on? The moment of inspiration came when I looked at the definition of Shoes::Basic in lib/shoes.rb. Basic is defined as a module within the Shoes class, and then included on a large list of classes. What I'd forgotten was the way that Modules exist in the Ruby inheritance chain. When a module is included, it creates a pseudo-class known as an Iclass that sits directly above the including class in the inheritance chain, and points to the module's methods. Thus by including a module, you are changing around the ancestors of your class. A good reference can be found here, and a quick quiz to test your knowledge here.

Including modules in the display of the hierarchy might be desirable, but I'm not sure at all how to do it. By doing so, instead of having a nice tree form, suddenly we have a DAG,
which is a much more complex structure to display. Both the introspection code and display code currently expect a tree, so this is the source of the errors/lack of completeness in the current manual.

Its not clear how useful having the modules displayed in the hierarchy is, either. The Shoes::Basic module doesn't do very much, its much more useful to know which classes are descended from the Shoes::Native class.

So the simple fix, to change the manual to showing what you might expect, is to add a single line to the loop over ancestors, skipping modules.

     k.ancestors[1..-1].each do |sk|
       break if [Object, Kernel].include? sk
+      next unless sk.is_a? Class #don't show mixins
       (tree[sk.name] ||= []) << c
       c = sk.name
     end

Now the introspected class hierarcy looks exactly like the hierarchy I constructed by hand. I've pushed the change to my fork of shoes on Github. Dunno when/if they'll get merged.

If someone wants to do the more complex fix of changing the structure and display to be a DAG, go for it!

P.S.

If anyone knows a way to get syntax highlighting for ruby code snippets on blogger, please let me know! Thanks!

Exploring the Shoes class hierarchy

I've spent a lot of time this week attempting to get to know the Shoes ruby GUI kit better. Shoes is an open source project, and one that is pretty early in its evolution.

This has a couple of implications:

1. Its very raw. It doesn't take too much tinkering to find something thats broken.


2. It is incomplete. Shoes has a few fundamental abstractions that make basic layout straightforward, and generally makes simple things simple. However, if you try to do something that _why hasn't tried to do yet, it probably won't work.
   


3. You can open it up, learn how it works, and fix things!

I've spent most of my train/commute time (3 hours a day) this week digging into the source, fixing minor bugs, and trying to understand how it all works. While Shoes is a ruby toolkit, most of it is written in C. From what I've been able to determine, there are essentially three layers.

The uppermost layer is the set of Ruby classes and their methods, most defined in C and then exposed to Ruby. These live in lib/shoes.rb, the various files in lib/shoes, shoes/ruby.[ch], and shoes/canvas.[ch].

Below that there is a set of abstract windowing code. This defines the way that Shoes layouts occur, how the different objects relate to each other within the visible canvas, etc. This is located primarily in shoes/app.[ch] and shoes/world.[ch]. Also living in this middle layer are things like image handling, http handling, and special effects.

Finally, the bottom layer is the interface to the native graphics libraries... a gtk interface, a windows interface, and an interface to Cocoa for OS X. These are located in shoes/native.h and the shoes/native directory. These are typically called into from the middle layer, and issue event-related callbacks back up into the middle layer.

As I learn more about Shoes, I hope to understand and write about all of these pieces, but for now I'm focusing most on learning about the interface Shoes exposes to ruby. This allows me to better make sense of what works and what doesn't, and gain an intuition for how things should work in Shoes programs. Since a lot of the code is written in C, I've spent some time figuring out just the Ruby class hierarchy.

Here's the basic hierarchy... I haven't included anything about what methods are defined on what classes here, but I've put my notes at http://drop.io/shoesclasshierarchy if you want to look at them. I'll probably update them as I do more learning.

class Window; end
class Mouse; end
class Canvas; end
class Shoes < Canvas
  # Canvas-like things
  class App < ::Shoes; end
  class Flow < ::Shoes; end
  class Stack < ::Shoes; end 
  class Mask < ::Shoes; end
  class Widget < ::Shoes; end

  # Standalones with no children
  class Shape; end
  class Effect; end
  class Video; end

  # Patterns
  class Pattern; end
  class Background < Pattern; end
  class Border < Pattern; end

  # TextBlocks
  class TextBlock; end
  class Para < TextBlock; end
  class Banner < TextBlock; end
  class Title < TextBlock; end
  class Subtitle < TextBlock; end
  class Tagline < TextBlock; end
  class Caption < TextBlock; end
  class Inscription< TextBlock; end

  # Text
  class Text; end
  class Code < Text; end
  class Del < Text; end
  class Em < Text; end
  class Ins < Text; end
  class Span < Text; end
  class Strong < Text; end
  class Sup < Text; end
  class Sub < Text; end
  class Link < Text; end
  class LinkHover < Text; end

  # Natives
  class Native; end
  class Button < Native; end
  class EditLine < Native; end
  class EditBox < Native; end
  class ListBox < Native; end
  class Progress < Native; end
  class Check < Native; end
  class Radio < Native; end

  # Timing related stuff
  class TimerBase; end
  class Animation < TimerBase; end
  class Every < TimerBase; end
  class Timer < TimerBase; end

  class Color; end

  class Download
    class Response; end
  end
  
  # Errors
  class InvalidModeError < StandardError; end
  class NotImplementedError < StandardError; end
  class VideoError < StandardError; end
end

So at least now I know that if I've figured out something that works pretty well flows, it should work for stacks or the entire app as well. Similarly if I'm comfortable working with para, its likely similar techniques will work for banners, titles, subtitles, captions etc.

Its a work in progress; let me know if I got anything wrong, or if you want to know more about the methods available for each class.

Running with shoes

I've been playing off and on with Shoes for the last couple of weeks. Shoes is a simple ruby GUI toolkit designed for beginners. It seems like a really cool way to do the GUI aspects of simple desktop applications, designed to reduce the barriers to entry.

Instead of the intimidating scope of something like learning the GTK API, with Shoes you only need to learn a few core abstractions, and you're off and running with simple and beautiful code.

Want to add a button? Its as simple as

button "My Button"

To add a handler on that button?

b = button "My Button"
b.click do
  alert "You clicked my button!"
end

I'm pretty excited about playing with this. Unfortunately, however, the framework is very young and still pretty buggy. Documentation is reasonably good, but definitely incomplete. I'm trying to get a development environment together so I can debug some of the problems in the framework that I've run into, but so far have been unsuccessful at building a working version of Shoes; I get a mostly functioning version, but the text is showing up as boxes. I think I'm getting something wrong in the dependencies... I dunno. I've got a question in to the mailing list about this, so we'll see what happens.

In the meantime, I've built a little tool to make my turnaround time for testing how shoes code will look a bit faster. Shoes Preview lets you enter code & see what it will look like all without restarting Shoes. Its still pretty clunky, but already speeding me up from the modify/save/reopen cycle that I had been using. If you try it out, feel free to send me patches or suggestions! Happy hacking!

Camping

I started playing with Camping yesterday. Camping is ruby web microframework with the explicit goal of taking less than 4k of space. This implies a sparse, simple interface, and for those (like me) who are fitting their web apps into a 256M slice at slicehost, makes for much less memory usage than a rails app, or even a merb app. The idea is that for a simple application, rails is overkill. I'm thinking of it as a way to build json-accessible REST microapps.

I really like the simplicity of routes being defined with a regular expression right at the controller definition.

module Hello::Controllers
  class Index < R '/', '/snips'
    def get
      @snippets = Snippet.find :all
      render :index
    end
  end
end

That said, it took me more code than I would have liked to get a CRUD interface for a simple model. Not even looking at the views, here is the controller code I ended up with:

module Hello::Controllers
  class Index < R '/', '/snips'
    def get
      @snippets = Snippet.find :all
      render :index
    end
  end
  class New < R '/snips/new'
    def get
      @snippet = Snippet.new
      render :new
    end
    def post
      snip = Snippet.create(:title => input.title,
                            :body => input.body)
      redirect Show, snip.id
    end
  end
  class Show < R '/snips/(\d+)'
    def get(id)
      @snippet = Snippet.find(id)
      render :show
    end
  end
  class Edit < R '/snips/edit/(\d+)'
    def get(id)
      @snippet = Snippet.find(id)
      render :edit
    end
    def post(id)
      Snippet.find(id).update_attributes(:title => input.title,
                                          :body => input.body)
      redirect Show, id
    end
  end
end

I'd like to make this a little tighter, and make it a little more purely RESTful (especially for my idea of using these as REST-based microapps), but I'm not sure the right way to do that yet. Camping doesn't inherently have anything like the rails _method hack for the missing put/delete http verbs, and the only thing googling has been able to turn up is a plugin that hasn't been touched in over a year called Sleeping Bag. I'll take a look at that and see how it goes.