The Plex “App Store” which while it isn’t really a store, is still pretty slick. Tried several plug-ins, but will just hit the highlights here. Couldn’t get Hulu plugin to work at all – probably wrong credentials. The pandora plug-in did work but feels hackish as the display is a cut section from safari viewing pandora. I ended up watch “Living With Father (1947)” while testing the iMovie plug-in. I noticed one burp that might have been network streaming and one cut section that is probably in the source material. The video quality was not the best, but most was probably due to the source. The National Geographic plug-in was a little disappointing in that the info boxes disappear before I could completely read them. Also not a lot of photos. I installed several others but haven’t given them a try yet.

I did put new batteries in my Gyration keyboard and that works well with the exception that the media keys don’t do anything and I haven’t found how to configure them yet.

Watched an iso movie from the NAS and the player was perfect. OK, it was a drama (Fat Man & Little Boy) so was not a stress test.

Still trying to move the media content from the old XBMC box to the NAS. I bumped a switch which lost power and interrupted the first copy. I then tried using cp -ur but that started copying the same files over. Ended up writing a ruby script to finish the sync with and that is still in process.

So the network reconfiguration is postponed until next weekend. But I did find that MacPorts has dnsmasq. So installed XCode so I could install MacPorts. I’ve installed dnsmasq but have not yet configured or started it.

Did some research on skins as I really miss being able to view the content by genre. There are a few skins that I need to give a try (Aeon, Alaska), but the really good news is there is a confluence skin in the works.

What’s really sweet is that after a day and a half, I’m feeling comfortable leaving the new system for the family to use for the week.

Have a great week!

With a DVD library of over 1200 titles, I’ve been on a multi-year quest for the perfect media center system. Today I took the next major step on that journey.

Here’s a brief recap of my journey to date. For several years I’d toyed with the idea of a media center, then the release of LinuxMCE caused me to take the plunge. Built up a system. Hit a few hardware issues, rebuilt it with a new motherboard. Distributed the media on three systems via NFS. But was never really happy with LinuxMCE, probably because I just wanted a media center and not a home control center. Just too complex and persistent player problems drove me to try XBMC. I’ll admit it, I just loved XBMC, it’s way cool. I wanted a plug in replacement for LinuxMCE which meant the system would be a router with firewall and DHCP/DNS server for my media LAN, so went with a gentoo system, which, IMO, is easier to setup and configure than a ubuntu system. But with no time to maintain the gentoo hosted system, and my growing comfort with Macbook laptops, I just decided to jump ship to a Mac Mini running Plex (a fork of XBMC for the mac).

So day one.

Purchased a 2.4GHz, 4GB, 2010 mac mini (with HDMI port). Also went by Radio Shack to get a HDMI cable and was surprised at how proud they are of their cables – ouch.

Initial setup using one of my LG246 HDMI monitors instead of the plasma TV. Mini boots right up, real simple snow leopard config. First issue is the display needed some underscan adjusting under Settings, Displays. Then update the system to the latest and it’s time to have fun.

Download Plex and install. Install the recommended Candelair driver for Apple Remote support. Candelair recommended/advertised Remote Buddy which looked pretty cool so installed the 30 day trial.

Also installed iTerm and Firefox.

My current network has four media sources: a kubuntu workstation with 3TB for media, the old XBMC system with 1TB, a gentoo workstation with 5TB, and a new Thecus N4100Pro NAS Raid5 with 4 2TB drives. I’ve moved my m4v files to the Thecus and started the copying of the iso files from the XBMC system to the Thecus (currently still in progress) in prep for decommissioning the XBMC system. I intend to eventually transcode all the iso images to m4v and have the Thecus as the primary media server.

So for now, just mounted the NFS volumes from the kubuntu and Thecus systems on the mac mini using Disk Utility. Here’s the first difference between XBMC linux and Plex mac – on the XBMC when adding the source, I normally just specify the path to the autofs mount point. But on the mac Plex, I needed to just select the mounted volume as a source. The first gotcha is that the source must be mounted. Currently I’m doing that with Finder, but need to find the mac way of making sure it is mounted on boot.

I had a little frustration with Plex trying to use the default keyboard only. Eventually enabled the mouse primarily so I could right click and remove a bad source definition. For now, I’m planning on leaving the mouse enabled.

Activated the Remote Buddy and configured it to start at login. Set the AJAX password. Then brought up a safari on my iphone and pointed it at the mac mini. Very nice. It did take me a while to figure out that the “Menu” button is functionally equivalent to the ESC key. One cool hint, use the ‘+’ button in safari to add a link to your home screen.

Went to Plex’s app store and added a few neat sounding extensions. Primarily the HDHomeRun app, which is pretty cool. It does take a while to verify each TV channel, so did just a few for now.

Moved the system to the home theater and connected the HDMI to my Yamaha RX-V861 A/V receiver. The video was perfect but no sound. Changed Plex’s system audio preference from built-in output to HDMI and that was solved.

Also hooked up Gyration air mouse to the mac mini.

Finally added Plex to the login.

So ended the first day with a functional mac mini running Plex by default which accesses media from two different NAS systems controlled by my iPhone, iPad and airmouse, sending video and audio over HDMI to my home theater.

TODO list includes reconfiguring my media LAN to remove the XBMC machine. This will necessitate setting up and running a replacement DHCP/DNS server (hopefully either dnsmasq or something equivalent). Also will need to change the mode of my wireless access point to be a gateway to my full home LAN.

Example code available at: http://github.com/royw/creating-gem-example

 ∴ git clone git://github.com/royw/creating-gem-example.git

First, I really like Ruby Version Manager so I highly recommend installing it and the versions of ruby you prefer. Keeping it simple for this article:

 ∴ rvm install 1.8.7
 ∴ rvm 1.8.7

Second, install the prerequisites Jeweler and Bundler :

 ∴ gem install jeweler
 ∴ gem install bundler

Now it’s time to code generate the project. Let’s see what jeweler can do for us:

 ∴ jeweler --help
Usage: jeweler [options] reponame
e.g. jeweler the-perfect-gem
        --directory [DIRECTORY]      specify the directory to generate into

        --rspec                      generate rspec code examples
        --shoulda                    generate shoulda tests
        --testunit                   generate test/unit tests
        --bacon                      generate bacon specifications
        --testspec                   generate test/spec tests
        --minitest                   generate minitest tests
        --micronaut                  generate micronaut examples
        --riot                       generate riot tests

        --cucumber                   generate cucumber stories in addition to the other tests

        --reek                       generate rake task for reek
        --roodi                      generate rake task for roodi

        --[no-]gemcutter             setup project for gemcutter
        --rubyforge                  setup project for rubyforge
        --summary [SUMMARY]          specify the summary of the project
        --description [DESCRIPTION]  specify a description of the project

        --user-name [USER_NAME]      the user's name, ie that is credited in the LICENSE
        --user-email [USER_EMAIL]    the user's email, ie that is credited in the Gem specification

        --github-username [GITHUB_USERNAME]
                                     name of the user on GitHub to set the project up under
        --github-token [GITHUB_TOKEN]
                                     GitHub token to use for interacting with the GitHub API
        --git-remote [GIT_REMOTE]    URI to set the git origin remote to
        --homepage [HOMEPAGE]        the homepage for your project (defaults to the GitHub repo)
        --create-repo                create the repository on GitHub

        --yard                       use yard for documentation
        --rdoc                       use rdoc for documentation
    -h, --help                       display this help and exit

Wow! Pretty impressive. As you can see, jeweler let’s you define your project your way. For this article:

∴ jeweler --rspec --no-gemcutter --summary "example gem for blog article" --description "This is an example gem for my blog article showing how to create a gem" --rdoc --user-name "Roy Wright" --user-email "roy@wright.org" example
        create  .gitignore
        create  Rakefile
        create  LICENSE
        create  README.rdoc
        create  .document
        create  lib
        create  lib/example.rb
        create  spec
        create  spec/spec_helper.rb
        create  spec/example_spec.rb
        create  spec/spec.opts
Jeweler has prepared your gem in example
∴ cd example
∴ ls
LICENSE         README.rdoc     Rakefile        lib             spec
 ∴ rake -T
(in /Users/royw/views/example)
rake build                           # Build gem
rake check_dependencies              # Check that runtime and development dependencies are installed
rake check_dependencies:development  # Check that development dependencies are installed
rake check_dependencies:runtime      # Check that runtime dependencies are installed
rake clobber_rcov                    # Remove rcov products for rcov
rake clobber_rdoc                    # Remove rdoc products
rake gemspec                         # Generate and validates gemspec
rake gemspec:debug                   # Display the gemspec for debugging purposes
rake gemspec:generate                # Generates the gemspec, using version from VERSION
rake gemspec:validate                # Validates the gemspec
rake git:release                     # Tag a release in Git
rake github:release                  # Release Gem to GitHub
rake install                         # Install gem using sudo
rake rcov                            # Run specs using RCov
rake rdoc                            # Build the rdoc HTML Files
rake release                         # Release gem
rake rerdoc                          # Force a rebuild of the RDOC files
rake spec                            # Run specs
rake version                         # Displays the current version
rake version:bump:major              # Bump the gemspec by a major version.
rake version:bump:minor              # Bump the gemspec by a minor version.
rake version:bump:patch              # Bump the gemspec by a patch version.
rake version:write                   # Writes out an explicit version.

As you can see, most of the infrastructure is ready to use. Ah, you caught the “most” did you? OK, let’s complete it.

First, let’s get a VERSION file installed:

 ∴ rake version:write
(in /Users/royw/views/example)
Updated version: 0.0.0
 ∴ ln -s ../VERSION VERSION

This puts a copy of the version in our gem so we can have a –version feature.

If your gem is going to furnish an executable, then do something like:

 ∴ mkdir bin
 ∴ mate bin/example # add the code below, changing "example" to your project name
 ∴ chmod +x bin/example
 ∴ cat bin/example

#!/usr/bin/env ruby

require File.expand_path(File.dirname(__FILE__) + "/../lib/example")
exit CLI.execute

Now let’s switch over to the lib directory and do a little infrastructure improvements.

 ∴ mkdir lib/example

And this directory is where our code goes. To access our code we modify the lib/example.rb to handle our requires:

 ∴ cat lib/example.rb

# not the best solution, probably should use require_relative in ruby 1.9 or the backported
# version from Programming Ruby v3 for ruby 1.8.
$:.unshift(File.dirname(__FILE__)) unless
  $:.include?(File.dirname(__FILE__)) || $:.include?(File.expand_path(File.dirname(__FILE__)))

# a few of the gems I use in most projects
require 'rubygems'
require 'extlib'
require 'versionomy'
require 'log4r'
require 'configliere'
require 'singleton'
require 'pp'

# require all of the library files
# note, you may need to specify these explicitly if there are any load order dependencies
Dir['example/**/*.rb'].each do |name|
  require name
end

And we will toss the start of a command line interface object (with support for command line arguments, environment variables, config files, and logging) into lib/example/cli.rb:

 ∴ cat lib/example/cli.rb

# Command Line Interface to the Example application
#
# Usage:
#
#  exit CLI.execute
#
# Note, this is a Singleton class, only use CLI.execute to access this class.
#
class CLI
  include Singleton

  # The application can monitor this attribute to know if an interrupt (^c) has been signalled.
  attr_reader :interrupt_signalled

  APP_NAME = 'example'

  # The main entry point for the CLI
  # returns the exit code
  def self.execute
    instance.execute
  end

  # protected by the Singleton.  I.e., do not attempt CLI.new
  def initialize
    @interrupt_signalled = false
  end

  # execute the app from the CLI
  # returns the exit code
  def execute
    exit_code = 0
    # ^c handler
    Signal.trap("INT") { @interrupt_signalled = true }
    begin
      setup_settings
      Settings[:app_name] = APP_NAME
      if Settings[:version]
        puts Example.version.to_s
      else
        Settings[:logger] = init_logger
        app = Example.new
        app.execute
      end
    rescue SystemExit
      # exit() calls from within the block actually raise SystemExit
    rescue Exception => e
      Settings[:logger].error e.to_s if Settings[:logger]
      exit_code = 1
    end
    exit_code
  end

  # log4r setup
  def init_logger
    # Initial setup of logger
    logger = Log4r::Logger.new(APP_NAME)
    level_map = {'DEBUG' => Log4r::DEBUG, 'INFO' => Log4r::INFO, 'WARN' => Log4r::WARN}

    # console messages
    logger.outputters = Log4r::StdoutOutputter.new(:console)
    Log4r::Outputter[:console].formatter  = Log4r::PatternFormatter.new(:pattern => "%m")
    Log4r::Outputter[:console].level = Log4r::INFO
    Log4r::Outputter[:console].level = Log4r::WARN if Settings[:quiet]
    Log4r::Outputter[:console].level = Log4r::DEBUG if Settings[:debug]
    Log4r::Outputter[:console].formatter = Log4r::PatternFormatter.new(:pattern => "%m")
    logger.trace = true if Settings[:trace]

    if Settings[:logfile]
      logfile_outputter = Log4r::RollingFileOutputter.new(:logfile, :filename => Settings[:logfile], :maxsize => 1000000 )
      logger.add logfile_outputter
      Settings[:loglevel] ||= 'INFO'
      Log4r::Outputter[:logfile].formatter = Log4r::PatternFormatter.new(:pattern => "[%l] %d :: %M")
      Log4r::Outputter[:logfile].level = level_map[Settings[:loglevel].upcase] || Log4r::INFO
    end
    logger
  end

  # Configliere setup
  def setup_settings
    Settings.use :all

    Settings.define :logfile, :type => String, :description => 'Log filename, default is: media2nfo.log', :required => false
    Settings.define :loglevel, :type => String, :description => 'Logfile logging level, must be one of: DEBUG, INFO, WARN, ERROR', :required => false
    Settings.define :debug, :type => :boolean, :description => 'Log debug messages to console', :required => false
    Settings.define :quiet, :type => :boolean, :description => 'Only log errors to console', :required => false
    Settings.define :trace, :type => :boolean, :description => 'Trace logging', :required => false
    Settings.define :version, :type => :boolean, :description => 'Application Version', :required => false

    Settings({
              :logfile => "#{APP_NAME}.log",
              :loglevel => 'debug'
             })

    Settings.read("#{APP_NAME}.yaml") if File.exist?(File.expand_path("~/.configliere/#{APP_NAME}.yaml"))
    Settings.resolve!
  end
end

And a really basic application:

 ∴ cat lib/example/example.rb

class Example
  def self.version
    ver = Versionomy.parse(IO.read(File.expand_path(File.dirname(__FILE__) + "/../VERSION")).strip)
    ver
  end
  def execute
    Settings[:logger].debug "Example.execute"
  end
end

And to be good, an rspec:

 ∴ cat spec/example_spec.rb

require File.expand_path(File.dirname(__FILE__) + '/spec_helper')

describe "Example" do
  it "should return a Versionomy object as the version" do
    ver = Example.version
    ver.instance_of?(Versionomy::Value).should be_true
  end
  it "should return the version stored in the VERSION file" do
    v1 = Example.version.to_s.strip
    v2 = IO.read(File.expand_path(File.dirname(__FILE__) + "/../VERSION")).strip
    v1.should == v2
  end
end

And the last change to the infrastructure is to bundle the gems for two reasons, first to ease deployment, but second, more importantly to resolve dependencies of all gems together instead of individually (which can lead to dependency problems). So here’s a real simple Gemfile to get started with:

 ∴ cat Gemfile

source :gemcutter

gem 'configliere',  '>=0.0.9'
gem 'extlib',       '>=0.9.15'
gem 'versionomy',   '>=0.4.0'
gem 'log4r',        '>=1.1.7'

gem 'jeweler',      '>=1.4.0'
gem 'rake',         '>=0.8.7'
gem 'rspec',        '>=1.3.0',  :require => 'spec'

Then to install the gems:

 ∴ bundler install

Try it

 ∴ ls *.log
ls: *.log: No such file or directory
 ∴ bin/example
 ∴ ls *.log
example000001.log
 ∴ cat example000001.log
[DEBUG] 2010-06-07 01:41:12 :: Example.execute
 ∴ bin/example --version
0.1.0
 ∴ bin/example --help
usage: example command [...--param=val...]

Params:
  --debug:                    Log debug messages to console
  --logfile:                  Log filename, default is: media2nfo.log
  --loglevel:                 Logfile logging level, must be one of: DEBUG, INFO, WARN, ERROR
  --quiet:                    Only log errors to console
  --trace:                    Trace logging
  --version:                  Application Version

Environment Variables can be used to set:

Commands:

Workflow

And now the work flow for building and publishing our gem:

∴ rake spec
(in /Users/royw/views/example)
/Users/royw/.rvm/gems/ruby-1.8.7-p174/gems/jeweler-1.4.0/lib/jeweler/commands/check_dependencies.rb:13:Warning: Gem::Dependency#version_requirements is deprecated and will be removed on or after August 2010.  Use #requirement
All dependencies seem to be installed.
..

Finished in 0.004048 seconds

2 examples, 0 failures
 ∴ rake version:bump:patch
(in /Users/royw/views/example)
Current version: 0.0.0
Updated version: 0.0.1
 ∴ echo "*.log" >> .gitignore
 ∴ git status
# On branch master
# Changed but not updated:
#   (use "git add ..." to update what will be committed)
#
#       modified:   .gitignore
#       modified:   lib/example.rb
#       modified:   spec/example_spec.rb
#
# Untracked files:
#   (use "git add ..." to include in what will be committed)
#
#       Gemfile
#       bin/
#       example.gemspec
#       lib/VERSION
#       lib/example/
no changes added to commit (use "git add" and/or "git commit -a")
 ∴ git add Gemfile bin example.gemspec lib .gitignore spec
 ∴ git status
# On branch master
# Changes to be committed:
#   (use "git reset HEAD ..." to unstage)
#
#       modified:   .gitignore
#       new file:   Gemfile
#       new file:   bin/example
#       new file:   example.gemspec
#       new file:   lib/VERSION
#       modified:   lib/example.rb
#       new file:   lib/example/cli.rb
#       new file:   lib/example/example.rb
#       modified:   spec/example_spec.rb
#
 ∴ git commit -m "initial coding"
Created commit 3aa37db: initial coding
 9 files changed, 208 insertions(+), 2 deletions(-)
 create mode 100644 Gemfile
 create mode 100755 bin/example
 create mode 100644 example.gemspec
 create mode 120000 lib/VERSION
 create mode 100644 lib/example/cli.rb
 create mode 100644 lib/example/example.rb
 ∴ rake build
(in /Users/royw/views/example)
Generated: example.gemspec
example.gemspec is valid.
WARNING:  no rubyforge_project specified
  Successfully built RubyGem
  Name: example
  Version: 0.0.1
  File: example-0.0.1.gem

And optionally if you are using github as your master and/or releasing to gemcutter:

 ∴ rake release

Conclusion

While we have just touched the surface of these powerful tools, hopefully I have shown that creating your project as a gem is pretty easy. Just a few minor infrastructure enhancements and you’ve run out of excuses not to gemify your next project.

Have fun,
Roy

So here are my basic requirements:

• Support using multiple DRb objects
• Support local and remote hosting of the DRb server(s)
• Application to install current code on host machine(s)
• No application code to have to be manually installed on host machine(s)
• Minimal software requirements on host machine(s)
• Application to control the DRb server(s) (start/stop)
• Linux and Mac support (don’t care about windows)

The result is Drbman.

Installing Drbman

sudo gem install royw-drbman –source http://gems.github.com

Using Drbman

Let’s start by coding up the primes example (drbman/examples/primes).

I was wanting something simple for an example so thought the Sieve of Eratosthenes would make a good one. It turns out it’s not really a good candidate for parallelizing via DRb, but is still a good example.

After coding up the sieve and using the ruby-prof profiler, the most time consuming chunk was the code that calculates the multiples of a prime which was taking 42% of the time, so I decided to parallelize it. Moving it to it’s own class yields:

class PrimeHelper
  # Find the multiples of the give prime number that are less than the
  # given maximum.
  # @example
  #  multiples_of(5,20) => [10, 15]
  # @param [Integer] prime the prime number to find the multiples of
  # @param [Integer] maximum the maximum integer
  # @return [Array] the array of the prime multiples
  def multiples_of(prime, maximum)
    a = []
    2.upto((maximum - 1) / prime) { |i| a << (i * prime) }
    a
  end
end

After getting the PrimeHelper working in the original code, it’s time to make it into a DRb server:

require 'drbman_server'

# A helper object for calculating primes using the Sieve of Eratosthenes
#
# == Usage
# ruby prime_helper.rb foo.example.com 1234
# will run the service as: druby://foo.example.com:1234
#
# ruby prime_helper.rb foo.example.com
# will run the service as: druby://foo.example.com:9000
#
# ruby prime_helper.rb
# will run the service as: druby://localhost:9000
#
class PrimeHelper
  include DrbmanServer

  # Find the multiples of the give prime number that are less than the
  # given maximum.
  # @example
  #  multiples_of(5,20) => [10, 15]
  # @param [Integer] prime the prime number to find the multiples of
  # @param [Integer] maximum the maximum integer
  # @return [Array<Integer>] the array of the prime multiples
  def multiples_of(prime, maximum)
    a = []
    2.upto((maximum - 1) / prime) { |i| a << (i * prime) }
    a
  end
end

DrbmanServer.start_service(PrimeHelper)

As you can see, not much to it. Add a require ‘drbman_server’, then a include DrbmanServer in your class, and finally start the service for your class with a DrbmanServer.start_service(YourClassName). One thing to note here is when we run the helper, it will not have a console connection (no stdin/stdout/stderr) so we would need to include some form of logger that supports alternate output such as syslog, file, email, etc. (personally I’m partial to the log4r gem).

Ok, now let’s see how we hook into the client application. Here’s the sieve in our example:

# The Sieve of Eratosthenes prime number finder
# Note, uses the Command design pattern
class SieveOfEratosthenes
  attr_reader :primes_elapse_time

  # Use the Sieve of Eratosthenes to find prime numbers
  #
  # @param [Integer] maximum find all primes lower than this maximum value - REQUIRED.
  # @option choices [Hash<String,String>] :dirs hash of local directories to copy to the host
  #   machines where key is local source and value is directory on host machine - REQUIRED.
  # @option choices [String] :run the name of the file to run on the host machine - REQUIRED.
  #  This file should start the drb server.  Note, this file will be daemonized before running.
  # @option choices [Array<String>] :hosts (['localhost']) array of host machine descriptions "{user{:password}@}machine{:port}".
  # @option choices [Integer] :port (9000) default port number used to assign to hosts without a port number,
  #  the port number is incremented for each host.
  # @option choices [Array<String>] :gems array of gem names to verify are installed on the host machine.
  #  Note, 'daemons' is always added to this array.
  # @param [Logger] logger the logger to use
  def initialize(maximum, choices, logger)
    @maximum = maximum.to_i
    @choices = choices
    @logger = logger

    # we need at least one host that has a drb server running
    @choices[:hosts] = ['localhost'] if @choices[:hosts].blank?

    # specify the directories to copy to the host machine
    @choices[:dirs] = {File.join(File.dirname(__FILE__), '../drb_server') => 'drb_server'}

    # set the file to be ran that contains the drb server
    @choices[:run] = 'drb_server/prime_helper.rb' if @choices[:run].blank?

    # specify gems required by the drb server object
    # each host will be checked to make sure these gems are installed
    @choices[:gems] = ['log4r']

  end

  # Calculate the primes
  # @return [Array<Integer] the primes in an Array
  def execute
    result = []
    @logger.debug { @choices.pretty_inspect }

    Drbman.new(@logger, @choices) do |drbman|
      @primes_elapse_time = elapse do
        result = primes(@maximum, drbman)
      end
    end
    result
  end

  private

  # recursive prime calculation
  # @param maximum (see #initialize)
  # @param [Drbman] drbman the drb manager instance
  # @return [Array<Integer>] the array of primes
  def primes(maximum, drbman)
    indices = []
    if maximum > 2
      composites = calc_composites(maximum, drbman)
      flat_comps = composites.flatten.uniq
      indices = calc_indices(flat_comps, maximum)
    end
    indices
  end

  # find the composites array
  # @param maximum (see #initialize)
  # @param drbman (see #primes)
  # @return [Array<Integer>] the composites array
  def calc_composites(maximum, drbman)
    # when n = 20
    # sqr_primes = [2,3]
    # composites = [[2*2, 2*3, 2*4,...,2*9], [3*2, 3*3, 3*4,...,3*6]]
    sqr_primes = primes(Math.sqrt(maximum).to_i, drbman)
    composites = []
    threads = []
    mutex = Mutex.new
    sqr_primes.each do |ip|
      # parallelize via threads
      # then use the drb object within the thread
      threads << Thread.new(ip, maximum) do |prime, max|
        drbman.get_object do |prime_helper|
          prime_multiples = prime_helper.multiples_of(prime, max)
          mutex.synchronize do
            composites << prime_multiples
          end
        end
      end
    end
    threads.each {|thrd| thrd.join}
    composites
  end

  # sift the indices to find the primes
  # @param [Array<Integer>] flat_comps the flattened composites array
  # @param maximum (see #initialize)
  def calc_indices(flat_comps, maximum)
    indices = []
    flags = Array.new(maximum, true)
    flat_comps.each {|i| flags[i] = false}
    flags.each_index {|i| indices << i if flags[i] }
    indices.shift(2)
    indices
  end

end

Let’s start with the @choices. The example application uses UserChoices in cli.rb to handle command line, environment, and/or config file parameters. UserChoices are close to a hash in that they have [] and []= hash like methods. If you don’t use UserChoices, then just pass the parameters in a regular Hash.

Back to the initializer, we first make sure at least one host is define in @choices[:hosts]. Then we set the @choices[:dirs] key to the local directory that contains the files we want copied to each host machine and the value to the relative directory on the host machine we want the files copied into. Next we point to our DRb server file with @choices[:run]. Finally we specify any gems we want to make sure are installed on each host with the @choices[:gems] choice.

    # we need at least one host that has a drb server running
    @choices[:hosts] = ['localhost'] if @choices[:hosts].blank?

    # specify the directories to copy to the host machine
    @choices[:dirs] = {File.join(File.dirname(__FILE__), '../drb_server') => 'drb_server'}

    # set the file to be ran that contains the drb server
    @choices[:run] = 'drb_server/prime_helper.rb' if @choices[:run].blank?

    # specify gems required by the drb server object
    # each host will be checked to make sure these gems are installed
    @choices[:gems] = ['log4r']

Next, take a look at the execute method. Here’s where we run Drbman. Note that we pass a Drbman instance into the primes method.

    Drbman.new(@logger, @choices) do |drbman|
      @primes_elapse_time = elapse do
        result = primes(@maximum, drbman)
      end
    end

Finally skip down to the calc_composities method and inside the threading block:

    sqr_primes.each do |ip|
      # parallelize via threads
      # then use the drb object within the thread
      threads << Thread.new(ip, maximum) do |prime, max|
        drbman.get_object do |prime_helper|
          prime_multiples = prime_helper.multiples_of(prime, max)
          mutex.synchronize do
            composites << prime_multiples
          end
        end
      end
    end
    threads.each {|thrd| thrd.join}

The drbman.get_object do |prime_helper| returns a DrbObject that references one of PrimeHelper instances on one of our host machines. Yep, we don’t know which nor really care. The threading handles the parallel execution.

And that pretty much covers how to use Drbman.

Oh, why is this example of the Sieve of Eratosthenes not a good fit for DRb parallelization? It turns out that the return value from PrimeHelper.multiples_of is an Array that can grow quite large. This has three problems. Memory usage can get quite large, the network bandwidth used in returning the marshaled array, and that DRb appears to try to place the entire marshaled return object into a single packet. While calculating primes of 10,000,000 finds 664,579 primes, trying with 20,000,000 generates the error “too large packet 51578437″.

So in choosing what to parallelize, it’s probably best to find parts with reasonable sized inputs and outputs.

Running the Primes Example

Using 3 machines:

• royw-macbook 2GHz Intel Core 2 Duo, 2GB
• royw-gentoo 2.5GHz Intel Q9300, 4GB
• dad-kubuntu 2.4GHz Intel Q6600, 4GB

royw-macbook:primes royw$ bin/primes 10000000 -H 'royw-gentoo,royw-gentoo,royw-gentoo,royw-gentoo,dad-kubuntu,dad-kubuntu,dad-kubuntu,dad-kubuntu'
664579 primes found
calculation elapsed time: 49.745109
total elapsed time: 62.603881
royw-macbook:primes royw$ bin/primes 10000000 -H 'royw-gentoo,royw-gentoo,royw-gentoo,royw-gentoo'
664579 primes found
calculation elapsed time: 55.525954
total elapsed time: 58.73239
royw-macbook:primes royw$ bin/primes 10000000 -H 'royw-gentoo'
664579 primes found
calculation elapsed time: 107.114372
total elapsed time: 109.655318
royw-macbook:primes royw$ bin/primes 10000000 -H 'localhost'
664579 primes found
calculation elapsed time: 102.872163
total elapsed time: 107.949127
royw-macbook:primes royw$ bin/primes 10000000 -H 'localhost,localhost'
664579 primes found
calculation elapsed time: 68.445687
total elapsed time: 71.763174

The last two runs should raise an eyebrow or two. The wire cost is pretty high.

Internals

There’s a little bit of the internals to Drbman that you really ought to be aware of.

First, Drbman uses net-ssh to connect to each host machine. It also uses net-scp to copy files to the host machines. This means you must be able to directly log into the host machine using ssh from your client machine (currently gateway sessions are not supported). I’ve tested in two scenarios: passwordless certificates and normal password logins.

Next, Drbman will create a directory, ~/.drbman, on each host and leave it. Then each time your client uses the host, it will create a ~/.drbman/{uuid} directory where it will upload your choices[:dirs] directories. It will also create a controller file in this directory, “#{File.basename(@choices[:run],’.*’)}_controller.rb”, that is used to daemonize and control your DRb server. This controller takes the normal daemon commands (start, stop, status) as well as command line parameters it should pass to your DRb server (after a double hyphen ‘–’). The command line for our prime example is:

~/.drbman/{uuid} # ruby foo_controller start — example.com 1234

So basic setup flow is:

1. open ssh connection to host machine
2. create ~/.drbman/{uuid}
3. scp @choices[:dirs] to ~/.drbman/{uuid}
4. create the controller in ~/.drbman/{uuid}
5. daemonize the @choices[:run] file by starting the controller

The cleanup flow is:

1. stop the DRb server on the host machine using DRbObject.stop_service
2. stop the daemon by issuing a controller stop command
3. delete the project’s directory (~/.drbman/{uuid}).

Note, if you set @config[:leave] to true, then the project’s directory will not be deleted. This can be useful for debugging. I’ve also found this little script useful to gracefully stop any running daemons and delete all the project files in ~/.drbman:

#!/usr/bin/env ruby

# place this script in the ~/.drbman directory
# then use it to stop all daemons and delete
# all projects

Dir.glob("**/*_controller.rb").each do |controller|
  `ruby #{controller} stop`
  unless `ruby #{controller} status` =~ /running\s+\[pid\s\d+\]/
    `rm -rf #{File.dirname(controller)}`
  end
end

Have fun,
Roy

My wife, Lynna, and I attended the Katy Tea Party today. For a rural county event, it was well attended. I’m not good at estimating crowd sizes but will guess around a thousand. The Katy Tea Party organization was pretty aggressive at getting an accurate head count (they counted the number of people driving in and did a registration) so hopefully they will publish the real number on their web site, http://katytea.com, soon.

Being a libertarian, I was a little surprised that the Tea Party was not a fiscal conservative event. Instead it was a religious right event, with all the evangelical violence propaganda. This preference for using violence to solve problems such as immigration, abortion, and foreign policy are what drove me away from the Republican party (along with their talk the talk, but don’t bother walking the walk hypocrisy). I really wished the Tea Party would have concentrated on old time fiscal conservatism versus the neo con, holier than thou flavour.

We arrived a little late so missed the first few speakers. A Lt. Colonel was speaking about flying P-38s in WWII, but the PA system’s volume was a little too low. The MC’s corrected the volume, then talked about the RagingElephant organization, which sounds like a “vote religious right” organization.

Next we listened to the excellent Kevin Black, who I’ll admit is one of my favourite local country western singers.

Next up was another Lt. Colonel from the Katy VFW. Besides having trouble reading his speech, his message was a mixture of traditional values mixed with it’s good to use violence to spread American/Christian beliefs all delivered as a sermon.

The best speaker was a lady of Egyptian ancestry whose main message was we need to fight for our freedoms because they are worthy ones. I don’t recall her quoting the bible.

Finally the MC’s got on their pulpit to deliver a call to arms sermon, quiet literally, complete with biblical references.

We finished the event listening to Kevin Black, among the few who stayed to the end.

Unfortunately the fireworks were cancelled, our guess is a burn ban, which was probably smart considering how dry the adjacent fields looked.

Overall, the event was mixed. The call to elect principled fiscal conservatives is good, the need to hold public servants accountable is good, and the call to get the grass roots involved is good. The call for using violence is major bad, but expected from christians.

In Liberty,
Roy
4 July 2009

Classic rebase

For completeness, the command is:

git rebase master developer_branch


Cool.

But for grins, let’s turn the problem around. Let’s say that you use a code generator to jump start your project. What usually happens is the code generators are only used until you invest time and effort into changing the code. Wouldn’t it be nice to be able to use the code generators through out the development cycle?

The easiest way is to create a ‘generated’ branch for the code generators to use then rebase to master.

Generator Rebasing

In this case the command would be:

git rebase generated master


While you will have to manually refactor any name changes on the master branch introduced by the generated branch, you now have the option of using the generators over the life of the project.

Have fun,
Roy

While in the early requirements phase of a new open source project, I was wanting to encourage discussion on the database schema and web API. So I wrote some rspecs to capture the models, their attributes, and their relationships. As we played around with the schema, it was difficult to manually keep the specs accurate, particularly with the relationships. So I wrote a script to generate a merb project from the rspecs to catch relationship problems. This was good and also let me generate model and controller diagrams using railroad_xing.

After a few iterations I was happy and ready to add a little test data so I could play with the RESTful routes as the web API. I was still reluctant to give up the flexibility of generating the project directly from the rspecs, so after a couple of iterations, ended up with two features: first having a files/directory that is copied over the generated project; and second preserving changes by using git and generating the project on a branch then rebasing back to the master branch.

The first feature is simply to recursively to copy the structure in a files/ directory to the project directory after generation. The provides replacement and addition capabilities. Actually most of this stuff (rest controller,…) might ought to be refactored into slices.

One note on the RESTful route generation, it is limited to a depth of 2 (ex: /foos/1/bars/3) because the complexity of my original database would cause route compilation to run out of memory when the depth was set to 3. You can edit the config/router.rb as needed. And yes, the script replaces the normal merb generated controllers and views with ones that support REST, although the views are geared more towards debugging than a final product.

Installation

1. Install git on your system (http://git-scm.com/)
2. Install ruby on your system (http://www.ruby-lang.org)
3. Install rubygems on your system (http://www.ruby-lang.org/en/libraries/)
4. Update rubygems (gem update system)
5. Install SQLite3 on your system (http://www.sqlite.org/)
6. Optionally install ImageMagick (http://www.imagemagick.org) recommended
7. Optionally install Curl (http://curl.haxx.se)
8. Optionally install RestClient (http://rest-client.googlecode.com)

Now you have a choice, either just install the gem with:

gem install royw-spec2merb

or clone the repository:

1. Change to the directory where you want to keep your repositories
   (ex: cd ~/views)
2. Clone the repository (git clone git://github.com/royw/spec2merb.git)
   If you are intended to commit changes to the spec2merb, then fork it
   and clone your fork instead.
3. Change to the repository directory (cd spec2merb)
4. Install required gems (refer to or run the install file)

So let’s generate a project

First there will be two “project” directories:

• A project definition directory which will contain the spec/, and files/ directories used to create the merb project. This should be under version control.
• The created project directory which will be a git repository with two branches: ‘master’ and ‘generated’.

The basic work flow will be:

1. Create the project definition directory.
2. Create the defining spec in the definition directory
3. Run the script to create the merb project.
4. Rebase master from generated branch.
5. Refine the spec until the created merb project runs
6. Examine the generated diagrams and classes
7. Edit the spec and repeat the creation until you are satisfied
8. Proceed with normal web app development on the ‘master’ branch.

As you can see from the work flow, all we are doing is adding a few iterations of using the project generation tools to a normal work flow. The goal being to spin the project quickly, see what you got, then refine it until you are satisfied.

So let’s get started (note, this example is located in examples/addressbook).


cd ~/views
mkdir addressbook-definition
cd addressbook-definition
mkdir spec
vi spec/addressbook-db-schema_spec.rb


And type into addressbook-db-schema_spec.rb:

# This is a stub used to attach information about a model to the spec.
def synopsis(*args)
end
# NOTES
# * NVARCHAR should be used for fields that can contain non-english content.
# * Database should be configured for Unicode
describe("Locationbook Database Schema") do
  describe("Person Model") do
    synopsis("This model describes a person in the address book")
    # attributes
    it "should have a name [TEXT]"
    it "should have a honorific [NVARCHAR(4)]"
    # relationships
    it "should have a relationship of zero or more companies [has 0:n Company]"
    it "should have a relationship of zero or more locations [has 0:n Location]"
    it "should have a relationship of zero or more phones [has 0:n Phone]"
    it "should have a relationship of zero or more emails [has n Email]"
    # additional required methods
  end
  describe("Company Model") do
    synopsis("This model describes a company or business in the address book")
    # attributes
    it "should have a name [NVARCHAR(255)]"
    # relationships
    it "should have a relationship of zero or more people [has 0:n Person]"
    it "should have a relationship of zero or more locations [has 0:n Location]"
    it "should have a relationship of zero or more phones [has 0:n Phones]"
    it "should have a relationship of zero or more emails [has n Email]"
    # additional required methods
  end
  describe("Location Model") do
    synopsis("This model describes a mailing address or location.",
                  "Note a bug in datamapper is letting me use the name that I would prefer, 'Address'")
    # attributes
    it "should have a street_location (including apt or suite number) [NVARCHAR(255)]"
    it "should have a city [NVARCHAR(80)]"
    it "should have a state [NVARCHAR(2)]"
    # relationships
    it "should have a relationship of zero or more people [has 0:n Person]"
    it "should have a relationship of zero or more companies [has 0:n Company]"
    # additional required methods
    it "should return the full mailing location"
  end
  describe("Phone Model") do
    synopsis("This model encapsulates phone numbers")
    # attributes
    it "should have a number [NVARCHAR(14)]"
    # relationships
    it "should have a relationship of zero or more people [has 0:n Person]"
    # it "should have a relationship of zero or more companies [has 0:n Company]"
    # additional required methods
    it "should return the area code"
    it "should return the exchange"
  end
  describe("Email Model") do
    synopsis("This model encapsulates email addresses")
    # attributes
    it "should have an location [NVARCHAR(255)]"
    # relationships
    it "should have a relationship to a person [has 1 Person]"
    it "should have a relationship to a company [has 1 Company]"
    # additional required methods
    it "should return the account (ex: return 'foo' when location is 'foo@bar.com')"
    it "should return the host service (ex: return 'bar.com' when address is 'foo@bar.com')"
  end
end

As you have probably guessed, the String parameter to the it() method has some key structures:

• “should have” denotes a model property
• “should have a relationship” denotes an association
• “should reference” denotes a belongs_to or a has 1 relationship
• “should … variable_name [...]” or “should … variable_name (comment) [...]” defines the variable name and an optional comment. The comment will be appended to the corresponding generated code.
• what is in square brackets are database types (see lib/model_editor::to_dmtype for supported SQL types) or relationship types (has 1, has n, has 0..n, belongs_to). The format is “[relationship model]” where the model is the class name of the model the relationship is with (see lib/spec2_merb::it for parsing details)

Any it “…” line that does not match the rules is added as a comment to the model’s class.

The observant may have noticed that I commented out the company relationship in the Phone model. That’s on purpose to demonstrated missing half of a relationship a little later.

OK now that we have a spec, let’s generate the project:

spec2merb --project addressbook --spec spec/addressbook-db-schema_spec.rb


Hint, I like to put the above command line in a build script file so I don’t have keep retyping it…

It is a good practice to look carefully at the output for any error messages. If you see any, then you can change the spec and generate again.

In this case the generation succeeds. So first do a rebase:


cd addressbook
git rebase generated master


then fire up you favorite SVG viewer (I use Firefox for viewing and GIMP for printing) and open up the generated doc/models.svg (if you have ImageMagick installed, then you should also have a doc/models.gif):

Models Diagram Missing Phone Relationship to Company

Notice that the CompanyPhone join model only has one connection to Company and none to the Phone model. So go back and uncomment the company relationship in the Phone model spec and regenerate. This time the models diagram should look like:

Models Diagram

Looking good, but does it work?

Smoke Test Time

Change to the generated project directory and fire up merb

cd addressbook
bin/merb


Now fire up your browser and look at http://localhost:4000

No route defined for /

While the script defined the rest routes, it did not define the home page route. You can add the home page at the end of config/router.rb. The title is the project name and the menu bar is the set of controllers (defined in config/init/app_config.rb). For now click on People, Companies, Locations and you should see:

Addressbook showing history

At the bottom of the page is the history navigation.

Fire up your curl and get the people index:


curl -H "Content-Type: text/xml; charset=UTF-8" http://localhost:4000/people.xml


Still not very interesting without any data.

Add Some Data

So let’s add some data via rest using curl. Note you may prefer using a rest client like the one at http://rest-client.googlecode.com

Let’s create a work area in the project definition directory:


mkdir rest-requests


Now create the request xml by adding the following to rest-requests/add-person.xml:

<person>
  <name>Roy Wright</name>
  <honorific>Mr.</honorific>
</person>

And lets run it:


cd rest-requests
curl -H "Content-Type: text/xml; charset=UTF-8" --data-binary @add-person.xml http://localhost:4000/people.xml


And look at the data again:


curl -H "Content-Type: text/xml; charset=UTF-8" http://localhost:4000/people.xml


which should return:

<people type='array'>
  <person>
    <id type='datamapper::types::serial'>1</id>
    <name type='datamapper::types::text'>Joe Bob</name>
    <honorific type='datamapper::types::text'/>
  </person>
</people>

From Spec to Web Service

Wow! What we have just done is created a web service directly from specs! As a bonus we have a primitive HTML interface too.

Now admittedly all we have really done is exposed our models via REST. We will probably want to add code to have the interface do something.

Adding Code

Simply edit away on the master branch. When you find you need a change in the database schema, simply commit your master branch, edit the db schema spec, regenerate using spec2merb, then rebase the generated branch back to master.

Conclusion

Overall I think the experiment is showing promise by defining the database schema in one location in a readable format (rspec). Also by using git rebasing to extend using the merb generators over the entire project life is a major benefit.

Where Next?

I like the simple rspec organization of: it “should…” statements. These are pretty easy for anybody to grasp. At the same time I do not like imposing structure on the string. I’m thinking of extending the rspec DSL by adding a shall() method that would let me pass in the line to put in the model, something like:

def shall(comment, model_line)
  'should ' + comment
end

describe('Foo Model') do
  synopsis('This is an example model')
  it shall('have a name attribute', 'property :name, String, :length => 40, :unique => true')
  it shall('reference several bars', 'has n, :bars, :through => Resource')
end

Currently both sides of a relationship must be specified. It would be nice to only have to spec a relationship once. I’m not sure the best way to accomplish this as having the relationship defined in the model seems most natural and all I can think of is to have a separate describe block for all of the relationships. Thoughts?

Finally what other parts of the application can we spec? Maybe whether or not to use RESTful vs. ad hoc routes? Maybe more on the views? Ideas?

Have Fun,
Roy