Specifications to Generated Merb Project

Introduction

While in the early requirements phase of a new open source project, I was wanting to encourage discussion on the database schema and web API. So I wrote some rspecs to capture the models, their attributes, and their relationships. As we played around with the schema, it was difficult to manually keep the specs accurate, particularly with the relationships. So I wrote a script to generate a merb project from the rspecs to catch relationship problems. This was good and also let me generate model and controller diagrams using railroad_xing.

After a few iterations I was happy and ready to add a little test data so I could play with the RESTful routes as the web API. I was still reluctant to give up the flexibility of generating the project directly from the rspecs, so after a couple of iterations, ended up with two features: first having a files/directory that is copied over the generated project; and second preserving changes by using git and generating the project on a branch then rebasing back to the master branch.

The first feature is simply to recursively to copy the structure in a files/ directory to the project directory after generation. The provides replacement and addition capabilities. Actually most of this stuff (rest controller,…) might ought to be refactored into slices.

One note on the RESTful route generation, it is limited to a depth of 2 (ex: /foos/1/bars/3) because the complexity of my original database would cause route compilation to run out of memory when the depth was set to 3. You can edit the config/router.rb as needed. And yes, the script replaces the normal merb generated controllers and views with ones that support REST, although the views are geared more towards debugging than a final product.

Installation

  1. Install git on your system (http://git-scm.com/)
  2. Install ruby on your system (http://www.ruby-lang.org)
  3. Install rubygems on your system (http://www.ruby-lang.org/en/libraries/)
  4. Update rubygems (gem update system)
  5. Install SQLite3 on your system (http://www.sqlite.org/)
  6. Optionally install ImageMagick (http://www.imagemagick.org) recommended
  7. Optionally install Curl (http://curl.haxx.se)
  8. Optionally install RestClient (http://rest-client.googlecode.com)

Now you have a choice, either just install the gem with:

gem install royw-spec2merb

or clone the repository:

  1. Change to the directory where you want to keep your repositories
    (ex: cd ~/views)
  2. Clone the repository (git clone git://github.com/royw/spec2merb.git)
    If you are intended to commit changes to the spec2merb, then fork it
    and clone your fork instead.
  3. Change to the repository directory (cd spec2merb)
  4. Install required gems (refer to or run the install file)

So let’s generate a project

First there will be two “project” directories:

  • A project definition directory which will contain the spec/, and files/ directories used to create the merb project. This should be under version control.
  • The created project directory which will be a git repository with two branches: ‘master’ and ‘generated’.

The basic work flow will be:

  1. Create the project definition directory.
  2. Create the defining spec in the definition directory
  3. Run the script to create the merb project.
  4. Rebase master from generated branch.
  5. Refine the spec until the created merb project runs
  6. Examine the generated diagrams and classes
  7. Edit the spec and repeat the creation until you are satisfied
  8. Proceed with normal web app development on the ‘master’ branch.

As you can see from the work flow, all we are doing is adding a few iterations of using the project generation tools to a normal work flow. The goal being to spin the project quickly, see what you got, then refine it until you are satisfied.

So let’s get started (note, this example is located in examples/addressbook).


cd ~/views
mkdir addressbook-definition
cd addressbook-definition
mkdir spec
vi spec/addressbook-db-schema_spec.rb

And type into addressbook-db-schema_spec.rb:

# This is a stub used to attach information about a model to the spec.
def synopsis(*args)
end
# NOTES
# * NVARCHAR should be used for fields that can contain non-english content.
# * Database should be configured for Unicode
describe("Locationbook Database Schema") do
  describe("Person Model") do
    synopsis("This model describes a person in the address book")
    # attributes
    it "should have a name [TEXT]"
    it "should have a honorific [NVARCHAR(4)]"
    # relationships
    it "should have a relationship of zero or more companies [has 0:n Company]"
    it "should have a relationship of zero or more locations [has 0:n Location]"
    it "should have a relationship of zero or more phones [has 0:n Phone]"
    it "should have a relationship of zero or more emails [has n Email]"
    # additional required methods
  end
  describe("Company Model") do
    synopsis("This model describes a company or business in the address book")
    # attributes
    it "should have a name [NVARCHAR(255)]"
    # relationships
    it "should have a relationship of zero or more people [has 0:n Person]"
    it "should have a relationship of zero or more locations [has 0:n Location]"
    it "should have a relationship of zero or more phones [has 0:n Phones]"
    it "should have a relationship of zero or more emails [has n Email]"
    # additional required methods
  end
  describe("Location Model") do
    synopsis("This model describes a mailing address or location.",
                  "Note a bug in datamapper is letting me use the name that I would prefer, 'Address'")
    # attributes
    it "should have a street_location (including apt or suite number) [NVARCHAR(255)]"
    it "should have a city [NVARCHAR(80)]"
    it "should have a state [NVARCHAR(2)]"
    # relationships
    it "should have a relationship of zero or more people [has 0:n Person]"
    it "should have a relationship of zero or more companies [has 0:n Company]"
    # additional required methods
    it "should return the full mailing location"
  end
  describe("Phone Model") do
    synopsis("This model encapsulates phone numbers")
    # attributes
    it "should have a number [NVARCHAR(14)]"
    # relationships
    it "should have a relationship of zero or more people [has 0:n Person]"
    # it "should have a relationship of zero or more companies [has 0:n Company]"
    # additional required methods
    it "should return the area code"
    it "should return the exchange"
  end
  describe("Email Model") do
    synopsis("This model encapsulates email addresses")
    # attributes
    it "should have an location [NVARCHAR(255)]"
    # relationships
    it "should have a relationship to a person [has 1 Person]"
    it "should have a relationship to a company [has 1 Company]"
    # additional required methods
    it "should return the account (ex: return 'foo' when location is 'foo@bar.com')"
    it "should return the host service (ex: return 'bar.com' when address is 'foo@bar.com')"
  end
end

As you have probably guessed, the String parameter to the it() method has some key structures:

  • “should have” denotes a model property
  • “should have a relationship” denotes an association
  • “should reference” denotes a belongs_to or a has 1 relationship
  • “should … variable_name […]” or “should … variable_name (comment) […]” defines the variable name and an optional comment. The comment will be appended to the corresponding generated code.
  • what is in square brackets are database types (see lib/model_editor::to_dmtype for supported SQL types) or relationship types (has 1, has n, has 0..n, belongs_to). The format is “[relationship model]” where the model is the class name of the model the relationship is with (see lib/spec2_merb::it for parsing details)

Any it “…” line that does not match the rules is added as a comment to the model’s class.

The observant may have noticed that I commented out the company relationship in the Phone model. That’s on purpose to demonstrated missing half of a relationship a little later.

OK now that we have a spec, let’s generate the project:

spec2merb --project addressbook --spec spec/addressbook-db-schema_spec.rb

Hint, I like to put the above command line in a build script file so I don’t have keep retyping it…

It is a good practice to look carefully at the output for any error messages. If you see any, then you can change the spec and generate again.

In this case the generation succeeds. So first do a rebase:


cd addressbook
git rebase generated master

then fire up you favorite SVG viewer (I use Firefox for viewing and GIMP for printing) and open up the generated doc/models.svg (if you have ImageMagick installed, then you should also have a doc/models.gif):

Models Diagram Missing Phone Relationship to Company
Models Diagram Missing Phone Relationship to Company

Notice that the CompanyPhone join model only has one connection to Company and none to the Phone model. So go back and uncomment the company relationship in the Phone model spec and regenerate. This time the models diagram should look like:

Models Diagram
Models Diagram

Looking good, but does it work?

Smoke Test Time

Change to the generated project directory and fire up merb

cd addressbook
bin/merb

Now fire up your browser and look at http://localhost:4000

No route defined for /
No route defined for /

While the script defined the rest routes, it did not define the home page route. You can add the home page at the end of config/router.rb. The title is the project name and the menu bar is the set of controllers (defined in config/init/app_config.rb). For now click on People, Companies, Locations and you should see:

Addressbook showing history
Addressbook showing history

At the bottom of the page is the history navigation.

Fire up your curl and get the people index:


curl -H "Content-Type: text/xml; charset=UTF-8" http://localhost:4000/people.xml

Still not very interesting without any data.

Add Some Data

So let’s add some data via rest using curl. Note you may prefer using a rest client like the one at http://rest-client.googlecode.com

Let’s create a work area in the project definition directory:


mkdir rest-requests

Now create the request xml by adding the following to rest-requests/add-person.xml:

<person>
  <name>Roy Wright</name>
  <honorific>Mr.</honorific>
</person>

And lets run it:


cd rest-requests
curl -H "Content-Type: text/xml; charset=UTF-8" --data-binary @add-person.xml http://localhost:4000/people.xml

And look at the data again:


curl -H "Content-Type: text/xml; charset=UTF-8" http://localhost:4000/people.xml

which should return:

<people type='array'>
  <person>
    <id type='datamapper::types::serial'>1</id>
    <name type='datamapper::types::text'>Joe Bob</name>
    <honorific type='datamapper::types::text'/>
  </person>
</people>

From Spec to Web Service

Wow! What we have just done is created a web service directly from specs! As a bonus we have a primitive HTML interface too.

Now admittedly all we have really done is exposed our models via REST. We will probably want to add code to have the interface do something.

Adding Code

Simply edit away on the master branch. When you find you need a change in the database schema, simply commit your master branch, edit the db schema spec, regenerate using spec2merb, then rebase the generated branch back to master.

Conclusion

Overall I think the experiment is showing promise by defining the database schema in one location in a readable format (rspec). Also by using git rebasing to extend using the merb generators over the entire project life is a major benefit.

Where Next?

I like the simple rspec organization of: it “should…” statements. These are pretty easy for anybody to grasp. At the same time I do not like imposing structure on the string. I’m thinking of extending the rspec DSL by adding a shall() method that would let me pass in the line to put in the model, something like:

def shall(comment, model_line)
  'should ' + comment
end

describe('Foo Model') do
  synopsis('This is an example model')
  it shall('have a name attribute', 'property :name, String, :length =&gt; 40, :unique =&gt; true')
  it shall('reference several bars', 'has n, :bars, :through =&gt; Resource')
end

Currently both sides of a relationship must be specified. It would be nice to only have to spec a relationship once. I’m not sure the best way to accomplish this as having the relationship defined in the model seems most natural and all I can think of is to have a separate describe block for all of the relationships. Thoughts?

Finally what other parts of the application can we spec? Maybe whether or not to use RESTful vs. ad hoc routes? Maybe more on the views? Ideas?

Have Fun,
Roy

Advertisements

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s