While in the early requirements phase of a new open source project, I was wanting to encourage discussion on the database schema and web API. So I wrote some rspecs to capture the models, their attributes, and their relationships. As we played around with the schema, it was difficult to manually keep the specs accurate, particularly with the relationships. So I wrote a script to generate a merb project from the rspecs to catch relationship problems. This was good and also let me generate model and controller diagrams using railroad_xing.
After a few iterations I was happy and ready to add a little test data so I could play with the RESTful routes as the web API. I was still reluctant to give up the flexibility of generating the project directly from the rspecs, so after a couple of iterations, ended up with two features: first having a files/directory that is copied over the generated project; and second preserving changes by using git and generating the project on a branch then rebasing back to the master branch.
The first feature is simply to recursively to copy the structure in a files/ directory to the project directory after generation. The provides replacement and addition capabilities. Actually most of this stuff (rest controller,…) might ought to be refactored into slices.
One note on the RESTful route generation, it is limited to a depth of 2 (ex: /foos/1/bars/3) because the complexity of my original database would cause route compilation to run out of memory when the depth was set to 3. You can edit the config/router.rb as needed. And yes, the script replaces the normal merb generated controllers and views with ones that support REST, although the views are geared more towards debugging than a final product.
- Install git on your system (http://git-scm.com/)
- Install ruby on your system (http://www.ruby-lang.org)
- Install rubygems on your system (http://www.ruby-lang.org/en/libraries/)
- Update rubygems (gem update system)
- Install SQLite3 on your system (http://www.sqlite.org/)
- Optionally install ImageMagick (http://www.imagemagick.org) recommended
- Optionally install Curl (http://curl.haxx.se)
- Optionally install RestClient (http://rest-client.googlecode.com)
Now you have a choice, either just install the gem with:
gem install royw-spec2merb
or clone the repository:
- Change to the directory where you want to keep your repositories
(ex: cd ~/views)
- Clone the repository (git clone git://github.com/royw/spec2merb.git)
If you are intended to commit changes to the spec2merb, then fork it
and clone your fork instead.
- Change to the repository directory (cd spec2merb)
- Install required gems (refer to or run the install file)
So let’s generate a project
First there will be two “project” directories:
- A project definition directory which will contain the spec/, and files/ directories used to create the merb project. This should be under version control.
- The created project directory which will be a git repository with two branches: ‘master’ and ‘generated’.
The basic work flow will be:
- Create the project definition directory.
- Create the defining spec in the definition directory
- Run the script to create the merb project.
- Rebase master from generated branch.
- Refine the spec until the created merb project runs
- Examine the generated diagrams and classes
- Edit the spec and repeat the creation until you are satisfied
- Proceed with normal web app development on the ‘master’ branch.
As you can see from the work flow, all we are doing is adding a few iterations of using the project generation tools to a normal work flow. The goal being to spin the project quickly, see what you got, then refine it until you are satisfied.
So let’s get started (note, this example is located in examples/addressbook).
And type into addressbook-db-schema_spec.rb:
# This is a stub used to attach information about a model to the spec. def synopsis(*args) end # NOTES # * NVARCHAR should be used for fields that can contain non-english content. # * Database should be configured for Unicode describe("Locationbook Database Schema") do describe("Person Model") do synopsis("This model describes a person in the address book") # attributes it "should have a name [TEXT]" it "should have a honorific [NVARCHAR(4)]" # relationships it "should have a relationship of zero or more companies [has 0:n Company]" it "should have a relationship of zero or more locations [has 0:n Location]" it "should have a relationship of zero or more phones [has 0:n Phone]" it "should have a relationship of zero or more emails [has n Email]" # additional required methods end describe("Company Model") do synopsis("This model describes a company or business in the address book") # attributes it "should have a name [NVARCHAR(255)]" # relationships it "should have a relationship of zero or more people [has 0:n Person]" it "should have a relationship of zero or more locations [has 0:n Location]" it "should have a relationship of zero or more phones [has 0:n Phones]" it "should have a relationship of zero or more emails [has n Email]" # additional required methods end describe("Location Model") do synopsis("This model describes a mailing address or location.", "Note a bug in datamapper is letting me use the name that I would prefer, 'Address'") # attributes it "should have a street_location (including apt or suite number) [NVARCHAR(255)]" it "should have a city [NVARCHAR(80)]" it "should have a state [NVARCHAR(2)]" # relationships it "should have a relationship of zero or more people [has 0:n Person]" it "should have a relationship of zero or more companies [has 0:n Company]" # additional required methods it "should return the full mailing location" end describe("Phone Model") do synopsis("This model encapsulates phone numbers") # attributes it "should have a number [NVARCHAR(14)]" # relationships it "should have a relationship of zero or more people [has 0:n Person]" # it "should have a relationship of zero or more companies [has 0:n Company]" # additional required methods it "should return the area code" it "should return the exchange" end describe("Email Model") do synopsis("This model encapsulates email addresses") # attributes it "should have an location [NVARCHAR(255)]" # relationships it "should have a relationship to a person [has 1 Person]" it "should have a relationship to a company [has 1 Company]" # additional required methods it "should return the account (ex: return 'foo' when location is 'firstname.lastname@example.org')" it "should return the host service (ex: return 'bar.com' when address is 'email@example.com')" end end
As you have probably guessed, the String parameter to the it() method has some key structures:
- “should have” denotes a model property
- “should have a relationship” denotes an association
- “should reference” denotes a belongs_to or a has 1 relationship
- “should … variable_name […]” or “should … variable_name (comment) […]” defines the variable name and an optional comment. The comment will be appended to the corresponding generated code.
- what is in square brackets are database types (see lib/model_editor::to_dmtype for supported SQL types) or relationship types (has 1, has n, has 0..n, belongs_to). The format is “[relationship model]” where the model is the class name of the model the relationship is with (see lib/spec2_merb::it for parsing details)
Any it “…” line that does not match the rules is added as a comment to the model’s class.
The observant may have noticed that I commented out the company relationship in the Phone model. That’s on purpose to demonstrated missing half of a relationship a little later.
OK now that we have a spec, let’s generate the project:
spec2merb --project addressbook --spec spec/addressbook-db-schema_spec.rb
Hint, I like to put the above command line in a build script file so I don’t have keep retyping it…
It is a good practice to look carefully at the output for any error messages. If you see any, then you can change the spec and generate again.
In this case the generation succeeds. So first do a rebase:
git rebase generated master
then fire up you favorite SVG viewer (I use Firefox for viewing and GIMP for printing) and open up the generated doc/models.svg (if you have ImageMagick installed, then you should also have a doc/models.gif):
Notice that the CompanyPhone join model only has one connection to Company and none to the Phone model. So go back and uncomment the company relationship in the Phone model spec and regenerate. This time the models diagram should look like:
Looking good, but does it work?
Smoke Test Time
Change to the generated project directory and fire up merb
Now fire up your browser and look at http://localhost:4000
While the script defined the rest routes, it did not define the home page route. You can add the home page at the end of config/router.rb. The title is the project name and the menu bar is the set of controllers (defined in config/init/app_config.rb). For now click on People, Companies, Locations and you should see:
At the bottom of the page is the history navigation.
Fire up your curl and get the people index:
curl -H "Content-Type: text/xml; charset=UTF-8" http://localhost:4000/people.xml
Still not very interesting without any data.
Add Some Data
So let’s add some data via rest using curl. Note you may prefer using a rest client like the one at http://rest-client.googlecode.com
Let’s create a work area in the project definition directory:
Now create the request xml by adding the following to rest-requests/add-person.xml:
<person> <name>Roy Wright</name> <honorific>Mr.</honorific> </person>
And lets run it:
curl -H "Content-Type: text/xml; charset=UTF-8" --data-binary @add-person.xml http://localhost:4000/people.xml
And look at the data again:
curl -H "Content-Type: text/xml; charset=UTF-8" http://localhost:4000/people.xml
which should return:
<people type='array'> <person> <id type='datamapper::types::serial'>1</id> <name type='datamapper::types::text'>Joe Bob</name> <honorific type='datamapper::types::text'/> </person> </people>
From Spec to Web Service
Wow! What we have just done is created a web service directly from specs! As a bonus we have a primitive HTML interface too.
Now admittedly all we have really done is exposed our models via REST. We will probably want to add code to have the interface do something.
Simply edit away on the master branch. When you find you need a change in the database schema, simply commit your master branch, edit the db schema spec, regenerate using spec2merb, then rebase the generated branch back to master.
Overall I think the experiment is showing promise by defining the database schema in one location in a readable format (rspec). Also by using git rebasing to extend using the merb generators over the entire project life is a major benefit.
I like the simple rspec organization of: it “should…” statements. These are pretty easy for anybody to grasp. At the same time I do not like imposing structure on the string. I’m thinking of extending the rspec DSL by adding a shall() method that would let me pass in the line to put in the model, something like:
def shall(comment, model_line) 'should ' + comment end describe('Foo Model') do synopsis('This is an example model') it shall('have a name attribute', 'property :name, String, :length => 40, :unique => true') it shall('reference several bars', 'has n, :bars, :through => Resource') end
Currently both sides of a relationship must be specified. It would be nice to only have to spec a relationship once. I’m not sure the best way to accomplish this as having the relationship defined in the model seems most natural and all I can think of is to have a separate describe block for all of the relationships. Thoughts?
Finally what other parts of the application can we spec? Maybe whether or not to use RESTful vs. ad hoc routes? Maybe more on the views? Ideas?