Gus and I will be presenting on "Effective Testing on Grails" at the London Groovy and Grails User Group next week Wed 1st of October, @18.30. Please feel free to attend if this seems relevant to you. A synopsis of the talk and registration for the event are all available at the following link.
http://skillsmatter.com/event/java-jee/ggug
Thursday, 25 September 2008
The UI decisions behind a typical web form
I stumbled across a very useful article recently and wanted to share it around. The article discusses the best placement of form submit/cancel buttons for a typical web form and backs this up with Heat Maps showing where the user spent time looking at elements on the page etc.
Check out the Primary & Secondary Actions in Web Forms blog post yourself.
There are a lot of very useful posts on that blog - all of them focused on useability and the "dark art" of UI decision making.
Inspecting Response Headers in the browser
I needed to inspect the headers being sent to various browsers today. Specifically we wanted to find out how Yahoo was able to deliver a "fresh" CAPTCHA image when you used the browser back button to return to the page with the CAPTCHA image on it.
We observed that this happened even with Javascript disabled, and so came to the conclusion that it must be something in the response headers.
In Firefox (v1.5+) there is a really useful extension that I have mentioned before (the Web Developer Toolbar extension) which can show the headers using Tools -> Web Developer -> Information -> View Response Headers.
When using Internet Explorer a solution is to use the IE HTTP Headers plugin (for IE v5.0+) which is available at http://www.blunck.se
For those interested, the solution to the problem was to add the following headers to the response:
static void doNotCacheResponse(HttpServletResponse response) {
response.setHeader('Pragma', 'no-cache')
response.setHeader('Cache-Control', 'no-cache,no-store')
response.setHeader('Expires', '-1')
}We've tested that this works to force a re-request using Firefox 2, Firefox 3, IE6, IE7, Chrome, Safari 3 and Opera 9 - which is nice!
Tuesday, 23 September 2008
Careful with what you Query.
The F5 BigIP load balancers have a feature called iRules. They allow you to manipulate and manage traffic in the Load Balancer.
They are really useful for checking or modifying cookies [HTTP::cookie], the host domain [HTTP::host], path [HTTP::path] and query parameters [HTTP::query] of URLs and requests.
A sligth word of warning though.... any checks made on the QUERY of a HTTP request actually search the URI of the referer as well. As an example, the following iRule would send a HTTP 200 response if you were either requesting a URL with a query parameter of "ThereIsAQuery" OR if you're refer page's URL included a query parameter of "ThereIsAQuery".
Other attributes such as [HTTP::host], [HTTP::path], and [HTTP::uri] are only valid for the requested URL. They are not checked in the referer header.
They are really useful for checking or modifying cookies [HTTP::cookie], the host domain [HTTP::host], path [HTTP::path] and query parameters [HTTP::query] of URLs and requests.
A sligth word of warning though.... any checks made on the QUERY of a HTTP request actually search the URI of the referer as well. As an example, the following iRule would send a HTTP 200 response if you were either requesting a URL with a query parameter of "ThereIsAQuery" OR if you're refer page's URL included a query parameter of "ThereIsAQuery".
when HTTP_REQUEST {
if {
([HTTP::QUERY] eq "ThereIsAQuery")
}
{
HTTP::respond 200 content "<html><body>There is a Query</body></html>"
}
}Other attributes such as [HTTP::host], [HTTP::path], and [HTTP::uri] are only valid for the requested URL. They are not checked in the referer header.
Monday, 15 September 2008
Command Objects Revisited
I posted yesterday about command objects but a conversation I had this morning made me realise I'd neglected to mention one of the interesting and less obvious ways to use them.
Imagine a scenario where you have a form that is basically for saving domain object instances but has a few other fields, not directly properties of the domain object in question but things like choices of how to set property values (e.g. a radio button to choose whether or not to update the timestamp on the object), workflow type instructions to the controller (e.g. preview this object after saving it), etc. This is a very common scenario.
You can handle this neatly with command objects by nesting an instance of the domain class inside the command object class, e.g.
Imagine a scenario where you have a form that is basically for saving domain object instances but has a few other fields, not directly properties of the domain object in question but things like choices of how to set property values (e.g. a radio button to choose whether or not to update the timestamp on the object), workflow type instructions to the controller (e.g. preview this object after saving it), etc. This is a very common scenario.
You can handle this neatly with command objects by nesting an instance of the domain class inside the command object class, e.g.
class MyCommand {
boolean updateTimestamp
boolean previewAfterSave
MyDomain domainObj
}Then name your form fields accordingly... <g:checkBox name="updateTimestamp" value="${command.updateTimestamp}" />
<g:checkbox name="previewAfterSave" value="${command.previewAfterSave}" />
<input type="hidden" name="domainObj.id" value="${command.domainObj.id}" />
<input type="text" name="domainObj.name" value="${command.domainObj.name}" />
<g:select name="domainObj.otherDomainObj" from="${OtherDomainClass.list()}" value="${command.domainObj.otherDomainObj.id}" optionKey="id" /> and so on.
Sunday, 14 September 2008
Command Objects
We have a bunch of horrible code in our codebase to do with binding form fields to objects. To be fair it dates from a time when we were a lot newer to Grails and Grails and its documentation was a lot less mature. The other day I had to add a date picker to one of our forms and write the binding code for it. The g:datePicker tag creates some select boxes and a hidden field. Unlike other implementations I've used in the past it doesn't use Javascript to compose the select box values into the hidden field on form submit. Instead the hidden field has the value "struct" which I guess is a hint to GrailsDataBinder. To cut a long story short the code I ended up writing is pretty nasty. Especially considering that GrailsDataBinder is perfectly capable of handling this kind of binding without any code being written.
Instead of all this tedious messing about with params Grails controllers can, as most of you probably know, bind form submissions to domain objects or command objects. Binding to domain objects works by doing domainObj.properties = params or using the bindData method. So much most of us are used to (even if you wouldn't think so from looking at some of our code).
To have your controller action use a command object you simply specify it as an argument to the action closure. e.g.
A form that contained...
Unit testing is easy because you can simply construct the command object and pass it in to the controller action. You don't have to worry about setting a bunch of esoteric key/value pairs on params to get your controller's hairy binding logic to work. In an integration test, if you really want to, you can set params and test that the command binding works as you expect.
The only non trivial thing in unit tests (in integration tests this is not a problem) is testing command error handling. In unit tests commands won't have an errors property or hasErrors method like they do in a running app or integration test. I've come up with the following which I've raised as an enhancement request for the experimental testing plugin:
Command objects can also be used to handle multipart upload forms...
The weak area in binding at the moment seems to be one-to-many domain class relationships (which is why I didn't jump into refactoring the code I mentioned at the start of this post). That does require you to write hairy binding code and is probably worth a post in and of itself.
Instead of all this tedious messing about with params Grails controllers can, as most of you probably know, bind form submissions to domain objects or command objects. Binding to domain objects works by doing domainObj.properties = params or using the bindData method. So much most of us are used to (even if you wouldn't think so from looking at some of our code).
To have your controller action use a command object you simply specify it as an argument to the action closure. e.g.
def save = { SaveCommand command ->
if (command.hasErrors()) {
render(view: 'index', model: [command: command])
}
// do some stuff
} The binding is done for you. As you may guess from the command.hasErrors() above, command objects can have constraints exactly as domain objects do. They can even have services and other Spring application context artefacts wired in automatically just like controllers and services can.A form that contained...
<input type="text" name="name" value="${command?.name}" />
<g:datePicker name="birthday" value="${command?.birthday}" />
<g:select name="sandwich.id" from="${Sandwich.list()}" value="${command?.sandwich?.id" optionKey="id" />Would bind to a command like this... class MyCommand {
String name // simple property
Date birthday // bound from multiple fields submitted by date picker
Sandwich sandwich // bound domain object
}Unit testing is easy because you can simply construct the command object and pass it in to the controller action. You don't have to worry about setting a bunch of esoteric key/value pairs on params to get your controller's hairy binding logic to work. In an integration test, if you really want to, you can set params and test that the command binding works as you expect.
The only non trivial thing in unit tests (in integration tests this is not a problem) is testing command error handling. In unit tests commands won't have an errors property or hasErrors method like they do in a running app or integration test. I've come up with the following which I've raised as an enhancement request for the experimental testing plugin:
def createCommand(Class clazz) {
def command = clazz.newInstance()
def commandErrors = new BeanPropertyBindingResult(command, GrailsClassUtils.getLogicalPropertyName(clazz.name, ''))
command.metaClass.getErrors = {-> commandErrors }
command.metaClass.hasErrors = {-> commandErrors.hasErrors() }
}
void testCommandWithBindingErrors() {
def command = createCommand(MyCommand)
command.errors.reject('birthday', 'nullable')
controller.action(command)
// assert the form is re-rendered with the command in the model, etc.
}Command objects can also be used to handle multipart upload forms...
<input type="file" name="avatar" />Maps to...
class MyCommand {
byte[] avatar
}If you need access to the uploaded file's metadata (content type, original filename, etc.) instead of declaring the property as type byte[] use MultipartFile. Either way the binding works seamlessly and allows you to apply constraints to the uploaded file.The weak area in binding at the moment seems to be one-to-many domain class relationships (which is why I didn't jump into refactoring the code I mentioned at the start of this post). That does require you to write hairy binding code and is probably worth a post in and of itself.
Friday, 12 September 2008
None of the rest of us can fathom CSS either.
Dave Minter has written a blog entry Float like a wasp that goes on about the inadequacies and problems faced with implementing CSS and the frustrations of HTML. I think that he's got a point when he says:
You are not alone. None of the rest of us can fathom CSS either.
He has a hit-list of things he'd like to see made more simple... and they include the usual bug-bears that we have all experienced at some stage (some more than others, obviously).
There are solutions to all the things he would like to see... but the key thing is that these solutions are invariably not intuitive and require arcane knowledge (and the patience of a saint) to ensure conformity.
I particularly liked this quote:
One of my tips for fixing all that irks with CSS would be to buy two or three of those books describing machiavellian ways to conspire against the deficiencies of CSS (anything with "hacks" or "tips" in the title) and then beat up the standard until the easiest approach to implementing everything in the book could be covered in a pamphlet.
I, too, would like to see a more structure simplification (rather that the current direction of CSS which seems to be adding to the mess).
Sunday, 7 September 2008
Pessimistic Locking With Grails
By default grails uses optimisic locking with versioning. In a nutshell this means that each of your domain objects is blessed with a version field and hibernate throws an exception if you ever try to update or lock an object with an old version number. For scenarios where you are happy to handle the exception, or just report the error to the user via a UI this is fine, however when this behaviour isn't ok you need to switch to pessimistic locking. This is where the fun begins.
Unfortunately the Grails pessimistic locking support is incomplete. There are scenarios when it will work, but scenarios when it wont. As of 1.03 the docs don't make this clear. For example, let's say you have a domain object called Account...
Obviously updates to the account balance need to be atomic. The Grails docs suggest the following...
The resulting SQL will look something like
The above will work fine PROVIDING the account entity is not already in the hibernate session. If the account entity was previously retrieved the resulting SQL only re-selects the id.
The db row is still locked, but because the account entity has not been refreshed your application could be working with stale data. The annoyance with the Account.lock(id) approach is that you always need to know the entity's id without obtaining the entity. So your code can never do
My initial workaround to the above is to duck into the hibernate API and do the following
Which appears ensures you have fresh data and a db lock. However this is ONLY true if your entity does not have any eager relationships (one-to-ones are eager by default). Incredibly hibernate retrieves the eager relationships before calling select for update on the main entity, meaning that the relationship data could be stale. I've tried fixing this with cascade: 'lock' but it did SFA. It now appears the best (and I use that term loosely) way to ensure fresh, locked data is to do the following
An alternative is to call "discard" on the object before you lock it
An interesting aside, which has nothing to do with grails but everything to do with concurrency is how you prepare to update multiple objects, e.g.
The above code might look OK at first glance, but what if two threads were to update an intersecting set of accounts?
Thread 1: accountA, accountB
Thread 2: accountB, accountA
If you were unlucky Thread 1 may obtain a lock on accountA at the same time that Thread 2 obtained a lock on accountB resulting in deadlock. The solution is to sort both sets of accounts using the same algorithm anywhere you attempt to locks multiple entites, e.g.
Another concurrency scenario is when you want to modify the many side of a one-to-many relationship, e.g.
In this case because a subscription is never shared between two people it may be ok to just lock the person object, e.g.
The reason I say 'may', is that this approach is dependent on all your other code taking the same approach when updating subscriptions and that it's not a performance bottleneck to lock the person. Choosing to lock the person is certainly an easy way to avoid creating duplicate subscriptions without the risk of a unique constraint violations.
Unfortunately the Grails pessimistic locking support is incomplete. There are scenarios when it will work, but scenarios when it wont. As of 1.03 the docs don't make this clear. For example, let's say you have a domain object called Account...
class Account {
String name
long balance
static mapping {
version false // Required to avoid stale object exceptions when hibernate attempts a lock
}
}Obviously updates to the account balance need to be atomic. The Grails docs suggest the following...
Account account = Account.lock(id)
account.balance += amount
account.save()
The resulting SQL will look something like
select id, name, balance from account where id = 123 for update;
update account set balance = 101 where id = 123;
The above will work fine PROVIDING the account entity is not already in the hibernate session. If the account entity was previously retrieved the resulting SQL only re-selects the id.
select id from acount where id = 123 for update;
update account set balance = 101 where id = 123;
The db row is still locked, but because the account entity has not been refreshed your application could be working with stale data. The annoyance with the Account.lock(id) approach is that you always need to know the entity's id without obtaining the entity. So your code can never do
Account account = Account.findByName(name) // Maybe in a controller or other service
...
account = Account.lock(account.id)
account.balance += amount
account.save()
My initial workaround to the above is to duck into the hibernate API and do the following
sessionFactory.currentSession.refresh(account, LockMode.UPGRADE)
account.balance += amount
account.save()
Which appears ensures you have fresh data and a db lock. However this is ONLY true if your entity does not have any eager relationships (one-to-ones are eager by default). Incredibly hibernate retrieves the eager relationships before calling select for update on the main entity, meaning that the relationship data could be stale. I've tried fixing this with cascade: 'lock' but it did SFA. It now appears the best (and I use that term loosely) way to ensure fresh, locked data is to do the following
account.lock() // versioning must be disabled
account.refresh()
account.balance += amount
An alternative is to call "discard" on the object before you lock it
account.discard()
account = Account.lock(account.id)
account.balance += amount
account.save()
An interesting aside, which has nothing to do with grails but everything to do with concurrency is how you prepare to update multiple objects, e.g.
accountsToBeUpdated.each { Account account ->
account.lock()
account.refresh()
account.balance += amount
account.save()
}The above code might look OK at first glance, but what if two threads were to update an intersecting set of accounts?
Thread 1: accountA, accountB
Thread 2: accountB, accountA
If you were unlucky Thread 1 may obtain a lock on accountA at the same time that Thread 2 obtained a lock on accountB resulting in deadlock. The solution is to sort both sets of accounts using the same algorithm anywhere you attempt to locks multiple entites, e.g.
def sortedAccounts = accountsToBeUpdated.sort { a, b -> a.name <=> b.name }
sortedAccounts.each { Account account ->
account.lock()
account.refresh()
account.balance += amount
account.save()
}Another concurrency scenario is when you want to modify the many side of a one-to-many relationship, e.g.
class Person {
Set subscriptions
} In this case because a subscription is never shared between two people it may be ok to just lock the person object, e.g.
person.lock()
person.refresh()
person.addToSubscriptions(subscription)
person.save()
The reason I say 'may', is that this approach is dependent on all your other code taking the same approach when updating subscriptions and that it's not a performance bottleneck to lock the person. Choosing to lock the person is certainly an easy way to avoid creating duplicate subscriptions without the risk of a unique constraint violations.
Sunday, 31 August 2008
Up The Limit
Recently while using the searchable plugin with Grails I've had exceptions citing too many open files. Linux systems typically limit resources on a per-user basis by various criteria such as number of processes and number of open files. Running ulimit -a I see the following:
core file size (blocks, -c) 0On this system I can only open 1024 files at once - I can increase this limit in /etc/security/limits.conf by adding something similar to the following:
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 36864
max locked memory (kbytes, -l) 32
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 36864
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
gus soft nofile 16384To ensure that PAM actually takes notice of these limits on login check /etc/pam.d/login for an entry like:
gus hard nofile 16384
session required pam_limits.soCan haz filez!
Thursday, 28 August 2008
Unit Testing Controllers With The Testing Plugin
Grails JUnit tests are, as you are no doubt aware, divided into unit tests and integration tests. Unit tests run fast, aren't dependent on the database or Spring context but mean you have to live without all the dynamic magic that Grails attaches to your artefacts (controllers, taglibs, domain classes, etc.) This can certainly lead to some sub-optimal test code.
Lets consider an example. I want to write some test coverage for a controller that looks up a domain object based on a parameter and places it in the model. Usual disclaimer: of course, in the real world I'd have written the test first, etc. Nothing much to the controller:
Okay, we could mock the Lolrus class, but the code for that isn't much cleaner and alarm bells start ringing when you start using mocks in an integration test. Shouldn't this be a unit test? Sure, that would be nice but we're using the render dynamic method in our controller, we'll have to mock that out on the metaClass, mock out the findByName method on Lolrus, etc. etc. Too much hassle right? Not any more.
The mockDomain method allows you to set up some domain objects for which all the usual Grails domain class dynamic methods will work - including findBy..., addTo..., etc. The collection passed as the second argument to mockDomain defines all the instances that exist and they can be retrieved just as a real domain object could. The mockDomain method co-exists happily with regular Groovy mocks and the testing plugin also provides some lighter mocking capabilities via its own mockFor method.
Additionally the controller's render method and params property work without any work on our part. This is true of all the dynamic properties and methods of the controller, request, response, params, controllerName, redirect, etc.
Not only that, but this test runs faster than the integration test as it's not configuring the Spring container or an in-memory database. That speed difference only grows as the project becomes more complex - it's a couple of seconds between the two tests here, but in a more complex project it's a lot more.
There are a couple of gotchas (the plugin isn't fully mature). First off you need to remember to call super.setUp() in order that all the Grails stuff happens. The ControllerUnitTestCase class will figure out based on the class naming convention which controller class it needs to mock up. Secondly, the domain class get method doesn't seem to work with non-standard identifier types at the moment (i.e. anything other than a Long). However, this unit testing support is intended to get rolled into Grails 1.1 so it should improve rapidly.
This is only part of what the testing plugin can do. Support for unit testing domain class constraints, etc. is also included. I've also been working on a TagLibUnitTestCase that works along the same lines as ControllerUnitTestCase.
Lets consider an example. I want to write some test coverage for a controller that looks up a domain object based on a parameter and places it in the model. Usual disclaimer: of course, in the real world I'd have written the test first, etc. Nothing much to the controller:
class LolrusController {
def show = {
def myLolrus = Lolrus.findByName(params.lolrusName)
render(view: 'lolrus', model: [lolrus: myLolrus])
}
}and here's the domain class:class Lolrus {
String name
String mood
static hasMany = [posessions: Item]
static constraints = {
name(unique: true)
posessions(validator: { posessions ->
return posessions.any { it.type == 'bukkit' }
})
}
String toString() { "Lolrus[$name]" }
boolean equals(Object o) { return o instanceof Lolrus && o.name == name }
int hashCode() { return 37 * name.hashCode() }
}Okay, so a reasonable test (ignoring edge cases for now) could be to create a couple of domain objects, make a request to the controller and verify that the correct domain object is retrieved. Simple, right?:class LolrusControllerTests extends GroovyTestCase {
def controller
def lolrus1, lolrus2
void setUp() {
controller = new LolrusController()
lolrus1 = new Lolrus(name: 'Hugh')
lolrus2 = new Lolrus(name: 'Alan')
[lolrus1, lolrus2]*.save(flush: true)
}
void testShowActionFindsCorrectLolrusAndSticksItInModel() {
controller.params.lolrusName = lolrus1.name
def model = controller.show()
assertEquals('/lolrus/show', controller.modelAndView.viewName)
assertEquals(lolrus1, controller.modelAndView.model.lolrus)
}
}Unfortunately, not so simple. The test fails and reports expected:<Lolrus[Hugh]> but was:<null>What we've forgotten of course is that Lolrus has a mandatory field mood and a custom validation on its posessions association that requires it to have a bukkit. The two Lolrus instances we tried to create in setUp couldn't be saved. Great. Well, let's just add that to the test set up:
void setUp() {
controller = new LolrusController()
lolrus1 = new Lolrus(name: 'Hugh', mood: 'worryingly cheerful')
lolrus1.addToPosessions new Item(type: 'bukkit')
lolrus2 = new Lolrus(name: 'Alan', mood: 'mildly distressed')
lolrus2.addToPosessions new Item(type: 'bukkit')
[lolrus1, lolrus2].each {
assert it.save(flush: true), it.errors
}
}Okay, it works but look how much of what's going on in setUp is actually nothing to do with anything the test cares about. This is a very simple example with a single test case. Imagine how much worse that setUp method is going to get when we add more tests or the Lolrus domain object gets more complex... Imagine the refactoring required when the Lolrus domain object acquires other constraints... Imagine the query is more complex requiring more domain objects to be set up to really prove it works...Okay, we could mock the Lolrus class, but the code for that isn't much cleaner and alarm bells start ringing when you start using mocks in an integration test. Shouldn't this be a unit test? Sure, that would be nice but we're using the render dynamic method in our controller, we'll have to mock that out on the metaClass, mock out the findByName method on Lolrus, etc. etc. Too much hassle right? Not any more.
grails install-plugin testingAllows us to (among other things) write controller unit test cases like this:
class LolrusControllerUnitTests extends grails.test.ControllerUnitTestCase {
def controller
def lolrus1, lolrus2
void setUp() {
super.setUp()
controller = new LolrusController()
lolrus1 = new Lolrus(name: 'Hugh')
lolrus2 = new Lolrus(name: 'Alan')
mockDomain(Lolrus, [lolrus1, lolrus2])
}
void testShowActionFindsCorrectLolrusAndSticksItInModel() {
controller.params.lolrusName = lolrus1.name
controller.show()
assertEquals('show', renderArgs.view)
assertEquals(lolrus1, renderArgs.model.lolrus)
}
}Much neater. We're not worrying about the constraints required to set up valid Lolrus instances, but neither are we having to mock out the findByName method.The mockDomain method allows you to set up some domain objects for which all the usual Grails domain class dynamic methods will work - including findBy..., addTo..., etc. The collection passed as the second argument to mockDomain defines all the instances that exist and they can be retrieved just as a real domain object could. The mockDomain method co-exists happily with regular Groovy mocks and the testing plugin also provides some lighter mocking capabilities via its own mockFor method.
Additionally the controller's render method and params property work without any work on our part. This is true of all the dynamic properties and methods of the controller, request, response, params, controllerName, redirect, etc.
Not only that, but this test runs faster than the integration test as it's not configuring the Spring container or an in-memory database. That speed difference only grows as the project becomes more complex - it's a couple of seconds between the two tests here, but in a more complex project it's a lot more.
There are a couple of gotchas (the plugin isn't fully mature). First off you need to remember to call super.setUp() in order that all the Grails stuff happens. The ControllerUnitTestCase class will figure out based on the class naming convention which controller class it needs to mock up. Secondly, the domain class get method doesn't seem to work with non-standard identifier types at the moment (i.e. anything other than a Long). However, this unit testing support is intended to get rolled into Grails 1.1 so it should improve rapidly.
This is only part of what the testing plugin can do. Support for unit testing domain class constraints, etc. is also included. I've also been working on a TagLibUnitTestCase that works along the same lines as ControllerUnitTestCase.
Subscribe to:
Posts (Atom)
