Viewing Liquibase's Generated Changelog At Runtime

I know grails 2.x has a migrations plugin, but if you still using / prefer the liquibase one then read on...

I use liquibase in all environments "north" of development to manage the database schema and reference data, but still use create-drop while developing locally. It can be a pain to construct changesets by hand, but it's also awkward to run grails generate-changelog (especially if development normally uses an in memory database).  Much better if you could see the generated changelog while the application is running. Here's how...

import liquibase.database.*
import liquibase.diff.*
import java.sql.Connection

class LiquibaseController {

    def dataSource
   
    def index() {
        Connection connection = dataSource.connection
       
        DatabaseFactory factory = DatabaseFactory.getInstance()
        Database database = factory.findCorrectDatabaseImplementation(connection)
        database.setDefaultSchemaName('schemaName')
       
        Diff diff = new Diff(database, 'schemaName')
        DiffResult diffResult = diff.compare()
        diffResult.printChangeLog(new PrintStream(response.outputStream), database)
               
        response.contentType = 'text/xml'
        response.outputStream.flush()
    } 
}

S.

Techie Books Suggestions

(Or "How I Learned To Stop Worrying About Gramatically Correct Blog Titles")

I'm starting to add to the dev side of my techie library which, as you can appreciate, is pretty biased toward sysadmin stuff at the moment (DNS, firewall, exim, Jabber and so on).

So far I've added the two GinAs, and recently jQuery in Action, Javascript: The Good Bits, Javascript: The Definitive Guide and The Java Developer's Guide to Eclipse.

So what else should I have on my wish list? Freely downloadable is good too - it doesn't have to be expensive dead-tree format - especially if it can be Kindleised.

Would love to hear your thoughts and suggestions.

Remote Controlled Geb Functional Tests

I've just started on a new project where Geb and Spock are our main functional testing tools. They're nice tools to work with but sometimes it's hard to write tests quickly when you can't see what's going on. I want to be able to step through tests like in Selenium IDE but without all the shortcomings of using fixture controllers.

One way to develop functional tests under Grails is to keep the app running while you launch tests from another jvm and use -baseUrl to target the remote jvm. You can do this from the command line or use the Functional Test Development plugin. Another way is to keep the app running and run the tests from within an IDE like IntelliJ; a bonus is that you can run individual tests/features which is great if you have a fairly large spec.

Whichever way you choose, you're going to need to setup data on the target jvm. On my last project we used fixture controllers that we would call from our functional tests but these grew into thousands of lines because re-use after a while became difficult. A better way to do it is to bundle the test data with the actual test which is what the Remote Control plugin allows us to do - the plugin documentation has a better explanation of what I mean.

With all that said, what I want to be able to do is:

• Run my app up
  
• Create my data in my setup block as though I was in an integration test.
  
• Right click on the test from intellij and hit run (or use debug and step through line by line)

Take a look at the plugin docs, it's pretty straight forward but we had to override getBaseUrl and create our own BuildSettingsHolder because remote control doesn't like running outside of a grails command line. So our parent Spec looked like:

public ParentSpec extends GebSpec {
  String getBaseUrl() {
       "http://localhost:8080/monkeytails/"
  }

   def setupSpec() {

       if (BuildSettingsHolder.getSettings() == null) {
           BuildSettings bs = new BuildSettings()
           bs.metaClass.getFunctionalTestBaseUrl = { getBaseUrl() }

           BuildSettingsHolder.setSettings(bs)
       }
   }

   def cleanupSpec() {
       resetDatabase()
   }

   void resetDatabase() {
       def remote = new RemoteControl()
       remote {
           User.list().each{ it.delete(flush:true) }
           // Other teardown stuff goes here
           // ...
           return true
       }
  }
}

and then your tests:

    def "Test something"() {

      setup:
          def remote = new RemoteControl()
          remote {
              User user = new User()
              user.save(flush: true)
              return true //have to return something serializable
          }
      .
      .
  }

It's handy to have firebug in the firefox that webdriver opens so use the profile manager to create a profile called test and then change your IntelliJ JUnit Run configuration VM Parameters to include '-Dgeb.driver=firefox -Dwebdriver.firefox.profile=test'

Now you when you right click and run your test, IntelliJ will first compile everything and hopefully run your test. I say hopefully because I have a bunch of inline plugins so I have to exclude them from the compile path and hope for the best. Setting breakpoints and choosing debug lets you step through your test and let you can use firebug within the browser.

There's a few things to be aware of. Groovy's property setters don't seem to work so you can't do stuff like user.username, instead you have to use the setter or if there isn't one you can use setProperty(). Spread operators don't work too well either, so you can't do User.list()*.delete().

Update: forgot to mention that you should run the app in test mode or Remote Control will not set up its listener unlesss you set remoteControl.enabled to true in your app config

Testing cron expressions in Grails Jobs

One big problem I have is I can never remember what the cron expression is supposed to be and once I've got it in the code if it is actually correct.

Very simple test below. I create a CronTrigger and pass in the expression from the job. I then check that it will fire by passing in a Calendar object. The only key here is you can't test something in the past hence why I am adding another day to the LocalDate object.

void testJobSchedule() {

def trigger = new CronTrigger("name", "group", JobName.cronExpression)

assertTrue "job should fire at 1:15", trigger.willFireOn(time(1, 15))

}

private Calendar time(int hour, int minute) {

new LocalDate().plusDays(1).toDateTime(new LocalTime(hour, minute, 0, 0)).toCalendar(Locale.UK)

}

Hopefully this helps someone out.

Unexpected search results

Bit of friday fun - just tried searching for Node GString, and Google helpfully suggested...

Did you mean: nude g string

Using Liquibase DropAll Automagically 2

A while back gus posted about how to get liquibase to drop all on application start up. He had to jump through some hoops because dependsOn wasn't working properly. This seems to be OK now so I think the process can be simplified a bit...

Assuming the changelog is specified in grails-app/conf/liquibase/master.xml

resources.groovy

if (config.liquibase.on) {
   liquibaseDropAll(LiquibaseDropAll) { bean ->
      dataSource = dataSource
      changeLog = "classpath:liquibase/master.xml"
      bean.initMethod = 'init'
   }

   liquibase(SpringLiquibase) { bean ->
      dataSource = dataSource
      changeLog = "classpath:liquibase/master.xml"
      bean.dependsOn = ['liquibaseDropAll']
   }
}

LiquibaseDropAll.groovy

import liquibase.spring.SpringLiquibase
import org.codehaus.groovy.grails.commons.ConfigurationHolder

class LiquibaseDropAll extends SpringLiquibase {

   void init() {
      if (ConfigurationHolder.config.liquibase.dropAll) {
         super.createLiquibase(dataSource.connection).dropAll()
      }
   }

}




Disclaimer: I've only just started using it so it may still thow up some gotchas.

Reducing backend load

I've recently been looking at alternatives to F5 and I remember from this year's QCon a lot of people mentioning Zeus so I thought I'd give them a look.

So far they look promising; being able to run Zeus on your own generic hardware could be a cost saving in the long run. For example we had to buy some spare PSUs out of fear we wouldn't be able to buy them in the future.

One feature I really like the sound of is the "webcache!refresh_time". It smooths out the load to your backend by only sending one request to your appserver while serving the rest from cache. For sites that must have low cache times this makes a lot of sense. If you get 30 requests per second for an item, you only send one request to your backend as it expires from cache.

Turns out F5 also have a similar sounding feature called Fast Cache but for us it would be an additional module (read additional cost).

If anyone has experience with either I'd love to hear their thoughts.

Stubbing g.message

It's easy to mock grails taglibs using gmock and hamcrest. You can do something like...

    ...
    def g

    void setup() {
        g = mock()        
        mock(tagLib).getG().returns(g).stub()
    }

    void someTest() {
        g.message(hasEntry('code', 'foo.bar')).returns('FOO BAR')
        String result
        play {
            result = tagLib.someMethod([:]).toString()
        }
        assert result.contains('FOO BAR')
    }
}

But when you've got a lot of calls to g.message this can become noisy and usually ends up being repeated in multiple tests. One option is to be relax the argument matching and switch to using a stub.

        g.message(instanceOf(Map)).returns('FOO BAR').stub()

This is fine for the tests where you don't really care about the message, but too loose for the ones you do. I'm playing with the an alternative which gives you both stubbing and the option of explicitly asserting g.message was called with the correct arguments. Be warned though it may raise a WTF exception on first glance.

    ...
    def g

    void setup() {
        g = mock()        
        mock(tagLib).getG().returns(g).stub()
        stubMessages(g)
    }

    void someTest() {
        String result
        play {
            result = tagLib.someMethod([:]).toString()
        }
        assert result.contains('code:foo.bar')
    }
}

class MessageMatcher extends BaseMatcher {

    String code = ''

    static void stubMessages(def g) {
        MessageMatcher matcher = new MessageMatcher()
        g.message(matcher).returns(matcher).stub()
    }

    boolean matches(Object o) {
        code = ((Map) o).containsKey('code') ? o.code : ''
        return true;
    }

    void describeTo(Description description) { }

    String toString() {
        return "code:${code}"
    }
}

The tricky bit was getting the g.message stub to return a value derived from the matcher arguments. I couldn't just do

g.message(matcher).returns(matcher.code).stub()

because matcher.code won't have a value at this point. The solution is to override the matcher's toString() method to return the code and rely on groovy / grails invoking toString() when adding the matcher to the output. This certainly violates the rule of least surprises, but I think I'm OK with that if it reduces duplication and the noise level of my tests - at least until I find a better way.

Clean TagLib Tests

Despite my best efforts I've always found it hard to write clean taglib tests. Now thanks to Spock and GroovyShell things are getting easier...

    void setup() {
        mockDomain Invoice
    }

    def "Attachments icon has correct markup"() {
        given:
        Invoice invoice = new InvoiceBuilder().buildAndSave()

        when:
        renderAttachmentsIcon([target: invoice])

        then:
        valueOf('img.@id') == "toggle-attachments-${invoice.id}"
        valueOf('script') == "\$('#toggle-attachments-${invoice.id}').bind('click', Books.attachments.toggle);"
    }

Because I want to use GPath make assertions about the resulting HTML I've overriden TagLibSpec's methodMissing closure as follows...

    def methodMissing(String name, args) {
        String html = super.methodMissing(name, args)
        createDocument(html)
    }
    
    void createDocument(String html) {
        String xml = "<results>${html}</results>"
        document = new XmlSlurper().parseText(xml)
    }    

And added helper methods for evaluating GPath expressions...

    String valueOf(String gPath) {
        evaluate(gPath).text()
    }

    GPathResult evaluate(String gPath) {
        new GroovyShell(new Binding(document: document)).evaluate("document.${gPath}")
    }

There's a little bit more to this story unfortunately...

Firstly my taglibs use the MarkupBuilder and when run from my Spock test this only outputs the opening tag of the first element I genererate. I haven't had a chance to look into this yet, but a workaround is to add "out << '' to the end of the taglib method

Secondly Spock interactions aren't yet as powerful as gmock, so I usually end up adding code to (g)mock grails taglibs.

Thirdly another one of my tests outputs &nbsp; in the HTML which causes the XML parsing to barf. The solution is to map the &nbsp entity to a known character (in this case underscore).

Finally there was a bug in Grails 1.3.3 / Spock 0.5 which breaks mockDomain. This is reportedly fixed in 1.3.4 and the latest Spock code, but I haven't tried upgrading yet.

Here's how things really look...

@WithGMock
class MetaAttachmentsTagLibSpec extends TagLibSpec {

    def g
    def document

    void setup() {
        g = mock()        
        mock(tagLib).getG().returns(g).stub()
        mockDomain Invoice

        PluginManagerHolder.pluginManager = [hasGrailsPlugin: { String name -> true }] as GrailsPluginManager // Workaround for JIRA GRAILS-6482       
    }

    def cleanup() {
        PluginManagerHolder.pluginManager = null // Workaround for JIRA GRAILS-6482
    }

    def "Attachments icon has correct markup"() {
        given:
        Invoice invoice = new InvoiceBuilder().buildAndSave()
        g.resource(instanceOf(Map)).returns '/foo.jpg'

        when:
        renderAttachmentsIcon([target: invoice])

        then:
        valueOf('img.@id') == "toggle-attachments-${invoice.id}"
        valueOf('img.@src') == "/foo.jpg"
        valueOf('script') == "\$('#toggle-attachments-${invoice.id}').bind('click', Books.attachments.toggle);"
    }
    
    def methodMissing(String name, args) {
        String html
        play {
            html = super.methodMissing(name, args)
        }
        createDocument(html)
    }

    def createDocument(String html) {
        String xml = """<!DOCTYPE html [<!ENTITY nbsp "_">]>\n<results>${html}</results>"""
        document = new XmlSlurper().parseText(xml)
    }
}

And the method under test...

    def renderAttachmentsIcon = { Map attrs, def body ->
        String targetId = attrs.target.id
        String iconId = "toggle-attachments-${targetId}"
        String imgSrc = g.resource(dir:'/images/skin', file:'paperclip.png')

        MarkupBuilder builder = new MarkupBuilder(out)
        builder.img(id: iconId, src: imgSrc)
        builder.script(type:'text/javascript') {
            mkp.yield "\$('#${iconId}').bind('click', Books.attachments.toggle);"
        }
        out << '' // flush for unit tests        
    }    

Internet Explorer and the 1 item remaining bug

We've started using Selenium 2 / WebDriver to soak test our application, and were caught once again by a bug in ie7 and ie8 that randomly causes the browser to hang, waiting for some resource to finish loading, even though the page is visible and works fine.

I hit the problem a year or so ago and solved it by hacking something in prototype.js, but can't remember exactly what. Thankfully there's now a more recent version of prototype (we're stuck on grails 1.1.1 which is bundled with prototype 1.6.0), so I upgraded to prototype 1.6.1 and problem solved :)