My Octopress Blog

A blogging framework for hackers.

Moving the Mysql Data Directory? Beware of AppArmor

Recently, my 8 year old linux server I was running mysql on died. My new computer is more than capable of running mysql in the background, but I didn’t want to mess with moving everything over. Now I’m forced to. It’s extremely easy on Ubuntu to install mysql. However, I run my operating system and all development related applications off of a 64GB SSD. There isn’t room, or a need to run it on the SSD. I have a second platter based hard drive as well. It should be fairly straightforward to store MySql data there. Most instructions for doing so will tell you to do the following:
  1. Copy the current data directory to your new partition. (Current directory is usually /var/lib/mysql on Ubuntu)
  2. Make sure it is still owned by the mysql user. ‘chown -R mysql:mysql newdatadir’
  3. Change the mysql configuration to point to the new directory. Inside of my.cnf is a value datadir, which is what needs to be changed to the new directory.
Sounds easy, right? It is, for the most part. If you try these steps on ubuntu, it won’t work. MySql won’t start up. If you’re like me, the first thing you’ll look is syslog, where you’ll find something like:

[1064644.473299] type=1400 audit(1305502269.192:364):
apparmor="DENIED" operation="open" parent=1 profile="/usr/sbin/mysqld"
name="/media/data1/mysql-data/" pid=24578 comm="mysqld" requested_mask="r"
denied_mask="r"fsuid=1001 ouid=1001
Your first instinct is to check the permissions, but they’re all fine. So, what’s going on? AppArmor is preventing access. If you type ‘ls -la /etc/apparmor.d’ you’ll see the following:
drwxr-xr-x   8 root root  4096 2011-05-15 19:18 ./
drwxr-xr-x 141 root root 12288 2011-05-15 18:20 ../
drwxr-xr-x 3 root root 4096 2011-04-11 18:38 abstractions/
drwxr-xr-x 2 root root 4096 2011-05-15 17:38 cache/
drwxr-xr-x 2 root root 4096 2010-10-07 11:06 disable/
drwxr-xr-x 2 root root 4096 2010-08-06 23:19 force-complain/
-rw-r--r-- 1 root root 986 2010-09-13 03:07 gdm-guest-session
drwxr-xr-x 2 root root 4096 2011-05-15 17:38 local/
-rw-r--r-- 1 root root 2052 2010-08-06 23:18 sbin.dhclient3
drwxr-xr-x 3 root root 4096 2011-04-11 18:38 tunables/
-rw-r--r-- 1 root root 2052 2010-09-27 17:58 usr.bin.evince
-rw-r--r-- 1 root root 4006 2011-04-29 04:00 usr.bin.firefox
-rw-r--r-- 1 root root 4158 2010-10-01 04:58 usr.sbin.cupsd
-rw-r--r-- 1 root root 989 2010-11-10 00:51 usr.sbin.mysqld
-rw-r--r-- 1 root root 1172 2010-08-06 12:45 usr.sbin.tcpdump
If you open up usr.sbin.mysqld, and add the new directory to the list, your problems will be solved. Something like:

 /mysql-data/ r,
/mysql-data/** rwk,

Comments

Chris Caldwell
would NEVER have found this. thank you thank you.
Unknown
I had this problem and wouldn't have been able to solve it without this post, many thanks.
johank
I had the same problem and it boiled down to one silly mistake. The directory where I mounted the disk to hold the MySQL data files didn't have the right permissions. Making it group+world readabe did the trick. I realised my error when starting mysqld "by hand" and actually got to see MySQL complain. With apparmor, I got no logs from MySQL…
Lucas Ward
Glad I could help.
Serge
Hey !
I was in the exact same situation than you.
Thank you very much for this piece of information, i'm pretty unaware of apparmor and wasn't aware of this configuration (i usually put all my application datas files on a separate partition, including the mysql datas).

Tips for Connecting to Github Private Repositories With Hudson/Jenkins

I recently created a private repository on GitHub, and wanted to create a build for it on my Jenkins server. I ran into some interesting problems that I couldn’t find reasonable solutions to via google, so I thought I would write about my experience.

The first and most important thing to keep in mind is that there are no settings in Jenkins to control or manage SSH keys for Git. You might get confused when you see the SSH section in the general settings, thinking you can upload your git RSA key there and it will work. It won’t. The only thing that will work is setting it in the .ssh directory of the user that is running Jenkins. If you’re running it standalone, it will be the Jenkins user. In the case of a standard Ubuntu install, this was in /var/lib/jenkins/.ssh. If you are running Jenkins inside of a tomcat container, then it will be the tomcat user. (tomcat6 generally in Linux)

The second thing I learned is: If you’re running Jenkins on Windows, it’s going to be painful. Mind you, I’m not running it that way, but nearly every single Google result on this issue was coming back with issues for Windows. It’s not really even Jenkins fault either. I’ll stop short of bashing Windows, but if you’ve ever had to deal with running any type of service on Windows, you’re already familiar with the pain anyway.

And the final lesson I learned? You can’t use a passphrase in your RSA key file. I lost hours to this one. I thought I had some other issue with the key. Until I stumbled on this blog post. It makes sense when you think about it. When you’re running git on your machine you might get prompted for access to a keychain, but after that its seamless so it can be easy to forget that its there. If you follow the default instructions for creating the key on github, it will guide you to create the passphrase. Ignore that and follow the advice on the previously mentioned post. Git should work without issue after that. I would still recommend using a passphrase for your personal account though.

Comments

Inori
For Windows, this blog helped me: http://element34.ca/blog/jenkins-remote-windows-nodes-and-private-github-repos

Grails Envers Plugin

***Update - 4/11/2013***

I finally got around to updating and publishing this plugin: http://www.lucasward.net/2013/04/grails-envers-plugin-update.html



I recently needed to add auditing to my grails application and decided to use envers, mostly for its ability to keep updates to multiple hibernate objects tied together in a way that is query-able afterwards. Unfortunately, because Grails exerts somewhat tight control of Hibernate, and because of how Envers was implemented, getting it to work with grails can be a bit tricky. I did a cursory glance around and couldn’t find anyone that had created a plugin to help with Envers and Grails. So while implementing it in my application, I wrote the envers specific code as a plugin, which I have published on github:

https://github.com/lucaslward/grails-envers-plugin

Eventually, I need to write actual documentation for it, and try and get it in the official grails plugin repository. However, for now, I’ll write up how to use it based on the integration tests.

If you’re pulling it down from the git repo above, you need only do a simple ‘grails package-plugin’, which will create a zip file, which can be then be installed into your grails application via: ‘grails install-plugin /path/to/zip/grails-envers-plugin-0.1.x.zip’. The plugin sets up envers via Hibernate event listeners, and provides gorm style dynamic methods that can be used to query for revisions.

In order to configure your domain objects to use Envers, you need to annotate them with @Audited. (I haven’t written any code to try and make that work with static members a la gorm) I’ll use the sample domain from the plugin:
@Audited
class Customer {

String email
String name
Address address
SortedSet orders = new TreeSet()

static constraints = {
address (blank: true, nullable: true)
address component: true
}

static mapping = {
orders cascade: "all,delete-orphan"
}
}

@Audited
class Address {
String city
String zip
}

@Audited
class OrderEntry implements Comparable{

Date date
double amount
int numberOfItems
Customer customer

static belongTo = [customer:Customer]

@Override
int compareTo(OrderEntry o) {
date.compareTo(o.date)
}
}
In this example, I have three domain classes: Customer, Address, and OrderEntry. Customer in this class is the main class, who has a one to one relation ship with an Address, and a collection of Orders. Now, let’s assume that we create a customer with an address, then modify the customer and the address twice in separate transactions:

Customer customer
Customer.withTransaction {
def address = new Address(city: "Chicago", zip: "60640")
address.save()
customer = new Customer(name: "PureGorm", email: "tester@gorm.org", address: address)
customer.save(flush: true)
}

Customer.withTransaction {
customer = Customer.findByName("PureGorm")
customer.email = "tester2@gorm.org"
customer.address.city = "New York"
customer.save(flush: true)
}

Customer.withTransaction {
customer = Customer.findByName("PureGorm")
customer.email = "tester3@gorm.org"
customer.address.zip = "10003"
customer.save(flush: true)
}

Note: If you’re trying to test envers in your application you have to turn off transactions for your test and wrap individual calls in transactions as above. (with an appropriate tear-down to clean up afterwards of course)

In this scenario, we should have 3 entries in the audit tables for Customer and Address (customer_aud and address_aud respectively). We can query for this methods with the findAllRevisions dynamic method:

def results = Customer.findAllRevisions()

assert results.size() == 3
def r = results[0]
assert r.name == "PureGorm"
assert r.email == "tester@gorm.org"
assert r.address.city == "Chicago"
assert r.address.zip == "60640"
assert r.revisionType == RevisionType.ADD
r = results[1]
assert r.email == "tester2@gorm.org"
assert r.address.city == "New York"
assert r.revisionType == RevisionType.MOD
r = results[2]
assert r.email == "tester3@gorm.org"
assert r.address.zip == "10003"
assert r.revisionType == RevisionType.MOD


As you can see, findAllRevisions returns an array of three results. The plugin will automatically take the envers RevisionEntity and RevisionType, and set them on the returned revisions, which is how revisionType is being checked. As you can see, all three revisions are present. By default its sorted by the earliest revision first. First is an add, followed by two ‘mods’ (update). The revision entity is also accessible via the revisionEntity property: r.revisionEntity.getRevisionDate(). It’s also worth noting that even a custom envers RevisionEntity will be accessible this way. (See the envers documentation for more details on how to do this)

findAllRevisions also supports sorting in the same way as gorm queries. Assuming the same audit trail as above, you could query it this way:

def results = Customer.findAllRevisions(sort:"email",order:"asc")

assert results.size() == 3
assert results[0].email == "tester2@gorm.org"
UserRevisionEntity entity = results[0].revisionEntity
assert entity.getUserId() == currentUser.id
assert results[0].revisionEntity.getUserId() == currentUser.id
assert results[1].email == "tester3@gorm.org"
assert results[2].email == "tester@gorm.org"


You still get back 3 results. But this time the first result is the first update, followed by the second, following by the initial create. This is because its being sorted by email ascending.

As with sorting, you can also use max and offset:

    
def results = Customer.findAllRevisions(max:1)

assert results.size() == 1
assert results[0].email == "tester@gorm.org"

results = Customer.findAllRevisions(max:1,offset:1)

assert results[0].email == "tester2@gorm.org"

results = Customer.findAllRevisions(max:1,offset:2)

assert results[0].email == "tester3@gorm.org"


Besides sorting and paginating, you can also search for revision by property name:

Customer.findAllRevisionsById(customer.id)

Customer.findAllRevisionsByAddress(customer.address)

Customer.findAllRevisionsByName("PureGorm",[sort:"email",order:"asc", max:2])


The first case above is searching by id, and requires a long. The second is searching by the address, which requires an attached hibernate entity. The final is searching by a simple property name, in this case: “PureGorm”, with some additional sorting, etc tacked on. As with gorm, you can use any property name on the object to query by. At this point, the plugin only supports querying by one property. If you need to query by more, you will need to use the Envers classes directly. (i.e. AuditReader)

The final method available statically is getCurrentRevision():

Customer.getCurrentRevision()


Which gets the current revision number. Meaning, what is the last revision of any customer?

There are also two methods that are applicable on domain class instances as well:

     
List revisions = customer.getRevisions()
assert revisions != null
assert revisions.size() == 3

Customer oldCustomer = customer.findAtRevision(revisions[1])
assert oldCustomer.email == "tester2@gorm.org"
assert oldCustomer.address.city == "New York"


The first method will return all revisions of a particular class. (The same as Customer.findAllRevisionById(customer.id)) And the second will find a particular customer at a particular revision.

There is certainly some functionality I’m missing here, as I’m using the integration tests as an example. I will add more blog entries on the subject as I discover the corner cases I have missed. Hopefully, I can also get time to write actual documentation, and get the plugin in the official repository as well.

Comments

bdbull
Has anyone come up with an efficient way to add a "revision" property to the domain class? It seems a little involved if I want to actually attach the revision to the domain class itself.
Prashant Potluri
I am using @Audited on getter methods instead of at class level for auditing specific properties only. However i am unable to use any of the dynamic finder methods findAllRevisionsById() etc. It says cannot resolve property revision. If i annotate the class @Audited, it works.
Is there a work around for this without annotating the class?
Prashant Potluri
There is @Transactional on controller. Also tried the programmatic way using Entity.withTransaction.
I still don't see any data written to audit tables. DataSource.groovy contains multiple data sources:

development {
dataSource {
dbCreate = "update"
url = "jdbc:mysql://localhost/dev"
username = "xxxxx"
password = "xxxx"
}
dataSource_audit {
dbCreate = "update"
url = "jdbc:mysql://localhost/audit"
username = "xxxx"
password = "xxxx"
}
}
Lucas Ward
Usually when it's not pushing to the AUD tables it's a transactional issue, because it only pushes those entries on transactional boundaries.
Prashant Potluri
If there are multiple data sources defined in DataSource.groovy, the plugin is not saving any data in the audit tables. Any suggestions?
Tomasz Kalkosiński
Hi Lucas

I wrote an interesting blog post about working with Grails 2.1 and Hibernate Envers together. It'd be great if you can read, comment and spread a word about my work. My post can be found here:

http://refaktor.blogspot.com/2012/08/hibernate-envers-with-grails-210.html

Greetings,
Tomasz Kalkosiński
Manuel B.
Jay Hogan has forked the repository and upgrade envers plugin for grails 2.0

URL: https://github.com/jayhogan/grails-envers-plugin

Regards
Lucas Ward
korsbecker,

Getting the plugin to work with Grails 2.0 is a whole other problem. Grails 2 jumped the hibernate version up to 3.6 which is something like 4 years worth of Hibernate. I haven't looked into that at all yet (but probably will in 3 or 4 months) but there's no telling what issues might pop up. The error you have looks more related to issues cause by this upgrade than from anything else.
korsbecker
Smita,

Did you solve your problem with

Caused by ClassCastException: org.codehaus.groovy.grails.orm.hibernate.cfg.GrailsAnnotationConfiguration cannot be cast to org.hibernate.cfg.AnnotationConfiguration

I have exact the same problem with Grails 2.0. I have upgraded the plugin to to grails 2.0 and change to Hibernate 3.6.7.Final and explicitly added a runtime for hibernate-envers:3.6.7.Final

But still with the same problem??

cheers
henrik
Lucas Ward
Sorry for the delay in responding to some of these. I took a long vacation and have just now gotten back.

@Smita - The plugin definitely won't work below grails 1.3, and even that can get tricky at times. The problem is that Envers was integrated with hibernate a couple of years ago. Grails is really behind on hibernate releases (for obvious reasons). It will all be much more straightforward with Grails 2.0 I think.

@Gmacmullin, I'm glad it worked out for you. I really need to get this thing into the official plugin repo…
gmacmullin
Looks like I had a typo where I declared the Envers plugin as a compile plugin dependency. It works great now.

Thanks for creating the plugin Lucas!

Glen
gmacmullin
I've been using the Envers plugin with Graisl 1.3.7, but I have to download the source code and upgrade it to Grails 1.3.7

The weird thing is the plugin works perfectly when I run "grails run-app", but when I deploy it as a war file in Tomcat, the _aud tables are not created. I turned on logging for Envers and the plugin, but I don't see any errors/problems.

Lucas, have you had any problems running the plugin in a WAR file?

Thanks,
Glen
Smita Saraswat
Anyone as run this plug in with the latest version of grails ?
Thanks.
Smita Saraswat
Lucas,

I am running grails 2.0M2 version and when I installed your plugin, I get the following error :

Caused by ClassCastException: org.codehaus.groovy.grails.orm.hibernate.cfg.GrailsAnnotationConfiguration cannot be cast to org.hibernate.cfg.AnnotationConfiguration

Thanks.
Lucas Ward
@euvitudo

Transactions in Grails are a bit weird anyway, but yes, nothing will be written to the audit tables until a commit. That's because a common revision ties together all the work done in a single transaction. Of course, if you do this work in a service, that method is transactional, you can use the @transactional annotation as you already mentioned, and there is a transactional controller plugin that let's you set which methods on a controller should be transactional. (which is what I use in my app)
euvitudo
Regarding the transactional context:

I've been experimenting with your plugin (thanks for the plugin, btw), and have only simplistic examples using grails scaffolding. I found that after adding @Transactional (org.springframework.transaction.annotation.Transactional) to the controllers, updates to the audit tables are committed.

The drawback to this method (at least, while using scaffolding) is that all methods will be transactional.

Annotating a service or service method in this way will do the same.
morungos
Thanks Lucas. I'd be OK doing that if I was a little more Groovy-savvy. Couldn't find much in the way of annotation at anything other than the class level. I was about 2/3 way through improvising auditing with Groovy MOP hacks, an event listener, and a static field, so I might for now go that way, but I'll certainly track this plugin, which appears to be much better for the long term.
Lucas Ward
@morungus

The plugin doesn't actually address configuration, and relies on envers itself. As I mentioned in a previous comment, I would love to do more gorm style mappings, but will likely wait until grails 1.4.

This problem is kind of tricky for groovy in general (and why I suspect grails uses static fields). There's a few posts you can find on the topic, but because groovy generates your getters and setters, etc, annotations on a field don't do what you expect (or in this case what envers expects). Your best bet is to create the getters and setters yourself, and annotate them. It's still tricky, but that's your best bet for individual field annotations on gorm classes. Of course, it will work straight away on java classes configured with hibernate xml or entity annotations. Although, I'm sure that's not a great option for many. You can test this yourself by annotating your groovy classes and using reflection in java to see if you can get at them.
morungos
This is almost ideal, but I need to be selective about which fields to track for audit. I tried the plugin but it always seemed to cough on a relation to a non-audited object. is this known to work?
Lucas Ward
@Fabio

Sorry for the delayed response.

Regarding the UserRevisionEntity. It's really tricky. The weirdest thing is it will only work if you make it a java class and register it directly with hibernate. If you look at the plugin code, you'll see a hibernate config file,where the revision entity is the only configured class. I have no idea why @Audit works fine on gorm classes, but @RevisionEntity doesn't. The listener via the annotation is also really silly of them. It prevents the listener from wiring it up. Once the plugin is released completely, and seems to be stable, my next big release will likely be to write my own configuration for it, probably using a combination of normal plugin config in config.groovy and static methods a la gorm on domain classes. But I probably won't even try that until grails 1.4, since envers changes to a first class citizen in hibernate 3.5. (1.4 will go to 3.6 I believe)

Oh, and it isn't necessary to use Spring Security. That's just what I use for my app at work, but really whatever you're using for authentication will work.
Fábio Miranda
Hi Lucas!

Nice post and nice plugin!

The only thing that is not clear is how to configure UserRevisionEntity to track the user that performed changes to the database.

First, only 'rev' and 'revtstmp' columns were created at the database.

Second, even after I found SpringSecurityRevisionListener, SpringSecurityServiceHolder and StubSpringSecurityService in the sources, it's not clear how to use them.

Third, is it necessary to integrate with a particular technology (Spring Security)?

Thanks!
Fábio Miranda.
daniel
Lucas,
that was the problem. When using a grails service, everything works as expected.

Thanks a lot for your help and the plugin ;-)

Daniel
Lucas Ward
@daniel

I'm assuming by empty _aud tables you mean that when doing an update to a database table, you aren't seeing a corresponding entry in the corresponding audit table?

My first thought is that it has to be something to do with transactions. For example, I have seen this happen with MySql myisam tables. You can write to them without transactions, and since Envers is tied completely to transactional hooks, it will never be called. I'm will to be if you breakpointed the Envers AuditEventListener class, which is what is actually listening to Hibernate events. If you replicate the tests I have in the blog post, using .withTransaction around your calls, you should see the entries. If that works in an integration test, then you need to probably take a look at your Grails controllers. Remember, by default they are not transactional.
Lucas Ward
@DM Sorry about that, I guess I left out the imports when I cut and pasted.
daniel
Hi Lucas,
I was recently integrating envers myself into Grails, but had a problem with empty _aud tables. This way I stumbled upon your plugin. Trying it with your plugin unfortunately yields the same result (empty _aud tables). I upgraded your plugin to Grails version 1.3.7 though, but I don't think this is causing the problem.

Any hint for me what is going wrong?

Cheers,
Daniel
DM
Okay - sorted. Just need to add:
"import org.hibernate.envers.Audited".
DM
G'day Lucas
Using eclipse (STS) and running through the blog, after installing the plugin, and attempting to implement the Customer class, the @Audited annotation is marked as an error … is there some extra configuration I need to do?

When Working With Ext JS, Pay Attention to the Cookies

We use Ext Js as our main Javascript library at Fundspire. There’s a couple of really cool features we use a lot. One is the GridPanel. It gives you a nice looking ‘table’ with sorting, column resizing, etc. There are also some interesting ways to make multiple ‘panes’ in your application. One example being the border layout. (The layout browser shows all the layouts pretty well) A good example is the official ‘forum example’ from Ext Js: http://dev.sencha.com/deploy/dev/examples/forum/forum.html I’m still relatively new to the framework, but have been able to do some interesting things with it. After a couple more months I hope to write up a more detailed review.

I’ve been working with a lot of these components this week and was having a lot of trouble manipulating the column and border size defaults. My changes just didn’t seem to have an impact. We use jawr, which renames the javascript when there’s changes to force a reload, so that wasn’t the issue. It turns out that the border and column size information is stored in a cookie on the user’s browser! It’s something that’s easily setup via an Ext Js CookieProvider Once you have a cookie provider setup, your state across a lot of components is saved automatically. Which, if you think about it is pretty slick. If someone is using a smaller monitor they can change the sizes to be what they want and not have to worry about it again. However, I went down a lot of rabbit holes when my changes didn’t appear to be having an affect, and I haven’t seen this behavior documented anywhere in the Api docs for the components I’m using. I knew we were saving state in cookies, but didn’t realize that it included these components automatically. So, if you’re using Ext Js and your layout changes don’t appear to be having an effect, make sure that it’s not storing a cookie to save that information.

Accessing the Identifier of a Lazy Loaded Association in Grails Without Another Database Call

I ran into a bit of a weird scenario recently with grails. I have a couple of classes similar to those below:

class Foo {
Bar bar
}

class Bar {
static hasMany = [foos: Foo]
}


Essentially, I have a many to one relationship. Now, consider the following code:

Foo foo = Foo.get(1)
Bar bar = foo.bar


Assuming that there is actually a bar set on this particular foo, what will you get?

You’ll get the actual Bar, completely instantiated. Now, imagine the exact same scenario in normal java with the exact same equivalent hibernate settings via annotations. (I’m too lazy to model what that would look like) What would you get back for a similar call? You would get back a proxy! Then, if you called something on the proxy, hibernate would go and fetch the actual object for you.

Gorm unwraps the proxy when the object itself is called via a property.

If you take a look at the HibernatePluginSupport class, which configures all of the dynamic Gorm methods (and the first place I go when I want to see how a particular method is actually working) you will see this in action:

static final LAZY_PROPERTY_HANDLER = { String propertyName ->
def propertyValue = PropertyUtils.getProperty(delegate, propertyName)
if (propertyValue instanceof HibernateProxy) {
return GrailsHibernateUtil.unwrapProxy(propertyValue)
}
return propertyValue
}

/**
* This method overrides a getter on a property that is a Hibernate proxy
* in order to make sure the initialized object is returned hence avoiding
* Hibernate proxy hell
*/
static void handleLazyProxy(GrailsDomainClass domainClass,
GrailsDomainClassProperty property) {
String propertyName = property.name
def getterName = GrailsClassUtils.getGetterName(propertyName)
def setterName = GrailsClassUtils.getSetterName(propertyName)
domainClass.metaClass."${getterName}" =
LAZY_PROPERTY_HANDLER.curry(propertyName)
domainClass.metaClass."${setterName}" = {
PropertyUtils.setProperty(delegate, propertyName, it)
}

for (GrailsDomainClass sub in domainClass.subClasses) {
handleLazyProxy(sub, sub.getPropertyByName(property.name))
}
}
So, anytime you access your properties, the above code is going to unwrap it.

What whoever wrote this is trying to avoid is the ‘proxy hell’ mentioned in the comment. The hibernate docs have a good explanation of it:

http://docs.jboss.org/hibernate/core/3.3/reference/en/html/performance.html#performance-fetching-lazy

As does the Grails docs:

http://grails.org/doc/latest/guide/5.%20Object%20Relational%20Mapping%20%28GORM%29.html#5.5.2.8%20Eager%20and%20Lazy%20Fetching

They’re more eagerly fetching the lazy relationships than Hibernate does to avoid some of the instanceof check issues outlined in the above links.

Now, back to my issue. What happens if you want to obtain the identifier of your relationship without actually resulting in a database call to get it? The proxy has the identifier already, that’s how it loads it up when you call it. In pure hibernate there’s a few ways to do it:
  1. Session.getIdentifier
  2. entity.getId() (assuming you are using property based configuration)
  3. Cast it to a HibernateProxy, get the LazyLoadInitializer and get the Identifier from it
The problem with all of these solutions is that you have to get around the Gorm code above that unwraps any properties when you call a getter. You can’t just add setBar and getBar. However, you can add another method that can get ‘raw’ access to the field. And before anyone posts a comment about it you can’t use load either because it requires the id, and that’s the whole point of this exercise (getting the id that is).

I didn’t find this out until I had wasted quite a few hours, but apparently this was a known issue: http://jira.grails.org/browse/GRAILS-2570 I never saw this when googling for it originally, but you can essentially say:

foo.barId


And you will get the id of the bar without the proxy being unwrapped. I’m not sure this is referenced anywhere in the docs though.

Comments

timbonicus
Thanks for putting this information out there, you saved me a few hours of frustration!
Manuel
Great post, thanks! Already bookmarked this for future reference!
Amit
Thanks for posting this!

New Desktop

In my new gig at Fundspire, I generally work from home. For awhile this has been on my 4 year-old Macbook Pro. However, during my time at ThoughtWorks, I got extremely accustomed to using extremely powerful desktop machines running Ubuntu. In fact, I think the ones I used on my last project were running two xeon processors. Regardless, rather than get a new laptop, since I’m working from home I decided to build a new desktop. I’ve built a number of machines over the years, and it’s always been a favorite hobby of mine. I enjoy spending time getting back up to speed with what all the major chipsets are, and the actual build can be a lot of fun too. Below, I’m going to list out everything I bought and why, with a total at the end. All of the prices and links are to NewEgg. I’ve been using them for years and have always been happy with their prices, selection and service.

Case

There’s two areas I never scrimp on when building a new computer: The motherboard and the case. The case can make the build and future upgrades to the machine simple, or a nightmare. When looking for a case, three things are important to me:
  1. Cable management. Besides the whole airflow thing, it makes it a lot easier to work inside of the case, or do simple upgrades when the cables aren’t in a huge jumble in front of the motherboard. I can’t count the number of times I accidentally knocked a cable loose and didn’t realize it.
  2. Sound proofing. Maybe I’ve just been using laptops too long, but I’m really over having a tower that sounds like a jet taking off when I turn it on.
  3. Lots of space for additional drives, etc.
I have no intentions of overclocking, so I’m not particularly worried if it has things like water cooler hookups or whatever. And I really don’t want a light show in my office. So, I went with a full tower by Fractal Design.

CPU

Since the CPU really determines the motherboard nowadays, I had to decide that next. However, since the new Intel Sandy Bridge processors came out a couple of months ago, it was a no-brainer. I went with the Core i7 Sandy Bridge 2600K. The K in this instance means the chip has an unlocked modifier, which is why all the overclockers you see tend to get either it or the i5. And while I’m not going to overclock it, there was a combo deal on my motherboard for the K version, which would normally be $20 more than the regular 2600 variant, making it the same price, so why not?

Motherboard

This was the most difficult spot for me. As I said above, it’s probably the most important piece in the system, and completely affects how you can upgrade in the future, so it’s important. The first thing I noticed when getting back into the latest hardware was how much it seems like every generation of new processors from either Intel or even AMD seems to have a completely different socket. Maybe they’re more interchangeable than I think, but it at least seems pretty specific. For instance, take a look at this wikipedia page: http://en.wikipedia.org/wiki/List_of_Intel_Core_i7_microprocessors. Here’s the sockets of all the different Intel i7 chips:

Lynnfield: LGA1156
Bloomfield/Gulftown: LGA 1366
Sandybridge: LGA1155

Obviously, I wanted to go with an LGA1155 socket motherboard, since that’s the chip I got, but it’s really easy to get confused. There’s also a noticeable difference in the motherboards for Bloomfield processors. Most of the them are triple channel. While all the newer Sandy Bridge are all dual channel. It’s a bit confusing at first, but SB represented a pretty big change in the memory controller. Namely, it’s on the processor itself now and not the northbridge. That being the case, the only option for motherboards is whatever the chip has, which is dual channel. Also, from the numbers I’ve read, it seems that SB chipsets running 1333 DDR3 have higher memory thoroughputs than the previous generation triple channel setups, so it seems to work out.

I went with Gigabyte for my motherboard: http://www.newegg.com/Product/Product.aspx?Item=N82E16813128478

I really just wanted something that would support a lot of memory for the future, (This one goes up to 32 GB) SATA III (6.0 GBs) and USB 3.0 ports. I also tend to favor the ‘enthusiast’ motherboards a bit more, as their tendency to be used in overclocking setups means they have gratuitous amounts of heatsinks on all the chips. Having overheated more than one motherboard in the past, it’s nice to have that even if you’re not going to overclock.

Video Card

I’m going to be running Ubuntu, so the video card selection is extremely important. Video cards and linux have always had a checkered past, and even in 2011 you can easily pick up a card that doesn’t work right with a particular kernel or something. (It happened to me last year with a Fedora setup I had) Personally, I think the only option is an NVidia setup. I used to use Radeons exclusively, but starting with something like the Geforce 8xxx chipsets, Nvidia added this thing called PureVideo. This has support for VDPAU (Video Decode and Presentation API for Unix) Which allows *nix machines to use hardware decoding of video on the video cards themselves. With that in mind, I bought a Geforce GT 430. It was introduced last fall, is extremely cheap and focuses on being able to push video, which is all I want. I watch mlb.tv on another screen while coding sometimes, without hardware video decoding, it would take up a lot of resources.

Everything else is pretty straightforward.

Powersupply: OCZ ModXStream Pro 700W You can now get ‘modular’ PSUs, which means that you only have the cables you plug into the unit, and don’t have a tangle of wires for all the connections you’re not currently using.

SSD: Crucial RealSSD 64GB Who in their right mind would build a system today and not use an SSD? Especially if you’re a programmer. I went with 64 Gb to save money. It’s enough to run ubuntu on the SSD, and any code projects. For everything else, I picked up a normal platter drive for $40.

Ram: 4 x 4GB DDR3 PC1333 (I did a quantity of 2 on the setup I linked to for a total of 16 GB) Most of the nicer SandyBridge setups support faster ram, but since the sandy bridge architecture is at 1333, I’m assuming they’re getting those speeds by overclocking, and it wasn’t something I wanted to deal with.

Everything else is pretty normal, I picked up a new DVD drive for $20, mostly because I needed one with a SATA hookup. (All my existing ones are PATA) I also picked up a heatsink, thermal paste, etc as well. They’re generally cheap and worth it to make sure you don’t fry your cpu. With all of that, the total including shipping was: $1,337.62

It’s not too bad for what is essentially a top of the line system. The only way to go higher is to get into server grade territory, with two CPUs and 12 ram slots. You could reduce the cost on the setup I have by going i5 and halving the ram, but that only saves you $200, and as a programmer, most of my work is CPU and Ram bound so I think it’s money well spent. You can also go cheaper on the motherboard and the case, but you’ll always regret it. Of course, if you also want to play games on the computer, you’ll probably want to dump way more money than I did in a video card.

Excluding Jars From a Grails Generated War File

Yesterday, I posted about how to upgrade hibernate versions in grails. This solution worked great for me locally. (i.e. using run-app) But we were still getting weird issues in our staging environment. It’s not uncommon for this kind of thing to happen, and I used my local tomcat setup to try and nail down the problem. When I cracked open the WAR, I found two version of hibernate, 3.3.1 and 3.3.2. Upon closer inspection of the problem, it turned out to be the same original issue I had upgraded Hibernate to fix, somewhat hidden by some additional ‘noise’ in the log file. I tried about 10 different things to get the war to generate correctly, until I stumbled onto this blog post:

http://www.anyware.co.uk/2005/2009/01/21/excluding-files-from-a-war-with-grails-the-right-way/

Grails gives you the option to declare a closure to modify the war resources. I added the following to the top of my buildConfig.groovy, and it fixed the issue:


grails.war.resources = { stagingDir ->
delete(file:"${stagingDir}/WEB-INF/lib/hibernate-core-3.3.1.GA.jar")
}

Upgrading Grails 1.3.x to Use Hibernate 3.3.2

While using Hibernate Envers, I ran into an issue that was fixed in Hibernate 3.3.2. Unfortunately, Grails is using Hibernate 3.3.1. Luckily, it’s fairly straightforward to exclude and declare a new dependency in your BuildConfig.groovy:


inherits("global") {
excludes 'hibernate'
}
dependencies {
compile ('org.hibernate:hibernate-core:3.3.2.GA'){
excludes 'ehcache', 'xml-apis', 'commons-logging'
}
}
Update: turns out it’s not as straightforward as I thought. The above will work fine with ‘grails run-app’ but the 3.3.1 version will still be in the War file: http://www.lucasward.net/2011/03/excluding-jars-from-grails-generated.html You can fix that by adding the following to your BuildConfig.groovy:


grails.war.resources = { stagingDir ->
delete(file:"${stagingDir}/WEB-INF/lib/hibernate-core-3.3.1.GA.jar")
}


Unfortunately, in the 3.3.2 release, ““org.hibernate.dialect.DialectFactory” was moved to “org.hibernate.dialect.resolve.DialectFactory”. Which was part of the following issue: http://opensource.atlassian.com/projects/hibernate/browse/HHH-2933

This is apparently an issue many have been seeing: http://grails.1312388.n4.nabble.com/update-hibernate-version-td3002449.html

While it may seem crazy that the Hibernate team moved something to a new package in a point release. I’m pretty sure they consider this class to be internal and not part of their API. The issue the change was made in was to expose to users a way to resolve their own dialects programmatically. Which, if you look at where the Grails Hibernate plugin is creating the offending spring bean, is the same thing Grails is doing:


if (ds && ds.dialect) {
if (ds.dialect instanceof Class) {
hibProps."hibernate.dialect" = ds.dialect.name
}
else {
hibProps."hibernate.dialect" = ds.dialect.toString()
}
}
else {
dialectDetector(HibernateDialectDetectorFactoryBean) {
dataSource = ref("dataSource")
vendorNameDialectMappings = vendorToDialect
}
hibProps."hibernate.dialect" = dialectDetector
}


The above is taken from HibernatePluginSupport.groovy, line 128. (At least in Grails 1.3.6) Essentially they’ve written a Spring bean to programmatically determine the dialect by calling the DialectFactory themselves. You can bypass this issue completely by specifying your dialect, rather than having it detected. If you’re using mysql, it looks something like:


dataSource {
driverClassName = "com.mysql.jdbc.Driver"
dialect = "org.hibernate.dialect.MySQLDialect"
}

Comments

Lucas Ward
I'll see if I can dig for the issue I found that was causing issues in Envers. I also wrote a plugin as well that I haven't had a time to post. I'll be pushing it to github within the next couple of days.
stephen westfall
Can you provide any details on the Envers issue that triggered this upgrade? I'm trying to use Envers on a Grails project and having issues.

The Programmer’s Time Paradox

At the beginning of this week I started work on a new feature. For the purposes of this post it doesn’t matter what the feature was, but I started in on it on Monday and things went extremely well. I was in the zone. It’s hard to describe, but I think if you’re a programmer you’ll understand the feeling. I started out with just the right amount of testing in the right places, built something simple and added on top of it, while refactoring periodically as I discovered new angles on the problem. Everytime I moved forward, I was able to run my tests and tell immediately that my last fix had broken something. I was moving forward in small steps and it was all falling together nicely.

And then I got greedy. I suppose it makes me a dork to admit it, but I was having fun. I had been somewhat behind on the feature because I was debating on a couple different paths, but now that I had chosen, I was moving forward with wreckless abandon. It’s somewhat intoxicating to be in the zone. So, I pushed myself hard. I worked from around 9 AM until 1 or 2 in the morning every night for 3 days straight. (I work from home, so it’s a bit easier to work later like that)

This morning I woke up begrudgingly. The comforter was very, very heavy. It was the weird programmer kind of fatigue, where you’ve done nothing but sit on your but all day, but your mind has been going too fast for too long, so you feel mentally exhausted but physically fine. When you hit that point, no amount of coffee is going to make it better. Now, I’ve been on some pretty hellish death marches before. Some of them spanned months, so it’s not like working hard for 3 days is all that terrible. But the programming gods are vengeful. I might have been tired, but I was excited to get the feature working. I was in the home stretch. My sister is coming to town this weekend and it’s her first time in Chicago since turning 21, so I wanted to get this thing done and party like it was…2003?

So I got sloppy. I ran into a weird bug. It was a NullPointerException with no stack trace. Which sometimes interchanged itself with an InovcationTargetException. When you see something like that happen, it’s time to stop the production line. I should have stopped, immediately, and attempted to recreate the conditions in my testing setup. I had a perfect, isolated, test suite setup where I could have recreated this thing in a controlled environment, but I didn’t do that. Instead, I fiddled with idea after idea. They were all well informed, reasonable things to be trying. But when you’re doing this kind of thing inside your application, there can be too many factors. Production code is complicated, and there’s just too many variables.

At around 6PM I stopped myself, and did what I should have done in the first place. In an integration test, I slowly recreated the problem, piece by piece. Within 15 minutes I had found the problem. I forgot to put an annotation on a particular class.

I. forgot. an. annotation…..$#$%#$%#$!!!!!!!!!!!

I suppose I could get mad at the guy who wrote the open source framework I was using. But I would be one heck of a hypocrite if I did. I can’t imagine how many people have been in the same situation with Spring Batch, and probably felt the same way about me. I hit a scenario he didn’t think of, so there wasn’t specific error handling for it, and the results were nondeterministic.

No, I was tired. I pushed myself too hard and I made a stupid mistake, and the funny thing is, I lost all of the time I had saved by working so much! Awesome.

And when has that not been the case? I can’t think of time where I worked too many hours and this exact same thing hasn’t happened. I was tired and made a stupid mistake. A mistake that I would have easily caught if fully rested. And that mistake had cost all the time I had spent working late. And it’s compounded by the number of people overworking themselves. The more people doing it, the worse the effect is. Because your stupid mistake made while trying to fix someone else’s stupid mistake quickly snowballs.

But sometimes, it’s just hard to grok. I don’t think the human mind is fully capable of thinking that way. If you’re running somewhere, running faster gets you there quicker, baring the limits of endurance. But overall, if you can run faster, running faster will get you there sooner. Or at least is seems that way when you’re coding at 1 in the morning on an interesting problem. And I know all about sustainable pace, and I agree with it fully. And I wasn’t under the spell of an evil project manager either. It’s a small company, and I’m the CTO. The only drive I have is to get features done quicker to support our sales efforts. And I know, from personal experience, what happens when programmers work unsustainable paces.

You know what it’s like? It’s like white castle. At least once or twice a year I’ll decide eating at White Castle is a good idea. I may or may not be sober when I make this decision, but I make it anyway. And I pay for it. The next day you wake up and you *know* you made a terrible mistake. And for days, maybe even weeks, you’ll swear to never eat at white castle again. But then you’ll be at your cousin’s wedding and there just so happens to be a white castle within walking distance, and the terrible cycle repeats itself.

Grails Testing Issue When Rendering as ‘Text/json’

I ran into this issue while unit testing my controllers (many of which only return JSON) in Grails. I’ve reported the issue to grails: http://jira.codehaus.org/browse/GRAILS-7339 but it’s also worth noting here, so anyone else left scratching their head as to why the rendering doesn’t appear to be working in their unit tests can find the workaround as well:

If you are writing a unit or integration test in grails and mocking the controller, which can be done either by extending ControllerUnitTestCase (or using mockController) you might run into this issue. As you probably know, there are multiple ways to render JSON in grails. One is to take a normal map and render as JSON:


def jsonMap = [success:true]
render jsonMap as JSON


This will work fine everywhere, because you’re effectively passing it in as a converter. However, if you use the more verbose (and in many ways more powerful) rendering, you will run into issues:


render(contentType: "text/json") {
success(true)
results {
result.each {
def period = it.getPeriod()
b(id: it.id,name: it.name)
}
}
}


In my case, my test looks like this:


assert JSON.parse(this.controller.response.contentAsString)?.success == true


This won’t work, because the content on the mock response will be null. With a little digging into how mockController() works, you will find the problem lies in MockUtils. Specifically, in the method that handles a map with render options and a closure:


clazz.metaClass.render = {Map map, Closure c ->
renderArgs.putAll(map)

switch(map["contentType"]) {
case null:
break

case "application/xml":
case "text/xml":
def b = new StreamingMarkupBuilder()
if (map["encoding"]) b.encoding = map["encoding"]

def writable = b.bind(c)
delegate.response.outputStream << writable
break

default:
println "Nothing"
break
}
}


As you can see, the case of ‘text/json’ isn’t handled. I was able to workaround this issue by creating a custom unit test case class to extend from the Grails ControllerUnitTestCase:


class CustomControllerTestCase extends ControllerUnitTestCase{

protected void setUp() {
super.setUp()
controller.class.metaClass.render = {Map map, Closure c ->
renderArgs.putAll(map)

switch(map["contentType"]) {
case null:
break

case "application/xml":
case "text/xml":
def b = new StreamingMarkupBuilder()
if (map["encoding"]) b.encoding = map["encoding"]

def writable = b.bind(c)
delegate.response.outputStream << writable
break

case "text/json":
new JSonBuilder(delegate.response).json(c)
break
default:
println "Nothing"
break
}
}
}

}


Since it’s groovy, you could pretty much replace this anywhere by overriding your mock controller’s meta class.

Comments

olikaf
you saved my day :-)
Robert
Thanks Lucas, this problem looked like it was going to start eating up the rest of my afternoon. Your solution did the trick.