The Sorensens

In a reversal of fortune, I recently found myself wishing Ruby was more like Java. Java has the ideas of abstract base classes and final methods (methods which should not be overridden in child classes). Such ideas don’t really exist in Ruby.

My problem was this. I had a base class Cronjob which represented some job which was going to run under cron. This class managed stuff like setting up logging, db connections, etc. I then wanted other jobs to be able to extend Cronjob and take advantage of the base class:

class Cronjob
  def initialize
    # do useful stuff here, setting up db connections, logging, etc.
  end

  def run   # method which no child class should override
    start = Time.now
    puts "starting at #{start}"
    run_job    # method which child class should override
    stop = Time.now
    puts "finished at #{stop}, took #{stop-start} seconds"
  end
end

class MyJob < Cronjob
  def run_job
    # real work goes here
  end
end 

job = MyJob.new
job.run

All well and good. If people use the base class correctly, they get some nice bits of functionality. However, I eventually noticed that someone had written this class:

class TheirJob < Cronjob
  def run
    # real work goes here
  end
end 

class Cronjob
  final :run

  def run
  end
end

class Object
  @@final_methods = {}

  class < < self
    def prevent_override?(method_name)
      @@final_methods.each do |class_name, final_methods|
        ancestors = self.ancestors
        ancestors.shift # remove myself from the list
        if ancestors.include?(class_name) and
           final_methods.include?(method_name)
          raise "Child class '#{self}' should not override parent class method '#{class_name}.#{method_name}'."
        end
      end
    end

    def method_added(method_name)
      prevent_override?(method_name)
    end

    def final(*names)
      @@final_methods[self] = names
    end

  end
end 

in `prevent_override?': Child class 'TheirJob' should not
override parent class method 'Cronjob.run'.(RuntimeError)

How does it work? The magic is possible because Ruby has a method called method_added. This gets called when a method is added to a class. So, when a source code file is being processed, if a method is defined with “def foo”, after the method has been added to the class this method_added method gets fired with “foo” as the argument. We can then implement the method with our desired behavior. In my case, I just wanted to blow up with an exception which is easy enough to do.

>> true.class.ancestors
=> [TrueClass, Object, Kernel]
>> [].class.ancestors
=> [Array, Enumerable, Object, Kernel]

So, the logic becomes simple. The final method just stores a hash of class => [methods which you cannot override]. Then, on method_added we do a check to see if the method being added is in this hash, and Bob’s your uncle!

So, while it was mildly surprising to find Ruby missing a language feature that I wanted, the language is powerful enough that you can “add to” the language! I’m also half expecting people to weigh in with suggestions of a better way to do this. I would be pleased to hear better variations. This solution definitely doesn’t make it impossible to override the method in a child class; a determined person could get around it. But it does solve my problem of someone inadvertently overriding the method.

Update: Now available via gems, thanks to Dr. Nic’s newgem magic:

gem install finalizer

This week I have been poking through the innards of a web application trying to find out why we were leaking memory (in the permanent generation) like crazy. After a bit of digging I isolated it down to a line that looked like this:

Document doc = SAXParser.new().parse( stringContainingXML );

My first inclination was to blame the parser. Everyone knows that XML parsers are troublemakers, right? But, in the end I had to conclude the leak was entirely the fault of our code. But I learned a bit along the way! The details:

Permgen space – what is it?

The memory a jvm uses is split up into three “generations”: young (eden), tenured, and permanent. This is done to improve the performance of garbage collection. Most objects are short lived (local variables, etc), and so they come and go in the young generation. Some objects (like things in caches) stick around for a while and get promoted from the young to the tenured generation. Some things live “forever”, like the classes themselves, and “interned” strings. These go straight into the permanent generation.

Most memory leaks involve normal objects, and you run out of heap space by filling up the young and tenured memory spaces. Sometimes though, you might see “java.lang.OutOfMemoryError: PermGen space failure”. The most common cause is that you simply don’t have enough space to load up all your classes. Use the param ‘-XX:MaxPermSize=100m’ to adjust to a desired value. You may also find that doing a hot deploy of a war into tomcat eventually uses up permgen space. That is a different issue which I won’t discuss here.

If you observe that your app is leaking permgen space just while it is running (and not because you are hot deploying a war), then you have an interesting problem. The issue is most likely to be either an errant ClassLoader, or String.intern gone awry. ClassLoaders are an interesting beast, but our problem was with interned strings.

What is String.intern?

String.intern is an optimization feature. Doing a double equals (==) compare of two strings is a common mistake people make, as they forget that this is doing an identity comparison. (a == b) is checking if a and b are in fact the same object. Usually, what you really want to do is check if (a.equals(b)). This does the character by character comparison that you probably want.

The thing is, the latter comparison is much slower than an identity comparison. So, a nice performance optimization can be to maintain a canonical list of strings that allow you to do the fast identity comparisons instead. It would be easy enough to write such a thing for yourself, but it is included in Java these days with the String.intern method . So Java maintains a pool of these “canonical” strings to allow you to get some better performance when dealing with strings. But, this pool lives in the permgen space!

Why not intern all strings?

A natural question might be why one shouldn’t just intern every string. Well, there are two reasons why this wouldn’t work. One, you have finite memory. If you stored every string you ever saw into permgen space with intern, you would run out of memory reasonably quickly. Secondly, the reason you are using intern in the first place is as a performance optimization. It happens to be faster to retrieve the canonical string from the intern string pool than it is to do a character by character string comparison. However, as the intern string pool grows infinitely large, the cost to find your string in the pool would probably eventually become more expensive than to just do the character comparison. So, you only want to intern strings which you use frequently throughout the life of your app.

XML parsers seem to use String.intern (or something similar)

XML parsing just happens to be a whole lot of string parsing. So, it is not surprising to find that they take advantage if intern. But, we just said that you probably don’t want to intern every string you see, so what does a parser like Xerces intern? According to (http://xerces.apache.org/xerces2-j/features.html), “All element names, prefixes, attribute names, namespace URIs, and local names are internalized using the java.lang.String#intern(String):String method”. These are all the strings that are going to be seen repeatedly when parsing multiple xml documents with the same DTD. Notice, that they don’t intern attribute values, and tag contents. These elements are what change from document to document; they are your actual data. To intern these would be to intern your entire data space, and we would be facing the previously mentioned problem of effectively interning all strings.

Our problem

At last we arrive at our problem. We were parsing XML documents and finding that our permgen was steadily growing. At first we just enlarged permgen, assuming we had a lot of classes to load. But when we were blowing up with 500 megs of permgen space used up, it was time to find the problem.

After a bunch of digging, what we found was this. The XML we were parsing was not really XML. It was well formed (tags opened and closed properly, nested properly, etc). But, it was XML for which it would be impossible to write a DTD because the data lived in the tag space. An example will show it best. We had tags that looked like:

<data.6541237895.field1>field one val</data.6541237895.field1>
<data.6541237895.field2>field two val</data.6541237895.field2>
<data.7813329781.field1>field one val</data.7813329781.field1>
<data.7813329781.field2>field two val</data.7813329781.field2>
...

The numbers inside of the tag itself was data! So, there was no limited, finite number of tags that could exist in an XML document of this form. Rather, you could have as many tags as could be represented by a ten digit number. To make it worse, there were different values like “foobar” and “name” and many others for each number. The details are boring, but the important bit was that our tag space was as big as our data space. The XML parser was merrily interning every tag string it saw as a reasonable performance optimization. But, as our XML was not true XML, everything came crashing down.

So how to fix it?

1. Maybe the best solution would be to fix the bad XML. In this case, we were not the source of the XML so this ideal option was not practical.
2. Supposedly one can turn off the “feature” of interning via the SAX parser interface in Java. In practice, none of the parsers we tried allowed us to turn it off (let me know if you find one that does!).
3. It would be nice if the interned strings could just be garbage collected like any other Java memory. I’ve seen conflicting reports on this. This article appears to show that interned strings can be collected.
4. Don’t use an XML parser if you aren’t really parsing XML.

Number 4 may seem like a copout, but it is the option we landed on. We now use a few regular expressions to pull the data we need from the “XML” document. This happens to both fix our memory problem, and result in a performance improvent. Apparently selecting just the parts of the document we need with a regex is faster than parsing the whole thing with an XML parser.

How to find these problems

Tracking down these problems can be challenging:

1. Profiling is your friend. Find a good profiler and learn how to use it (JProfiler works nicely).
2. jmap and jstat are useful tools that come with the jdk. They give you info about memory usage, etc.
3. visualgc (jvmstat) is a nice tool for seeing an overall picture of your memory usage.
4. understand how garbage collection works
   (http://java.sun.com/docs/hotspot/gc5.0/gc_tuning_5.html)
5. get familiar with jvm args that help with this kind of debugging and performance optimizations. Verbose gc logging, tracing of class loading, etc.
   (http://java.sun.com/docs/hotspot/gc1.4.2/faq.html, http://www.brokenbuild.com/blog/2006/08/04/java-jvm-gc-permgen-and-memory-options/)

~haakon

This evening I was enjoying my common pastime these days, trying to settle Master Trygve to sleep. A little burping, tying up (I mean swaddling), rocking, etc. Whatever it takes to bring on the elusive state of sleep. I have tried telling him to just close his eyes and pretend he is asleep, or maybe count sheep, but he doesn’t seem to understand the finer points of reason yet. Finally success arrived, he was sleeping quietly and peacfully. One thing we’ve found that seems to help is an astonishingly loud clock. It is the ol’ manual wind up kind that gives you dreams that you are Captain Hook and that a crocodile is relentlessly pursuing you night and day. Not having read Peter Pan, Trygve seems to find it to be a pleasant bit of steady white noise.

As I stealthily stood up to leave the room, the stillness was shattered by the self same clock ringing itself mightily in joyous abandon, sound reverberating through the room for what seemed an eternity. I desperately grabbed it and attempted to muffle it as best I could but to no avail. Trygve had experienced his first alarm clock, and with good success. In the words of Mr. Castanza, “Serenity now!”

I thought I would post a picture of the offending clock and it wasn’t until I saw the picture that I chuckled at the irony of the situation.

~haakon

We are pleased to announce the arrival of the newest member of the Sorensen family. Trygve Anders Peter made his arrival into the world at 9:05 (BST) at the Royal Infirmary, Edinburgh, weighing in at 7 lbs 10 oz. and a whopping 22 inches long! He has been a joy so far and loves to sleep and eat. Our lives have been forever changed and we feel blessed greatly by it.

As some have wondered at the name, here is a quick run-down on what it means, why we chose it, and the all important…how to pronounce it!

Name guide

Trygve is a old-school Norwegian name that means “True or Trustworthy”. Anders is the Scandinavian form of Andrew (the patron saint of Scotland) and it means “Strong and Manly”. Peter is one of Haakon’s middle names and means “Rock”. So all put together it roughly means “true/trustworthy strong rock”. A lot to live up to, we know…we expect his personality to be very gentle, compliant, etc. :)

Also, we have had some requests for a pronunciation guide. We are pronouncing his name ‘trigvee’, though the proper scandinavian is a bit different. You can listen to http://www.thesorensens.org/stuff/trygve.wav for an example. We mostly plan on calling him Tryg (trig), except I’m sure later in life when the whole Trygve Anders Peter will come out at appropriate times. :) We also find ourselves using the nick name ‘Trigger’, so don’t feel like you can’t call him that at times. For you maths heads out there, Haakon’s dad suggested we can make reference to ‘Trig functions’ at appropriate times. :) Goofy engineering/surveyor joke. I have always loved the name Trygve! It’s funny because most people have never heard this name, but I’ve come across two different people with this name in my life and both were well respected and role models for those around them. Trygve Johnson from my high school was an amazing Christian who stood up for his beliefs, loved others around him unconditionally, and still managed to be a stellar student, ASB president, and respected by people of all walks of life. I haven’t kept up with him, but from what I hear, he’s serving God as a pastor now. I hope our son has the same love for others and Christlike attitude to people as him.

~Jenny

Now for a (related) tangent. Should we be “thinking beyond salvation” when sharing God’s good news? How should we be “selling” Christianity? In my context, we (the church) seem to be selling salvation. Focusing on the classic “You have sinned and will face judgement when you die, but Jesus has died for you so if you believe in him God won’t be mad at you anymore and you will go to heaven when you die”. I have to be careful looking back on this line to make sure I am not making the position too farcical (I suspect I’ve failed). Now these statements are pretty much true, and sometimes they might even be the right thing to say. I recently saw a poster that embodies the idea of selling life salvation. It read something like “Believe in the Lord Jesus Christ…and you shall be saved. Instant Life Insurance!” This disturbs me somehow; let me see if I can put this in more every day terms and highlight what concerns me.

Say you are a young man, pretty normal, a few years out of college. Your working a vanilla job, and some new girl just got hired a couple weeks ago and you were talking over lunch. During the conversation the topic of relationships come up and she starts to ask you if you are seeing anyone, etc., etc. You admit that you are single, and this person starts to speculate about why. “Maybe it is your temper”, or “yeah, I noticed you are late to things pretty often, sloppy, etc…” I suspect your hackles would rise, and you would write off what they said, and probably make an effort not to take lunch at the same time anymore. A person probably would not change themselves based upon such a conversation. Contrast this to a different scenario. Take the same young man, a few months later. Your life has suddenly got interesting because you have met the girl of your dreams. But, the more you get to know her, you begin to realize that she is just a step beyond you. Dang it, she is smarter, nicer, and better looking. She seems to spend more time thinking about other people than herself. You know she loves you better than you love her and that is confusing. The more you think about it, the more you want to be a better person just for her sake. You want to shed your bad habits, work on those character flaws and generally improve yourself.

So what is the difference? I think it is relationship. We don’t go to much effort to please strangers, we rarely care what random people think about us. “I will never see them again anyway”. But, it is vitally important to us that the people close to us think well of us. We want our family, our spouse, our closest friends to think well of us. It is a bitter twist to find out that someone you care about dislikes some part of you.

Surely this should play a part in how we tell people about Jesus. Telling people they need to shape up for God doesn’t carry much weight until people understand how much God loves them, and they start to love God themselves. When we “peddle” Jesus, surely the main point is not that when we die we go here or there. Is not the biggest news that the greatest lover the world has ever known loves me, even though he knows all the flaws in my past? And better yet, he loves me knowing how I am going to continue to mess up in the future? Isn’t salvation just the icing on the cake, not the substance?

For the past week or so I have been trying to “think beyond salvation”. Some verses from Matthew (25:31-46) keep popping up, stirring my thoughts. I will ruthlessly paraphrase. When Jesus returns everyone will be brought before him, and he will divide people to his left and his right. In the passage, the criteria for where you fall is the truth of this statement: “I was thirsty and you gave me something to drink, I was a stranger and you invited me in, I needed clothes and you clothed me, I was sick and you looked after me, I was in prison and you came to visit me.” The words are pretty strong, so strong that I am almost uncomfortable to put them up, but here is how it ends just the same “Then they will go away to eternal punishment, but the righteous to eternal life.”

When confronted with such a bald statement connecting actions (caring for the hungry, the poor, the prisoners) to salvation one can imagine the church saying “but we are saved by grace, you know!” While without a doubt we are saved through the grace of God rather than through our own actions, my concern is with the implicit conclusion that since we are not saved by “works” that they are then unimportant. So when I say “thinking beyond salvation” I mean that we should avoid debating issues like this, as such debate ties our hands and feet. If I try to distill God’s commands down to the things I need to do to be “saved” then I am going to naturally try to make that list small, and it is going to escape the whole point. My mindset shouldn’t be “what is the minimum I can do to be saved”, but rather, “how can I please my Father?”. You can think of it in terms of marriage (or any relationship for that matter). What odds would you lay on a relationship where one party spends most of its time trying to clearly lay out the requirements for the relationship rather than loving and pleasing the other?

Now it is possible to go too far in this direction. It is important to understand relationships, to know what makes them tick. In christianity we should know what we believe. But it is my opinion that both myself and the church today falls a bit heavily on the side of legality. As christians we should be thinking further than minimum requirements. As a church, don’t you think most would agree with little difficulty that we should be feeding the hungry, giving water to the thirsty? If we know it would please our Father, we should jump right in on it together.

~haakon

Benjamin Franklin said, “The definition of insanity is doing the same thing over and over and expecting different results. ” Sometimes, though, a retry is exactly what you need. You may need to call something that is not reliable. Maybe it is a network connection that might be down temporarily. How often have you built retry logic around a bit of code that just might fail on occasion? This snippet takes care of that situation nicely.

class Fixnum
  def tries(message)
    current_try_num = 1
    begin
      yield current_try_num
    rescue => e
      if current_try_num >= self
        raise
      else
        puts "Try #{current_try_num} failed (#{message}): #{e}"
        current_try_num = current_try_num.next
        retry
      end
    end
  end
end

10.tries('doing work') do
  raise 'network connection failed' if rand() < 0.7
  puts 'success'
end

output:
>ruby tries.rb
Try 1 failed (doing work): network connection failed
Try 2 failed (doing work): network connection failed
success

Update! 2007-06-01

This is now available as a ruby gem! To install:

sudo gem install retry

http://retry.rubyforge.org

Jenny and I have been talking a bit about the fact that we are really good about sitting on the fence. By that I mean that we talk big about what we believe, about what is important to us without letting it affect our lives in any significant way. We are quite adept at maintaining this division between “belief” and action. We would like to do something about this. On Saturday we had a good conversation in which we threw out some thoughts:

1. Just Do Something

Part of the problem is that we are looking for “the big thing” that we should be doing. What should we be spending all our effort on? What drastic life change to we need to make? We are coming to the realization that we need to just start, do something small and go from there. This thought is echoed in a few things we have read recently. “We are called not to be successful but to be faithful” (Mother Theresa). Or, this gem, “Anything worth doing is worth doing badly” (G. K. Chesterton).

For us, we are beginning to think more about poverty and justice. I just finished reading the book “Irresistible Revolution, living as an ordinary radical” by Shane Claiborne. He writes, “I had come to see that the great tragedy in the church is not that rich Christians do not care about the poor but that rich Christians do not know the poor” (113). This hits home to us. We think we would like to be about hospitality, giving food to people who don’t have it, sharing our home, etc. But in reality we don’t even know anyone who has needs like this. So, for us the first step is to get to know those people. Fortunately we have a connection through church to Edinburgh City Mission and we hope to get involved there and begin meeting people that we can be in community with.

2. Living in Scotland has been a good opportunity to get off the fence.

In many ways life in Seattle was a fast track towards the American Dream. When we were first married, both of us were working, earning good money, enjoying a comfortable life in up and coming Ballard. Our first chance to deviate from normal was Jenny stopping her work as a teacher. This decision flew a bit in the face of the American Dream. Yes, it meant we would not be able to buy a house anytime soon. Couldn’t have two sweet cars, can only buy so many books / cds. But, it also meant more time to spend together, more time to spend with others. That our house was free to invite people over. We continue to be really happy with that decision.

In the same way moving to Edinburgh has been another detachment from the American Dream. We have again traded progress financially for experience, learning, and growth. What we fear is that we might not make enough of this opportunity. As we discuss how long we should be here, how much we miss family and friends, one thing we talk about things like “Why we are here? What we are supposed to be learning. Have we accomplished what we came here for?” Jenny put it more strongly yesterday, saying that we can’t leave Scotland until we have learned / changed enough (begs the question, ‘how do you define enough?’). And even then, should we return to our Seattle life? By then we will have three people in our family, and it would be even more natural to follow the normal way of things. Who knows? But we do think we have more to learn here.

3. At some point, we need to try out life that is 100% other / God focused.

While point 1 is true (just do something, however small), sometimes it is necessary to make more drastic changes. Again quoting from The Irresistible Revolution “If you ask most people what Christians believe, they can tell you, ‘Christians believe that Jesus is God’s Son and that Jesus rose from the dead.’ But if you ask the average person how Christians live, they are struck silent. We have not shown the world another way of doing life. Christians pretty much live like everybody else; they just sprinkle a little Jesus in along the way” (117). If we only ever make small changes we risk having lives just like everyone else, with a little bit of Jesus sprinkled here and there. So we find ourselves thinking that at some point we are going to need to try something a bit more radical, and that it won’t lend itself nicely to the life we have right now.

This is a all a bit nebulous right now. What does it mean? How would we do it? What is “it”? For how long? Where? I suppose the closest thing we could describe it to is missions work, or community living, or something of that sort. There are lots of logistical challenges, but we are keeping this in the back of our heads.

So, these are our thoughts while we sit on the fence. Meanwhile, we will try to get started with something small!

~haakon

This weekend I had the pleasure of a weekend away with the Bruntsfield group, a social gathering from church. Lest you think this was some hasty gathering, let me inform you that the group will be celebrating 25 weekends away next year. The gathering was held at the Badenoch Christian Centre, in the Cairngorm national park. I think when I travel in future I may have to search for such retreat centres, as the prices are quite reasonable.

The gathering was a pleasant size, around 18 people. Initially I was the youngest, but later in the weekend some girls (9 and 10) arrived so I did not have to eat at the kids table by myself. I think the oldest were in their sixties or so. The weekend was delightfully relaxed, consisting mostly of taking intermittent walks during the day around the surrounding lochs, then relaxing in the evening. On Sunday we attended the local church, picturesquely situated on a small hill looking down the length of the loch towards the mountains. The church itself was very Scottish Presbyterian in its simplicity. A simple rectangle, white on the outside. Inside, abut 25 wooden pews seating six or so. Overhead a simple wooden ceiling. The decoration was understated – no stained glass, but the large window at the head of the church had an etched Celtic cross in the glass. The only colour I could see was a simple bouquet of yellow flowers at the front. At the rear of the church was a simple cross designed to look like a road sign. Pointing to the left “He came”; to the right “He is coming”; and in the centre, “He is here”.

The other hit of the weekend was geocaching. One couple has recently taken up the hobby in their travels around the country. They came armed with a gps and a few sites to visit. So, we tromped over the heather, up the moss, down the sandy beach, and found 4 or so over the course of the weekend.

The company was delightful; I fully enjoyed getting to know several people further than is possible from chatting after church. A grand way to spend the weekend!

~haakon

A couple of days ago, I was at the local pool for my morning swim. I’ve been trying to keep up with the swimming during my pregnancy—”they say” that it is good exercise and feels great to be “weightless” in the pool. I haven’t experienced any weightlessness just yet. Who are “they” anyway? Have “they” swam pregnant? Mostly I’m just excited to still be able to do a flip turn without my bump getting in the way too much (imagine a 6 month pregnant lady doing a somersault).

Despite my awkwardness and feeling like I’m not as streamlined as I used to be, I had a small victory on Thursday morning. There was a gentleman in the pool who was decently good (I think he was a triathelete). He could pass me when I was doing a kick set with my kickboard, but when I switched to swimming, he couldn’t pass me and I was keeping up with him. I was proud of myself for still having enough energy to be able to keep up. When he finished with his workout, he got out and went to dry off at the side of the pool. I was doing 100 IM sets, so I stopped after my 100 and stood up at the shallow end (cheeky jenny) and my bump, in all its glory, was above the water. I think he was stunned that I was a pregnant lady! Poor guy, I probably gave him a complex and he will swim much harder next time he works out. I count it as a small victory for the bump—I might feel frumpy and large most of the time (especially going into a cute, trendy, clothing shop where the “big girl” sizes are like size 6), but that morning in the pool I had a moment of feeling great!

~jenny