Category Archives: Software

Log4R In A Rails Application

Logging is arguably one of the best things you can do for your future self. Getting the best and clearest picture of what is happening inside your application isn’t always easy with the default Rails logger. In this post I will show you how to setup log4r in a Rails application. Specifically, a Rails 3 app.

First let me explain why I am dissatisfied with the default logging that comes with Rails. One of the biggest peeves I have with the default Rails logger is that it doesn’t indicate the severity level of the log message. Nor does it include the PID. Nor does it include a consistent timestamp on every log message. After all, this sort of stuff is the tip of the iceberg. It seems logical to me that you’d always want to see this sort of detail when your bug hunting. Luckily, log4r is easily configurable and provides all these things and much much more.

In order to get started, create a file called “log4r_init.rb” under your $RAILS_ROOT/config folder. As a sample to get started try something like this inside your log4r_init.rb file.

require 'log4r'
require 'log4r/yamlconfigurator'
require 'log4r/outputter/fileoutputter'
require 'log4r/outputter/datefileoutputter'
 
log4r_config = YAML.load(ERB.new(File.read(File.expand_path(File.dirname(__FILE__) + "/log4r.yml"))).result)
Log4r::YamlConfigurator.decode_yaml(log4r_config['log4r_config'])
NameOfYourProject::Application.config.logger = Log4r::Logger[Rails.env]

You’ll notice that I am loading a YAML file called “log4r.yml”. It is in here that you setup the formatting and outputs for your logger. Here is a sample of what I typically configure:

log4r_config:
  loggers:
    - name : development
      level : DEBUG
      additive : 'false'
      trace : 'true'
      outputters:
      - standard_log_outputter
    - name : test
      level : DEBUG
      additive : 'false'
      trace : 'true'
      outputters:
      - standard_log_outputter
    - name : staging
      level : INFO
      additive : 'false'
      trace : 'true'
      outputters:
      - standard_log_outputter
    - name : production
      level : INFO
      additive : 'false'
      trace : 'true'
      outputters:
      - standard_log_outputter
 
  outputters:
    - type : StderrOutputter
      name : stderr_outputter
      level : INFO
      formatter:
        date_pattern: '%Y-%m-%d %H:%M:%S'
        pattern : "[%d] PID:%p %l %m"
        type : PatternFormatter
    - type : FileOutputter
      name : standard_log_outputter
      trunc : 'false'
      filename : <%= "#{Rails.root}/log/#{Rails.env}.log" %>
      formatter :
        date_pattern: '%Y-%m-%d %H:%M:%S'
        pattern : "[%d] %l %m"
        type : PatternFormatter
    - type : FileOutputter
      name : standard_plus_trace_log_outputter
      trunc : 'false'
      filename : <%= "#{Rails.root}/log/#{Rails.env}.log" %>
      formatter :
        date_pattern: '%Y-%m-%d %H:%M:%S'
        pattern : "[%d] PID:%p %t %l %m"
        type : PatternFormatter

The name of the loggers match the “Rails environment” just like the default Rails logger. Of course, you can name them whatever you’d like, but I find it best not to veer too far off the path of convention.

The next thing you’ll observe is the minimum severity level to appear in your logs. In development and test I like to see everything so I set it at “DEBUG”, but then tone it down to “INFO” for staging and production.

Take note of the “outputters” part of the configuration. This is another advantage log4r has over the default Rails logger. log4r allows your application to have many outputters. Beyond standard file output, there are “outputters” for email, UDP, or you can roll your own! I’ve heard many hipster hackers are putting their logs in document stores. I actually agree that aggregating all of your logging data into a central data store is both convenient and smart. Rather than SSH’ing and grep’ing on a ton of servers you can go to one place and apply some type of SQL to intelligently extract the info. Not only that, all your logged data is now archived for free.

Lastly each “outputter” needs a formatter. I like to include a proper timestamp, the PID, the severity, and the actual message. You can also include the “trace” that gives you insight to exactly where the logger was called.

Alright, the final piece of the puzzle is adding the “log4r_init.rb” to your “application.rb”. Notice that I do not put it inside the scope of the “NameOfYourProject::Application”, but instead I add to the last line of the “application.rb”. Also, notice that I intentionally didn’t add the code “log4r_init.rb” to an “initializer”.
I’ve tried all sorts of combinations, and this is the only way I can ever get log4r and Rails to work together.
If anyone has figured out a cleaner way of doing log4r setup, please let me know.

module NameOfYourProject
  class Application < Rails::Application
    ##
    # all the things
  end
end
 
require File.expand_path(File.dirname(__FILE__) + "/log4r_init.rb")

Hope this helps and happy logging!

Clojure Applications as Daemons

Running your Clojure app as a daemon is very useful especially when you’re ready to take it live for several reasons.

  • You need a way to easily and consistently “background” or disassociate from the controlling tty for every deployment.
  • Systems often start daemons at boot time (think about when your server is rebooted intentionally and more often…unintentionally).
  • Something a monitor script can execute if your application becomes unresponsive or unruly.

There may be others, but those are the main reasons.

Those unfamiliar with the Java ecosystem may feel especially lost in this area, but fear not because here is an awesome solution. I am going to walk you through getting your application running as a daemon using a great piece of software made by Tanuki. I do not work for Tanuki, nor do I even know anyone at Tanuki, and so trust me this isn’t some lame plug for a product. I’ve used it for many of my Java applications and now I use it with my Clojure applications. It is quite simply a great piece of software. They call it a “Java Service Wrapper”, but that is a bit of a misnomer. It is really a JVM Service Wrapper.

In order to start using it, you’ll need to bundle your application as a Jar file. As you’ll read from the documentation on the Tanuki site there are several ways to “integrate” with the service wrapper. However, I find that using their fourth approach is the best. It is not  coupled in any way to your application and there is no code to be written. Generating a Jar file is of course a breeze if you’re using Leiningen or Cake. I’m most familiar with Leiningen so you’ll want to build an “uberjar” or an equivalent Jar file that contains all of its dependencies. Wait, you’re already quandering…”will it work if I have a web app?” Yes! You can make it a War file, if you are using an embedded Jetty server. In fact, you can make your web application a Jar file if you’re using the Ring Jetty adapter and skip the complexity of a War file. Of course this is beyond the scope of the post. Before you jump off into the deep end with the service wrapper be sure that your application starts properly. Remember you’ll need a “main” function registered in the manifest of your Jar file. On the command line start it up like this:

java -jar your-nifty-clojure-app.jar

Perfect. Now you’ll need to download the service wrapper from here. You’ll see that they have thought of just about every OS distribution. If your development and production OS are the same, simply choose the right download. Or if you develop on a Mac and deploy to a Linux server like me, then choose the “Delta Pack”. The “Delta Pack” is cross platform support for those of us who develop  and deploy to a multitude of OS’s. I forgot to mention I use the community edition and that is what I will be using for this setup.

I prefer to create a folder in my project called “daemon” and unzip the service wrapper contents there. Once unzipped you’ll see a directory structure like this. Begin trimming the unnecessary files by deleting the README’s, doc, and jdoc folders.

Now go into the src/bin directory and find a file called “sh.script.in”. Copy it into the bin folder of daemon directory, rename it, and lastly make it executable.

cd src/bin
cp sh.script.in ../../bin/your-nifty-clojure-app
cd ../../bin
chmod +x your-nifty-clojure-app

Open the executable and edit the following three variables to match your application name.

APP_NAME="your-nifty-clojure-app"
APP_LONG_NAME="Your Nifty Clojure App"
...
PIDDIR="./"

Naturally you can change the PIDDIR to write where ever you want or conditionally change it based on the target environment. For starters though just put the PID file where you start the script.

Now go into the src/conf directory and find a file called “wrapper.conf.in”. Copy it into the conf folder of daemon directory, and rename it “wrapper.conf”.

cp src/conf/wrapper.conf.in conf/wrapper.conf

You’ll to make several edits to the wrapper.conf file.

Begin by telling the wrapper script that it should use the fourth integration method. Specifically it will use the WrapperJarApp as the main method and then subsequently execute your main method.

...
wrapper.java.mainclass=org.tanukisoftware.wrapper.WrapperJarApp
...

I like to set a reusable variable that sets the path to your application Jar file.

...
# Reusable path to the main JAR file for your application
set.APP_JAR_PATH=../../your-nifty-clojure-app.jar
...

Now you’ll need to add your Jar file to the classpath as defined in the wrapper.conf.

# Java Classpath (include wrapper.jar)  Add class path elements as
#  needed starting from 1
wrapper.java.classpath.1=../lib/wrapper.jar
wrapper.java.classpath.2=%APP_JAR_PATH%

You should adjust the JVM minimum and maximum heap sizes.

# Initial Java Heap Size (in MB)
wrapper.java.initmemory=256 # Set it to whatever you see fit...
 
# Maximum Java Heap Size (in MB)
wrapper.java.maxmemory=512 # Set it to whatever you see fit...

Lastly, tell the wrapper script what Jar file to execute. These work like command line arguments. If your application requires arguments, feel free to enumerate them

...
wrapper.app.parameter.1=%APP_JAR_PATH%
#wrapper.app.parameter.2=some argument
#wrapper.app.parameter.3=another argument
...

Now you should be able to start,stop, and restart your application using the bin/your-nifty-clojure-app

./bin/your-nifty-clojure-app start|stop|restart

Tanuki offers a bunch of other options that I haven’t even scratched the surface on and they are worth a look. I hope this helps you daemonize your Clojure application!

Short and Sweet Disclojure

After experimenting and bouncing back and forth on several relatively new languages, I was undecided on what new language I wanted to commit myself to. It wasn’t until I began reading Clojure in Action that I really started to grok and see through its otherwise intimidating syntax. Admittedly, Clojure just looked and sounded too exotic at first glance. Do I really want to learn Lisp? Aren’t those parentheses crazy? Shouldn’t I be programming the next million dollar idea for the iPhone?

It turns out that the answers to those questions are: Yes. No. What money?

I wanted a challenge. I wanted something that would transcend the way I’ve been thinking about programming. I wanted something that I knew would make me a better software engineer. I wanted something more expressive and powerful.

Luckily, the past two weeks I’ve had an excellent opportunity at work to apply and push my learnings of Clojure further. I thought it would be an opportune time to share the good, the bad, and whatever else in between. A reflective and honest “disclojure”.

It seems that the best way to learn a new language is finding well documented source code and blogs from likeminded people. Clojure is a “newer” burgeoning language so it goes without saying that there are far fewer comprehensive online resources to tap into. The blogosphere is actually a bit sparse on Clojure. I’d strongly recommend having a couple Clojure e-books at your fingertips for reference. Speaking of books, don’t expect to learn Clojure by reading a stack of books for 5 months and think that it will magically flow from your digits. It’ll never ever happen. The best recipe for me has been a 1:1 ratio of time spent reading and coding. If you’re in pursuit of books that might help you on your quest to learning Clojure, remember Clojure is a Lisp. The Little Schemer and The Seasoned Schemer have been priming Lisp programmers for decades. If anything reading these two books will make you hungrier for more, wink wink nudge nudge.

Github has been an invaluable asset when looking for good source code examples. One of the questions I continue to ask myself while coding, “Is the code I’m writing idiomatic?” That is important to me because one of my main objectives is to transcend my thinking and programming style. So before you add that dependency to your Leiningen project.clj have a peek at the author’s source. In reality, you might often find yourself sifting through source because of the lack of documentation or examples. Actually I read other developers source code daily for this very reason. So let me re-phrase that, while you’re trying to figure out how the heck some library or “clojar” works, be mindful of the coding style. Undoubtedly you will pick up something useful.

I had a perplexing problem the other day and for the life of me I couldn’t figure it out. I simply lacked the context necessary that other more advanced Clojure developers have. I found my answer after posting my question on the Clojure forum hosted on Google Groups. More specifically, I was using lein run to execute a simple database preparation task, but the process was hanging on well after it completed. It wasn’t until someone informed me that I should use (shutdown-agents) after the task I was running finished because of the way the agent thread pool delays shut down. See my gist here. The turn around time on the Clojure forum varies. I think it took 2 or 3 days before my post was moderated and then received a response. Most if not all of the Clojure core developers answer questions in this forum so the responses are of high quality and are super helpful.

Sometimes you’re totally stuck. Even after using your e-book as a reference and reading the source code it just doesn’t make sense. Enter Clojure IRC! During the daytime it is bustling with chatter. Lots of the folks who answer questions on the Clojure forum hang out here too. In my experiences, the community that gathers here is friendly and willing to help.

Clojure is as expressive, powerful, and succinct as advertised. The Java interop with Clojure is about the simplest and cleanest I’ve seen thus far and that includes JRuby and Scala. I’m impressed with the clean HTTP API Ring has to offer and using the Clojure JDBC API is really nice. If only JDBC were that easy in Java there would be no need for heavyweight clunky ORMs. In terms of testing, speclj is a pretty awesome testing framework that mirrors Ruby’s RSpec API. I also have enjoyed rolling my own solutions for things that aren’t available yet. And one of the most important parts…it is fast and performs very well under load. I don’t make these claims falsely either, we ran a load test today at work and we were absolutely thrilled with the response times and throughput. I look forward to posting more about Clojure in the coming weeks. Stay tuned!

Monitoring a remote JVM process

Monitoring a remote JVM process has never been easier. As part of the default JDK installation you have a utility called jconsole at your fingertips. jconsole provides a simple way to attach to a running JVM process either locally or remotely. There are basically two ways to attach to the process, either choosing to attach to the PID or through JMX. Personally, I find that attaching to a PID is the easiest either remotely or locally.

Attaching to a process remotely might seem difficult or impossible, but if you are able to use X11 window forwarding through SSH it is a nice option. Check your /etc/ssh/ssh_config and make sure that you see ForwardX11 is set to “yes”, sometimes it is X11Forwarding, check your Linux distribution to be sure.

Host *
#   ForwardAgent no
   ForwardX11 yes

Once you have made this change, restart your ssh service /etc/init.d/ssh restart or service ssh restart. Voila. Now ssh onto your remote server with the X option

ssh -X user@remote.server.com

So once you’ve SSH’d into your remote server startup jconsole, assuming it is in your PATH you should be able to just type it straight away. You should then immediately see that jconsole has started up through X11 as if it were running on your local machine. Just select the PID and you will be able to start monitoring memory, threads, garbage collection, etc.

jconsole

jconsole

Generally when I am load testing an application (I use JMeter) on the JVM I will crack open jconsole to see if there are any blaring memory leaks or memory consumption problems.

Fat chunky ugly bloated “models”

I’m not referring to the runway model variety, I’m talking about code.

Observation

I’ve had the opportunity to work with many Ruby code bases the past few years. One reoccurring theme I keep seeing is bloated models. Fat chunky ugly bloated models. Literally thousands of lines, mixing in of modules, and extensions galore. All of which are convoluting the notion of two completely separate things, data access and business logic. The phrase “fat models” has been taken too literally. I have one question. Why?

I am officially declaring it an anti-pattern for the Ruby community. A “model” is supposed to be an object representation of the underlying data your application needs. So when working with a _data_ model, you ought to stay laser focused on the DML behavior. Get your data and get out. Move on to the next set of objects that are responsible for determining, strategizing, composing, decorating, and displaying (these action verbs should sound familiar) the object representations of your data.

Remarkably most Java projects I’ve seen get this one right. Typically there is DAO, there is an entity, and beyond those two objects that are focused on DML, there are other objects that responsible for doing “things” with that data. I say “things”, but I am referring to whatever your business or domain does with the data beyond creating, reading, updating, or deleting it.

Real World Example

Here’s a solid example for the sake of discussion. Every e-commerce site has some notion of a product or service. Suppose you are modeling a “Product”. Your product has some attributes regarding description, size, weight, color, etc. You get the gist. Let’s not forget pricing, every product has some type of pricing. I see pricing go sideways almost every time. Pricing always starts out simple. You might have a “base” price and “selling” price. The “base” price is your company’s cost, and the “selling” price is your customers cost. Shortly after you deliver that functionality your product development team will undoubtedly come up with all kinds of pricing strategies.

These should all sound familiar:

  • site wide sales
  • single use coupons
  • volume discounts
  • bundling discounts
  • affiliate pricing tiers
  • employee discounts

Wrong

This is where it goes sideways. It’s so easy, right? Just fetch the “selling” price within an instance of the “Product” model and begin applying the business rules inside your model.

Right

I hinted a couple times in that this is a perfect application for the strategy pattern. Not familiar? Each individual pricing type should have it’s own strategy. It is safe to let a “Price” object flow though the strategies. The pricing strategies classes can then apply the discounts and the appropriate pricing separately. One of the golden rules is that composition is almost always better than inheritance and that is the construct the strategy pattern embraces.

That should look and feel better on many different levels. You should observe the freedom to add or remove pricing features without the tight coupling to any particular model. In the case that your business evolves, and it does inevitably, your pricing strategies should be flexible enough to handle any sort of pricing irrespective of whether it has anything to do with a “Product” or not. It should also be easier to test. There are no worries about the creation of database objects or fixtures, simply assert the strategies of pricing with a “Pricing” object. Also as the pricing strategies grow in size and complexity it will certainly be easier to abstract away into a separate library.

Facetious Wonderment

Here is where my “wonderment” comes into action. How is it possible that so many “Rubyists” are ignoring traditional OO programming techniques and patterns when coding at the data access layer? Is it possible that the suggestive nature of the most popular and opinionated Ruby framework has created some psychological voodoo on developers making them believe that classes only come in 4 flavors (models, views, controllers, and helpers)? Is it really that scary to add new directories or categorize your classes differently? Perhaps it is the dynamic nature of the language? Can it be the power and “expressiveness” of the language? A language wherein the lack of true interfaces, strict type checking, and the magical injection of behaviors and methods has lead its developers down a path of wicked and corrupt habits. Gosh…why should I create a new class when I can just inject a few unrelated wacky ass methods to the String class? Is it laziness? Could it be that Ruby has given way and lowered the barrier to entry for people who shouldn’t be programming at all? Or perhaps it is a lack of classic OO programming training?

Conclusion

Draw your own conclusions. The major takeaway for me is that I’m sick of it; or rather I’m sickened by it. Unwavering I will remain steadfast on an ever-quest to right the wrong where ever my fingertips land regarding this matter. You should too. There are some great code katas that focus on this very problem set. Give the Gilded Rose kata a whirl. You might be surprised by the harshness of my befuddlement and/or the generalizations I’ve made about the Ruby community. Don’t get me wrong. There are many super intelligent people out there that swear by Ruby, who are doing the right thing, and you know who you are. I am just saying based on my observations and experiences there are many more people doing the wrong thing, specifically in the data access realm, that outweigh the right-doers. But hey that’s the case with most things in life, isn’t it?

Implementing Ruby’s inject method

I spent a little time tinkering with some of the “newer” RSpec 2 features while re-implementing the inject method that belongs to the Enumerable module. It is a fun and simple exercise because it reinforces the understanding of blocks and the sometimes confusing nature of the inject method. Here it is.

 
require 'rubygems'
require 'rspec'
 
module Enumerable
  def inject(*args, &block)
    if(args.size == 2)
      inject_binary_operation(args[0], args[1], self)
    elsif(args.size == 1 && !block_given?)
      inject_binary_operation(self.first, args[0], self.drop(1))
    else
      the_memo = args[0] || self.first
      self.each { |item| the_memo = yield(the_memo, item) if block_given? }
      the_memo
    end
  end
  alias :reduce :inject
 
  private
 
  def inject_binary_operation(the_memo, operator, enum)
    raise TypeError.new("#{operator} is not a symbol") unless operator.is_a?(Symbol)
    enum.each { |item| the_memo = the_memo.send(operator, item) }
    the_memo
  end
end
 
shared_examples_for "an Enumerable object" do
 
  describe "inject" do
 
    describe "given there is a specified memo value" do
 
      it('should return the newly injected "memo" value if using a block') do
        enumerable.inject([3,4,5]) { |memo, enum_item| memo.include?(enum_item) ? memo : memo << enum_item }.should == [3, 4, 5, 1, 2]
      end
 
      describe "when it is a binary operation" do
 
        it('should return a TypeError because the single argument is not a Symbol') do
          lambda { enumerable.inject(2) }.should raise_error(TypeError)
        end
 
        it('should add the values') do
          enumerable.inject(2, :+).should == 8
        end
 
        it('should multiply the values') do
          enumerable.inject(2, :*).should == 12
        end
 
        it('should subtract the values') do
          enumerable.inject(2, :-).should == -4
        end
 
    end
 
    end
 
    describe "given there isn't a specified memo value" do
 
      it('should return the newly injected "memo" value if using a block') do
        enumerable.inject { |memo, enum_item| memo > enum_item ? memo : enum_item }.should == 3
      end
 
      describe "when it is a binary operation" do
 
        it('should add the values') do
          enumerable.inject(:+).should == 6
        end
 
        it('should multiply the values') do
          enumerable.inject(:*).should == 6
        end
 
        it('should subtract the values') do
          enumerable.inject(:-).should == -4
        end
      end
 
    end
  end  
end
 
describe Range do
 
  it_should_behave_like "an Enumerable object" do
    let(:enumerable) { (1..3) }
  end
 
end
 
describe Array do
 
  it_should_behave_like "an Enumerable object" do
    let(:enumerable) { [1,2,3] }
  end
 
end

I especially like the RSpec documentation format because it provides such great readability. When appropriate the “shared_examples_for” is also nice to keep your test suite DRY.

$ rspec injectable.rb --format documentation
 
Range
  it should behave like an Enumerable object
    inject
      given there is a specified memo value
        should return the newly injected "memo" value if using a block
        when it is a binary operation
          should return a TypeError because the single argument is not a Symbol
          should add the values
          should multiply the values
          should subtract the values
      given there isn't a specified memo value
        should return the newly injected "memo" value if using a block
        when it is a binary operation
          should add the values
          should multiply the values
          should subtract the values
 
Array
  it should behave like an Enumerable object
    inject
      given there is a specified memo value
        should return the newly injected "memo" value if using a block
        when it is a binary operation
          should return a TypeError because the single argument is not a Symbol
          should add the values
          should multiply the values
          should subtract the values
      given there isn't a specified memo value
        should return the newly injected "memo" value if using a block
        when it is a binary operation
          should add the values
          should multiply the values
          should subtract the values
 
Finished in 0.007 seconds
18 examples, 0 failures

And of course the gist…
https://gist.github.com/1096650