Thursday, May 10, 2012

Common try-with-resources idiom won't clean up properly

So I was re-reading some blogs on the JDK 7 language changes and I noticed the following idiom seems popular:

//
try (BufferedReader br = new BufferedReader(
        new FileReader("/tmp/click.xml"))) {
   System.out.println(br.readLine());
}

That seems all well and good; but consider what happens should something goes wrong constructing the BufferedReader. The contract for try-with-resources will only try to close and tidy up the declared field, if something goes wrong then FileReader is never closed.

So consider this example where the constructor of BufferedReader throws an Error, it could well be an OutOfMemoryError or any number of other failure modes.

//
package client;

import java.io.*;

public class BufferedFileExample {

    public static void main(String[] args) throws FileNotFoundException, 
                                                  IOException {

        try (BufferedReader br = new MyBufferedReader(
            new MyFileReader("/tmp/click.xml"))) {
            System.out.println(br.readLine());
        }
    }
    
    public static class MyFileReader extends FileReader 
    {

        public MyFileReader(String in) throws FileNotFoundException {
            super(in);
        }
        
        public void close() throws IOException {
            super.close();
            System.out.println("Close called");
        }
    }
    
    
    
    public static class MyBufferedReader extends BufferedReader 
    {

        public MyBufferedReader(Reader in) {
            super(in);
            throw new Error();
        }
    }
}

This gets us the following output and as such the close is not performed on the FileReader which until the finaliser is called. You are stuck until the gc wakes up in the future to tidy it up.

Exception in thread "main" java.lang.Error
 at client.BufferedFileExample$MyBufferedReader.(BufferedFileExample.java:38)
 at client.BufferedFileExample.main(BufferedFileExample.java:12)
Process exited with exit code 1.

Luckily there is an easy fix for this, the language change supports multiple entries in try statement, so you can modify this code to be:

//
try (FileReader fr = new MyFileReader("/tmp/click.xml");
     BufferedReader br = new MyBufferedReader(
        fr)) {
   System.out.println(br.readLine());
}

So if you run the code with the modification above then you find that close is now called:

Close called
Exception in thread "main" java.lang.Error
 at client.BufferedFileExample$MyBufferedReader.(BufferedFileExample.java:38)
 at client.BufferedFileExample.main(BufferedFileExample.java:12)
Process exited with exit code 1.

So problem solved, well unfortunately there is another wrinkle here, if we remove the code that throws the Error then we will see the following output:

Close called
Close called
Process exited with exit code 0.

So this is fine for classes that implement the existing java.io.Closeble interface because calling close for a second time is fine; but the more generic interface that is used for other closeable resources unfortunately doesn't have this restriction, so quote the spec

Note that unlike the close method of Closeable, this close method is not required to be idempotent. In other words, calling this close method more than once may have some visible side effect, unlike Closeable.close which is required to have no effect if called more than once. However, implementers of this interface are strongly encouraged to make their close methods idempotent.

So in order to using a split-try-with-resources idiom you first need to make sure that the resources have a idempotent close method. I suspect that most API will do this; but you need to check first and perhaps need to further wrap your objects to make this the case.

One final point with try-with-resource is the handling of exceptions when the close and construction fails. JDK 7 introduced the concept of suppressed exceptions, so you it make the split version of the code throw an exception both in the constructor or the MyBufferedReader and in the close of MyFileReader you get to see the following novel stack trace.

Close called
Exception in thread "main" java.lang.Error
 at client.BufferedFileExample$MyBufferedReader.(BufferedFileExample.java:38)
 at client.BufferedFileExample.main(BufferedFileExample.java:12)
 Suppressed: java.lang.Error: close
  at client.BufferedFileExample$MyFileReader.close(BufferedFileExample.java:27)
  at client.BufferedFileExample.main(BufferedFileExample.java:14)
Process exited with exit code 1.

This might not be what you calling code is expecting - certainly it might differ slightly from what the try/finally code you are replacing would produce, worth know thing that exceptions are handled slightly differently. For example if there is just an Error thrown during the close method would would see this odd self-referential stack trace.

Close called
Close called
Exception in thread "main" java.lang.Error: close
 at client.BufferedFileExample$MyFileReader.close(BufferedFileExample.java:28)
 at java.io.BufferedReader.close(BufferedReader.java:517)
 at client.BufferedFileExample.main(BufferedFileExample.java:15)
 Suppressed: java.lang.Error: close
  at client.BufferedFileExample$MyFileReader.close(BufferedFileExample.java:28)
  ... 1 more
Process exited with exit code 1.

So the lesson here is that the impact of some of the new JDK 7 constructs are going to a little bit subtler than we might expect. Many thanks to my valued colleague Mark Warner for talking this one through with me.

Tuesday, April 3, 2012

Off-loading test execution using Hudson part 3, skipping the queue

Previously I had looked at preflighting, (1, 2) with Hudson (And Jenkins) but forgot to mention a very important plugin that is essential if you want to convince your developer to preflight there work. The problem is that is the slave allotment/smallholder/farm is busy running other jobs then they are forced to wait before merging.

The easy solution to this problem is the priority sorter plugin. This allows you to specify that some jobs are more or less important then others. The default priority is 100, so seeing your preflight job to 1000 should see it jump to the front of the queue.

It is interesting how the length of tests affects my commuting behaviour, now that is take around an hour to run the long tests just for our components. It makes sense to start a work at home, kick of a job and you should have you nice shiny test results by the time we get into work. One day I might even get a blue ball if I am so very lucky with our dependencies. :-)

Friday, March 16, 2012

Catalog support for WADL client generation

I have just been fixing a few bugs in the WADL client generator that I have been working on recently. In particular one bug relates to being able to control the server the client uses at deploy time. (WADL-44)

So for example you have a standard packaged ?ar file that contains a client to a production service, in order to run automated testing you want to be able to use the same archive; but redirect the request to an internal server. Using a deployment plan, for example, you can update or introduce an extra deployment descriptor called META-INF/jax-rs-catalog.xml.

Only the base URI for each root resource in the WADL is updated; but this should be enough for most cases. (Would be interested to hear where people though this was limiting). This simple example redirects the client from a production service to a qa server.

<catalog xmlns="urn:oasis:names:tc:entity:xmlns:xml:catalog">
  <uri name="http://production.example.com/resource"
          uri="http://qa.example.com/resource"/>
</catalog>

If you want to play with this you are going to have to check out the tip of the WADL client project for the moment; but in the coming weeks it will be part of the 1.1.2 release. Feedback always welcome.

Monday, January 23, 2012

Off-loading test execution using Hudson part 2

Previously I have written about using Hudson to perform pre-flights using branches on source control systems; but sometimes you just have a patch file that you want to run your tests against.

It turns out this can be quite simple, you just need to make use of a job parameter that takes a File as it's input, this is then written to the specified location in the workspace. In my example I have the following two parameters, the Title allow me to name the job and this property is picked up later by the job Description plugin.

Once you know where the patch is located you can just apply the patch before you perform the required build steps, here is an example using SVN so we use the unix patch command, git users can use the git command instead for this.

And there you are, you can test a user submitted patch for testing without tying up your development machine. You can also run the tests more than once to check for any intermittent tests which can be very valuable in concurrent environments.

Friday, November 25, 2011

x!, a very simple xml library

A couple of weeks ago, having gotten annoyed about the amount of boilerplate you need in Java in order to do some very simple XML actions I decided to put something in place that would get me close to the fluency of the Groovy and JScript XML extensions with a very small interface. Indeed the original proposal can be contained in a tweet and the only methods I have added since are parent() and prefix() which doesn't quite fit in the standard 140 characters.

So there is basically only one class to worry about X, this allows you to read and write values, select children and perform the normal input and output operations. Everything throws unchecked exceptions, and where possible Attr and Elements are treated the same so for example set(val) and get() will work for both in a consistent manner.

You can find simpler examples in the unit tests in the github project, but the most interesting use cases for me are when used with JAX-RS to access and program REST resources that accept XML. XPath makes Duck Typing easy and if you are careful your code can ignore minor schema changes, this is in stark contrast to static JAX-B binding for example.

Take this example, it looks at a remote resource and sets it as being offline using the Jersey client. In the simplest case you need to explicitly tell Jersey about the X message body readers and writers; but other than that the code is quite straight forward.

   ClientConfig clientConfig = new DefaultClientConfig(
      XMessageBodyWriter.class,XMessageBodyReader.class);    
   Client client = Client.create(clientConfig);
   client.addFilter(new HTTPBasicAuthFilter(...));
        
   WebResource resource = client.resource("http://hudson...");

   resource.put(
      resource.get(X.class)
         .set("//n:offline", "true"));
        
   System.out.println(resource.get(X.class).get("//n:offline"));

Note in the last line we use the pre-defined namespace prefix n which is always populated by the first node in the tree. You can also perform the select, or selectList for a list, explicitly to get hold of the value:

   resource.put(
      resource.get(X.class)
         .select("//n:offline").set("true"));
        
   System.out.println(resource.get(X.class)
     .select("//n:offline").get());

You can also use x! in JAX-RS resources, here is a very simple hello world example using the same message body readers and writers as before. Since they are marked as @Provider your container should be able to pick them up for you.

@Path("/hello")
public class HelloMessage {
    @POST
    @Produces("text/xml")
    @Consumes("text/xml")
    public X hello(X input) {

        String name = input.select("//name").get();

        X response = X.in("http://www.example.com", "message");
        response
            .children().create("name")
                .set(name).parent()  // Think of .parent() like a CR
            .children().create("message")
                .set("Hello " + name)
                .set("@lang", "en");

        return response;
    }
}

I am not entirely sure about the flow to create new objects, feedback is always appreciated of course. The use of @ to set attributes is quick as internally this doesn't result in a XPath query. Currently you can only work on direct children when creating new attributes, for more complex paths you need to select the direct parent to create new attributes. You can still set values on attributes that do exist with complex xpath expressions though.

    // @attr exists
    x.select("....@attr").set("value");
    or 
    s.set("...@attr", "value");

Here is a simple request and response from this service just for comparison you can get the code to reflect the xml quite closely.

// Example input message
<message>
   <name>Bob></name>
</message>

// Example response

<message xmlns="http://www.example.com">
   <name>Bob></name>
   <message lang="en">Hello Bob></message>
</message>

This really was a thought experiment that got out of control; but I would welcome feedback, suggestions, and of course as it is on GitHub patches.

One interesting direction is replacing the current DOM implementation with a streaming version which might be possible for certain read and write options depending on the ordering of the data.

Wednesday, October 19, 2011

A little script to update job descriptions on Matrix jobs

We have started to use Matrix jobs in Hudson to try and reduce the amount of clutter on the front page; but we have run into problems in that the job description plugin we used to add the LABEL name to the job id only works for the subordinate tasks. This leaves the top level job without a description.

There are of course other ways you can do this; but since I was playing with the new REST api yesterday, I am going to use this. You could of course use the old RESTy API or indeed the command line tool to perform similar feats. This week I mostly just have a hammer.

Before you use the proxy classes you need to first create a client with the right security credentials configured:

   public static Client createClient() {
       Client client = Client.create();
       client.addFilter(new HTTPBasicAuthFilter("xxxxx@oracle.com","xxxxxxx"));
       return client;
   }

The first part of the job is going to be to create proxies for both the parent project and of the child variant:

   public static void main(String[] args) {

       if (args.length!=2) {
           System.err.println("Expecting two parameters project and variant");
       }

       String project = args[0];
       String variant = project + "%2F" + args[1]; 

       System.out.printf("Will copy missing description from %s to %s\n", variant, project);

       // Create a Jersey client
       Client client = createClient();

       // Create a suitable proxy using the client we configured
       Projects hudsonProjects = projects(client);

       // Handles to the right resources
       Projects.ProjectNameBuilds projectBuilds = hudsonProjects.projectNameBuilds(project);
       Projects.ProjectNameBuilds variantBuilds = hudsonProjects.projectNameBuilds(variant);

Then we can make a quick map of the variant build number to descriptions by getting the first build object:

       // Copy the descriptions out from the variant
       Map<Integer, String> variantToDescription = new HashMap<Integer,String>();
       BuildsDTO variantBuildsDTO = variantBuilds.getAsApplicationXml(BuildsDTO.class);
       for (BuildDTO build : variantBuildsDTO.getBuild()) {
           if (build.getDescription()!=null)
           {
               variantToDescription.put(
                   build.getNumber(), 
                   build.getDescription());
           }
       }

Now it turns out that the current version of the API doesn't allow you to update build descriptions; but we can easily read them using this API and then update them using a simple form POST using the same client.


       // Update the main project descriptions
       BuildsDTO projectBuildsDTO = projectBuilds.getAsApplicationXml(BuildsDTO.class);
       for (BuildDTO build : projectBuildsDTO.getBuild()) {

           String description = build.getDescription();
           // Update description if it has not already been set
           if (description == null || description.length()==0) {

               String variantDesc = variantToDescription.get(build.getNumber());
               if (variantDesc!=null && variantDesc.length()!=0) {

                   // We need to set the description; but nothing in the REST API
                   // so we can use the url property to perform a form submit that will
                   // update the description for us.

                   MultivaluedMap map = new Form();
                   map.add("description", variantDesc);
                   map.add("Submit", "Submit");
                   WebResource buildResource = client.resource(
                       build.getUrl() + "submitDescription");
                   ClientResponse response = buildResource.type(MediaType.APPLICATION_FORM_URLENCODED_TYPE)
                       .post(ClientResponse.class, map);
                   System.out.printf("Updating build %d with description %s gave response %d\n",
                                     build.getNumber(), variantDesc, response.getStatus());

               }
           }
       }

Monday, October 17, 2011

Getting hold of the schemas for the Hudson REST API, and generating a client

I have been playing with the newish REST-like interface in Hudson 2.1.2 and I wanted to generate a client; but as it only uses Jersey 1.5 so it doesn't contain my fix to publish and connect them up directly. Luckily you can just down load these files from the github repository. You ideally want to add these to the wadl description of the service so download the root WADL using a tool such as wget from http://hudson.my.com/rest/application.wadl and then you can add the required imports.

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<application xmlns="http://research.sun.com/wadl/2006/10">
    <doc xmlns:jersey="http://jersey.dev.java.net/" jersey:generatedBy="Jersey: 1.5 01/14/2011 12:36 PM"/>
    
    <grammars>
      <include href="build.xsd" />
      <include href="common.xsd" />
      <include href="fault.xsd" />
      <include href="project.xsd" />
    </grammars>
    
    <resources base="http://hudson/rest/">

    ....

Now you can either use the schemas as is and make sure you have annox on your class path when you generate the model classes, or you can find and replace jaxb:extensionBindingPrefixes="xjc annox" with jaxb:extensionBindingPrefixes="xjc".

This will give you a wadl that will generate the interface and the model classes in the same step; we will have to wait for later version of Hudson to connect the types up to the methods. For the moment though if you have a quick look at the WADL it is relatively obvious which types are for which.


/**
 *
 */
@Generated(value =
           { "wadl|file:/tmp/workspaceWorkDir5662539174912846886.abbot/Hudson/Hudson-WADL/src/wadl/application.wadl",
             "run|ca8e07f8-00d8-40ab-bde3-b15cee244091" }, comments = "wadl2java, http://wadl.java.net",
           date = "2011-10-17T14:31:17.818+01:00")
public class Hudson_Rest {


    public static void main(String[] args) {

        // Write out all projects status
        Hudson_Rest.Projects hudsonProjects = projects();
        ProjectsDTO projects = hudsonProjects.getAsApplicationXml(ProjectsDTO.class);
        for (ProjectDTO project : projects.getProject()) {
            System.out.printf("++++ %s +++++\n", project.getName());
            System.out.println(project.getHealth().getDescription());
        }

        // Query the enablement of one particular project
        Hudson_Rest.Projects.ProjectName hudsonAbbotSf = hudsonProjects.projectName("abbot_SF");
        ProjectDTO abbot_SF = hudsonAbbotSf.getAsApplicationJson(ProjectDTO.class);
        System.out.println("Abbot_SF is enabled " + abbot_SF.isEnabled());
   }

   ...
}

So this is a simple read operation, I will look at writing back through this API which will involve configuring the client for authentication in a later installment.