Monday, January 23, 2012

Off-loading test execution using Hudson part 2

Previously I have written about using Hudson to perform pre-flights using branches on source control systems; but sometimes you just have a patch file that you want to run your tests against.

It turns out this can be quite simple, you just need to make use of a job parameter that takes a File as it's input, this is then written to the specified location in the workspace. In my example I have the following two parameters, the Title allow me to name the job and this property is picked up later by the job Description plugin.

Once you know where the patch is located you can just apply the patch before you perform the required build steps, here is an example using SVN so we use the unix patch command, git users can use the git command instead for this.

And there you are, you can test a user submitted patch for testing without tying up your development machine. You can also run the tests more than once to check for any intermittent tests which can be very valuable in concurrent environments.

Friday, November 25, 2011

x!, a very simple xml library

A couple of weeks ago, having gotten annoyed about the amount of boilerplate you need in Java in order to do some very simple XML actions I decided to put something in place that would get me close to the fluency of the Groovy and JScript XML extensions with a very small interface. Indeed the original proposal can be contained in a tweet and the only methods I have added since are parent() and prefix() which doesn't quite fit in the standard 140 characters.

So there is basically only one class to worry about X, this allows you to read and write values, select children and perform the normal input and output operations. Everything throws unchecked exceptions, and where possible Attr and Elements are treated the same so for example set(val) and get() will work for both in a consistent manner.

You can find simpler examples in the unit tests in the github project, but the most interesting use cases for me are when used with JAX-RS to access and program REST resources that accept XML. XPath makes Duck Typing easy and if you are careful your code can ignore minor schema changes, this is in stark contrast to static JAX-B binding for example.

Take this example, it looks at a remote resource and sets it as being offline using the Jersey client. In the simplest case you need to explicitly tell Jersey about the X message body readers and writers; but other than that the code is quite straight forward.

   ClientConfig clientConfig = new DefaultClientConfig(
      XMessageBodyWriter.class,XMessageBodyReader.class);    
   Client client = Client.create(clientConfig);
   client.addFilter(new HTTPBasicAuthFilter(...));
        
   WebResource resource = client.resource("http://hudson...");

   resource.put(
      resource.get(X.class)
         .set("//n:offline", "true"));
        
   System.out.println(resource.get(X.class).get("//n:offline"));

Note in the last line we use the pre-defined namespace prefix n which is always populated by the first node in the tree. You can also perform the select, or selectList for a list, explicitly to get hold of the value:

   resource.put(
      resource.get(X.class)
         .select("//n:offline").set("true"));
        
   System.out.println(resource.get(X.class)
     .select("//n:offline").get());

You can also use x! in JAX-RS resources, here is a very simple hello world example using the same message body readers and writers as before. Since they are marked as @Provider your container should be able to pick them up for you.

@Path("/hello")
public class HelloMessage {
    @POST
    @Produces("text/xml")
    @Consumes("text/xml")
    public X hello(X input) {

        String name = input.select("//name").get();

        X response = X.in("http://www.example.com", "message");
        response
            .children().create("name")
                .set(name).parent()  // Think of .parent() like a CR
            .children().create("message")
                .set("Hello " + name)
                .set("@lang", "en");

        return response;
    }
}

I am not entirely sure about the flow to create new objects, feedback is always appreciated of course. The use of @ to set attributes is quick as internally this doesn't result in a XPath query. Currently you can only work on direct children when creating new attributes, for more complex paths you need to select the direct parent to create new attributes. You can still set values on attributes that do exist with complex xpath expressions though.

    // @attr exists
    x.select("....@attr").set("value");
    or 
    s.set("...@attr", "value");

Here is a simple request and response from this service just for comparison you can get the code to reflect the xml quite closely.

// Example input message
<message>
   <name>Bob></name>
</message>

// Example response

<message xmlns="http://www.example.com">
   <name>Bob></name>
   <message lang="en">Hello Bob></message>
</message>

This really was a thought experiment that got out of control; but I would welcome feedback, suggestions, and of course as it is on GitHub patches.

One interesting direction is replacing the current DOM implementation with a streaming version which might be possible for certain read and write options depending on the ordering of the data.

Wednesday, October 19, 2011

A little script to update job descriptions on Matrix jobs

We have started to use Matrix jobs in Hudson to try and reduce the amount of clutter on the front page; but we have run into problems in that the job description plugin we used to add the LABEL name to the job id only works for the subordinate tasks. This leaves the top level job without a description.

There are of course other ways you can do this; but since I was playing with the new REST api yesterday, I am going to use this. You could of course use the old RESTy API or indeed the command line tool to perform similar feats. This week I mostly just have a hammer.

Before you use the proxy classes you need to first create a client with the right security credentials configured:

   public static Client createClient() {
       Client client = Client.create();
       client.addFilter(new HTTPBasicAuthFilter("xxxxx@oracle.com","xxxxxxx"));
       return client;
   }

The first part of the job is going to be to create proxies for both the parent project and of the child variant:

   public static void main(String[] args) {

       if (args.length!=2) {
           System.err.println("Expecting two parameters project and variant");
       }

       String project = args[0];
       String variant = project + "%2F" + args[1]; 

       System.out.printf("Will copy missing description from %s to %s\n", variant, project);

       // Create a Jersey client
       Client client = createClient();

       // Create a suitable proxy using the client we configured
       Projects hudsonProjects = projects(client);

       // Handles to the right resources
       Projects.ProjectNameBuilds projectBuilds = hudsonProjects.projectNameBuilds(project);
       Projects.ProjectNameBuilds variantBuilds = hudsonProjects.projectNameBuilds(variant);

Then we can make a quick map of the variant build number to descriptions by getting the first build object:

       // Copy the descriptions out from the variant
       Map<Integer, String> variantToDescription = new HashMap<Integer,String>();
       BuildsDTO variantBuildsDTO = variantBuilds.getAsApplicationXml(BuildsDTO.class);
       for (BuildDTO build : variantBuildsDTO.getBuild()) {
           if (build.getDescription()!=null)
           {
               variantToDescription.put(
                   build.getNumber(), 
                   build.getDescription());
           }
       }

Now it turns out that the current version of the API doesn't allow you to update build descriptions; but we can easily read them using this API and then update them using a simple form POST using the same client.


       // Update the main project descriptions
       BuildsDTO projectBuildsDTO = projectBuilds.getAsApplicationXml(BuildsDTO.class);
       for (BuildDTO build : projectBuildsDTO.getBuild()) {

           String description = build.getDescription();
           // Update description if it has not already been set
           if (description == null || description.length()==0) {

               String variantDesc = variantToDescription.get(build.getNumber());
               if (variantDesc!=null && variantDesc.length()!=0) {

                   // We need to set the description; but nothing in the REST API
                   // so we can use the url property to perform a form submit that will
                   // update the description for us.

                   MultivaluedMap map = new Form();
                   map.add("description", variantDesc);
                   map.add("Submit", "Submit");
                   WebResource buildResource = client.resource(
                       build.getUrl() + "submitDescription");
                   ClientResponse response = buildResource.type(MediaType.APPLICATION_FORM_URLENCODED_TYPE)
                       .post(ClientResponse.class, map);
                   System.out.printf("Updating build %d with description %s gave response %d\n",
                                     build.getNumber(), variantDesc, response.getStatus());

               }
           }
       }

Monday, October 17, 2011

Getting hold of the schemas for the Hudson REST API, and generating a client

I have been playing with the newish REST-like interface in Hudson 2.1.2 and I wanted to generate a client; but as it only uses Jersey 1.5 so it doesn't contain my fix to publish and connect them up directly. Luckily you can just down load these files from the github repository. You ideally want to add these to the wadl description of the service so download the root WADL using a tool such as wget from http://hudson.my.com/rest/application.wadl and then you can add the required imports.

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<application xmlns="http://research.sun.com/wadl/2006/10">
    <doc xmlns:jersey="http://jersey.dev.java.net/" jersey:generatedBy="Jersey: 1.5 01/14/2011 12:36 PM"/>
    
    <grammars>
      <include href="build.xsd" />
      <include href="common.xsd" />
      <include href="fault.xsd" />
      <include href="project.xsd" />
    </grammars>
    
    <resources base="http://hudson/rest/">

    ....

Now you can either use the schemas as is and make sure you have annox on your class path when you generate the model classes, or you can find and replace jaxb:extensionBindingPrefixes="xjc annox" with jaxb:extensionBindingPrefixes="xjc".

This will give you a wadl that will generate the interface and the model classes in the same step; we will have to wait for later version of Hudson to connect the types up to the methods. For the moment though if you have a quick look at the WADL it is relatively obvious which types are for which.


/**
 *
 */
@Generated(value =
           { "wadl|file:/tmp/workspaceWorkDir5662539174912846886.abbot/Hudson/Hudson-WADL/src/wadl/application.wadl",
             "run|ca8e07f8-00d8-40ab-bde3-b15cee244091" }, comments = "wadl2java, http://wadl.java.net",
           date = "2011-10-17T14:31:17.818+01:00")
public class Hudson_Rest {


    public static void main(String[] args) {

        // Write out all projects status
        Hudson_Rest.Projects hudsonProjects = projects();
        ProjectsDTO projects = hudsonProjects.getAsApplicationXml(ProjectsDTO.class);
        for (ProjectDTO project : projects.getProject()) {
            System.out.printf("++++ %s +++++\n", project.getName());
            System.out.println(project.getHealth().getDescription());
        }

        // Query the enablement of one particular project
        Hudson_Rest.Projects.ProjectName hudsonAbbotSf = hudsonProjects.projectName("abbot_SF");
        ProjectDTO abbot_SF = hudsonAbbotSf.getAsApplicationJson(ProjectDTO.class);
        System.out.println("Abbot_SF is enabled " + abbot_SF.isEnabled());
   }

   ...
}

So this is a simple read operation, I will look at writing back through this API which will involve configuring the client for authentication in a later installment.

Tuesday, October 11, 2011

A few simple command line operations on zip files every java developer should know

Java developer spend a lot of time working with zip files it always surprises me that many reach for winzip when some really quite simple command line operation will do. These should all work fine on Unix or using Cygwin, I have consciously used unzip rather than jar as it is more readily available.

First is a quite simple question, does this file file contain a particular file?

unzip -l Example.jar | grep LookingFor.java
or the slightly prettier version with jar:
jar -tf Example.jar | grep LookingFor.java

Second is that you want to look at the content of a file in the zip.

unzip -pq Example.jar META-INF/MANIFEST.MF 
Thirdly you sometimes need to search a bunch of jar files looking for a particular class file: (Thanks a lot to Mark Warner for optimising this for me)
find . -name "*.jar" -type f -exec sh -c 'jar -tf {} | grep "ProtocolHandlerT3"' \; -exec echo ^^^^^^^ in {} \;

Finally the last operation is that you want to be able to compare two zip files, if you have a some UI tools for this then great but this little script will give you a quick command line diff:

#!/bin/bash

FIRST_DIR=`mktemp -d`
SECOND_DIR=`mktemp -d`

echo Unzipping $1 to ${FIRST_DIR}
unzip -q $1 -d ${FIRST_DIR}
echo Unzipping $2 to ${SECOND_DIR}
unzip -q $2 -d ${SECOND_DIR}

diff -r ${FIRST_DIR} ${SECOND_DIR} 

rm -rf ${FIRST_DIR}
rm -rf ${SECOND_DIR}

Monday, September 12, 2011

Replacment for Endpoint.publish when using Jersey

Sometimes when working with JAX-WS services it is nice to be able to test the service without having to deploy to a heavyweight server. You can do this using the Endpoint.publish method and just pass in an instance of your server class.

There is something similar in Jersey, but it isn't part of the JAX-RS specification. So you can publish a single resource as follows:

package client;


import com.sun.jersey.api.container.httpserver.HttpServerFactory;
import com.sun.jersey.api.core.ClassNamesResourceConfig;
import com.sun.net.httpserver.HttpServer;

import java.io.IOException;


public class Test {
    
    public static void main(String[] args) throws IOException {

        HttpServer create = HttpServerFactory.create(
            "http://localhost:9001/",new ClassNamesResourceConfig(Hello.class));
        create.start();
    }
}

Update: 12th December 2013: The code for this has slightly changed in Jersey 2.0, the following will publish a resource using the HttpServer that comes with the JDK (you will need to include a dep on the jersey-container-jdk-http component):


import com.sun.net.httpserver.HttpServer;

import java.net.URI;

import org.glassfish.jersey.jdkhttp.JdkHttpServerFactory;
import org.glassfish.jersey.server.ResourceConfig;


public class Test {
    public static void main(String[] args) {
        
        HttpServer d = JdkHttpServerFactory.createHttpServer(
            URI.create("http://localhost:9001"), 
            new ResourceConfig().register(new Hello()));  
   }

Wednesday, September 7, 2011

Improvements in the WADL client generator

I have recently been working on bringing the useful WADL client generator as originally penned by Marc Hadley more in line with currently technology. I have updated it so that it now uses the Jersey client and also I have made a few modifications to make it easier to use. There is plenty of documentation on the WADL site on how to run the tool in various configurations, but I thought it worth while just to show a simple example of what you might expect.

You can download a snapshot of version 1.1 from here, and hopefully soon a final version of 1.1 will be avaliable here if I have guessed the URL correctly. If you pick the version after the 6th of September it should contain all the bits your need.

So as an example we have a really simple hello world service that has a path param to select the correct languauge, the WADL and the Schema generated in Jersey 1.9 are as follows:


<!-- application.wadl -->

<?xml version = '1.0' encoding = 'UTF-8' standalone = 'yes'?>
<application xmlns="http://wadl.dev.java.net/2009/02">
    <doc xmlns:jersey="http://jersey.java.net/" jersey:generatedBy="Jersey: 1.9 09/02/2011 11:17 AM"/>
    <grammars>
        <include href="application.wadl/xsd0.xsd">
            <doc title="Generated" xml:lang="en"/>
        </include>
    </grammars>
    <resources base="http://localhost:7101/cr/jersey/">
        <resource path="webservice">
            <resource path="{lang}">
                <param xmlns:xs="http://www.w3.org/2001/XMLSchema" name="lang" style="template" type="xs:string"/>
                <resource path="greeting">
                    <method id="hello" name="GET">
                        <response>
                            <representation xmlns:ns2="http://example.com/hello" element="ns2:bean" mediaType="*/*"/>
                        </response>
                    </method>
                </resource>
            </resource>
        </resource>
    </resources>
</application>

<!-- application.wadl/xsd0.xsd -->

<?xml version = '1.0' standalone = 'yes'?>
<xs:schema version="1.0" targetNamespace="http://example.com/hello" xmlns:tns="http://example.com/hello" xmlns:xs="http://www.w3.org/2001/XMLSchema">

  <xs:element name="bean" type="tns:bean"/>

  <xs:complexType name="bean">
    <xs:sequence>
      <xs:element name="message" type="xs:string" minOccurs="0"/>
    </xs:sequence>
  </xs:complexType>
</xs:schema>

Running this through the wadl tool you end up with a java class that look like this along with the relevant JAX-B classes for the schema type Bean.

package client;

import java.util.HashMap;

import javax.annotation.Generated;

import javax.ws.rs.core.UriBuilder;

import com.example.hello.Bean;

import com.sun.jersey.api.client.Client;
import com.sun.jersey.api.client.GenericType;
import com.sun.jersey.api.client.WebResource;


/**
 *
 */
@Generated(value = { "wadl|http://localhost:7101/cr/jersey/application.wadl" },
           comments = "wadl2java, http://wadl.java.net", date = "2011-09-07T11:53:39.206+01:00")
public class Localhost_CrJersey {


    public static Localhost_CrJersey.Webservice webservice(Client client) {
        return new Localhost_CrJersey.Webservice(client);
    }

    public static Localhost_CrJersey.Webservice webservice() {
        return webservice(Client.create());
    }

    public static class Webservice {

        private Client _client;
        private UriBuilder _uriBuilder;
        private HashMap<String, Object> _templateAndMatrixParameterValues;

        /**
         * Create new instance using existing Client instance
         *
         */
        public Webservice(Client client) {
            _client = client;
            _uriBuilder = UriBuilder.fromPath("http://localhost:7101/cr/jersey/");
            _uriBuilder = _uriBuilder.path("webservice");
            _templateAndMatrixParameterValues = new HashMap<String, Object>();
        }

        /**
         * Create new instance
         *
         */
        public Webservice() {
            this(Client.create());
        }

        public Localhost_CrJersey.Webservice.Lang lang(String lang) {
            return new Localhost_CrJersey.Webservice.Lang(_client, lang);
        }

        public static class Lang {

            private Client _client;
            private UriBuilder _uriBuilder;
            private HashMap<String, Object> _templateAndMatrixParameterValues;

            /**
             * Create new instance using existing Client instance
             *
             */
            public Lang(Client client, String lang) {
                _client = client;
                _uriBuilder = UriBuilder.fromPath("http://localhost:7101/cr/jersey/");
                _uriBuilder = _uriBuilder.path("webservice");
                _uriBuilder = _uriBuilder.path("{lang}");
                _templateAndMatrixParameterValues = new HashMap<String, Object>();
                _templateAndMatrixParameterValues.put("lang", lang);
            }

            /**
             * Create new instance
             *
             */
            public Lang(String lang) {
                this(Client.create(), lang);
            }

            /**
             * Get lang
             *
             */
            public String getLang() {
                return ((String)_templateAndMatrixParameterValues.get("lang"));
            }

            /**
             * Set lang
             *
             */
            public Localhost_CrJersey.Webservice.Lang setLang(String lang) {
                _templateAndMatrixParameterValues.put("lang", lang);
                return this;
            }

            public Localhost_CrJersey.Webservice.Lang.Greeting greeting() {
                return new Localhost_CrJersey.Webservice.Lang.Greeting(_client,
                                                                       ((String)_templateAndMatrixParameterValues.get("lang")));
            }

            public static class Greeting {

                private Client _client;
                private UriBuilder _uriBuilder;
                private HashMap<String, Object> _templateAndMatrixParameterValues;

                /**
                 * Create new instance using existing Client instance
                 *
                 */
                public Greeting(Client client, String lang) {
                    _client = client;
                    _uriBuilder = UriBuilder.fromPath("http://localhost:7101/cr/jersey/");
                    _uriBuilder = _uriBuilder.path("webservice");
                    _uriBuilder = _uriBuilder.path("{lang}");
                    _uriBuilder = _uriBuilder.path("greeting");
                    _templateAndMatrixParameterValues = new HashMap<String, Object>();
                    _templateAndMatrixParameterValues.put("lang", lang);
                }

                /**
                 * Create new instance
                 *
                 */
                public Greeting(String lang) {
                    this(Client.create(), lang);
                }

                /**
                 * Get lang
                 *
                 */
                public String getLang() {
                    return ((String)_templateAndMatrixParameterValues.get("lang"));
                }

                /**
                 * Set lang
                 *
                 */
                public Localhost_CrJersey.Webservice.Lang.Greeting setLang(String lang) {
                    _templateAndMatrixParameterValues.put("lang", lang);
                    return this;
                }

                public Bean getAsBean() {
                    UriBuilder localUriBuilder = _uriBuilder.clone();
                    WebResource resource =
                        _client.resource(localUriBuilder.buildFromMap(_templateAndMatrixParameterValues));
                    WebResource.Builder resourceBuilder = resource.getRequestBuilder();
                    resourceBuilder = resourceBuilder.accept("*/*");
                    return resourceBuilder.method("GET", Bean.class);
                }

                public <T> T getAs(GenericType<T> returnType) {
                    UriBuilder localUriBuilder = _uriBuilder.clone();
                    WebResource resource =
                        _client.resource(localUriBuilder.buildFromMap(_templateAndMatrixParameterValues));
                    WebResource.Builder resourceBuilder = resource.getRequestBuilder();
                    resourceBuilder = resourceBuilder.accept("*/*");
                    return resourceBuilder.method("GET", returnType);
                }

                public <T> T getAs(Class<T> returnType) {
                    UriBuilder localUriBuilder = _uriBuilder.clone();
                    WebResource resource =
                        _client.resource(localUriBuilder.buildFromMap(_templateAndMatrixParameterValues));
                    WebResource.Builder resourceBuilder = resource.getRequestBuilder();
                    resourceBuilder = resourceBuilder.accept("*/*");
                    return resourceBuilder.method("GET", returnType);
                }

            }

        }

    }

}

This looks little bit wordy, but things become a little bit simpler if you just look at the interface that has been generated. For example you can now access each part of the path directly using the accessor methods. In this example we are asking for just the english version of the greeting:

public class Test
{
   public static void main(String[] args) {
 
      Bean bean = Localhost_CrJersey.webservice().lang("en").greeting().getAsBean();
      System.out.println(bean.getMessage());

   }
}

Finally for each method that returns a JAX-B element there are also two that take Class.class or GenericType as parameters. This allows the client to access the ClientResponse object, or any other mapping in a way that just wasn't possible in the old API:

public class Test
{
   public static void main(String[] args) {
 
      ClientResponse response = Localhost_CrJersey.webservice().lang("en").greeting().getAs(ClientResponse.class);
      System.out.println(response.getStatus());

   }
}

Now this client currently has limitations in that it doesn't know about HATEOAS and you can't control the URL with deployment descriptors; but it is a start that can hopefully evolve into a more generally useful tool. Also I have played with the constructors for Wadl2Java a little bit to make easier to integrate into IDE tooling, it is the work of a couple of afternoons to wrap this up in a wizard and integrate this into your IDE of choice.