Friday, February 22, 2013

Using UriTemplate to change path parameters in a URI

I have been recently working on a client generator in the wadl.java.net project and I was solving a bug where I wanted to change path parameter on a URI. This turned out to be quite easy with existing classes that are part of the Jersey project.

First of all you need a instance of UriTemplate, depending on the very of Jersey this is in a slightly different package - you IDE's automatic import will do the work for you.

  String template = "http://example.com/name/{name}/age/{age}";
  UriTemplate uriTemplate = new UriTemplate(template);


Then you use match(...) to extract them:

  String uri = "http://example.com/name/Bob/age/47";
  Map<String, String> parameters = new HashMap<>();

  // Not this method returns false if the URI doesn't match, ignored
  // for the purposes of the this blog.
  uriTemplate.match(uri, parameters);

Then you can replace the parameters at will and rebuild the URI:

  parameters.put("name","Arnold");

  UriBuilder builder = UriBuilder.fromPath(template);
  URI output = builder.build(parameters);

A little temporal feedback when running tests

I have been working a lot recently with a bunch of test jobs that are intermittently getting stuck. Now it is possible to work out the point where the problem is occurring by looking at the date labels on the LOG output; but it is hard visually to pick these discontinuities out.

In order improve the output in a way that my brain can really quickly parse and identify I thought I would try my hand a bit of simple ASCII animation. I wrote a simple fixture that writes out a line every 1 minute while the test is running, here is a very simple implementation - I am sure there are better ways to do the animation!

import java.util.Timer;
import java.util.TimerTask;
import java.util.concurrent.TimeUnit;

public class TickTockFixture
{
   private Timer timer;
   

   @Before
   public void setUp()
   {
      // Clean out any old timers
      tearDown();
      
      // Start a new timer
      
      timer = new Timer();
      timer.scheduleAtFixedRate(new TimerTask()
      {
         int counter = 0;
         
         @Override
         public void run()
         {
            StringBuilder sb = new StringBuilder(". TickTock : ");
            int animation = counter++ % 10;
            boolean direction = animation < 5;
            int count = direction ?  animation : 9 - animation;
            sb.append("     ", 0, count);
            sb.append(direction ? "/" : "\\");
            sb.append("     ",0, 5-count);
            sb.append(counter);
            sb.append(" min(s)");
            System.err.println(sb);
         }
      }, 0, TimeUnit.MINUTES.toMillis(1));
      
   }

   @After
   public void tearDown()
   {
      if (timer!=null)
      {
         timer.cancel();
         timer = null;
      }
   }
}

So does this help, well take a look at the - made up example - output below and try to spot the section of the log output where everything slows down. It is a lot easier with the ticking output which I think sticks out better because of the animation.
22-Feb-2013 11:54:30 Items in drs, going to tidy up
22-Feb-2013 11:54:40 Removing file:/scratch/gdavison/view_storage/gdavison_lt/oracle/jdeveloper/system12.1.2.0.40.65.92/o.j2ee/drs/Application1/Project1WebApp.war/WEB-INF/weblogic.xml
22-Feb-2013 11:54:40 Removing file:/scratch/gdavison/view_storage/gdavison_lt/oracle/jdeveloper/system12.1.2.0.40.65.92/o.j2ee/drs/Application1/Project1WebApp.war/WEB-INF/web.xml
22-Feb-2013 11:54:40 Removing file:/scratch/gdavison/view_storage/gdavison_lt/oracle/jdeveloper/system12.1.2.0.40.65.92/o.j2ee/drs/Application1/Project1WebApp.war/WEB-INF/classes/project1/Hello.class
22-Feb-2013 11:54:40 Removing file:/scratch/gdavison/view_storage/gdavison_lt/oracle/jdeveloper/system12.1.2.0.40.65.92/o.j2ee/drs/Application1/Project1WebApp.war/WEB-INF/classes/project1/
22-Feb-2013 11:54:40 Removing file:/scratch/gdavison/view_storage/gdavison_lt/oracle/jdeveloper/system12.1.2.0.40.65.92/o.j2ee/drs/Application1/Project1WebApp.war/WEB-INF/classes/
22-Feb-2013 11:54:40 Removing file:/scratch/gdavison/view_storage/gdavison_lt/oracle/jdeveloper/system12.1.2.0.40.65.92/o.j2ee/drs/Application1/Project1WebApp.war/WEB-INF/
22-Feb-2013 11:54:40 Removing file:/scratch/gdavison/view_storage/gdavison_lt/oracle/jdeveloper/system12.1.2.0.40.65.92/o.j2ee/drs/Application1/Project1WebApp.war/
22-Feb-2013 11:54:40 Removing file:/scratch/gdavison/view_storage/gdavison_lt/oracle/jdeveloper/system12.1.2.0.40.65.92/o.j2ee/drs/Application1/.module_marker
22-Feb-2013 11:54:40 Removing file:/scratch/gdavison/view_storage/gdavison_lt/oracle/jdeveloper/system12.1.2.0.40.65.92/o.j2ee/drs/Application1/META-INF/weblogic-application.xml
22-Feb-2013 11:54:40 Removing file:/scratch/gdavison/view_storage/gdavison_lt/oracle/jdeveloper/system12.1.2.0.40.65.92/o.j2ee/drs/Application1/META-INF/application.xml
22-Feb-2013 11:54:40 Removing file:/scratch/gdavison/view_storage/gdavison_lt/oracle/jdeveloper/system12.1.2.0.40.65.92/o.j2ee/drs/Application1/META-INF/
22-Feb-2013 11:54:40 Removing file:/scratch/gdavison/view_storage/gdavison_lt/oracle/jdeveloper/system12.1.2.0.40.65.92/o.j2ee/drs/Application1/.adrs-module.properties
22-Feb-2013 11:54:40 Removing file:/scratch/gdavison/view_storage/gdavison_lt/oracle/jdeveloper/system12.1.2.0.40.65.92/o.j2ee/drs/Application1/
22-Feb-2013 11:54:53 oracle.jdevimpl.webservices.tcpmonitor.config.AnalyzerInstance stop
WARNING: AnalyzerInstance aborting a stop as already appears to be stopped
22-Feb-2013 11:54:53 weblogic.logging.ServerLoggingHandler publish
WARNING: Container weblogic.wsee.jaxws.WLSContainer$BasicContainer@1232e17 doesn't support class com.sun.xml.ws.api.server.Module
22-Feb-2013 11:54:53 weblogic.logging.ServerLoggingHandler publish
WARNING: Container weblogic.wsee.jaxws.WLSContainer$BasicContainer@1232e17 doesn't support class com.sun.xml.ws.api.server.Module
trigger seeding of SecureRandom
. TickTock :  \    9 min(s)
. TickTock : \     10 min(s)
. TickTock : /     11 min(s)
. TickTock :  /    12 min(s)
. TickTock :   /   13 min(s)
22-Feb-2013 11:59:50 done seeding SecureRandom
22-Feb-2013 11:59:53 weblogic.logging.ServerLoggingHandler publish
WARNING: Container weblogic.wsee.jaxws.WLSContainer$BasicContainer@1232e17 doesn't support class com.sun.xml.ws.api.server.Module
22-Feb-2013 12:00:01 Removing file:/scratch/gdavison/view_storage/gdavison_lt/oracle/jdeveloper/system12.1.2.0.40.65.92/o.j2ee/drs/Application1/Project1WebApp.war/WEB-INF/classes/project1/Hello.class
22-Feb-2013 12:00:01 Removing file:/scratch/gdavison/view_storage/gdavison_lt/oracle/jdeveloper/system12.1.2.0.40.65.92/o.j2ee/drs/Application1/Project1WebApp.war/WEB-INF/classes/project1/
22-Feb-2013 12:00:01 Removing file:/scratch/gdavison/view_storage/gdavison_lt/oracle/jdeveloper/system12.1.2.0.40.65.92/o.j2ee/drs/Application1/Project1WebApp.war/WEB-INF/classes/
22-Feb-2013 12:00:01 Removing file:/scratch/gdavison/view_storage/gdavison_lt/oracle/jdeveloper/system12.1.2.0.40.65.92/o.j2ee/drs/Application1/Project1WebApp.war/WEB-INF/
22-Feb-2013 12:00:01 Removing file:/scratch/gdavison/view_storage/gdavison_lt/oracle/jdeveloper/system12.1.2.0.40.65.92/o.j2ee/drs/Application1/Project1WebApp.war/

Monday, February 11, 2013

Proxying a request to localhost in Java

It can be very convenient when developing to server based application to run them using "localhost" in order to maintain consistency between developer machines. This is normally a good idea but there is a small case where this can cause problems.

Consider if you are running a local http proxy on your machine in order to capture your HTTP traffic. (Cough perhaps even the one in JDeveloper I work on). Then you might run into Java bug 6737819. Basically by default JDK 1.6 was hard coded not to send any request to localhost via a proxy which of course was a bit of a pain. Luckily a workaround was put in where you could put the string "~localhost" in your nonProxyHosts entry to turn of this feature:

java -client -classpath classes -Dhttp.proxyHost=localhost -Dhttp.proxyPort=8099 -Dhttp.nonProxyHosts=~localhost -Dhttps.proxyHost=localhost -Dhttps.proxyPort=8099 client.Example

Now moving forward to JDK 1.7 this workaround no longer works; but you need to take care to define nonProxyHosts as an empty string:

java -client -classpath classes -Dhttp.proxyHost=localhost -Dhttp.proxyPort=8099 -Dhttp.nonProxyHosts= -Dhttps.proxyHost=localhost -Dhttps.proxyPort=8099 client.Example

If you define this any anything other than an empty string the DefaultProxySelector though beware because internally it will append / or use the http.nonProxyHosts value from ../jre/lib/net.properties".

Just a minor complication that is not obvious from the published API.

Monday, November 19, 2012

JSON-Schema generation in Jersey

So in my previous post I talked about a proposal to allow the use of JSON-Schema in a WADL, this post looks at how to get this working with a recently build of Jersey. You are going to have to download / reference 1.16SNAPSHOT until 1.16 is released.

If you are using Maven this should be quite straight forward to update your dependencies assuming you already have jersey and jersey-json. You are just going to need to add a dependency on the "jersey-wadl-json-schema" artefact from the "com.sun.jersey.contribs" group to get the new feature.

If you are outside of Maven the easiest thing to do would be to download the latest jersey-archive and then the jersey-wadl-json-schema jar. How you deploy these is tool specific, but if you are using WLS then here are some specific notes on how to upgrade the version of Jersey.

Once you have this working, you need to create a WadlGeneratorConfig class in order to enable this new grammar generation:

package jersey;

import com.sun.jersey.api.wadl.config.WadlGeneratorConfig;
import com.sun.jersey.api.wadl.config.WadlGeneratorDescription;
import com.sun.jersey.wadl.generators.json.WadlGeneratorJSONGrammarGenerator;

import java.util.List;

public class JsonGeneratorConfig extends WadlGeneratorConfig {
 
    @Override
    public List configure() {
        return generator(WadlGeneratorJSONGrammarGenerator.class).descriptions();
    }
}

This can then be registered in a variety of ways, here is an example using a servlet init param. Note also that to make this example simple that we are using the Jersey POJO mapping; but whilst writing this blog I noticed that this setting affected the format of the JSON version of the WADL in case you try this.

<?xml version = '1.0' encoding = 'ISO-8859-1'?>
<web-app xmlns="http://java.sun.com/xml/ns/javaee" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://java.sun.com/xml/ns/javaee http://java.sun.com/xml/ns/javaee/web-app_3_0.xsd"
         version="3.0">
  <servlet>
    <servlet-name>jersey</servlet-name>
    <servlet-class>com.sun.jersey.spi.container.servlet.ServletContainer</servlet-class>
    <init-param>
      <param-name>com.sun.jersey.config.property.WadlGeneratorConfig</param-name>
      <param-value>jersey.JsonGeneratorConfig</param-value>
    </init-param>
    <init-param>
      <param-name>com.sun.jersey.api.json.POJOMappingFeature</param-name>
      <param-value>true</param-value>
    </init-param>
    <load-on-startup>1</load-on-startup>
  </servlet>
  <servlet-mapping>
    <servlet-name>jersey</servlet-name>
    <url-pattern>/resources/*</url-pattern>
  </servlet-mapping>
</web-app>

So I put together a really simple echo service, just to check this is all working:

package jersey;

import javax.ws.rs.Consumes;
import javax.ws.rs.GET;
import javax.ws.rs.POST;
import javax.ws.rs.Path;
import javax.ws.rs.Produces;

@Path("/echo")
public class EchoResource {
     

    @GET
    @Produces("application/json")
    public EchoBean echo() {
        EchoBean bean = new EchoBean();
        bean.setMessage("Hello");
        return bean;
    }

    @POST
    @Consumes("application/json")
    @Produces("application/json")
    public EchoBean echo(EchoBean echo) {
        return echo;
    }
    
}

and

package jersey;

public class EchoBean {
    public EchoBean() {
        super();
    }
    
    private String message;

    public void setMessage(String message) {
        this.message = message;
    }

    public String getMessage() {
        return message;
    }


}

This very simple example results in the following WADL with the JSON-Schema elements referenced:

<?xml version = '1.0' encoding = 'UTF-8'?>
<ns0:application xmlns:ns0="http://wadl.dev.java.net/2009/02">
   <ns0:doc xmlns:ns1="http://jersey.java.net/" ns1:generatedBy="Jersey: 1.16-SNAPSHOT 11/19/2012 12:59 AM"/>
   <ns0:grammars/>
   <ns0:resources base="http://localhost:7103/Jersey/resources/">
      <ns0:resource path="/echo">
         <ns0:method id="echo" name="GET">
            <ns0:response>
               <ns0:representation mediaType="application/json" xmlns:ns2="http://wadl.dev.java.net/2009/02/json-schema" ns2:describedby="application.wadl/echoBean"/>
            </ns0:response>
         </ns0:method>
         <ns0:method id="echo" name="POST">
            <ns0:request>
               <ns0:representation mediaType="application/json" xmlns:ns3="http://wadl.dev.java.net/2009/02/json-schema" ns3:describedby="application.wadl/echoBean"/>
            </ns0:request>
            <ns0:response>
               <ns0:representation mediaType="application/json" xmlns:ns4="http://wadl.dev.java.net/2009/02/json-schema" ns4:describedby="application.wadl/echoBean"/>
            </ns0:response>
         </ns0:method>
      </ns0:resource>
   </ns0:resources>
</ns0:application>

The URI application.wadl/echoBean contains this simple JSON-Schema definition:

{
    "type" : "object",
    "properties" : {
        "message" : {
            "type" : "string"
        }
    },
    "name" : "echoBean"
}

Now there are a number of limitations in the current design, not least that the generated schema doesn't take into account any notation settings. But I thought this would be enough to provoke feedback on whether this feature would be useful generally. There appears to be a growing interest in JSON-Schema both around the net and internally to Oracle, so it would be interesting to see whether this description becomes more common.

Friday, November 16, 2012

JSON-Schema in WADL

In between other jobs I have been recently been reviewing the WADL specification with a view to fixing some documentation problems with a view to producing an updated version. One of the things that because apparent was the lack of any grammar support for languages other than XML - yes you can use a mapping from JSON<->XML Schema but this would be less than pleasant for a JSON purist.

So I began to look at how one would go about attaching a JSON-Schema grammar of a JSON document in a WADL description of a service. This isn't a specification yet; but a proposal of how it might work consistently.

Now I work with Jersey mostly, so lets consider what Jersey will current generate for a service that returns both XML and JSON. So the service here is implemented using the JAX-B binding so they both use a similar structure as defined by the XML-Schema reference by the include.

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<application xmlns="http://wadl.dev.java.net/2009/02">
    <doc xmlns:jersey="http://jersey.java.net/" jersey:generatedBy="Jersey: 1.16-SNAPSHOT 10/26/2012 09:28 AM"/>
    <grammars>
        <include href="xsd0.xsd">
            <doc title="Generated" xml:lang="en"/>
        </include>
    </grammars>
    <resources base="http://localhost/">
        <resource path="/root">
            <method id="hello" name="PUT">
                <request>
                    <representation xmlns:m="urn:message" element="m:requestMessage"  mediaType="application/json" />
                    <representation xmlns:m="urn:message" element="m:requestMessage" mediaType="application/xml" />
                </request>
                <response>
                    <representation xmlns:m="urn:message" element="m:responseMessage" mediaType="application/json"/>
                    <representation xmlns:m="urn:message" element="m:responseMessage" mediaType="application/xml" />
                </response>
            </method>
        </resource>
    </resources>
</application> 

So the first thing we considered was re-using the existing element property, which is defined as a QName, on the representation element to reference an imported JSON-Schema. It is shown here both with and another an arbitrary namespace to it can be told apart from XML elements without a namespace.

<grammars>
        <include href="xsd0.xsd" />
        <include href="application.wadl/responseMessage" />
    </grammars>


    <representation element="responseMessage" mediaType="application/json"/>

Or
    xmlns:json="http://wadl.dev.java.net/2009/02/json" 

    <representation 
        element="json:responseMessage" mediaType="application/json" />


The problem is that the JSON-Schema specification as it stands doesn't have a concept of a "name" property, so each JSON-Schema is uniquely identified by it's URI. Also from my read of the specification each JSON-Schema contains the definition for at most one document - not the multiple types / documents that can be contained in XML-Schema.

So the next best suggestion would be to just use the "filename" part of the URI as a proxy for the URI; but of course that won't necessarily be unique. I could see for example the US government and Yahoo both publishing there own "address" micro format.

The better solution to this problem is to introduce a new attribute, luckily the WADL spec was designed with this in mind, that is of type URI that can be used to directly reference the JSON-Schema definitions. So rather than the direct import in the previous example we have a URI property on the element itself. The "describedby" attribute name comes from the JSON-Schema proposal and is consistent with the rel used on atom links in the spec.

xmlns:json="http://wadl.dev.java.net/2009/02/json-schema" 
    xmlns:m="urn:message" 


    <grammars>
        <include href="xsd0.xsd" />
    </grammars>

    <representation 
        mediaType="application/json"
        element="m:responseMessage" 
        json:describedby="application.wadl/responseMessage" />


The has the secondary advantage in that this format is backwardly compatible with tooling that was relying on the XML-Schema grammar. Although this is probably only of interesting to people who work in tooling / testing tools like myself.

Once you have the JSON-Schema definition then some users are going to want to do away with the XML all together, so finally here is a simple mapping of the WADL to a JSON document that contains just the JSON-Schema information. It has been suggested by Sergey Breyozkin the JSON mapping would only show the json grammars and I am coming around to that way of thinking. I would be interested to hear of a usecase for the JSON mapping that would want access to the XML Schema.

{
   "doc":{
      "@generatedBy":"Jersey: 1.16-SNAPSHOT 10/26/2012 09:28 AM"
   },
   "resources":{
      "@base":"http://localhost/",
      "resource":{
         "@path":"/root",
         "method":{
            "@id":"hello",
            "@name":"PUT",
            "request":{
               "representation":[
                  {
                     "@mediaType":"application/json",
                     "@describedby":"application.wadl/requestMessage"
                  }
               ]
            },
            "response":{
               "representation":[
                  {
                     "@mediaType":"application/json",
                     "@describedby":"application.wadl/responseMessage"
                  }
               ]
            }
         }
      }
   }
}

I am currently using the mime type of "application/vnd.sun.wadl+json" for this mapping to be consistent with the default WADL mime type. I suspect we would want to change this in the future; but it will do for starters.

So this is all very interesting but you can't play with it unless you have an example implementation. I have something working for both the server side and for a Java client generator in Jersey and wadl2java respectively and that will be the topic of my next post. I have been working with Pavel Bucek on the Jersey team on these implementations and the WADL proposal, thanks very much to him for putting up with me.

Thursday, October 11, 2012

Using Hudson/Jenkins to diagnose that intermittent failure


I have been working on one of those intermittent bugs that just won't reproduce on my machine; but will reproduce intermittently on other machines while they are running automated testing. I filled the code with trace statements, now I suspect the problem is in code that I don't control and doesn't appear to have much in the way of diagnostics in the place I am working on.

So I did the obvious thing which is to run the tests on a loop on my machine overnight, 12 hours later and 8 test iterations later, no test failures and I am no further forward.

Since the tests are failing in the Hudson farm, it make sense to try to connect the debugger up to those jobs; but I don't want to hang around to attach the remove debugger to each. Thankfully there is a workaround that will allow me to set suitable breakpoints and manage the debugger connection for me.

First of all you need to configure you IDE to accept incoming debugger connections, here are some notes on configuring JDeveloper for a listening debugger, in Netbeans you need to use the Debug->Attach menu item and select "SocketListen" as the connector and configure as per JDeveloper. In Eclipse you need to configure the debug type as "Socket Listen".

The second step is modifying your build system so that there is a target you can call that will start the test cases in debug mode. This is an example of the parameters for one of our CI jobs that passes in the right information. Note of course the blacked out text the the name of then machine you are trying to connect back to. (The java tests are started with the parameter -agentlib:jdwp=transport=dt_socket,address=xxxx.oracle.com:5000,server=n) Make sure that you don't have any firewalls running on that machine that will block the in-coming connections.




You probably will want to run to run multiple jobs at the same time if you have the nodes available, so consider checking this concurrent build box. Always a good idea to bring cakes / cookies into the office if you are going to tie up all the preflight nodes for the day.




And then all that remains is to run a bunch of jobs and wait for your breakpoint to be hit, might take a little while; but it is going be quicker than running these jobs in series on your own machine. And if your farm is Heterogeneous so much the better for reproducing intermittent failures.



You can sit back and then wait for your code to fail..... may I suggest some sessions from JavaOne while you wait?

Thursday, October 4, 2012

wadl2java 1.1.3 released

So after a little bit of a pause we now have a new release of the wadl2java client generation tool. This is a wide range of improvements in this release; but the main feature is the new support for generating JAX-RS 2.0 client code.

This generated code is nearly identical to the Jersey 1.x interface so most simple code should compile straight after a swap without too many problems. If the client is making use of ClientResponse these references will have to be replaced with Response; but otherwise the interface will be consistent. I would welcome suggestions as to how I could improve what is generated.

So here is a list of items of changes in the release:
  • WADL-25, a patch to allow the passing in of argument to xjc, as provided by Brian Chapman thanks for that..
  • There is a problem in when there was not response content type, this type of method now returns the relevant Response object.
  • Classes that are @XmlType were being wrapped incorrectly with JAXBElement and the information was being taken from the return class.
  • The Proxy objects are now immutable, modifying any property created a new instance.
  • There is a now a programmatic interface to override the base URI at each level. (Suggested by Luigi Tagliamonte)
  • Fix for Oracle Bug 13804542, generation would fail if inner and outer class names match
  • JAX-RS 2.0 generation support available in all tool modes
  • Fix for Oracle Bug 14534583, where a fault element was being incorrectly upgraded from a '2006 WADL. This was causing problems with the examples SOAPUI were providing.
  • Fix for Oracle Bug 1462282, where the generated exception classes were not actually used in the generated code. All exceptions now extend WebAppilcationException to make migration from 1.x to 2.x easier.
  • Improved method generation, removed "application" from media types, removed duplication when method takes and returns the same type (putApplicationXmlAsApplicationXml -> putXml), moved "As" to correct location when we have a type, removed excessive method combinations where we have symmetrical media types. (eg xml,json->xml,json)
  • Initial work looking at how to support JSON-Schema type generation, non functional
  • Adding functional testing for generated clients as was previously only examining the generated classes. Now they are run for both 1.x and 2.x type clients.
 Thanks again as usual help from Pavel Bucek in getting this software out the door.