Thursday, April 25, 2013

Building a simple java client for forecast.io

I like to be outdoors when life allows so weather is always an interest for me. So i was very pleases to see forecast.io becoming available with global coverage after being jealous of the US only "DarkSkys" app. It was also nice to see that there is a nice REST api for the service which I could play with.

Unfortunately there was only a textual description of service, nothing I could consume with the Java client tools I have been working with. But I though it would be an interesting experiment to try to document an existing service from scratch.

The first thing I did was to get an example of the data that the service would respond with and run it through a JSON->JSON Schema tool. I used JSONSchema.net which gave me a rough v3 JSON Schema; but it wasn't able to take into account elements that are reused. A little bit of hand editing later I ended up with three JSON Schema files, one for the forecast itself and one for a list of weather details and and one for each particular data point.

From there is was pretty simple to read the developer documentation and write up a simple WADL to describe the two resources provided by the service. Luckily Jason LaPort from the DarkSkys was on hand to fix a few mistake and generally verify my description.

Once I had this all in place I of course found a couple of bugs in my own code, so it was time to update and fix wadl.java.net and release a 1.1.5 version with a revised depdency on the jsonschema2pojo project. This is why a little bit of adhoc expert testing pay dividends.

So taking the WADL from github it is possible to generate a nice clean Java API using the wadl.java.net generator. I won't reproduce the steps to generate the client here, but this is the end result and the kind of code you can get out of it.

// Create a Jersey client and setup for POJO JSON mapping

    ClientConfig cc = new DefaultClientConfig();
    cc.getFeatures().put(JSONConfiguration.FEATURE_POJO_MAPPING, true);
    Client c = Client.create(cc);


    // Get current weather for location and check for any alerts 
    
    Forecast f = ApiForecastIo_Forecast
        .apikey(c, "xxxxxxx")                  // Get own key from developer.forecast.io
        .latitudeLongitude(51.32d, -1.244d)    // Some location near where I live
           .getAsForecast(
              null,                            // Not JSONP
              Units.SI,                        // Return values in SI units
              "hourly,minutely,daily,flags");  // Just return current weather and any alerts
    
    // Output a summary

    System.out.println(
        f.getCurrently().getSummary() + " " +
        f.getCurrently().getTemperature() + " C");

    // Output any alerts
  
    List<Alert> alerts = f.getAlerts();
    for (Alert altert : alerts) {
        System.out.println(
            altert.getTitle() + " " + altert.getUri());
    }

The nice thing here was just how pleasantly easy describing a service that already exists was, yes I could have written it by hand in Java; but then when I wanted to move to Jersey 2.x I would have to re-write by hand rather that just regenerating from the metadata to a new technology target. (Also did the same for a BBC REST Schedule API, that is in the same git project)

Friday, March 22, 2013

wadl2java 1.1.4 released

We have released a new version of the wadl2java tool today with a bunch of new features and bug fixes. Again thanks for Pavel Bucek from the Jersey team for his continual help, and for the contribution from my JDeveloper team mate Michael Bachand.

New Features:

  • Added support for generating binding classes for WADL & JSON Schema decribed services.
  • Refactored upgrade transform to wadl-xslt module and added a WADL->HTML transform.
  • Added support for constructors that take a URI, this require use of an internal class UriTemplate in Jersey 2.x to have this work in the RS 2.0 version breaking general compatibility.
  • Fixes with regards to the requirement to use a later version of JAX-B than provided with JDK 6, some examples now required endorsed directory to run / build.
  • The default message for this exception with just a status was just an empty string, so we generate some boilerplate to fix this. (See JAX_RS_SPEC-312)

WADL Bugs:

  • WADL-65 Exception if non-required query parameters are passed in as null.
  • WADL-66 If the XML Schema didn't start with <?xml it would be treated as a JSON Schema.
  • WADL-52 UriBuilder impl dependency missing.

Oracle Bugs:

  • 16234340 matrix parameters were not showing up in the test URL because of inconsistent usage of the resourceTypes and the resourceObjects structures.
  • 16429792, where you would get duplicate methods if there were matrix parameters. These are dealt with like template parameters and as such are dealt with differently and needed to be filtered from the optional list.
  • 16069855 we are modifying the WADL stylesheet (wadl_2009-02.xsl) since it's resulting XHTML document is now properly depicting non ascii characters (such as a french accent) the fix involves adding a element specifying UTF-8 as the charset.
  • 16061180 Cannot process WADL with regex in template parameters, templates needed to be take out for code gen.
  • 15878417 Generate template method to make it easier for IDE tooling to customize client creation
  • 14831111 The client was no longer throwing an exception for a server side failure.
  • 14710107 where the generated javadoc would would not be rendered properly do a XML copying issue.

Future Work:

Going forward we are looking to generate "java.lang.reflect.Proxy" based clients based on the Jersey 2.x module or a future part of the JAX-RS 2.x specification. We are also going to look at transforming other REST service descriptions into XML, for example Swagger, so that they can be fed into the client generation tool.

Friday, February 22, 2013

Using UriTemplate to change path parameters in a URI

I have been recently working on a client generator in the wadl.java.net project and I was solving a bug where I wanted to change path parameter on a URI. This turned out to be quite easy with existing classes that are part of the Jersey project.

First of all you need a instance of UriTemplate, depending on the very of Jersey this is in a slightly different package - you IDE's automatic import will do the work for you.

  String template = "http://example.com/name/{name}/age/{age}";
  UriTemplate uriTemplate = new UriTemplate(template);


Then you use match(...) to extract them:

  String uri = "http://example.com/name/Bob/age/47";
  Map<String, String> parameters = new HashMap<>();

  // Not this method returns false if the URI doesn't match, ignored
  // for the purposes of the this blog.
  uriTemplate.match(uri, parameters);

Then you can replace the parameters at will and rebuild the URI:

  parameters.put("name","Arnold");

  UriBuilder builder = UriBuilder.fromPath(template);
  URI output = builder.build(parameters);

A little temporal feedback when running tests

I have been working a lot recently with a bunch of test jobs that are intermittently getting stuck. Now it is possible to work out the point where the problem is occurring by looking at the date labels on the LOG output; but it is hard visually to pick these discontinuities out.

In order improve the output in a way that my brain can really quickly parse and identify I thought I would try my hand a bit of simple ASCII animation. I wrote a simple fixture that writes out a line every 1 minute while the test is running, here is a very simple implementation - I am sure there are better ways to do the animation!

import java.util.Timer;
import java.util.TimerTask;
import java.util.concurrent.TimeUnit;

public class TickTockFixture
{
   private Timer timer;
   

   @Before
   public void setUp()
   {
      // Clean out any old timers
      tearDown();
      
      // Start a new timer
      
      timer = new Timer();
      timer.scheduleAtFixedRate(new TimerTask()
      {
         int counter = 0;
         
         @Override
         public void run()
         {
            StringBuilder sb = new StringBuilder(". TickTock : ");
            int animation = counter++ % 10;
            boolean direction = animation < 5;
            int count = direction ?  animation : 9 - animation;
            sb.append("     ", 0, count);
            sb.append(direction ? "/" : "\\");
            sb.append("     ",0, 5-count);
            sb.append(counter);
            sb.append(" min(s)");
            System.err.println(sb);
         }
      }, 0, TimeUnit.MINUTES.toMillis(1));
      
   }

   @After
   public void tearDown()
   {
      if (timer!=null)
      {
         timer.cancel();
         timer = null;
      }
   }
}

So does this help, well take a look at the - made up example - output below and try to spot the section of the log output where everything slows down. It is a lot easier with the ticking output which I think sticks out better because of the animation.
22-Feb-2013 11:54:30 Items in drs, going to tidy up
22-Feb-2013 11:54:40 Removing file:/scratch/gdavison/view_storage/gdavison_lt/oracle/jdeveloper/system12.1.2.0.40.65.92/o.j2ee/drs/Application1/Project1WebApp.war/WEB-INF/weblogic.xml
22-Feb-2013 11:54:40 Removing file:/scratch/gdavison/view_storage/gdavison_lt/oracle/jdeveloper/system12.1.2.0.40.65.92/o.j2ee/drs/Application1/Project1WebApp.war/WEB-INF/web.xml
22-Feb-2013 11:54:40 Removing file:/scratch/gdavison/view_storage/gdavison_lt/oracle/jdeveloper/system12.1.2.0.40.65.92/o.j2ee/drs/Application1/Project1WebApp.war/WEB-INF/classes/project1/Hello.class
22-Feb-2013 11:54:40 Removing file:/scratch/gdavison/view_storage/gdavison_lt/oracle/jdeveloper/system12.1.2.0.40.65.92/o.j2ee/drs/Application1/Project1WebApp.war/WEB-INF/classes/project1/
22-Feb-2013 11:54:40 Removing file:/scratch/gdavison/view_storage/gdavison_lt/oracle/jdeveloper/system12.1.2.0.40.65.92/o.j2ee/drs/Application1/Project1WebApp.war/WEB-INF/classes/
22-Feb-2013 11:54:40 Removing file:/scratch/gdavison/view_storage/gdavison_lt/oracle/jdeveloper/system12.1.2.0.40.65.92/o.j2ee/drs/Application1/Project1WebApp.war/WEB-INF/
22-Feb-2013 11:54:40 Removing file:/scratch/gdavison/view_storage/gdavison_lt/oracle/jdeveloper/system12.1.2.0.40.65.92/o.j2ee/drs/Application1/Project1WebApp.war/
22-Feb-2013 11:54:40 Removing file:/scratch/gdavison/view_storage/gdavison_lt/oracle/jdeveloper/system12.1.2.0.40.65.92/o.j2ee/drs/Application1/.module_marker
22-Feb-2013 11:54:40 Removing file:/scratch/gdavison/view_storage/gdavison_lt/oracle/jdeveloper/system12.1.2.0.40.65.92/o.j2ee/drs/Application1/META-INF/weblogic-application.xml
22-Feb-2013 11:54:40 Removing file:/scratch/gdavison/view_storage/gdavison_lt/oracle/jdeveloper/system12.1.2.0.40.65.92/o.j2ee/drs/Application1/META-INF/application.xml
22-Feb-2013 11:54:40 Removing file:/scratch/gdavison/view_storage/gdavison_lt/oracle/jdeveloper/system12.1.2.0.40.65.92/o.j2ee/drs/Application1/META-INF/
22-Feb-2013 11:54:40 Removing file:/scratch/gdavison/view_storage/gdavison_lt/oracle/jdeveloper/system12.1.2.0.40.65.92/o.j2ee/drs/Application1/.adrs-module.properties
22-Feb-2013 11:54:40 Removing file:/scratch/gdavison/view_storage/gdavison_lt/oracle/jdeveloper/system12.1.2.0.40.65.92/o.j2ee/drs/Application1/
22-Feb-2013 11:54:53 oracle.jdevimpl.webservices.tcpmonitor.config.AnalyzerInstance stop
WARNING: AnalyzerInstance aborting a stop as already appears to be stopped
22-Feb-2013 11:54:53 weblogic.logging.ServerLoggingHandler publish
WARNING: Container weblogic.wsee.jaxws.WLSContainer$BasicContainer@1232e17 doesn't support class com.sun.xml.ws.api.server.Module
22-Feb-2013 11:54:53 weblogic.logging.ServerLoggingHandler publish
WARNING: Container weblogic.wsee.jaxws.WLSContainer$BasicContainer@1232e17 doesn't support class com.sun.xml.ws.api.server.Module
trigger seeding of SecureRandom
. TickTock :  \    9 min(s)
. TickTock : \     10 min(s)
. TickTock : /     11 min(s)
. TickTock :  /    12 min(s)
. TickTock :   /   13 min(s)
22-Feb-2013 11:59:50 done seeding SecureRandom
22-Feb-2013 11:59:53 weblogic.logging.ServerLoggingHandler publish
WARNING: Container weblogic.wsee.jaxws.WLSContainer$BasicContainer@1232e17 doesn't support class com.sun.xml.ws.api.server.Module
22-Feb-2013 12:00:01 Removing file:/scratch/gdavison/view_storage/gdavison_lt/oracle/jdeveloper/system12.1.2.0.40.65.92/o.j2ee/drs/Application1/Project1WebApp.war/WEB-INF/classes/project1/Hello.class
22-Feb-2013 12:00:01 Removing file:/scratch/gdavison/view_storage/gdavison_lt/oracle/jdeveloper/system12.1.2.0.40.65.92/o.j2ee/drs/Application1/Project1WebApp.war/WEB-INF/classes/project1/
22-Feb-2013 12:00:01 Removing file:/scratch/gdavison/view_storage/gdavison_lt/oracle/jdeveloper/system12.1.2.0.40.65.92/o.j2ee/drs/Application1/Project1WebApp.war/WEB-INF/classes/
22-Feb-2013 12:00:01 Removing file:/scratch/gdavison/view_storage/gdavison_lt/oracle/jdeveloper/system12.1.2.0.40.65.92/o.j2ee/drs/Application1/Project1WebApp.war/WEB-INF/
22-Feb-2013 12:00:01 Removing file:/scratch/gdavison/view_storage/gdavison_lt/oracle/jdeveloper/system12.1.2.0.40.65.92/o.j2ee/drs/Application1/Project1WebApp.war/

Monday, February 11, 2013

Proxying a request to localhost in Java

It can be very convenient when developing to server based application to run them using "localhost" in order to maintain consistency between developer machines. This is normally a good idea but there is a small case where this can cause problems.

Consider if you are running a local http proxy on your machine in order to capture your HTTP traffic. (Cough perhaps even the one in JDeveloper I work on). Then you might run into Java bug 6737819. Basically by default JDK 1.6 was hard coded not to send any request to localhost via a proxy which of course was a bit of a pain. Luckily a workaround was put in where you could put the string "~localhost" in your nonProxyHosts entry to turn of this feature:

java -client -classpath classes -Dhttp.proxyHost=localhost -Dhttp.proxyPort=8099 -Dhttp.nonProxyHosts=~localhost -Dhttps.proxyHost=localhost -Dhttps.proxyPort=8099 client.Example

Now moving forward to JDK 1.7 this workaround no longer works; but you need to take care to define nonProxyHosts as an empty string:

java -client -classpath classes -Dhttp.proxyHost=localhost -Dhttp.proxyPort=8099 -Dhttp.nonProxyHosts= -Dhttps.proxyHost=localhost -Dhttps.proxyPort=8099 client.Example

If you define this any anything other than an empty string the DefaultProxySelector though beware because internally it will append / or use the http.nonProxyHosts value from ../jre/lib/net.properties".

Just a minor complication that is not obvious from the published API.

Monday, November 19, 2012

JSON-Schema generation in Jersey

So in my previous post I talked about a proposal to allow the use of JSON-Schema in a WADL, this post looks at how to get this working with a recently build of Jersey. You are going to have to download / reference 1.16SNAPSHOT until 1.16 is released.

If you are using Maven this should be quite straight forward to update your dependencies assuming you already have jersey and jersey-json. You are just going to need to add a dependency on the "jersey-wadl-json-schema" artefact from the "com.sun.jersey.contribs" group to get the new feature.

If you are outside of Maven the easiest thing to do would be to download the latest jersey-archive and then the jersey-wadl-json-schema jar. How you deploy these is tool specific, but if you are using WLS then here are some specific notes on how to upgrade the version of Jersey.

Once you have this working, you need to create a WadlGeneratorConfig class in order to enable this new grammar generation:

package jersey;

import com.sun.jersey.api.wadl.config.WadlGeneratorConfig;
import com.sun.jersey.api.wadl.config.WadlGeneratorDescription;
import com.sun.jersey.wadl.generators.json.WadlGeneratorJSONGrammarGenerator;

import java.util.List;

public class JsonGeneratorConfig extends WadlGeneratorConfig {
 
    @Override
    public List configure() {
        return generator(WadlGeneratorJSONGrammarGenerator.class).descriptions();
    }
}

This can then be registered in a variety of ways, here is an example using a servlet init param. Note also that to make this example simple that we are using the Jersey POJO mapping; but whilst writing this blog I noticed that this setting affected the format of the JSON version of the WADL in case you try this.

<?xml version = '1.0' encoding = 'ISO-8859-1'?>
<web-app xmlns="http://java.sun.com/xml/ns/javaee" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://java.sun.com/xml/ns/javaee http://java.sun.com/xml/ns/javaee/web-app_3_0.xsd"
         version="3.0">
  <servlet>
    <servlet-name>jersey</servlet-name>
    <servlet-class>com.sun.jersey.spi.container.servlet.ServletContainer</servlet-class>
    <init-param>
      <param-name>com.sun.jersey.config.property.WadlGeneratorConfig</param-name>
      <param-value>jersey.JsonGeneratorConfig</param-value>
    </init-param>
    <init-param>
      <param-name>com.sun.jersey.api.json.POJOMappingFeature</param-name>
      <param-value>true</param-value>
    </init-param>
    <load-on-startup>1</load-on-startup>
  </servlet>
  <servlet-mapping>
    <servlet-name>jersey</servlet-name>
    <url-pattern>/resources/*</url-pattern>
  </servlet-mapping>
</web-app>

So I put together a really simple echo service, just to check this is all working:

package jersey;

import javax.ws.rs.Consumes;
import javax.ws.rs.GET;
import javax.ws.rs.POST;
import javax.ws.rs.Path;
import javax.ws.rs.Produces;

@Path("/echo")
public class EchoResource {
     

    @GET
    @Produces("application/json")
    public EchoBean echo() {
        EchoBean bean = new EchoBean();
        bean.setMessage("Hello");
        return bean;
    }

    @POST
    @Consumes("application/json")
    @Produces("application/json")
    public EchoBean echo(EchoBean echo) {
        return echo;
    }
    
}

and

package jersey;

public class EchoBean {
    public EchoBean() {
        super();
    }
    
    private String message;

    public void setMessage(String message) {
        this.message = message;
    }

    public String getMessage() {
        return message;
    }


}

This very simple example results in the following WADL with the JSON-Schema elements referenced:

<?xml version = '1.0' encoding = 'UTF-8'?>
<ns0:application xmlns:ns0="http://wadl.dev.java.net/2009/02">
   <ns0:doc xmlns:ns1="http://jersey.java.net/" ns1:generatedBy="Jersey: 1.16-SNAPSHOT 11/19/2012 12:59 AM"/>
   <ns0:grammars/>
   <ns0:resources base="http://localhost:7103/Jersey/resources/">
      <ns0:resource path="/echo">
         <ns0:method id="echo" name="GET">
            <ns0:response>
               <ns0:representation mediaType="application/json" xmlns:ns2="http://wadl.dev.java.net/2009/02/json-schema" ns2:describedby="application.wadl/echoBean"/>
            </ns0:response>
         </ns0:method>
         <ns0:method id="echo" name="POST">
            <ns0:request>
               <ns0:representation mediaType="application/json" xmlns:ns3="http://wadl.dev.java.net/2009/02/json-schema" ns3:describedby="application.wadl/echoBean"/>
            </ns0:request>
            <ns0:response>
               <ns0:representation mediaType="application/json" xmlns:ns4="http://wadl.dev.java.net/2009/02/json-schema" ns4:describedby="application.wadl/echoBean"/>
            </ns0:response>
         </ns0:method>
      </ns0:resource>
   </ns0:resources>
</ns0:application>

The URI application.wadl/echoBean contains this simple JSON-Schema definition:

{
    "type" : "object",
    "properties" : {
        "message" : {
            "type" : "string"
        }
    },
    "name" : "echoBean"
}

Now there are a number of limitations in the current design, not least that the generated schema doesn't take into account any notation settings. But I thought this would be enough to provoke feedback on whether this feature would be useful generally. There appears to be a growing interest in JSON-Schema both around the net and internally to Oracle, so it would be interesting to see whether this description becomes more common.

Friday, November 16, 2012

JSON-Schema in WADL

In between other jobs I have been recently been reviewing the WADL specification with a view to fixing some documentation problems with a view to producing an updated version. One of the things that because apparent was the lack of any grammar support for languages other than XML - yes you can use a mapping from JSON<->XML Schema but this would be less than pleasant for a JSON purist.

So I began to look at how one would go about attaching a JSON-Schema grammar of a JSON document in a WADL description of a service. This isn't a specification yet; but a proposal of how it might work consistently.

Now I work with Jersey mostly, so lets consider what Jersey will current generate for a service that returns both XML and JSON. So the service here is implemented using the JAX-B binding so they both use a similar structure as defined by the XML-Schema reference by the include.

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<application xmlns="http://wadl.dev.java.net/2009/02">
    <doc xmlns:jersey="http://jersey.java.net/" jersey:generatedBy="Jersey: 1.16-SNAPSHOT 10/26/2012 09:28 AM"/>
    <grammars>
        <include href="xsd0.xsd">
            <doc title="Generated" xml:lang="en"/>
        </include>
    </grammars>
    <resources base="http://localhost/">
        <resource path="/root">
            <method id="hello" name="PUT">
                <request>
                    <representation xmlns:m="urn:message" element="m:requestMessage"  mediaType="application/json" />
                    <representation xmlns:m="urn:message" element="m:requestMessage" mediaType="application/xml" />
                </request>
                <response>
                    <representation xmlns:m="urn:message" element="m:responseMessage" mediaType="application/json"/>
                    <representation xmlns:m="urn:message" element="m:responseMessage" mediaType="application/xml" />
                </response>
            </method>
        </resource>
    </resources>
</application> 

So the first thing we considered was re-using the existing element property, which is defined as a QName, on the representation element to reference an imported JSON-Schema. It is shown here both with and another an arbitrary namespace to it can be told apart from XML elements without a namespace.

<grammars>
        <include href="xsd0.xsd" />
        <include href="application.wadl/responseMessage" />
    </grammars>


    <representation element="responseMessage" mediaType="application/json"/>

Or
    xmlns:json="http://wadl.dev.java.net/2009/02/json" 

    <representation 
        element="json:responseMessage" mediaType="application/json" />


The problem is that the JSON-Schema specification as it stands doesn't have a concept of a "name" property, so each JSON-Schema is uniquely identified by it's URI. Also from my read of the specification each JSON-Schema contains the definition for at most one document - not the multiple types / documents that can be contained in XML-Schema.

So the next best suggestion would be to just use the "filename" part of the URI as a proxy for the URI; but of course that won't necessarily be unique. I could see for example the US government and Yahoo both publishing there own "address" micro format.

The better solution to this problem is to introduce a new attribute, luckily the WADL spec was designed with this in mind, that is of type URI that can be used to directly reference the JSON-Schema definitions. So rather than the direct import in the previous example we have a URI property on the element itself. The "describedby" attribute name comes from the JSON-Schema proposal and is consistent with the rel used on atom links in the spec.

xmlns:json="http://wadl.dev.java.net/2009/02/json-schema" 
    xmlns:m="urn:message" 


    <grammars>
        <include href="xsd0.xsd" />
    </grammars>

    <representation 
        mediaType="application/json"
        element="m:responseMessage" 
        json:describedby="application.wadl/responseMessage" />


The has the secondary advantage in that this format is backwardly compatible with tooling that was relying on the XML-Schema grammar. Although this is probably only of interesting to people who work in tooling / testing tools like myself.

Once you have the JSON-Schema definition then some users are going to want to do away with the XML all together, so finally here is a simple mapping of the WADL to a JSON document that contains just the JSON-Schema information. It has been suggested by Sergey Breyozkin the JSON mapping would only show the json grammars and I am coming around to that way of thinking. I would be interested to hear of a usecase for the JSON mapping that would want access to the XML Schema.

{
   "doc":{
      "@generatedBy":"Jersey: 1.16-SNAPSHOT 10/26/2012 09:28 AM"
   },
   "resources":{
      "@base":"http://localhost/",
      "resource":{
         "@path":"/root",
         "method":{
            "@id":"hello",
            "@name":"PUT",
            "request":{
               "representation":[
                  {
                     "@mediaType":"application/json",
                     "@describedby":"application.wadl/requestMessage"
                  }
               ]
            },
            "response":{
               "representation":[
                  {
                     "@mediaType":"application/json",
                     "@describedby":"application.wadl/responseMessage"
                  }
               ]
            }
         }
      }
   }
}

I am currently using the mime type of "application/vnd.sun.wadl+json" for this mapping to be consistent with the default WADL mime type. I suspect we would want to change this in the future; but it will do for starters.

So this is all very interesting but you can't play with it unless you have an example implementation. I have something working for both the server side and for a Java client generator in Jersey and wadl2java respectively and that will be the topic of my next post. I have been working with Pavel Bucek on the Jersey team on these implementations and the WADL proposal, thanks very much to him for putting up with me.