Dealing With Promotion, Or How To Change Which Web Service Gets Called Without Altering The Code

On of the difficulties of working with web services is how to deal with external dependencies as you promote code from development to QA to production. Take this JAX-WS code below as an example. Here was have a SMS service from VoodoPhone that makes use of an external translation service from Noodle.com. (Names has been changed to protect the innocent)

The problem comes from that fact that whilst using @WebServiceRef make everything really quite tidy is that the location of the WSDL, and therefore the service endpoint, is hard coded in the annotations and constants on the TranslatorService class. This makes it harder to replace the translation service with an internal or "mock" implementation during development time. You probably wouldn't want to rely on an external server for your internal regression tests for example.

Previously in JAX-RPC it was relatively obvious that you could create the proxy class and then change the endpoint. Indeed in the JAX-WS SE case you can do the same but it requires you to write code to externalize the wsdl location and this is not very JEE. Another way around the problem is to regenerate the proxy for each target system; but this is a pain and it probably takes you away from the pleasant land of the IDE into the wilds of ant tasks and complex build scripts.

I turns out that in the EE case, when using @WebServiceRef, that each injection point can be modified at deployment time with a new WSDL location. This is all done using JNDI behind the scenes; but it is suitably abstracted away from the developer.

In the next screen grab you can see the JSR-88 "Deployment Plan" dialog that you will see when you try to deploy using JDeveloper 11. (Normal disclaimers as regards to this not being production software) As you can see it has picked up the the fact that there is an injection point as part of the deployment process. There is no "service-ref" entry in my web.xml but the project the deployment process has worked out that one might be needed.

As you can see you can load and save deployment plans so you can have different configurations depending on whether you are doing development or even promoting to a production system.

You can of course do this by just editing the web.xml file yourself to alter the enviroment. The JNDI name is the name of the class and the name of the field with a slash between then. This is the "com.voodophone.SMSService/translationService" node in the tree on the left of the panel. There is no reason why you cannot type this yourself into web.xml as required or add in later using the deployment tool of your choice.

<web-app>
  ...
  <service-ref>
    <service-ref-name>com.voodophone.SMSService/translationService</service-ref-namae>
    <wsdl-file>http://localhost:8099/mockTranslation</wsdl-file>
  </service-ref>
  ...
</web-app>

The main thing to take away from this is that whilst a lot of the new annotation stuff appears to work like magic, underneath there is a JNDI assistant doing all of the dirty work. You might not have to use JNDI directly but understanding how it works under the covers allows you to leverage the indirection in some really quite powerful ways. Just remember to make sure that the different WSDL define the same interface.....

Update: Sadly the deployment plan is broken in this specific area in JDeveloper TP3, hopefully it should be working in the next release. Or you will find similar features in other tools/servers that support JSR-88.

Have-A-Patent : Improvements to Use Case Modelling

The thing about patent applications is that they take so long to process that you tend to forget about them until you find out they have been granted. I just found out that a patent application I put in when I was working on the use case modeler has been granted.

Right back to preparing for UKOUG'07.

Yikes, Leopard took my account away

I just got around to upgrading my main mac to OS X 10.5; but rather strangely I found that my user account when missing after the first attempt a logging in. Took me a while to fix the problem; but in the end tracked it down to this little treat. Looks like if you have just upgraded since the original version of OS X there is a sting in the tail if you had a nice long secure password. Oh well fixed now.

XmlAdapter, or what you are looking for it you are missing custom serializers from JAX-RPC

JAX-B 2.0 has a nice little pattern to deal with hard to serialize types. Rather than forcing the user to create xml elements you instead just have to convert your java object into another set of java objects it does know how to persist. There is a clear example of using XmlAdapter here on java.net.

The importance of good certificates and sensible names

I was looking at the latest beta of SOAP UI today and decided to try the web start version to have a play with a few of the features. I dutifully accepted the first cert which was without a root key but appears to be from eviware who write this software.....

... but my nerve failed when I saw this certificate:

I am sure just downloading the software is about as secure; but it did remind me of the importance of using sensible names when you are trying to get somebody to trust you.

Update: A friend gave me a explanation for this rather strange certificate. It comes with this crypto library.

Changing the endpoint location for a JAX-WS SE Client

Some things JAX-WS makes easy; but other things are not so obvious. One of the quite common issues that confuses developer new to this area is how you change the endpoint location for a web service proxy/client without having to regenerate/recompile the code.

It turns out you have to make use of one of those maps I enumerated in my previous entry and your code needs to looks something like this:

 1  import java.net.URI;

 2  import java.net.URL;
 3  
 4  import java.util.Map;

 5  
 6  import javax.xml.ws.BindingProvider;
 7  import javax.xml.ws.WebServiceRef;

 8  
 9  import project2.proxy.Hello;
10  import project2.proxy.HelloService;
16  

17  public class HelloPortClient
18  {
19    @WebServiceRef
20    private static HelloService helloService;

21  
22    public static void main(String [] args)

23    {
24      helloService = new HelloService();
25      Hello hello = helloService.getHelloPort();

26      setEndpointAddress(hello, "http://some.new.addr/endpoint");
27   
28      hello.sayHello("Bob");

29    }
30   
31   
32   
33    public static void setEndpointAddress(Object port, String newAddress) {

34        assert port instanceof BindingProvider : "Doesn't appear to be a valid port";
35        assert newAddress !=null :"Doesn't appear to be a valid address";

36   
37        //
38   
39        BindingProvider bp = (BindingProvider)port;

40        Map<String, Object> context = bp.getRequestContext();

41        Object oldAddress = context.get(BindingProvider.ENDPOINT_ADDRESS_PROPERTY);

42        context.put(
43          BindingProvider.ENDPOINT_ADDRESS_PROPERTY,
44          newAddress);

45    }
46  }

Note that I extracted the relevant code into a separate method, this you would probably want to move to some kind of utility class. For some reason there isn't a nice helper method to do this that comes with JAX-WS; but hopefully they will fix this in a future revision.

Update: As noted by Kurt in the comments you will probably need to have a local copy of the WSDL in order for this to work properly. Most tools will have an option to help you do this.

Update 22nd Dec 2008: After considering the problem in the real world a little bit more in the cases where you might have policies defined I now consider it better to alter the WSDL location rather than just the endpoint. This means that any extra policies on the new server are read and processed. You could do this using web.xml or a jax-ws catalog file, both can be updated at deploy time with a deployment plan.

An alternative JDK 6 for OS X

Some bods have been working at porting over the FreeBSD version of JDK 1.6 to OS X. As you might expect it usees X11 rather than native swing bindings; but that is sufficient for a lot of service side stuff. Hello I spend a lot of my day running X11 apps anyhow. Also of interest is that they have this running on 10.4 and 10.5 which the apple version of the JDK certainly wont.

Take a look at Landon Fuller's site for more information. A bit more to do on hotsport stability; but this is a valuable resource when you just gotta have the latest and greatest JDK on your mac.

Daleks over Dingle (Off Topic)

Whilst on holiday in Ireland a few weeks back in my Dormobile Campervan* I was pleased to see what looks like a huge Dalek wondering over the hills opposite Dingle.

Here is a quick snap I took of it looking over the bay from Dingle:

Just in case you think I made this up, here is the same view on google maps:

View Larger Map

Back to more sensible stuff soon I hope. (And yes I do know what this is not a real Dalek, my brain is not that addled by bug fixing)

Finding the root certificate keystore in the JDK

If you look in your JDK installation directory you will find a keystore called "cacerts" in %JDK_ROOT%/jre/lib/security. This keystore contains all the root certificates used when validating regular web pages. This is the one used by default for operations such as opening SSL connections.

Now some tools, <cough> the HTTP Analyser in JDeveloper </cough>, might have a bug that prevents it from using a keystore that doesn't have a password. But it is a easy step to import all of the certificates into a keystore that contains a password.

keytool -importkeystore -srckeystore cacerts -destkeystore /tmp/output.keystore -deststorepass welcome

Building an async JAX-WS from the bottom up.

Now and then in JDeveloper land we have an organization wide bug hunt to improve the overall quality of our product. For the most recent one I decided to try to write a async web service from the bottom up.

For the purposes of this experiment I am going to create a simple HelloWorld example so first off lets created the primary interface:

package model;

import ...;

@WebService
public class Hello
{

  public void sayHello(String name)
  {
  ...
  }
}

And of course we need to define the interface for the callback service:

package model;

@WebService
public interface HelloCallback
{
  public void response(String message);
}

Simple enough, right? Now I we are going to create a static wsdl under public_html/wsdl. Use the normal wizard to create a blank document then cut and paste the results of "Show WSDL for annotations" from the Hello class. You also need to do something similar for the HelloCallback service. Unfortunately you can't show the WSDL for a SEI, although I have raised that bug, so the quickest way to generate the WSDL content is to temporarily convert HelloCallback to a class.

To complete the WSDL add a partnerLinkType, this is used by tools such as BPEL to link the two halves of the equation:

    <partnerLinkType name="HelloService"
                     xmlns="http://schemas.xmlsoap.org/ws/2003/05/partner-link/">
        <role name="In">
            <portType name="tns:Hello"/>
        </role>
        <role name="Out">
            <portType name="tns:HelloCallBack"/>
        </role>
    </partnerLinkType>

Then it is worth updating the Hello class to reference this WSDL in the annotations:

package model;

@WebService(wsdlLocation = "/wsdl/Hello.wsdl")
public class Hello 
{
  ...
}

Okay so the specifications are in place lets look at a trivial implementation of the sayHello method. Note that in this case the response can arrive before the original call has completed; but this will do for this example.

SOAPMessageContext smc = (SOAPMessageContext)_context.getMessageContext();
        
// Get hold of the AddressingProperties and look up the reply to address
// and later the return message id
//
        
AddressingProperties ap = AddressingUtils.getAddressingProperties(smc.getMessage());
EndpointReference endpointReference = ap.getReplyTo();
AttributedURI address = endpointReference.getAddress();
URI replyTo = address.getURI();
        
// 

try
{

  // Use the servlet context to get a reference to the local wsdl
  //  

  ServletContext sc = (ServletContext)_context.getMessageContext().get(
    "javax.xml.ws.servlet.context");
  URL found = sc.getResource("/wsdl/Hello.wsdl");
 
  // Create a new service instance on the callback

  Service s = Service.create(found, new QName(
                             "http://model/",
                             "HelloCallBackService"));
            
  // Create a port

  HelloCallBack acb = s.getPort(
           new QName(
               "http://model/",
               "HelloCallBackPort"),
               HelloCallBack.class); 
            
  BindingProvider bp = (BindingProvider)acb;
  Map requestProperties = bp.getRequestContext();

  // Change the endpoint address to the match the replyTo address
            
  requestProperties.put(
    BindingProvider.ENDPOINT_ADDRESS_PROPERTY,
    replyTo.toString());

  // Make sure that this request has the correct addressing headers
            
  AddressingUtils.setupClientSideWSAddressing(requestProperties);
  String t = requestProperties.keySet().toString();
  AddressingProperties rap = 
    AddressingUtils.getOutboundAddressingProperties(requestProperties);
  rap.setRelatesTo(
    ap.getMessageID());
       
  // Invoke the response address
  //
     
  acb.response("Hello " + name);
}
catch (Exception ex) {
  ex.printStackTrace();            
}
        

Part of the pleasure of using JAX-WS over JAX-RPC is the simplicity of invoking services dynamically in this way. As you can see above this is actually really quite straight forward.

So how would you go about testing this, well the HTTP Analyzer in JDeveloper doesn't support WS-Addressing headers properly although we are working on this. You can of course create your own headers in the HTTP Content view, it might look like this:

<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
   <soap:Header xmlns:ns1="http://www.w3.org/2005/08/addressing">
      <ns1:MessageID>ws:uniqueAddress</ns1:MessageID>
      <ns1:ReplyTo>
         <ns1:Address>http://returnAddress.com/example</ns1:Address>
      </ns1:ReplyTo>
   </soap:Header>
   <soap:Body xmlns:ns2="http://model/">
      <ns2:sayHello>
         <arg0>John</arg0>
      </ns2:sayHello>
   </soap:Body>
</soap:Envelope>

If you start up the HTTP Analyzer before the embedded server you can send his message to the endpoint and you should see the failed attempt to send the message to the returnAddress url in the analyzer log. It should look something like:

<env:Envelope xmlns:env="http://schemas.xmlsoap.org/soap/envelope/" xmlns:wsa="http://www.w3.org/2005/08/addressing">
   <env:Header>
      <wsa:To>http://returnAddress.com/example</wsa:To>
      <ns1:MessageID>ws:anotherUniqueAddress</ns1:MessageID>
      <wsa:RelatesTo xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" soap:mustUnderstand="0">ws:uniqueAddress</wsa:RelatesTo>
   </env:Header>
   <env:Body>
      <ns0:response xmlns:ns0="http://model/">
         <arg0 xmlns="">Hello John</arg0>
      </ns0:response>
   </env:Body>
</env:Envelope>

This is worth remembering this trick if you are trying to test a async web service and don't want to go to the bother of setting up a valid return address to connect to.

Next time I get a few minutes I will take a look at the client side of this equation. Back to fixing the bugs I have just found.

A viewlet showing Abbot being used to test relatively complex UI

As I have mentioned previously we have been playing with Abbot and Costello internally in order to automate testing of some of our UI. When you start working Abbot/Costello there is a steep learning curve with people not sure that they will ever get there tests run more than once. (Some of the reasons I have talked about before.) Once you get the hang of things though setting up tests becomes much easier with recording tests becoming a normal part of the working day.

I figured it would be usefully to publish a viewlet showing abbot testing some relatively meaty UI. So here is a recording of current smoke tests for JDeveloper web services tooling which were created in Costello and run using Abbot.

There are a few glitches in that some context menu are not recorded and the playback speed is slower than the tests actually run on my machine. In general though this gives a fair representation of a test run on my machine. Also if you look carefully you will notice some bugs that the tests are not picking up; but I will leave that as an exercise for the reader to figure out what is wrong.

More than meets the eye, or JAX-WS and the not so obvious interfaces.

I really enjoy using JAX-WS I have only come to web services relatively recently but I have found the programming model to be relatively easy to use. Certainly removing the need to create n xml mapping files has made the developer's life much easier.

One problem I have with the design of JAX-WS it the way that you end up having to cast common objects to other helper interfaces. Now most of the implementation is done using proxies so in most debuggers you are going to have a hard time understanding the objects you are passed. Lets just knock up a quick method to list all the interfaces for a bit of debugging:

public class Util
{
    public static void dumpInterfaces(Class t)
    {
       System.out.println("=== Interfaces for " + t.getSimpleName());
       List interfaces = new ArrayList();
       interfaces.add(t);
       for (int i=0; i < interfaces.size(); i++) {
           Class it = interfaces.get(i);
           for (Class sp : it.getInterfaces()) {
               if (!interfaces.contains(sp))
               {
                 interfaces.add(sp);
               }
           }
       }
       
       // List them out
       //
       
       for (Class sp : interfaces)
       {
         System.out.println(sp.getName());
       }
    }

    public static  void dumpMap(String name, Map map) 
    {
        System.out.println("=== " + name);
        for(Map.Entry e : map.entrySet()) {
            System.out.println(e.getKey() + " : " + e.getValue());
        }
    }
}

On the server side lets take a look at what interfaces that might not otherwise be obvious, take this simple web service example:

@WebService
public class Hello {
    
    @Resource
    private WebServiceContext context;
    
    
    public String sayHello(String name) {
        
        Util.dumpInterfaces(context.getClass());
        Util.dumpInterfaces(context.getMessageContext().getClass());
        
        Util.dumpMap("Dumping message context properties",context.getMessageContext());        
        
        return "Hello " + name;
    }
}

This gives the following output for the interfaces. The web service context is less interesting that the message context. This can be cast to SOAPMessageContext which can give you access to the original SOAP message along with the headers. This is an alternative to using Handlers in order to process headers.

=== Interfaces for WebServiceContextImpl
oracle.j2ee.ws.server.jaxws.WebServiceContextImpl
javax.xml.ws.WebServiceContext
=== Interfaces for SOAPMessageContextImpl
oracle.j2ee.ws.common.jaxws.SOAPMessageContextImpl
javax.xml.ws.handler.soap.SOAPMessageContext
javax.xml.ws.handler.MessageContext
java.util.Map

The since the message context is also a map it is relatively easy to list all of the properties defined in there. Here is what I see when I run the service defined above. Part of the problem with JAX-WS is that these properties are extensible so it is hard to get a definitive list of all of them. Something for a future blog I think.

=== Dumping message context properties
javax.xml.ws.binding.attachments.inbound : {}
javax.xml.ws.http.request.pathinfo : null
incomming.addressing.properties : amespace=http://www.w3.org/2005/08/addressing
javax.xml.ws.servlet.context : Application1-Project1-webapp web-app
javax.xml.ws.http.request.querystring : null
javax.xml.ws.http.response.code : null
javax.xml.ws.http.response.headers : null
javax.xml.ws.servlet.request : com.evermind.server.http.EvermindHttpServletRequest@109b23b
oracle.j2ee.ws.context.Message : oracle.j2ee.ws.saaj.soap.soap11.Message11@1945696
javax.xml.ws.binding.attachments.outbound : {}
javax.xml.ws.servlet.response : com.evermind.server.http.EvermindHttpServletResponse@1c0bec5
javax.xml.ws.http.request.headers : {SOAPACTION=[""], CONNECTION=[Keep-Alive, TE], CONTENT-LENGTH=[202], USER-AGENT=[Oracle HTTPClient Version 10h], TE=[trailers, deflate, gzip, compress], ACCEPT-ENCODING=[gzip, x-gzip, compress, x-compress], CONTENT-TYPE=[text/xml; charset=ASCII], HOST=[localhost:8988], ECID-CONTEXT=[HIINUXQV`[YegG@DKKGSLNWWZZ[Zbef@BKCAGSLNS[TQ]}
javax.xml.ws.http.request.method : POST
javax.xml.ws.handler.message.outbound : false

We can do something similar with the on the client side, so if we were to create a simple client:

public class Client {
    public Client() {
    }
    
    public static void main(String argv[]) {
        HelloService hs = new HelloService();
        Hello h = hs.getHelloPort();
        Util.dumpInterfaces(h.getClass());
        
        BindingProvider bp = (BindingProvider)h;
        Util.dumpInterfaces(bp.getBinding().getClass());
        Util.dumpMap("Request context",bp.getRequestContext());
        Util.dumpMap("Response context",bp.getResponseContext());
    }
}

We can see that the port is implemented using a proxy object that implement both the SEI and the BindingProvider interface. The BindingProvider interface is perhaps not that much of a secret, although I would have thought that a more usable design for the client generation would have seen the SEI extend this interface to make the programming model cleaner.

=== Interfaces for $Proxy12
$Proxy12
project1.proxy.Hello
javax.xml.ws.BindingProvider
=== Interfaces for SOAPBindingImpl
oracle.j2ee.ws.client.jaxws.SOAPBindingImpl
javax.xml.ws.soap.SOAPBinding
javax.xml.ws.Binding
=== Request context
javax.xml.ws.binding.attachments.outbound : {}
=== Response context

It is worth noting that again there is a SOAP variant that you need to cast to in order to get all of the functionality available to you. Of course you have to be careful that you understand the context of your service. For example if you were to using the HTTP binding than the Binding object would of course be HTTP not SOAPBinding.

Finally where I have dumped out a the contents of a map, these represent the current state when running on an interim build of OC4J 11. As noted before there are many more properties that are available that can be added depending on what platform and extensions you are using.

Update: A nice summary document for the standard MessageContext and BindingContext parameters can be found here.

Q: How do I monitor outgoing HTTP Traffic in OC4J, or any other java app for that matter

Make sure that -Dhttp.proxyHost or -Dhttp.proxyPort is set

When you use the HTTP Analyzer within JDeveloper you will find that all java processes started from then on will send there traffic via the HTTP Analyzer. This includes the embedded oc4j instance. This can be useful if your service is using external services such as a credit card validation system.

Is the relatively straight forward to do this trick for any java process that might have been started outside of JDeveloper. You simply need to supply something like -Dhttp.proxyHost=localhost -Dhttp.proxyPort=8099 to the java executable. You can of course alter the host name if the process is running on a different machine.

If you are running an instance of oc4j you tend to do this with with oc4j or oc4j.bat. These both read the environment variable OC4J_JVM_ARGS which can contain the parameters mentioned above.

This may not capture all traffic between SOA services, such as BPEL, as in some cases inter-process communicate will skip the "HTTP Protocol" stack and exchange infosets more directly. This is something we will of course look at in future releases.

"Wait, Don't Delay" or how to test complex async Java GUIs

I have been working with abbot and costello for some time now and people tend to have one of two problems getting there scripts to run reliably: the first is not being able to find a component again; and the second is the real time nature of most complex swing UIs.

The first issue can usually be solved by ensuring the that program is accessible and that use of the "name" property where required. Things do get more complicated with nested component; but they can generally be solved with a little bit of common sense.

The second issue is the real topic of this post and it sometimes takes quite a while to change around the mindset required to create tests that will run reliably on a range of platforms and systems.

Take for example an wizard that needs to query a database between pages. This means that the required components might not be enabled instantly after the user action. Now a naive user might try to include a "delay" step because they know how long the action might take; but they will find that eventually the test will fail on one platform or another. They might also try to use the waitForIdle action which is a better guess; but unfortunately not reliable and it might never return in the case of a dialog with an ongoing animation.

In all cases where delays and waits have been used in our tests we have seen then resulting in unstable test suites. The only thing I can think that delay should be used for in day to day tests is using it to ensure a piece of functionality doesn't take too long to run. Even then it will be hard to pick a value that will work on all platforms being tested upon.

Lets look at a few ideas on how you can detect whether the task has completed.

The first and most important thing you can look at is obvious UI changes that signal that the asynchronous task as finished. This might be as simple as waiting for a component to be showing using "Insert"->"Wait For"->"Component Showing". Or in the case where the UI component is always displayed you can use an assert step to ensure that a given component is enabled. In Costello this is done by selecting the "References" tab, then go to the "Properties" tab and find the "enabled" property, hold down Control and use the button that appears to insert a "Wait For" step.

The one exception to this are progress dialogs which are generally not a reliably way of tracking a task. If a progress dialog is implemented properly they might never show if the task is really quite quick.

You can also monitor lists and trees for the addition of new nodes. In costello you only need to add a selectRow step for a tree and code being the scenes will wait until that node path becomes available. (I did some of the work for this). You do find that when you have some experience with costello that you instinctively select elements in list and tree because they provide you will a cheap way to test the current state of the model.

If you are working in a product such as an IDE then you will find that many asynchronous tasks write to a log window of some kind. The example of this in JDeveloper is when we try to start and embedded oc4j instance and we have to monitor the log window to know when it has finished starting. To deal with this we use the "Call" step to invoke a static java method that does the work for us. We might also have achieved the same results with an "Expression" step; but they can be harder debug.

If the worst comes to the worst then you do end up having to provide some kind of behind the scenes API. In the work I have been doing on JDeveloper it proved to be easier to invoke the build system programatically using an Expression step rather than the UI directly. You can also use "Call" or "Expression" steps to query the state of UI components that you don't have tester for, although in the long term you are much better or writing the testers in the first place. An example I currently have on my queue waiting is a custom tester for the "gutter" in the code editor in JDeveloper. Without that it is very hard to work out when "Audit" has finished applying its rules in order to invoke a "Quick Fix". Currently a test is this area is using a very long delay which I am less than pleased with; but I hope to remove this soon.

Finally we have implemented a kind of "internal" log window using Logger for those cases where we can't think of something more creative. This provide a way for actions to say "I am finished" and for abbot to be able to pick this up. In general this is only for use as a sticking plaster until we can put the right tester in place.

Configuring the HTTP Analyzer to record HTTPS traffic

One of the cool new feature we added to the HTTP Analyzer in 11 is the ability to monitor https traffic. This gives you the ability to see all of the crossing traffic in plain text. The configuration for this though is a mite complicated and didn't make it into the documentation for the Technology Preview so I thought that I should write it up here. To make things simple I am just going to look at try to sniff my password from amazon.co.uk; but the same principles apply when working with other HTTP traffic such as SOAP messages.

The first thing you need to do is to use the keytool command to create a server side key. For various security and legal reasons we can't ship a default keystore with the product; but it is fairly easily to create one yourself:

keytool -genkey -alias oracle-server -dname "CN=Gerard Davison, OU=HTTPAnalyzer O=Oracle L=Reading, S=Berkshire, C=UK" -keyalg RSA -keypass welcome -storepass welcome -keystore server.keystore

You then need to configure the HTTP Analyzer, bring up the preferences dialog and set this keystore for both the client and the server. You can of course use two separate keystores if you wish. This is more likely to be useful when working with web services rather than standard web traffic.

In this example I am going to look at using the HTTP Analyzer to look at traffic going to amazon.co.uk. To do this I need to configure my web browser to use the analyzer as its proxy. This by default will be localhost with port 8099.

For the moment, although this is something I am working on, you also need to manually import any certificates for website you are going to visit. Some ideas on how your might go about this are in my previous blog entry on this topic. The snippet of java code appears to be the most reliable option. Update: Or you can use the root certificate keystore that comes with java with a workaround. He have made this much easier in more recent internal builds of JDeveloper.

Now start the HTTP Analyzer and try to log into www.amazon.co.uk using this proxy configuration. You will see a bunch of paranoid messages complaining about how the certificate is not right. This is normal and your should just accept the certificate for the session. (But it is a reminder of why you should never ignore these dialogs in the real world)

If you continue to log in the you should see that there is a bunch of requests, one of these will contain your username and password in plain text. (I am not going to show you a screen grab of this). You will also notice a whole bunch of error that relate to fetching images. This is because you need to import a certificate for each server you are talking to. In the case of amazon.co.uk images are served from "https://images-eu.ssl-images-amazon.com".

Being able to watch traffic in this way is very valuable tool when debugging interactions. AJAX services are really quite interesting to watch in this way. You can also record, replay and mock requests as you would un-encrypted traffic.

Have-A-Job Junior Test Engineer. Reading, UK

We have an opening for a junior test engineer working on Oracle JDeveloper in the Reading, UK who has an interest in Java and a desire to learn new things. The successful candidate will gain experience in testing a range of JDeveloper components both manually and using automated testing tools. They will also have the benefit of being co-located with the development team and be able to work closely with them on a day to day basis.

We have a lovely office overlooking the Thames which is currently in the process of flooding. Thankfully the lovely riverside and nature reserve provide a great flood plain so the office stays dry. We have a great benefits package including the ability to buy up to 35 days leave a year, not including bank holidays.

If you are interested, drop me a comment and I will put you in touch with the right people.

Setting a watchpoint to monitor the access of a field

Some times you really need to track how a field on a class is used and many a time I am working with somebody else's code and find there are not handy accessors to breakpoint. This might be down to poor code encapsulation, or in the case I came across today I just wanted to know who was using the value in given constant. Find usages would give me the static view; but I wanted to find out when this instance was being used at run-time to narrow my search even more.

Now by JDeveloper lets you set a "watchpoint" on a field that will by default break each and every time that the field is modified. I didn't occur to me until today that it would also allow me to monitor the access of a given field, at least until I was debugging a complex case and thought wouldn't it be nice if..... well it turns out that you simply have to open the breakpoints tab edit the breakpoint:

This will not of course work for scalar or String static final types where the compiler is free to copy the actual value between classes. In that case the debugger will not be able to detect a "read". But in this afternoons debugging session it was a real life saver.

Refurbishing your RoboMow Powerpack

So I made the foolish mistake of leaving my already old and decrepit battery pack in the shed over winter and a few very cold nights appear to have done for its ability to hold any charge. I decided that it was time to replace the battery pack and get myself an external charger so the little thing doesn't need to sit in the kitchen to feed. (I have no power outside the house)

A quick look at the uk distributor suggested that perhaps UKP140 for the battery pack and UKP99 for the quick charger was a mite expensive. Time for a little bit of DIY I think. A bit of applied pressure later and I had the top off the battery pack.

It turns out my pack was made up of two Yacel Y17-12 lead gel batteries. These are no longer made but the nice chaps at Yuasa suggested the replacement model NP17-12. In the end I went with NPC17-12 which is designed for cyclic applications such as the lawn mowers and golf buggies. (Notice the extra C) In the end I picked up a pair from BatteryMasters for UKP 68. You need two 12v cells rather then one fat 24v otherwise they wont fit in the packaging unless you are willing to do some more serious modifications to the holder. Depending on how your battery pack in configured you might also need to pick up some crimps as I need to convert from spades to bolts.

Finally while I was at the shop I picked up a charger; but in retrospect I should have gone with the 4amp version which is now 1 UKP less. Still it charges the entire battery overnight which is much better that the standard 22 hours charge time for the standard power adapter. You don't need any special connector, just use the supplied bulldog clips on the big bits of brass on the back. The red one should go to the slider nearest to the fuse. If you get it wrong the LEDs on the charger should flash. No more mower in the kitchen.

I would think that doing the battery upgrade should be doable by anybody who can wire a plug or do simple car maintenance. Just make sure you keep track of polarities when you take the old batteries out and put in the new. As with anything you read on the web though you do this at your own risk etc. etc. results might vary, you might well cause your robomow to burst into flames.

140UKP saving for a little bit of web searching, well worth it. Battery now lasts far longer than it takes to mow the lawn which is better than when I bought the thing refurbished to begin with. Oh and don't leave your battery out in the frosts!

Paper Accepted for UKOUG'07

Looks like I will be presenting at UKOUG this year, my paper titled "Testing and Debugging SOA and Web Service Applications" has been accepted for the afternoon of the 5th of December. Better get my thinking cap on then.

Q: How do I get a remote server's SSL Cert?

A: Not 100% securely over the internet, but there are a few options

So you have to work with a web service over SSL? In order to get this to work you are going to need a certificate to import into your local key store using the keytool command.

The problem is that communication over the internet can be very easily intercepted so ideally you would receive the certificate in person from the provider preferably with the providers parents and partner there to verify the authenticity of the certificate.

Having said that if you are not entirely paranoid you can normally rely on a certificate that is signed by a root authority. Your web browser will do a reasonable job of checking this and just accepting the certificate that the server gives out is likely to be secure enough for most low key operations. You would probably want to be more paranoid if you are exchanging millions of pounds.

The question comes as to how do I get hold of this in a handy format that I can then pass into the keytool? Well the first and easiest way is to ask your web browser although sadly this trick only works for IE7. So connect to the endpoint you are going to work with or just request the WSDL over SSL. Then click on the padlock icon in the url status bar and view the certificate. Then travel to the "Details" page and voila there is a "Copy To File..." button which invokes a wizard to generate a .CER file that keytool with understand.

So how do you do the same trick from Java? Well it is easy enough to get a list of server side certificates using java and write the first one directly into the key store:

 public static void main(String[] args)
    throws Exception
  {
    assert args.length == 3 : "Should have three parameters keystore, password, site";
    File keystoreFile = new File(args[0]);
    assert keystoreFile.exists();
    char[] password = args[1].toCharArray();
    URL url = new URL(args[2]);

    // Load the keystore
    //
    
    KeyStore keyStore = KeyStore.getInstance(KeyStore.getDefaultType());
    keyStore.load(
      new BufferedInputStream(new FileInputStream(keystoreFile)),
      password);    
  
    // Create a connection
    //

    HttpsURLConnection con = (HttpsURLConnection) url.openConnection();
    con.connect();
    Certificate certs[] = con.getServerCertificates();
    
    // Get the first certificate
    //
    
    Certificate firstCert = certs[0];
    
    
    keyStore.setCertificateEntry(
      url.getHost(), firstCert);
      
    // Store the keystore
    //
      
    keyStore.store(
      new BufferedOutputStream(
        new FileOutputStream(keystoreFile)), password);
    
  }

Again this is a way of getting a certificate is not 100% secure; but might well do for a bit of development. Also it is probably the only choice when dealing with entities such as google or amazon.

For more fun and games with HTTPS check out Tug's Blog on the topic of configuring HTTPS in oc4j and on the client side.

Q: Why won't my JDeveloper JAX-WS web service deploy to glassfish?

A: Not so standard standards

We have a few reports of customers building JAX-WS web services in JDeveloper 11 Technology Preview; but finding that they get some strange errors when trying to deploy these services to glassfish. (Also Tomcat which I haven't had time to play with; but I suspect that they both have the same root cause)

To investigate this I downloaded the latest beta of glassfish version 2 to try to work out what is going on. This prove to be really quite easy to do using the quick start guide. As detailed in the guide I dropped my simple example web service in the auto deploy directory and was rewarded with a pleasing _deployed response file.

Right I thought, lets use the web service. So I went to the management console to see if they have a tester like OC4J. Indeed they did with a handy page listing all deployed web services; but this is where things started to fall apart. Any attempt to test the service or view the WSDL would be met with a 404 response.

It turns out that if I had looked in the server.log file I would have found the following message:

[#|2007-07-06T11:07:03.891-0400|SEVERE|sun-appserver9.1|javax.enterprise.system.container.web|_ThreadID=19;_ThreadName=httpWorkerThread-4848-1;_RequestID=e309ad42-9897-4fcd-97cf-e1352c5766df;|WebModule[/WebServices]PWC1396:
Servlet /WebServices threw load() exception
java.lang.ClassCastException: org.kingsfleet.MathService cannot be cast to
javax.servlet.Servlet
       at
org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1132)
       at org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1007)
       at
org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:4800)
       at org.apache.catalina.core.StandardContext.start(StandardContext.java:5188)
       at com.sun.enterprise.web.WebModule.start(WebModule.java:324)

This relates to the servlet-class element in the web.xml that we, JDeveloper, generates when we create the service for the first time. This is a quick way to specify how the endpoint maps to a URL. Here is a simple example:

<?xml version = '1.0' encoding = 'US-ASCII'?>
<web-app xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://
java.sun.com/xml/ns/javaee http://java.sun.com/xml/ns/javaee/web-app_2_5.xsd" version="2.5" 
xmlns="http://java.sun.com/xml/ns/javaee">
    <description>Empty web.xml file for Web Application</description>
    <servlet>
        <servlet-name>MathServicePort</servlet-name>
        <servlet-class>org.kingsfleet.MathService</servlet-class>
        <load-on-startup>1</load-on-startup>
    </servlet>
    <servlet-mapping>
        <servlet-name>MathServicePort</servlet-name>
        <url-pattern>/mathserviceport</url-pattern>
    </servlet-mapping>
    <session-config>
        <session-timeout>35</session-timeout>
    </session-config>
    <mime-mapping>
        <extension>html</extension>
        <mime-type>text/html</mime-type>
    </mime-mapping>
    <mime-mapping>
        <extension>txt</extension>
        <mime-type>text/plain</mime-type>
    </mime-mapping>
</web-app>

Now it may not make sense to put a class that doesn't implement servlet in this place; but that is how the people who wrote JSR 109 decided it should work. I have to say that it is a nice clean and apparently platform non-specific way of working.

If you remove the servlet mappings or indeed the web.xml from the war file (Okay hands up; you can't create a war file without a web.xml in the technology preview logged as bug 6195725) then this web service will deploy, but at a slightly different URL to the one you actually wanted. To take control over the endpoint url then you will need to create sun-jaws.xml and modify web example as in this tutorial. This is less than optimal because it contains platform-specific implementation classes that might not be portable across different app servers.

I have raised a bug to track this issue but the glassfish people don't appear to be fussed about fixing this problem even though it appears to break the CTS compliance tests. They do seem to be fixing the admin console so that it won't show a web service that has failed to deploy which at least will reduce the level of confusion just a little bit.

Hopefully one day we will get WORA for web services; but it looks like we are not quite there yet.

Understanding Rules in the HTTP Monitor in JDeveloper 11

One of the problems with releasing technology early is that sometimes the documentation has trouble keeping up. This appears to have been a particular issue with the new HTTP Analyzer which as I have talked about previously has introduced the concept of Rules. Unfortunately the on-line help we shipped with the technology preview was not updated to describe the new rules. So I am going to give a quick little summary of the different types here:

Rule Name	Description
Tape	This rules allows you to intercept one or more URLs and response with pre-canned messages taken from a WSI-Log file. This is used to create effect in my mock paypal demo I showed before. Of course this can be useful for any HTTP traffic.
Forward	This rule will intercept all URL matched by the filter and forward the request on to a single URL. Useful for testing or perhaps for replacing your iPhone OS.
URL Substitution	This rule was envisioned to allow the user to re-host services by replacing parts of URL ranges. For example replacing the machine name is particularly useful when moving between the embedded and external oc4js.
Pass Through	This is the default rule and for the given range of URL process the request normally through the normal proxy settings. This is generally going to be the last rule in the list with a blank filter catch all that remaining messages.

One important thing to remember with rules is that they are processed from top to bottom and that a blank URL filter will match anything. Unfortunately the controls for adding Rules in the technology preview will tend to add the rules at the end of the list. This will put them after the default pass through rule which by default has a blank filter and therefore will match everything.

We have received some user feedback that the rules are hard to get right, So to that end in the nightly builds we have been playing with an inline filter tester which may or may not make it to production. I would be interested to get feedback on this:

This will also show you when the filter is invalid. Also, unlike the developer preview, we now correctly validate the rule panel and wont let you leave the dialog until they make some kind of sense.

Right hopefully that is enough to be going on with. We are quite proud of the HTTP Analyzer in JDeveloper and it compares very well with the functionality of other products. (For example the equivalent tool in Eclipse WTP or NetBeans) It can be applied to anything that makes use of the HTTP protocol so can be used to debug anything from

Using "Object Preferences" in JDeveloper 11's debugger to shed light on complex types

If you have programmed in Java for a long time you will probably have ended up in a debugger session with an exception class that either some other person has failed to log or just never get traced out before and another exception is thrown. Now you could just modify the code to put in a printStackTrace; but these can be forgotten which is a real pain when they end up in the production code. I ended up wondering if it was possible to get JDeveloper's debugger to display a little bit more information. Turns out you can.

What you need to do is to set up the "Object Preferences..." for the given type. The easiest way of doing this is to right click on a field in the debugger's data window. By default the combo box will display the settings for the "current" exception; but we want to make this change work for all exceptions so drop this down and then select "Throwable".

Now in the "What to show..." section select the expression check box and add the string "getStackTrace()". Note it has to use the method and not the field as this information is fetched lazily. When you have finished the dialog should look something like this:

Right so you should now see all the stack trace elements appearing under the exception node, the thing is that this doesn't really give you any useful information. To do this you need to make sure that the preferences for StackTraceElement are correct. If you take a look at the next screen grab you will see that I have set the "What to display..." section to use toString(): You should now see something like the following the debugger data window This was kind of promising so I decided to take a look at whether I could improve the debugger information displayed when working with DOM elements. You might find, that specially when working with Oracle's XDK, that the information provided to you in the debugger is less than usefull. You cannot easily for example examine a list of child nodes. This proved to be a little bit harder to display in the debugger as you have to get the debugger to understand the class "org.w3c.dom.NodeList". Turns out that you can happily call static method on other classes. I happened to have a method that converts a NodeList into an array of Node instance that looked like this:

 public static Node[] convert (NodeList list)
 {
   Node nodes[] = new Node[list.getLength()];
   for (int counter = 0; counter < list.getLength(); counter++)     
   {      
      nodes[counter] = list.item(counter);    
   }         
   return nodes;  
 } 

And it turned out to be a simple matter to plug this into the debugger: And here are the results shown in the data window: You could of course take this further an change the convert method so that it returns a structure with more information in it, such as what is the parent of the current element. Or indeed add methods to your objects to provide this information at debug time.

So this all sounds like fun, so what is the downside? Well it does make the debugger run a little bit slowly; but otherwise it does seem to be a win for programmer productivity.

Note this is a retread of a previous blog posting; but I think it makes sense to republish it as there is a now a public release to try this with.

XQuery API for Java, JSR 225, sadly missing external functions

The public review of of the XQuery interface for java is avaliable. The interface appears to be reasonable but it unfortunately has one major ommission: the ability to define external functions in Java. This is particularly anoying as this is a feature that I rely on day to day for my usage of XQuery in JDeveloper.

In my case I use XQuery to walk over WSDL and XSD documents to provide validation and contextual linking. In order to do this in synch with the user I need to be processing the in memory buffers rather than those in the file system. Obviously the standard "location" function doesn't know about these in memory copies. Instead I defined a new version of the location method which I implemented in java. My version can take multiple relative paths in order to make the code run more efficiently:

  (: Declare external functions  :)

  declare function xmlef_func:location($doc as item(), $location as xsd:string*) as node()* external;

This I can quite happily implement using some internal classes that I have access to in the Oracle internal implementation of the xquery api. I am told that most JSR 225 implementations will provide some extension mechanism for defining external functions; but that it wont be part of the standard. This of course will make it real hard to make portable code. It is also personally quite supprising because the XPath API already included in java does provide a mechanism for defining external functions.

Of course there might well be time constraints that I am not aware of; but it would seem a shame for the standard to not define an interface to this most useful of extension mechanisms. It would seem like the main focus for the JSR 225 expert group has been for XQuery to be used an a public API for querying data sources such as database connections. This is a real shame for me personally as I can get a lot of mileage out of XQuery for common XML processing actions. (And it is boon for those like me who find XSL harder to read back)

Having said that I wouldn't let this put you off using XQuery with Java, personally I find it by far much easier to write and read than XSL. There is always something to be said about using a different language notation when you are doing someting quiet different. With XQuery it is always immediately obvious what is XML and what is program construct which can really help with readibility.

Moving Day

Looks like it is time to move from the very comfortable orablogs onto something a bit more modern. I would like to thank Brian a lot for hosting my blog over the years. He tells me that due to HD issue that I cannot update my blog there so there will be no redirects for the moment; but he hopes to fix this in due course.

Well then, back to work I think.