Thursday, April 7, 2016

Converting string configuration properties to other types, with a bit of Optional.

Somedays you come across some code and think that's pretty, why didn't I think of that? So my long time colleague Mark Warner has a nice twist on the standard name/value store pattern using method references to deal with converting from a String.

int size = store.getProperty("cache.limit", 500, Integer::parseInt);
    boolean enabled = store.getProperty("cache.enabled", true, Boolean::getBoolean);

I took his example and refactored it slightly to return Optional, and I ended up with the following:

public Optional<String> getProperty(
        String propertyName) {
    return Optional.ofNullable(map.get(propertyName));
}
   
public <T> Optional<T> getProperty(
        String propertyName, 
        ThrowingFunction<String,? extends T,? extends Exception> func ) {

    return getProperty(propertyName).map(val -> {
        try {
            return func.apply( val );
        } catch ( Exception e ) {
            LOGGER.severe( () -> "Invalid property transform, will default " + e.getMessage() );
            return null;
        }
    });
}

This means that the default value ends up being provided by the Optional which is a nice application of OAOO.

int size = store.getProperty("cache.limit", Integer::parseInt).orElse(500);
    boolean enabled = store.getProperty("cache.enabled", Boolean::getBoolean).orElse(true);

I think this is even tidier; but it does depend on who you feel about using Optionals.

Thursday, March 10, 2016

Lambda of Lambda, if/else from an Optional

So I got frustrated with two limitations of the Optional interface in JDK 8. The first problem is that there is no obvious way to perform an else operation in a block as there is only a isPresent method unless you are using an old school if statement. The second problem is of course the old chestnut that even if you could do that the methods would not be able to throw a checked exception. (Yes you can wrap with a RuntimeException but it is not the prettiest.)

The workaround I found was to use the map function as the success case and the orElseGet to return the failure case. In both branches the code returns an instance of ThrowingRunnable by having a lambda return a lambda. The run() is then called at the end and it can throw any exception it wants to.

@FunctionalInterface
public interface ThrowingRunnable<E extends Throwable>  {
    
    public void run() throws E;
}



Optional<Credential> credential = ....

credential.<ThrowingRunnable<IOException>>map(auth -> () -> {
                PasswordWrapper pw = auth.getToken();
                ... // something that might throw an IOException
            }).orElseGet(() -> () -> {
                        response.setStatus(401);
                        LOGGER.log(Level.INFO, "credential is not found");
                    }
            ).run();


This is possibly excessive for this particular use case; but I can see this technique being useful elsewhere and it is worth knowing what it looks like so it is not a surprise in others code.

Update 14th March 2016: Thanks to Michael Rasmussen on the DZone version of this article he noted I could use orElse rather than orElseGet to remove the second double lambda.

credential.<ThrowingRunnable<IOException>>map(auth -> () -> {
                PasswordWrapper pw = auth.getToken();
                ... // something that might throw an IOException
            }).orElse(() -> {
                        response.setStatus(401);
                        LOGGER.log(Level.INFO, "credential is not found");
                    }
            ).run();



Wednesday, March 2, 2016

Getting hold of an un-used port when writing a test

A really quick post today, I was writing a unit test that needed to put up a temporary JAX-RS resource in order to test something very specific with out the need of a large testing framework. Unfortunately the code I was using didn't know about replacing zero with a random allocated port so I had to do something different. The nice tidy code that does this looks like the following:

try(ServerSocket ss = new ServerSocket(0)) {
   freePort = ss.getLocalPort();
}

server = createServerWithPort(freePort);

But of course there is a race condition here as there could be another process on the machine running the exact same code that gets lucky with its time on the CPU - note the API I was using couldn't use the ServerSocket directly so there is a gap where the port is not tied up. So to rule out the one in a million failure chance, which always comes on a Friday at 5pm, we need to wrap this code in a loop.

while(server==null) {
    try(ServerSocket ss = new ServerSocket(0)) {
       freePort = ss.getLocalPort();
    }

    server = createServerWithPort(freePort);
}

This may seem extreme; but if you are running some kind of CI system like Hudson or Jenkins it is not unusual to be running multiple executors on the same machine. You might have two developers running preflights at the same time causing an annoying intermittent failure as described above.

Finally a version using a JDK 8 stream with a helper method to convert the IOException from ServerSocket into a RuntimeException. Some would argue this is less readable; but I find it more explicit.

server = Stream.generate(convertException(() -> {
    try(ServerSocket ss = new ServerSocket(0)) {
        return ss.getLocalPort();
    }
})).map(TestClass::createServerWithPort)
.filter(Objects::nonNull)
.findFirst().get();

Wednesday, January 20, 2016

Removing multiple items from a JavaScript array, no for loops

So I was reviewing some code today that was using two for loops to remove elements from a JavaScript array, I won't relate the code here but I figured there as got to be a nicer way. So the first guess is to make use of the filter method; but that creates a new array which might be problem if the array is large but not if it is fairly small or not called very often.

providers = providers.filter(function(provider) {
     return provider.age > 35
}

For performance or API reasons you might want to perform an in place removal using splice, so you can simple map the values to indexes you want to remove, reverse the order then perform a sequence of splice operations:

providers.map(function(provider, index) {
    return provider.age > 35 ? index : -1;
}).filter(function(index) {
    return index >= 0;
}).reverse().forEach(function(index) {
    providers.splice(index,1);
});

You can play with this code in this jsfiddle.

Update 21 Jan 20125: The hazard of doing full stack development is that it is easy to accidentally duck type between languages whilst forgetting the impact. In Java the equivalent streaming code using map just returns another lazy step in the stream whereas in JavaScript you end up creating a new array of the same size as the original. This removes the performance improvement over the first example if you have lots of data. This is simple too fix though as you can use the reduce function to just create an array and populate it using the reduce operation:

providers.reduce(function(list, provider, index) {
    if (provider.age> 35) list.push(index);
    return list;
}, []).reverse().forEach(function(index) {
    providers.splice(index,1);
});

Here is the updated fiddle. I guess at some point I am going to have to benchmark these variants to see which is best; but that is for a less busy day.

Tuesday, January 12, 2016

Beware of slightly different behaviour between ConcurrentMap and ConcurrentHashMap for computeIfAbsent

So I was refactoring some code to use non-locking collection classes and I noticed the this significant different between ConcurrentMap.computeIfAbsent and ConcurrentHashMap.computeIfAbsent. They key different is that for the former default implementing the mapping function can be called many times for a particular key where as for the concrete implementation it will be called only once. This will likely affect whether the code in the function needs to be thread safe or not.

Right okay so armed with that you know that any collection that implements just ConcurrentMap will inherit this behaviour, further to that I found in particular guava would return a different implementation depending on the passed on parameters:

ConcurrentMap map1 = new MapMaker().makeMap();
System.out.println(map1.getClass());

ConcurrentMap map2 = new MapMaker().weakKeys().makeMap();
System.out.println(map2.getClass());

....

class java.util.concurrent.ConcurrentHashMap
class com.google.common.collect.MapMakerInternalMap

MapMakerInternalMap doesn't override computerIfAbsent therefore the behaviour of this function will be significantly different depending on the parameters to pass into the maker, something that might be apparent from the get go.

Wednesday, November 25, 2015

A very simple implementation of Suite to make it easy to run across a bunch of test machines at the same time.

So I was presented with a large suite that I need to distribute amongst an arbitrary number of Hudson slaves. Now my first guess would be to hand balance this into N sub suites; but that would require on-going work and would need to change depending on the number of nodes. (We have a target of 30 minutes to run all tests in parallel which is going to take some machines)

@RunWith(Suite.class)
@Suite.SuiteClasses({
    TableCRUD.class,
    TableCRUDChild.class,
    TableFilterSort.class,
    TableDefaultQuery.class,
    TableActions.class,
    ....
})
public class DevSuite {

}

So the quickest way I found to do this was to write different version of Suite that allocated the tests to different bins. This implementation doesn't ensure properly balanced bins as it is biased towards keeping the order stable and consistent which is useful for comparing tests results split into multiple jobs.

import java.util.Collection;
import java.util.List;
import java.util.Map;
import java.util.TreeMap;
import java.util.function.Function;
import java.util.function.ToIntFunction;
import java.util.stream.Collector;
import java.util.stream.Collectors;
import org.junit.runner.Runner;
import org.junit.runners.Suite;
import org.junit.runners.model.InitializationError;
import org.junit.runners.model.RunnerBuilder;

/**
 * Try to proportion a suites tests into "SPLIT" number of equals sections and
 * then just run "SECTION" rather than all of the tests. Defined by
 * "SplitSuite.split" and "SplitSuite.section" respectively
 */
public class SplitSuite
        extends Suite {

    private static int SECTIONS
            = Integer.getInteger("SplitSuite.split", -1);
    private static int SECTION
            = Integer.getInteger("SplitSuite.section", -1);

    public SplitSuite(RunnerBuilder builder, Class<?>[] classes) throws InitializationError {
        super(builder, classes);
    }

    public SplitSuite(Class<?> klass, RunnerBuilder builder) throws InitializationError {
        super(klass, builder);
    }

    public List<Runner> getChildren() {
        if (SECTIONS < 0) {
            return super.getChildren();
        } else if (SECTIONS == 0) {
            throw new IllegalArgumentException("SplitSuite.split needs to be a positive integer, it cannot be zero. A negative valid will disable this feature.");
        } else if (SECTION >= SECTIONS) {
            throw new IllegalArgumentException("SplitSuite.section parameter " + SECTION + " needs to be less than SplitSuite.split " + SECTIONS);
        }

        final Map<Integer, List<Runner>> collect = originalList.stream().collect(
                assignToGroups(SECTIONS, Runner::testCount, originalList));

        return collect.get(SECTION);
    }

    
    /**
     * Assigns object to buckets, moving to the next when filled, whilst preserving
     * the original order.
     * @param <Type> The type that has some kind of size property
     * @param sections The number of sections to split the code across
     * @param sizer A function to return the size of the given object
     * @param originalList The original list to provide a base line size
     * @return A map that contains an order list of sections
     */
    public static <Type> Collector<Type, ?, Map<Integer, List<Type>>> assignToGroups(
            int sections, 
            final ToIntFunction<Type> sizer,
            Collection<Type> originalList) {

        // Get length
        int count = originalList.stream().collect(Collectors.summingInt(sizer));
        // Workout section length
        int sectionLength = count / sections;
        
        Function<Type, Integer> grouping = new Function<Type, Integer>() {
            
            int counter = 0;
            
            @Override
            public Integer apply(Type t) {
                int section = Math.min(counter / sectionLength, sections - 1);
                counter += sizer.applyAsInt(t);                
                return section; 
            }
        };
        
        
        return Collectors.groupingBy(grouping, TreeMap::new, Collectors.toList());
    }
    
    
}



So a quick change to the suite class....

@RunWith(SplitSuite.class)
@Suite.SuiteClasses({
    TableCRUD.class,
    TableCRUDChild.class,
    TableFilterSort.class,
    TableDefaultQuery.class,
    TableActions.class,
    ....
})
public class DevSuite {

}

Now the actual test step looks like this, note that this is running under hudson but you should be able to achieve something similar on the CI server of your choice. In order to make my life easier I derive the section number from the job name for so 3 sections I have ..._0 ..._1 and ..._2. In hudson the later jobs all cascade from the the first sharing all properties which makes maintenance a lot easier.

java ... -DSplitSuite.sections=${SECTIONS} -DSplitSuite.section=`echo ${JOB_NAME} | sed -e s/.*_//` ...

And it appears to work, obviously the algorithm to split the suites could be better to produce more balanced results; but that is an effort for another day.....

Friday, July 31, 2015

Creating a shared library for Jersey 2.19 to use with Weblogic 12.1.3

Weblogic server comes with a shared library so you can deploy JAX-RS 2.0 applications; but is limited to Jersey version 2.5.1 and the instructions for changing this are not entirely obvious or straightforward. I have recently joined a new team at Oracle and one of the first things I did was to look at upgrading the dependent libraries. Now I have talked to the Jersey team and they don't support this combination; but it maybe useful enough to get you out of a bind until the next version of Weblogic is released.

I am going to do this using Maven because it means all the packaging and downloading is done for you. The basic structure of the project is as follows:

|-pom.xml
|-src
| |-main
| | |-java
| | |-resources
| | | |-META-INF
| | | | |-MANIFEST.MF
| | |-webapp
| | | |-WEB-INF
| | | | |-web.xml
| | | | |-weblogic.xml

I just generated a vanilla Maven project using Netbeans and then added in the Jersey dependencies I needed, it is likely this file could be cut down a little bit more with some determination. But it worked well enough for me:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    

    <groupId>com.example</groupId>
    <artifactId>JerseyLibrary</artifactId>
    <version>2.19</version>
    <packaging>war</packaging>

    <name>Jersey Library</name>

    <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
         
        <jersey.version>2.19</jersey.version>
    </properties>
    <dependencies>
        <dependency>
            <groupId>org.glassfish.jersey.containers</groupId>
            <artifactId>jersey-container-servlet</artifactId>
            <version>${jersey.version}</version>
            <scope>compile</scope>
        </dependency>
        <dependency>
            <groupId>org.glassfish.jersey.core</groupId>
            <artifactId>jersey-client</artifactId>
            <version>${jersey.version}</version>
        </dependency>
        <dependency>
            <groupId>org.glassfish.jersey.media</groupId>
            <artifactId>jersey-media-multipart</artifactId>
            <version>${jersey.version}</version>
            <type>jar</type>
        </dependency>
        <dependency>
            <groupId>org.glassfish.jersey.media</groupId>
            <artifactId>jersey-media-moxy</artifactId>
            <version>${jersey.version}</version>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>3.1</version>
                <configuration>
                    <source>1.7</source>
                    <target>1.7</target>
                </configuration>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-war-plugin</artifactId>
                <version>2.3</version>
                <configuration>
                    <failOnMissingWebXml>false</failOnMissingWebXml>
                    <archive>
                        <manifestFile>src/main/resources/META-INF/MANIFEST.MF</manifestFile>
                    </archive>
                </configuration>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-dependency-plugin</artifactId>
                <version>2.6</version>
                <executions>
                    <execution>
                        <phase>validate</phase>
                        <goals>
                            <goal>copy</goal>
                        </goals>
                        <configuration>
                            <outputDirectory>${endorsed.dir}</outputDirectory>
                            <silent>true</silent>
                            <artifactItems>
                                <artifactItem>
                                    <groupId>javax</groupId>
                                    <artifactId>javaee-endorsed-api</artifactId>
                                    <version>7.0</version>
                                    <type>jar</type>
                                </artifactItem>
                            </artifactItems>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>

</project>

The next most important file in the MANIFEST.MF file, this tells Weblogic when you try to deploy the .war that this is a library and also contains the versions it supplies:

Specification-Title: Weblogic JAX-RS 2.0
Specification-Version: 2.0
Implementation-Title: Weblogic JAX-RS 2.0 Reference Implementation
Implementation-Version: 2.19
Extension-Name: jax-rs

Finally, you have to include a weblogic.xml file to tell the server that some classes you need to take from here rather than the server class loader. I got the basis of this from the file that comes with the 2.5.1 shares library that ships with 12.1.3 and then added a few more lines to take in account how the code has moved on. Depending on what your code is doing you may have to add a few more.

<?xml version="1.0" encoding="UTF-8"?>
<weblogic-web-app
        xmlns="http://xmlns.oracle.com/weblogic/weblogic-web-app"
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:schemaLocation="http://xmlns.oracle.com/weblogic/weblogic-web-app http://xmlns.oracle.com/weblogic/weblogic-web-app/1.4/weblogic-web-app.xsd">

  <container-descriptor>
        <prefer-application-packages>
            <!-- apis -->
            <package-name>javax.ws.rs.*</package-name>

            <!-- guava -->
            <package-name>com.google.common.*</package-name>

            <!-- jersey providers -->
            <package-name>com.sun.jersey.*</package-name>
            <package-name>org.glassfish.jersey.*</package-name>
            
            <package-name>jersey.repackaged.*</package-name>
            
            <!-- hk2 -->
            <package-name>org.jvnet.hk2.*</package-name>
            <package-name>org.jvnet.hk2.tiger_types.*</package-name>
            <package-name>org.glassfish.hk2.*</package-name>

            <package-name>javassist.*</package-name>

            <!-- media providers -->
            <package-name>org.eclipse.persistence.jaxb.rs.*</package-name>
            <package-name>org.codehaus.jackson.jaxrs.*</package-name>

            <!-- wls -->
            <package-name>weblogic.jaxrs.api.client.*</package-name>
            <package-name>weblogic.jaxrs.internal.api.client.*</package-name>
            <package-name>weblogic.jaxrs.dispatch.*</package-name>
            <package-name>weblogic.jaxrs.monitoring.util.*</package-name>
        </prefer-application-packages>
    </container-descriptor>
</weblogic-web-app>

Now all this needs is a quick mvn install and in the target directory there will be a nice complete shared library called JerseyLibrary-2.19.war that you can deploy in the normal way. Remember of course to update the library entries for the .war that is going to depend on this to have the right versions in it so it does pick up the 2.5.1 version.