Showing posts with label jenkins. Show all posts
Showing posts with label jenkins. Show all posts

Thursday, June 2, 2016

Getting a name for someone to connect back to your server.

When doing test automation it is often the case you need to know the name of the current machine in order to prompt another machine to connect to it, particularly if you are running your tests in parallel. This week I was trying to get the server under test to connect back to a WireMock server running on the slave test machine.

The standard response on stack overflow is to use the following pattern to get a network address. In my version here if we can't resolve the name then we are assuming we are running on a developers laptop on VPN so all the tests are run on the same machine. (Hence localhost)

String hostName = "localhost";
try {
    InetAddress addr = InetAddress.getLocalHost();
    String suggestedName = addr.getCanonicalHostName();
    // Rough test for IP address, if IP address assume a local lookup
    // on VPN
    if (!suggestedName.matches("(\\d{1,3}\\.?){4}") && !suggestedName.contains(":")) {
        hostName = suggestedName;
    }
} catch (UnknownHostException ex) {
}

System.out.println(hostName);

The problem comes is that we have to trust the local machine settings, for example /etc/hostname, which can result in a network name that is not accessible from another machine. To counter this I wrote the following code to work over the available network interfaces to find a remotely addressable network address name that can be used to talk back to this machine. (I could use an IP address but they are harder to remember, particularly as we are moving towards IPv6)

String hostName = stream(wrap(NetworkInterface::getNetworkInterfaces).get())
        // Only alllow interfaces that are functioning
        .filter(wrap(NetworkInterface::isUp))
        // Flat map to any bound addresses
        .flatMap(n -> stream(n.getInetAddresses()))
        // Fiter out any local addresses
        .filter(ia -> !ia.isAnyLocalAddress() && !ia.isLinkLocalAddress() && !ia.isLoopbackAddress())
        // Map to a name
        .map(InetAddress::getCanonicalHostName)
        // Ignore if we just got an IP back
        .filter(suggestedName -> !suggestedName.matches("(\\d{1,3}\\.?){4}")
                                 && !suggestedName.contains(":"))
        .findFirst()
        // In my case default to localhost
        .orElse("localhost");

System.out.println(hostName);

You might notice there a are a few support methods being used in there to tidy up the code, here are the required support methods if you are interested.

@FunctionalInterface
public interface ThrowingPredicate<T, E extends Exception>{

    boolean test(T t) throws E;
}

@FunctionalInterface
public interface ThrowingSupplier<T, E extends Exception>{

    T get() throws E;
}

public static <T, E extends Exception> Predicate<T> wrap(ThrowingPredicate<T, E> th) {
    return t -> {
        try {
            return th.test(t);
        } catch (Exception ex) {
            throw new RuntimeException(ex);
        }
    };
}

public static <T, E extends Exception> Supplier<T> wrap(ThrowingSupplier<T, E> th) {
    return () -> {
        try {
            return th.get();
        } catch (Exception ex) {
            throw new RuntimeException(ex);
        }
    };
}

// http://stackoverflow.com/a/23276455
public static <T> Stream<T> stream(Enumeration<T> e) {
    return StreamSupport.stream(
            Spliterators.spliteratorUnknownSize(
                    new Iterator<T>() {
                public T next() {
                    return e.nextElement();
                }

                public boolean hasNext() {
                    return e.hasMoreElements();
                }
            },
                    Spliterator.ORDERED), false);
}

Wednesday, November 25, 2015

A very simple implementation of Suite to make it easy to run across a bunch of test machines at the same time.

So I was presented with a large suite that I need to distribute amongst an arbitrary number of Hudson slaves. Now my first guess would be to hand balance this into N sub suites; but that would require on-going work and would need to change depending on the number of nodes. (We have a target of 30 minutes to run all tests in parallel which is going to take some machines)

@RunWith(Suite.class)
@Suite.SuiteClasses({
    TableCRUD.class,
    TableCRUDChild.class,
    TableFilterSort.class,
    TableDefaultQuery.class,
    TableActions.class,
    ....
})
public class DevSuite {

}

So the quickest way I found to do this was to write different version of Suite that allocated the tests to different bins. This implementation doesn't ensure properly balanced bins as it is biased towards keeping the order stable and consistent which is useful for comparing tests results split into multiple jobs.

import java.util.Collection;
import java.util.List;
import java.util.Map;
import java.util.TreeMap;
import java.util.function.Function;
import java.util.function.ToIntFunction;
import java.util.stream.Collector;
import java.util.stream.Collectors;
import org.junit.runner.Runner;
import org.junit.runners.Suite;
import org.junit.runners.model.InitializationError;
import org.junit.runners.model.RunnerBuilder;

/**
 * Try to proportion a suites tests into "SPLIT" number of equals sections and
 * then just run "SECTION" rather than all of the tests. Defined by
 * "SplitSuite.split" and "SplitSuite.section" respectively
 */
public class SplitSuite
        extends Suite {

    private static int SECTIONS
            = Integer.getInteger("SplitSuite.split", -1);
    private static int SECTION
            = Integer.getInteger("SplitSuite.section", -1);

    public SplitSuite(RunnerBuilder builder, Class<?>[] classes) throws InitializationError {
        super(builder, classes);
    }

    public SplitSuite(Class<?> klass, RunnerBuilder builder) throws InitializationError {
        super(klass, builder);
    }

    public List<Runner> getChildren() {
        if (SECTIONS < 0) {
            return super.getChildren();
        } else if (SECTIONS == 0) {
            throw new IllegalArgumentException("SplitSuite.split needs to be a positive integer, it cannot be zero. A negative valid will disable this feature.");
        } else if (SECTION >= SECTIONS) {
            throw new IllegalArgumentException("SplitSuite.section parameter " + SECTION + " needs to be less than SplitSuite.split " + SECTIONS);
        }

        final Map<Integer, List<Runner>> collect = originalList.stream().collect(
                assignToGroups(SECTIONS, Runner::testCount, originalList));

        return collect.get(SECTION);
    }

    
    /**
     * Assigns object to buckets, moving to the next when filled, whilst preserving
     * the original order.
     * @param <Type> The type that has some kind of size property
     * @param sections The number of sections to split the code across
     * @param sizer A function to return the size of the given object
     * @param originalList The original list to provide a base line size
     * @return A map that contains an order list of sections
     */
    public static <Type> Collector<Type, ?, Map<Integer, List<Type>>> assignToGroups(
            int sections, 
            final ToIntFunction<Type> sizer,
            Collection<Type> originalList) {

        // Get length
        int count = originalList.stream().collect(Collectors.summingInt(sizer));
        // Workout section length
        int sectionLength = count / sections;
        
        Function<Type, Integer> grouping = new Function<Type, Integer>() {
            
            int counter = 0;
            
            @Override
            public Integer apply(Type t) {
                int section = Math.min(counter / sectionLength, sections - 1);
                counter += sizer.applyAsInt(t);                
                return section; 
            }
        };
        
        
        return Collectors.groupingBy(grouping, TreeMap::new, Collectors.toList());
    }
    
    
}



So a quick change to the suite class....

@RunWith(SplitSuite.class)
@Suite.SuiteClasses({
    TableCRUD.class,
    TableCRUDChild.class,
    TableFilterSort.class,
    TableDefaultQuery.class,
    TableActions.class,
    ....
})
public class DevSuite {

}

Now the actual test step looks like this, note that this is running under hudson but you should be able to achieve something similar on the CI server of your choice. In order to make my life easier I derive the section number from the job name for so 3 sections I have ..._0 ..._1 and ..._2. In hudson the later jobs all cascade from the the first sharing all properties which makes maintenance a lot easier.

java ... -DSplitSuite.sections=${SECTIONS} -DSplitSuite.section=`echo ${JOB_NAME} | sed -e s/.*_//` ...

And it appears to work, obviously the algorithm to split the suites could be better to produce more balanced results; but that is an effort for another day.....

Thursday, June 6, 2013

Problems running lots of X11 sessions concurrently when testing under Hudson/Jenkins

So we are running a quite a bit of automated UI testing and we have found over a large number of concurrent test nodes that even when the Xvnc plugin has correctly started that the environment is not always ready or reliable. This blog post looks at two changes we made that have improved our test stability in recent times.

The first and most important thing to settle when using the Xvnc plugin is to make sure that the user that is running the tests has a normal window manager. By default the default VNC server will start using "twm" as defined in the .vnc/xstartup file. This has significant behavioural differences from the more standard Gnome / KDE and indeed the window managers on Windows and a Mac. Rather than deal with this we have standardised on running with Gnome.

The problem is though is that in our experience if you have multiple gnome sessions starting at the same time that sometimes they will fail with odd error messages. My esteemed colleague Mark Warner came up with the following xstartup script which just puts a simple lock around the startup code.

#!/bin/bash
# It uses a lockfile in the user's .vnc directory to ensure that
# only one instance of gnome will start up for the given user
# at once. This should work around problems associated with
# multiple gnome instances starting and failing to lock certain
# files. It will start gnome after 5 mins of failing to lock as
# a fail safe so a gnome startup will always be attempted.
#

LOCK_FILE=${HOME}/.vnc/xstartup.lock
SESSION_VNC_NAME=VNC-`hostname -f`
LOCKED=0

function clean_up {
  # don't delete the lock file if we didn't create it
  if [ $LOCKED -eq 1 ]; then
    echo "$(date) All finished, deleting lock file."
    rm -f $LOCK_FILE
  fi
}

# make sure that we delete the lock file if we get killed
trap clean_up SIGHUP SIGINT SIGTERM

# retry for 5 mins then assume something died and delete
# the existing lockfile taking control of it for ourselves.
echo "$(date) Attempting to obtain startup lock."
if lockfile -l 300 $LOCK_FILE; then
  echo "$(date) Startup lock aquired."
  LOCKED=1
else
  echo "$(date) Lock failed... starting anyway."
fi

# give X a little breather before we start gnome
sleep 10 

# Start Gnome
gnome-session &

# just let gnome settle before we relinquish the lock
sleep 20
clean_up

Now this resolved spurious test failures that were due to a missing window manager, but as recently the number of nodes increased we found that we were still seeing troubling intermittent failures that took the form of failures to connect to the window server for no obvious reason.

This would happen even if previously in the job we had displayed a window successfully.

[exec] java.lang.InternalError: Can't connect to X11 window server using ':58' as the value of the DISPLAY variable.
     [exec]  at sun.awt.X11GraphicsEnvironment.initDisplay(Native Method)
     [exec]  at sun.awt.X11GraphicsEnvironment.access$200(X11GraphicsEnvironment.java:65)
     [exec]  at sun.awt.X11GraphicsEnvironment$1.run(X11GraphicsEnvironment.java:110)
     [exec]  at java.security.AccessController.doPrivileged(Native Method)
     [exec]  at sun.awt.X11GraphicsEnvironment.(X11GraphicsEnvironment.java:74)
     [exec]  at java.lang.Class.forName0(Native Method)
     [exec]  at java.lang.Class.forName(Class.java:186)
     [exec]  at java.awt.GraphicsEnvironment.createGE(GraphicsEnvironment.java:102)
     [exec]  at java.awt.GraphicsEnvironment.getLocalGraphicsEnvironment(GraphicsEnvironment.java:81)
     [exec]  at org.netbeans.core.startup.Main.start(Main.java:253)
     [exec]  at org.netbeans.core.startup.TopThreadGroup.run(TopThreadGroup.java:123)
     [exec]  at java.lang.Thread.run(Thread.java:722)
     [exec] Result: 2

This would also happen right at the start when gnome was being set up, as shown here by looking at the content of the logs under .vnc .

_XSERVTransmkdir: Owner of /tmp/.X11-unix should be set to root

Xvnc Free Edition 4.1.2
Copyright (C) 2002-2005 RealVNC Ltd.
See http://www.realvnc.com for information on VNC.
Underlying X server release 70101000, The X.Org Foundation


Thu May 30 03:51:34 2013
 vncext:      VNC extension running!
 vncext:      Listening for VNC connections on port 5974
 vncext:      Listening for HTTP connections on port 5874
 vncext:      created VNC server for screen 0
Thu May 30 03:51:37 BST 2013 Attempting to obtain startup lock.
Thu May 30 03:53:45 BST 2013 Startup lock aquired.
Start gnome for OL4 or OL5
AUDIT: Thu May 30 03:53:56 2013: 10280 Xvnc: client 1 rejected from local host
Xlib: connection to ":74.0" refused by server
Xlib: No protocol specified


(gnome-session:10543): Gtk-WARNING **: cannot open display:  
Thu May 30 03:54:15 BST 2013 All finished, deleting lock file.
AUDIT: Thu May 30 04:14:03 2013: 10280 Xvnc: client 1 rejected from local host
AUDIT: Thu May 30 04:14:22 2013: 10280 Xvnc: client 1 rejected from local host
FreeFontPath: FPE "unix/:7100" refcount is 2, should be 1; fixing.

After a little bit of playing around suspicion fell to the .Xauthority file we we knew could cause error messages like this and could become corrupted. Removing /zeroing the file didn't help so like the Gnome problems some kind of concurrency / contention issues was suspected.

After a bit of playing about we decided that for our test user it was going to be best to just have a copy of this file per host. This works for us as we only have the single executor per machine in order to get consistent testing results. In our case we just added the following line in our .bashrc for the NIS user we run our tests as and restarted all our of slave nodes.

# make sure we don't share the Xauthority file between hosts

export XAUTHORITY=~/.Xauthority.`hostname -f`

Since we have made this change we have seen no failures of the above type after nearly week, that of course doesn't rule out that we have just reduced the occurrence of the issue to a reasonably small level; but we are happy that we can run tests again. Of course the moment I post this blog I am going to loose that lovely run of blue balls. :-)



Thursday, October 11, 2012

Using Hudson/Jenkins to diagnose that intermittent failure


I have been working on one of those intermittent bugs that just won't reproduce on my machine; but will reproduce intermittently on other machines while they are running automated testing. I filled the code with trace statements, now I suspect the problem is in code that I don't control and doesn't appear to have much in the way of diagnostics in the place I am working on.

So I did the obvious thing which is to run the tests on a loop on my machine overnight, 12 hours later and 8 test iterations later, no test failures and I am no further forward.

Since the tests are failing in the Hudson farm, it make sense to try to connect the debugger up to those jobs; but I don't want to hang around to attach the remove debugger to each. Thankfully there is a workaround that will allow me to set suitable breakpoints and manage the debugger connection for me.

First of all you need to configure you IDE to accept incoming debugger connections, here are some notes on configuring JDeveloper for a listening debugger, in Netbeans you need to use the Debug->Attach menu item and select "SocketListen" as the connector and configure as per JDeveloper. In Eclipse you need to configure the debug type as "Socket Listen".

The second step is modifying your build system so that there is a target you can call that will start the test cases in debug mode. This is an example of the parameters for one of our CI jobs that passes in the right information. Note of course the blacked out text the the name of then machine you are trying to connect back to. (The java tests are started with the parameter -agentlib:jdwp=transport=dt_socket,address=xxxx.oracle.com:5000,server=n) Make sure that you don't have any firewalls running on that machine that will block the in-coming connections.




You probably will want to run to run multiple jobs at the same time if you have the nodes available, so consider checking this concurrent build box. Always a good idea to bring cakes / cookies into the office if you are going to tie up all the preflight nodes for the day.




And then all that remains is to run a bunch of jobs and wait for your breakpoint to be hit, might take a little while; but it is going be quicker than running these jobs in series on your own machine. And if your farm is Heterogeneous so much the better for reproducing intermittent failures.



You can sit back and then wait for your code to fail..... may I suggest some sessions from JavaOne while you wait?

Saturday, May 26, 2012

Ignoring the return value of a command in Hudson/Jenkins sh task

So I was was trying to run pylint and nosetest running on a Jenkins instance; but these python commands tend to return a non zero status code if they see a non zero return code. This is a problem as it means that any further steps are ignored. The most obvious thing to try was this:

pylint src/openhea -f parseable ; true

The problem is that both statements are treated as separate steps so when the first one fails it never does the return true. Instead, and pointed out by a helpful chap in the #efdhack2012 room was to do the following

pylint src/openhea -f parseable || true

This correctly ignores the false status code, the only wrinkle is that you need to be careful when piping the output, it has to be to the left of the bars.

pylint src/openhea -f parseable > output.report || true

Tuesday, April 3, 2012

Off-loading test execution using Hudson part 3, skipping the queue

Previously I had looked at preflighting, (1, 2) with Hudson (And Jenkins) but forgot to mention a very important plugin that is essential if you want to convince your developer to preflight there work. The problem is that is the slave allotment/smallholder/farm is busy running other jobs then they are forced to wait before merging.

The easy solution to this problem is the priority sorter plugin. This allows you to specify that some jobs are more or less important then others. The default priority is 100, so seeing your preflight job to 1000 should see it jump to the front of the queue.

It is interesting how the length of tests affects my commuting behaviour, now that is take around an hour to run the long tests just for our components. It makes sense to start a work at home, kick of a job and you should have you nice shiny test results by the time we get into work. One day I might even get a blue ball if I am so very lucky with our dependencies. :-)

Monday, January 23, 2012

Off-loading test execution using Hudson part 2

Previously I have written about using Hudson to perform pre-flights using branches on source control systems; but sometimes you just have a patch file that you want to run your tests against.

It turns out this can be quite simple, you just need to make use of a job parameter that takes a File as it's input, this is then written to the specified location in the workspace. In my example I have the following two parameters, the Title allow me to name the job and this property is picked up later by the job Description plugin.

Once you know where the patch is located you can just apply the patch before you perform the required build steps, here is an example using SVN so we use the unix patch command, git users can use the git command instead for this.

And there you are, you can test a user submitted patch for testing without tying up your development machine. You can also run the tests more than once to check for any intermittent tests which can be very valuable in concurrent environments.

Monday, July 4, 2011

Off-loading your test execution using Hudson

One of the annoyances if you have a load of tests for a particular project is that running them can tie up your machine for a while. There are also sometimes consistency issues when running on different machines with different window managers and operating system versions. One thing we do is for every main hudson job we create a copy just to run the tests for a particular developer. It solves the "it runs on my machine" dilemma and frees up the developer to get on with other work. In our part of the organization running the tests on another machine is a mandatory step before merging to main.

This example uses subversion; but it most likely work equally well with any other source control system with a similar interface. It also assumes that all the developers are working on branches, you are doing this right?.

First of all you take a copy of you original job, in my case I am running test for the abbot project so I am starting with "abbot_SF", and create a copy of this job with a suitable postfix. In our case the job is called "abbot_SF_TestTrans". You need to first disable any normal build triggers so this job only gets used when there is a manual action. We keep the results separate as it makes the normal build results much easier to interpret.

So to this new job add a new Parameter "BRANCH" of type string and update the repository URL to contain this parameter, so in my case https://abbot.svn.sourceforge.net/svnroot/abbot/abbot/branches/$BRANCH. You will also notice that I have the used the description setting plugin to capture the branch name automatically. (You could also just put the $BRANCH parameter in the description field and leave the RegEx blank). After you have finished you have something like this, ignore the red ink as that appears to be a local configuration issue.

Now if you click build on the job you get a nice interface where you can ask Hudson to build a particular branch, leaving the developer to get on with the next job while the build machine does it's business.

Speaking of that, it looks like I have some bugs to fix...

P.S. You might also want to tick the "Execute Concurrent Builds if necessary" flag so that if you have more than one developer you can make proper use of your farm.