So I was presented with a large suite that I need to distribute amongst an arbitrary number of Hudson slaves. Now my first guess would be to hand balance this into N sub suites; but that would require on-going work and would need to change depending on the number of nodes. (We have a target of 30 minutes to run all tests in parallel which is going to take some machines)
@RunWith(Suite.class)
@Suite.SuiteClasses({
TableCRUD.class,
TableCRUDChild.class,
TableFilterSort.class,
TableDefaultQuery.class,
TableActions.class,
....
})
public class DevSuite {
}
So the quickest way I found to do this was to write different version of Suite that allocated the tests to different bins. This implementation doesn't ensure properly balanced bins as it is biased towards keeping the order stable and consistent which is useful for comparing tests results split into multiple jobs.
import java.util.Collection;
import java.util.List;
import java.util.Map;
import java.util.TreeMap;
import java.util.function.Function;
import java.util.function.ToIntFunction;
import java.util.stream.Collector;
import java.util.stream.Collectors;
import org.junit.runner.Runner;
import org.junit.runners.Suite;
import org.junit.runners.model.InitializationError;
import org.junit.runners.model.RunnerBuilder;
/**
* Try to proportion a suites tests into "SPLIT" number of equals sections and
* then just run "SECTION" rather than all of the tests. Defined by
* "SplitSuite.split" and "SplitSuite.section" respectively
*/
public class SplitSuite
extends Suite {
private static int SECTIONS
= Integer.getInteger("SplitSuite.split", -1);
private static int SECTION
= Integer.getInteger("SplitSuite.section", -1);
public SplitSuite(RunnerBuilder builder, Class<?>[] classes) throws InitializationError {
super(builder, classes);
}
public SplitSuite(Class<?> klass, RunnerBuilder builder) throws InitializationError {
super(klass, builder);
}
public List<Runner> getChildren() {
if (SECTIONS < 0) {
return super.getChildren();
} else if (SECTIONS == 0) {
throw new IllegalArgumentException("SplitSuite.split needs to be a positive integer, it cannot be zero. A negative valid will disable this feature.");
} else if (SECTION >= SECTIONS) {
throw new IllegalArgumentException("SplitSuite.section parameter " + SECTION + " needs to be less than SplitSuite.split " + SECTIONS);
}
final Map<Integer, List<Runner>> collect = originalList.stream().collect(
assignToGroups(SECTIONS, Runner::testCount, originalList));
return collect.get(SECTION);
}
/**
* Assigns object to buckets, moving to the next when filled, whilst preserving
* the original order.
* @param <Type> The type that has some kind of size property
* @param sections The number of sections to split the code across
* @param sizer A function to return the size of the given object
* @param originalList The original list to provide a base line size
* @return A map that contains an order list of sections
*/
public static <Type> Collector<Type, ?, Map<Integer, List<Type>>> assignToGroups(
int sections,
final ToIntFunction<Type> sizer,
Collection<Type> originalList) {
// Get length
int count = originalList.stream().collect(Collectors.summingInt(sizer));
// Workout section length
int sectionLength = count / sections;
Function<Type, Integer> grouping = new Function<Type, Integer>() {
int counter = 0;
@Override
public Integer apply(Type t) {
int section = Math.min(counter / sectionLength, sections - 1);
counter += sizer.applyAsInt(t);
return section;
}
};
return Collectors.groupingBy(grouping, TreeMap::new, Collectors.toList());
}
}
So a quick change to the suite class....
@RunWith(SplitSuite.class)
@Suite.SuiteClasses({
TableCRUD.class,
TableCRUDChild.class,
TableFilterSort.class,
TableDefaultQuery.class,
TableActions.class,
....
})
public class DevSuite {
}
Now the actual test step looks like this, note that this is running under hudson but you should be able to achieve something similar on the CI server of your choice. In order to make my life easier I derive the section number from the job name for so 3 sections I have ..._0 ..._1 and ..._2. In hudson the later jobs all cascade from the the first sharing all properties which makes maintenance a lot easier.
java ... -DSplitSuite.sections=${SECTIONS} -DSplitSuite.section=`echo ${JOB_NAME} | sed -e s/.*_//` ...
And it appears to work, obviously the algorithm to split the suites could be better to produce more balanced results; but that is an effort for another day.....
