Ordering

ITC2011 Ordering

Here we explain here how the ordering works. For each solution there is a pair (d,s) composed by the infeasibility cost d and the objective value s. Remember that results are compared based on d first, and then on s in the case where competitors have the same value for d.

Consider the following example with 6 instances and 7 participants.

Instance 1 2 3 4 5 6
---------------------------------------------------------
Solver A: (0,34) (0,35) (1,42) (1,32) (0,10) (0,12)
Solver B: (3,32) (1,24) (1,44) (1,33) (0,13) (0,15)
Solver C: (0,33) (0,36) (2,30) (5,12) (1,11) (0,17)
Solver D: (0,36) (0,32) (1,46) (1,32) (0,12) (0,13)
Solver E: (0,37) (0,30) (1,43) (1,29) (0,9) (0,4)
Solver F: (2,68) (0,29) (1,41) (0,55) (0,10) (0,5)
Solver G: (0,36) (0,30) (2,43) (0,58) (0,10) (0,4)

The ranks are the following:

Instance 1 2 3 4 5 6
---------------------------------------------------------
Solver A: 2 5 2 4.5 3 4
Solver B: 7 7 4 6 6 6
Solver C: 1 6 6 7 7 7
Solver D: 3.5 4 5 4.5 5 5
Solver E: 5 2.5 3 3 1 1.5
Solver F: 6 1 1 1 3 3
Solver G: 3.5 2.5 7 2 3 1.5

We define for each solver the mean of the ranks. The winner of the competition will be the solver with the lowest mean ranks. In the example, the mean ranks are:

Mean rank
--------------------
Solver A: 3.42
Solver B: 6
Solver C: 5.66
Solver D: 4.5
Solver E: 2.66
Solver F: 2.5
Solver G: 3.25

Hence in this case the winner would be solver F.

The solvers will be run by the organisers, thus the participants should give support to the organisers in the process of compiling and running the solvers.

For each instance, the organisers will run 10 independent trails with seeds chosen at random. For each trial, we will compute the ranks and average them on all trials on all instances.

The winner is the one with the lowest mean rank. If there are several winners, the prize will be split.