# Ordering

Consider the following example with 6 instances and 7 participants.

Instance 1 2 3 4 5 6

---------------------------------------------------------

Solver A: (0,34) (0,35) (1,42) (1,32) (0,10) (0,12)

Solver B: (3,32) (1,24) (1,44) (1,33) (0,13) (0,15)

Solver C: (0,33) (0,36) (2,30) (5,12) (1,11) (0,17)

Solver D: (0,36) (0,32) (1,46) (1,32) (0,12) (0,13)

Solver E: (0,37) (0,30) (1,43) (1,29) (0,9) (0,4)

Solver F: (2,68) (0,29) (1,41) (0,55) (0,10) (0,5)

Solver G: (0,36) (0,30) (2,43) (0,58) (0,10) (0,4)

The ranks are the following:

Instance 1 2 3 4 5 6

---------------------------------------------------------

Solver A: 2 5 2 4.5 3 4

Solver B: 7 7 4 6 6 6

Solver C: 1 6 6 7 7 7

Solver D: 3.5 4 5 4.5 5 5

Solver E: 5 2.5 3 3 1 1.5

Solver F: 6 1 1 1 3 3

Solver G: 3.5 2.5 7 2 3 1.5

We define for each solver the mean of the ranks. The winner of the competition will be the solver with the lowest mean ranks. In the example, the mean ranks are:

Mean rank

--------------------

Solver A: 3.42

Solver B: 6

Solver C: 5.66

Solver D: 4.5

Solver E: 2.66

Solver F: 2.5

Solver G: 3.25

Hence in this case the winner would be solver F.

The solvers will be run by the organisers, thus the participants should give support to the organisers in the process of compiling and running the solvers.

For each instance, the organisers will run 10 independent trails with seeds chosen at random. For each trial, we will compute the ranks and average them on all trials on all instances.

The winner is the one with the lowest mean rank. If there are several winners, the prize will be split.