Sébastien Labbé

USA Ultimate recently annonced modifications made to their ranking algorithm for 2014 Season:

Boulder, Colo. (January 17, 2014) – USA Ultimate, the national governing body for the sport of ultimate in the United States, and the organization’s Algorithm Working Group announced today improvements to the algorithm used to calculate rankings in the college and club divisions. The new model, USA Ultimate Top 25 (version 2.0) was devised by the volunteer Algorithm Working Group with guidance from members of the USA Ultimate competition department.

The USA Ultimate Rankings (version 2.0) is fully described here and reactions already started on the ultiworld blog. To me, the modifications are technical improvement and fix some issues of the algorithm. But, in my opinion, the inherent problem of the algorithm remains. Below are my thoughs that I shared on the ultiworld blog.

To me the principal weekness of the actual USAU Ranking algorithm is that it doesn't see the structure of tournaments. For the algorithm used by USAU, a tournament is just a bunch of random games. The algorithm does not see the difference between the saturday morning game and the sunday afternoon final (apart the one day difference which won't affect much). Suppose the two finalist teams of a tournament appear to be in the same pool on the Saturday. The algorithm will consider both games to decide the best. Any human will say the best team is the one who wons the final!

This is why I believe that any Ranking Algorithms in Ultimate should not use the results of games but the results of tournaments. Ultimate frisbee is such a sport that games are played during tournaments and this fact must be considered. The output of a tournament is an ordering of teams (may not be a total order if not all placements games are played) and this is what the algorithm should take as an input.

Moreover, ranking is not only about ranking. It can be a great source of motivation for team to improve (this is why USAU, if they want team to improve, should look into this) but it will do so only if at least two conditions are satisfied : (1) the ranking must be predictable and (2) the ranking must reward what is the most difficult to acheive : winning the final of a tournament. And these are exactly the weaknesses of the actual ranking algorithm whose recursive computation part makes it completely unpredictable and which is blind on the importance of games.

If the ranking system is predictable, then you know in advance that winning a certain game will give you exactly 200 pts for example and this is the amount you need to become #1 in the country, then it will be one more motivation to win the game.

Other sports use such system : tennis ATP ranking, Formula 1, etc. Moreover, it exists in Ultimate as well : Quebec Ultimate Federation is using such a system since more than 5 years to rank Ultimate teams. Team get points according to their final placement. Winners of "Grand Slam" tournament get 1000 pts. Other smaller tournaments gives less points. The final ranking considers the sum of the best three results during the season. For more info, the system is described in this text.