Overview and Purpose

The AOS Masters League in Australia now has 7+ years of event data and player performance data, across over 200 events.  Each seasons National Masters, State Masters and Faction Master ladders serve the primary purpose of reflecting players performance for the current active season, and by nature reset to back to 0 starting base for everyone each season.  

However the seasonal ladders do not provide a mechanism to rank players across long term, reflecting their long term performance, as they are restricted to the current season performance only.    So what happens to all that historical data?  To date, it just kind of disappears.

By contrast, in other games and sports, this data is usable to develop long term player ranking mechanisms (example, the Chess ELO rating system).   Long term player relative rankings may be used for:

– An aspiration for a player to aim for; 

– fuelling *friendly and sportsmanlike* competitive rivalry in our community;

– conduct opponent research & matchup expectation analysis coming into an event (eg some hard data to reference and feed into list review show analysis!); and

– mechanism for TO’s to innovate with alternative event formats (for example, these rankings could be used as the data reference for “Seeded” or “Reverse-Seeded” style event structures).   

More detail on the current algorithm methodology can be found underneath the rankings ladder.

Algorithm Technical Details

This algorithm is ‘inspired’ by the principles of the “Elo” algorithm as used for Chess (search Wikipedia Elo Chess for details).  It has technical differences due to data limitations but is based on some of the underlying core philosophy.   

 –  Each player starts at 1000 points for their first event.  The model is “baynesian” with each players score then changing after the event based on results at that event.

 – At each event, the algorithm derives an ‘expected result’ based on the average strength of the field and their own starting score pre-event. The expected result is a formula to estimate the percentile of where this player “should” end up based on the field.      

Example 1: a new player joins an event with all other new players, their ‘expected result’ is tuned to be 50th percentile of that field.   

Example 2: A new player joins an event with a lot of strong experienced players, the ‘expected result’ will be lower (say 35th percentile of the field) (and vice versa).

Example 3: A highly experienced strongly rated player joining an event that has a lot of less experienced will have a high ‘expected result’ (say 80th percentile of the field).   

 – After each event, the players score is updated based on their actual result (converted to be a percentile rank of the field) minus their ‘expected result’.  This can be result in either a positive or negative ‘delta’ .     

– Various Multipliers or discounts to the ‘delta’ are then applied based on the size and length of the event as follows:

Events that are 32 players or higher get a 1.25x multiple

Events that are 24-31 players get a 1.0x multiple

Events that are <24 players get a 0.5x multiple   

Events that are 5+ rounds get 1.0x multiple

Events that less than 5 rounds get a 0.6x multiple.

(Events <12 players are not applicable)  

A flat 1.5x “acceleration multiple” also applies to all deltas.  The acceleration multiple is designed to make the player scores (and resulting ladder ranking) more dynamic as events are logged.

 – By design it is harder to accumulate additional points the higher a players starting score and also easier to lose points if a strong player posts a poor result in an event.    In effect, the algorithm is tuned such that a “soft cap” of 1500 is close to the maximum achievable result, and the only way to get to that result is very consistent top tier performance.    

Example 1: A ~1500 player can still continue to positively accumulate points if they continue to achieve results at the very top of the ladder every single event (at present, top ranked players need to finish in the 10% of the field better in any event to accumulate additional points);

Example 2:  A ~1400 player in a typical field will accrue positive points if they finish in the top ~15% of the field, and lose points if they perform poorer than this.

Example 3:   A ~1300 player in a typical field will accrue positive points if they finish in the top ~25% of the field, and lose points if they perform poorer than this.

Example 4:  A ~800 player in a typical field will accrue positive results if they finish in the top ~75% of the field.

The higher your starting ranking, the more points you also lose for any poorer-than-expected performance in an event.     The vice versa also applies -> a player with a low starting score can accumulate lots of points quickly with each strong performance.

– The algorithm is tuned such that anything over approx. 1200 points is evidence the player has achieved long term consistent competitive success at the game.    Overall it will take many events with good performance to accumulate to the 1200+  level, and players getting to 1300+, 1400+ level is difficult by design, requiring many events with consistently good performance.   A new player to the scene will take at least 7-10 events with solid performance to accumulate enough points to place towards the top of the table.    

– An ‘Active Player’ filter is applied such that only players who have played a recorded event in the last 720 days (~2 years) are included in the rankings.   The most notable individual consequence amongst the top ranked players from apply an ‘Active Player’ cut-off is “Dan Brewer” (Score 1428).  There is also roughly 500 other players who are currently filtered out by the Active Player filter.     If an inactive player becomes re-active, all their old results will still contribute to their score when it reactivates.   Future evolutions of the algorithm may attempt to refine a “depreciation model” where older results are given lower weighting compared to more recent results, but this has been difficult to technically achieve in test modelling so far.

– The current algorithm is tuned on results from nearly 200 events between 2016 to present.  Data limitations include event final standing are known, but individual game level data has not been captured.   At present there is no attempt to adjust for any sort of faction strength factor.

– No algorithm is ever going to be perfect.   Any score difference between around +/- 30 is likely the “margin of error”.   Each event can see a player accrue (or also lose) 50+ points depending on result and the multipliers that apply, which is hoped to lead to lots of dynamism in the ladder as events unfold.  

– It is also recognised that AOS as an overall hobby experience is very different to chess, in that players come to events all all sorts of goals (not just trying to winning maximum number of games), and it is an asymmetrical game.  It is also noted that players sometimes switch gears – sometimes they play in events for competitive reasons, but in other events they are participating for other reasons (eg experimental lists, hobby goals, social goals).  This could influence a players score going up and down over time and therefore their score may not fully reflect of their ‘true’ pure competitive strength as a player.   No algorithm can capture such aspects, and the algorithm is designed based solely on each players recorded results at events.   

Note: Data integrity -> the data is based on results as submitted by TO’s over time.  Various errors can creep in, such as same player exact name matching (eg “Matthew” vs “Matt” or “Dave” vs “David”).  Attempts have been made to identify but residual errors could remain.   Please contact Michael Clarke if you think you have found a some type of error.