Pilot ranking list - could we have something meaningful?

Moderator: BOD1

User avatar
BOD1
Posts: 2142
Joined: Mon Aug 07, 2006 2:45 pm
Location: Brittany, France
Contact:

Re: Pilot ranking list - could we have something meaningful?

Post by BOD1 » Sat Sep 25, 2021 4:11 pm

After intensive testing and observations, I finally obtained something coherent, but I had to modify Glicko algorithm, so I have a rating system halfway between Elo and Glicko.
The previous rating system (Glicko2) rated only competitions results was working fine but Witor insisted to rate ALL competition days individually :wink:, so I selected only valid days (in FAI meaning) since 2008, about 3300 days and ~2800 players.

First tries showed up a lot of volatility in rating, one pilot could go back and forth to the first place and to the last, with a rating going insanely high (8 000 000!) or low (-7 000 000 000). That's due to the mechanism used to rate:
BOD1 wrote:
Mon Sep 20, 2021 9:40 pm
a race is converted to a set of matches: p1 won p2, p1 won p3, p2 won p3 (the match is counted a single time, so there is not p3 lost p1 for example).
I've to amend myself here, match is effectively counted 2 times: p1 won p2, p1 won p3, p2 won p3, p3 lost p2, p3 lots p1, p2 lost p1
So, in a race with a lot of participants, a good pilot who unfortunately crashed when he usually wins will be last in day's ranking and so will be penalized by N lost matches (where N is nb of participants -1). If his previous rating was high, the new rating will decrease considerably more than if it was already previously low. The volatility of his performance will also increase (so much that Glicko hanged in infinite loop) and the rating deviation (the confidence in the measured rating, lower number is better) was going very high.

To reduce these side effects, I had to remove volatility calculations and restrict rating deviation to a maximum of 100. I had also to limit the rating evolution to +/- 300 (one can't gain or loose more than 300 "points" - that's not point, but it's easier for explanation), and if one obtains a rating above the best one, his rating is limited to best one+0,75%.

Code: Select all

All competitions rated in 6h. 12min. 19s. 451ms.
MaxRating: 2389.0033016863836
maxRd: 100.0
minRating: 759.8946465525196
minRd: 21.83514114006881
2945 players

Rating		# of pilots
[2200-2300]	9
[2100-2200]	16
[2000-2100]	39
[1900-2000]	64
[1800-1900]	141
[1700-1800]	194
[1600-1700]	304
[1500-1600]	407
[1400-1500]	525
[1300-1400]	571
[1200-1300]	472
[1100-1200]	170
[1000-1100]	24
[900-1000]	6
[800-900]	2
[700-800]	1

Code: Select all

Pilot name		Rating		Deviation	Days	Last day
Glider_josh		2260.2537	41.23		9	2021-09-11
PLP			2254.5942	35.10		310	2021-09-11
Maymar			2233.0181	26.23		176	2021-09-12
Wojciech Soltysik 	2232.4609	24.69		64	2021-09-12
Kaleu			2231.4658	100.00		7	2012-12-17
QuMaPa			2213.5206	100.00		127	2021-02-10
Cnorbee			2212.8287	23.36		414	2021-09-12
TAF			2209.9540	100.00		20	2021-02-20
Charon			2209.9272	100.00		113	2012-03-29
Dgtfer			2192.9246	22.43		275	2021-09-12
Philipp Kapferer	2191.8959	100.00		258	2021-04-23
Kest4s			2186.3080	100.00		77	2015-02-06
Frankls10		2177.8301	23.63		179	2021-09-12
Witor			2170.5248	71.17		705	2021-06-27
Floyd141		2164.6596	100.00		41	2020-06-26
Mauheb			2160.9789	74.49		87	2021-06-25
Wiek			2159.1292	23.44		64	2021-09-12
Jwr51			2146.3336	84.31		41	2021-09-04
Tonyh			2145.4175	100.00		87	2013-05-23
Alioth			2138.8914	83.51		66	2021-06-23

I'm not completely satisfied with these results, particularly with low frequency player rating. I'll have a try with EloR system to see what I get.
Image
Please use this button to send me a PM, Forum messaging is too limited.
Image

User avatar
dgtfer
Posts: 521
Joined: Mon Nov 19, 2007 9:49 am
Location: Marseille - france

Re: Pilot ranking list - could we have something meaningful?

Post by dgtfer » Sat Sep 25, 2021 7:24 pm

BOD1 wrote:
Sat Sep 25, 2021 4:11 pm
I'm not completely satisfied with these results, particularly with low frequency player rating. I'll have a try with EloR system to see what I get.
I didn't look closely at the details of your new calculations, but a ranking that gives the tenth position is necessarily much better than a ranking that gives the 118th!!!
Image

User avatar
Snapcase
Posts: 42
Joined: Tue May 11, 2010 4:51 pm
Location: Cracow, POLAND
Contact:

Re: Pilot ranking list - could we have something meaningful?

Post by Snapcase » Mon Sep 27, 2021 9:24 pm

dgtfer wrote:
Sat Sep 25, 2021 7:24 pm
BOD1 wrote:
Sat Sep 25, 2021 4:11 pm
I'm not completely satisfied with these results, particularly with low frequency player rating. I'll have a try with EloR system to see what I get.
I didn't look closely at the details of your new calculations, but a ranking that gives the tenth position is necessarily much better than a ranking that gives the 118th!!!
Agree with that! In the first calculations i wasn't even listed! Now im 4th :idea:

Just to help - im flying since 2020 so - quite new here.
Wojciech Sołtysik - SOL - Poland

witor
Posts: 200
Joined: Wed Dec 15, 2010 11:47 pm

Re: Pilot ranking list - could we have something meaningful?

Post by witor » Mon Sep 27, 2021 11:48 pm

If rating system proposed by Glicko doesn't work on it's own I am afraid the whole idea of having skill based rating in condor is not going to be easy. This rating systems (glicko, glicko 2, elo) are results of a lot of work done by mathematicians who are experts in their fields. Sorry for not being very enthusiastic, but I doubt that we can just combine two systems or do something else and hope that we will end up with something useful. Those rating systems are based on very delicate mathematics and any changes will just not produce what we need.

maybe there is a way to adjust parameters of the rating system (without actually changing the system itself)? Have you tried to play with different rating periods? Maybe instead of calculating rating after each race we could collect results for a month and then treat them all has happening at the same time? in rating calculations? maybe we could set tau to be very small?

Glicko wrote:

'The system constant, τ , which constrains the change in volatility over time, needs to be set prior to application of the system. Reasonable choices are between 0.3 and 1.2, though the system should be tested to decide which value results in greatest predictive accuracy. Smaller values of τ prevent the volatility measures from changing by large amounts, which in turn prevent enormous changes in ratings based on very improbable
results.

What happens if tau is something like 0.05?

Anyway, I know that it is easier said than done, but we should try to keep the original rating calculations math as it was design at all costs.

I wander what rating system Iracing is using to calculate the irating in esport racing game. They have a similar situation. Good driver can crash on lap one and loose with everyone, and then win the next race - just like in condor.

User avatar
BOD1
Posts: 2142
Joined: Mon Aug 07, 2006 2:45 pm
Location: Brittany, France
Contact:

Re: Pilot ranking list - could we have something meaningful?

Post by BOD1 » Tue Sep 28, 2021 4:00 pm

😭😭

I already tested tau with a very small value, it just delay the infinite loop a bit.
It was working great when the rating period is a competition (ie the final ranking).
The problem is that all these rating systems were created to rate a match between 2 people, not a race with several people.

To be scored, a race must be converted into a set of matches: [Player1 vs Player2], [P1 vs P3], [P1 vs P4], [P2 vs P3], [P2 vs P4], [P3 vs P4]

Here is your rating evolution with the current system. I still have to test EloR and maybe Elo.
Witor-rating.png

Lowest rating, after a bad day:
Witor-ratingfall.png
The breaker I added worked here, your lost is limited to 300. The regular rating without breaker should be around 900. In my opinion it would be a nonsense and reflect a wrong skill level.
Witor-rank.png
The rank dropped accordingly:
Witor-rankingfall.png
As far as I read, IRacing uses Elo,
You do not have the required permissions to view the files attached to this post.
Image
Please use this button to send me a PM, Forum messaging is too limited.
Image

witor
Posts: 200
Joined: Wed Dec 15, 2010 11:47 pm

Re: Pilot ranking list - could we have something meaningful?

Post by witor » Tue Sep 28, 2021 6:31 pm

That's a bummer... I can see the problem, which is that when there is a big race with 100 people flying and you crash then you have just lost almost 100 times in a row in glico calculations, which would be very very improbable in other 1v1 games like chess. High rated players just don't lose 100 times in a row...

Looks like glicko or other rating systems are just not suitable for condor. I can't find anything about what algorithm Iracing uses, but that could be something that we need as the nature of the competition in both games is quite similar.

I will keep looking and I will try to ask on Iracing forum. I will report if find something useful.

witor
Posts: 200
Joined: Wed Dec 15, 2010 11:47 pm

Re: Pilot ranking list - could we have something meaningful?

Post by witor » Tue Sep 28, 2021 10:59 pm

Iracing rating system is not public but I got this calculator from members' forum.
iRacing SOF iRating Calculator v1.1.xlsx.zip
Maybe you will find it useful. I just found it and didn't have much time to see how it works or what it is based on.
You do not have the required permissions to view the files attached to this post.

User avatar
BOD1
Posts: 2142
Joined: Mon Aug 07, 2006 2:45 pm
Location: Brittany, France
Contact:

Re: Pilot ranking list - could we have something meaningful?

Post by BOD1 » Tue Sep 28, 2021 11:47 pm

Wonderful, thank you!
As I wrote in my last post (see last line ;) ) it looks like modified ELO.
In a 2-players match, outcome is noted 1 for winner, 0 for loser and 0.5 for a draw (as Glicko does). Here, they convert race to a set of matches (like Glicko: P1 vs P2, P1 vs P3, P2 vs P3 ...) the outcome of player1's matches is distributed between 0.5 and 1 (they add a draw match for each player, P1 vs P1), hence the 0.50 outcome below.
Here, the outcome for Player1 vs 28 players:

Code: Select all

0.50	0.67	0.72	0.82	0.83	0.84	0.85	0.87	0.90	0.91	0.92	0.92	0.93	0.93	0.94	0.94	0.94	0.94	0.96	0.96	0.96	0.96	0.96	0.96	0.96	0.96	0.97	0.97
I'll try to update my glicko rating to distribute outcomes and see if it's better.
Image
Please use this button to send me a PM, Forum messaging is too limited.
Image

witor
Posts: 200
Joined: Wed Dec 15, 2010 11:47 pm

Re: Pilot ranking list - could we have something meaningful?

Post by witor » Thu Sep 30, 2021 12:18 am

I am glad that you found this Iracing rating calculator useful. The only thing that I don't like too much about elo is the fact that inactivity does not affect future changes in rating like RD number in glicko does. However, I think this is a relatively minor issue and elo based rating system should work just fine for our purposes. If someone with high rating stops flying and keeps the rating despite loosing skill so what... I think we can live with that. If skill really was lost as soon as the persons starts flying again his rating will go down.

Thank you for your effort and It is great to see that you are making some progress, and that the whole idea of having a meaningful rating system is looking promising again :D

User avatar
BOD1
Posts: 2142
Joined: Mon Aug 07, 2006 2:45 pm
Location: Brittany, France
Contact:

Re: Pilot ranking list - could we have something meaningful?

Post by BOD1 » Sat Oct 02, 2021 1:02 am

I know why you wanted iRating :lol:
witor-irating.PNG
iRating gives another ranking (sorry Dgtfer, you are now 22nd :lol: ), no more 2010's players (last flight was a loooong time ago) stuck at the top.

Code: Select all

MaxRating: 6463.89774320411
minRating: 176.00662388225604

2945 players
[6400-6500]	1
[5600-5700]	1
[5500-5600]	1
[5100-5200]	1
[5000-5100]	2
[4900-5000]	1
[4800-4900]	2
[4700-4800]	2
[4500-4600]	4
[4400-4500]	7
[4200-4300]	6
[4100-4200]	6
[4000-4100]	4
[3900-4000]	5
[3800-3900]	7
[3700-3800]	5
[3600-3700]	7
[3500-3600]	12
[3400-3500]	10
[3300-3400]	9
[3200-3300]	10
[3100-3200]	12
[3000-3100]	11
[2900-3000]	16
[2800-2900]	18
[2700-2800]	27
[2600-2700]	27
[2500-2600]	34
[2400-2500]	23
[2300-2400]	34
[2200-2300]	29
[2100-2200]	45
[2000-2100]	54
[1900-2000]	63
[1800-1900]	88
[1700-1800]	95
[1600-1700]	159
[1500-1600]	342
[1400-1500]	811
[1300-1400]	414
[1200-1300]	219
[1100-1200]	117
[1000-1100]	86
[900-1000]	36
[800-900]	31
[700-800]	23
[600-700]	12
[500-600]	10
[400-500]	2
[300-400]	4
A wider distribution here, with with a very scattered distribution in the top.

Code: Select all

1	6443	Witor
2	5617	PLP 
3	5511	EP | Erik
4	5124	Maymar 
5	5094	B.Fest
6	5078	TimoA
7	4989	CNorbee 
8	4835	Kuba
9	4809	UWE
10	4787	AM Guettler
11	4761	Knausz Mihaly
12	4539	Charon
13	4532	Norma
14	4503	Ackkie
15	4502	XL136
16	4483	Nikolai Neiva
17	4470	TimKuijpers
18	4460	Nicolas Pizzolante
19	4451	Philipp Kapferer
20	4447	Jonas Angenendt
21	4419	LSanyika 
22	4415	Dgtfer 
You do not have the required permissions to view the files attached to this post.
Image
Please use this button to send me a PM, Forum messaging is too limited.
Image

witor
Posts: 200
Joined: Wed Dec 15, 2010 11:47 pm

Re: Pilot ranking list - could we have something meaningful?

Post by witor » Sat Oct 02, 2021 3:28 am

I didn't want IRating any more than any other rating systems as long as we have something that actually works :)

I was always a big fan of Glicko 2 but you convinced me that it is not suitable for condor. This led me to thinking that why are we actually trying to reinvent the wheel? There must be other racing game that has a rating based on skill - this lead me to Iracing where all competition races are rated. By the way, there are some other online racing games out there, but I don't know if they use rating system and if it is any good. Iracing is in general a very professional game/simulator, used by real F1 drivers with lot's of prize money in top events. This makes me think that their rating system can not be rubbish and that maybe we could borrow :) it from them.

Anyway, if you don't see any strange behavior in the rating calculations Irating might be the winner.

User avatar
BOD1
Posts: 2142
Joined: Mon Aug 07, 2006 2:45 pm
Location: Brittany, France
Contact:

Re: Pilot ranking list - could we have something meaningful?

Post by BOD1 » Fri Oct 08, 2021 11:56 pm

It's online :)
Image
Have a look on competition tab->rating.
Each competition task has a 4th tab with the rating issued after the race, a click on pilot name show his historical rating.
Bonus: on upcoming competition day, competitors can see the probabilities of race outcome and the Strength of Field.
To do: each competition day gives points to competitors (not shown today), I'll use them for an annual ranking.

I had a hard time to translate "rating" ("Evaluation"), which has the same meaning has "ranking" in french, and particularly "Strength of Field" ("Force", where "Niveau" -Level- would be maybe better). French speakers are welcome to give advice :)
Image
Please use this button to send me a PM, Forum messaging is too limited.
Image

User avatar
wickid
Posts: 2429
Joined: Mon Dec 04, 2006 7:32 pm
Location: Venlo, NL
Contact:

Re: Pilot ranking list - could we have something meaningful?

Post by wickid » Sat Oct 09, 2021 6:44 am

Cool! Thanks

One thing that would be useful is to add the callsign to the table. The usernames don't say much. They usually differ from the in game name. We know most of the people by their callsigns anyway :wink:
PH-1504, KOE

6266
Posts: 925
Joined: Tue Aug 25, 2020 7:07 am

Re: Pilot ranking list - could we have something meaningful?

Post by 6266 » Sat Oct 09, 2021 1:17 pm

Thanks BOD1 for the fast and surely very correct calculated rating. That will be useful for all who likes it.

But I'm sorry to say, that I don't like to be rated. You are playing in public with my personal data without my explicit permission. On all my ftr uploads I deactivated that the result can be used for a ranking or best list (I don't remember the exact wording, but there is only one option and it is activated per default).

I have unregistered now the running NCC with deleting all my results there and I request you to delete all my other results on CC as soon as possible.

Thanks for that
Visit https://www.baleit.no

Vintage Series 24, The Journey
25 landscapes with free available textures, newest: Maine

Discord https://discord.gg/bAXJaRT6ZC

User avatar
BOD1
Posts: 2142
Joined: Mon Aug 07, 2006 2:45 pm
Location: Brittany, France
Contact:

Re: Pilot ranking list - could we have something meaningful?

Post by BOD1 » Sat Oct 09, 2021 3:19 pm

No problem, is it ok if your rating is not displayed at all? (charts and board) but still computed to get a coherent rating.
One player's rating is computed against other players' rating, so if one player doesn't want to be rated, all other pilots can't get evaluated correctly and, worse, in a 2 players match, the second player is not rated at all, that's a bit unfair IMHO.
Image
Please use this button to send me a PM, Forum messaging is too limited.
Image

Post Reply