Pilot ranking list - could we have something meaningful?

[BOD1]

To be valid day, at least 25% of pilots must finish the task OR do at least 100km (handicapped distance, can be less with older gliders), almost every tasks are valid (a couple of tasks on 3000+ were not).

I'll add a feature to filter rating in order to display only active pilots. I think I'll also generate a rating every first day of the year with evolution since the previous year. And concerning Witor's suggestion, sort option will be available later, for sure.

@Fox9, I think rating should not be used for ranking. Rating distributes points for each race (the more competitors and the higher the Strength of Field, the more points)., this sum of points will be used to generate a yearly ranking. If a pilot selects his preferred tasks, he'll earn less points and will be low in this ranking.

@EDB
About safety rating: actually I can't get sufficient information about pilot behavior from FTR (collision, wrong turn in thermal, dangerous flight, ...) to calculate accurately.
About banning crashed pilots: a solution could be to stop rating/giving points a player after first crash in competition (the crash is rated) until the end of the competition. It's a bit difficult to implement, I'll see if I have the courage to do it.

I will add an option for organizers to limit registration to a competition based on a minimum rating a player must have to participate. For unrated players, the pre-requisite could be to have obtained the 3 badges (silver, gold, diamond).
We must also keep in mind this community of competitors is very small: on a 12 years period, there are only 3,000 pilots rated, that's not much comparing to the 27,000 members. The average number of days per competitors is 34 (on 12 years!).

I finished testing systems and settings (I spent 5 weeks, I think it's enough for now) and got a very satisfying balance in what we have (see rating on CC, released yesterday afternoon).
Crashed pilots are seriously impacted, especially if they have a high rating or if there were few crashes during the task.

[witor]

I think rating should not be used for ranking. Rating distributes points for each race (the more competitors and the higher the Strength of Field, the more points)., this sum of points will be used to generate a yearly ranking. If a pilot selects his preferred tasks, he'll earn less points and will be low in this ranking.

I don't think you need to do anything with the old ranking system where the more you fly the more points you get. It could be left as it was and there is no need to change it. My initial idea was to have something (skill rating) in addition to the existing ranking not to replace it.

a solution could be to stop rating/giving points a player after first crash in competition (the crash is rated) until the end of the competition.

Please, please, please don't do that. As we all know we already don't have enough people participating in competitions. If one of top pilots crashes and is banned others can not compete with the best any more and the competition is pointless. There is no satisfaction in winning a competition just because the best pilot crashed once. And what about a blue screen? Are we going to ban people from competition because they had a blue screen on day one?

Like I already said and I think you agree with me; a crash will already be punished enough by a lots of rating points lost - there is no need to make it even more severe.

[BOD1]

a solution could be to stop rating/giving points a player after first crash in competition (the crash is rated) until the end of the competition.

Please, please, please don't do that. As we all know we already don't have enough people participating in competitions. If one of top pilots crashes and is banned others can not compete with the best any more and the competition is pointless. There is no satisfaction in winning a competition just because the best pilot crashed once. And what about a blue screen? Are we going to ban people from competition because they had a blue screen on day one?

The crashed pilot is not banned from the competition, he can still continue to fly all other days and be scored and ranked in this competition. It's only the rating which ignores it.

[witor]

It's only the rating which ignores it

If I understand you correctly this will again lead to some races in a rated competition not counting towards rating and as a result the rating will not really work correctly (rating will no longer reflect the skill). If I crash in a first race in a competition in which 15 out of total 20 races count towards the final standings I have already lost my rating, my rating will no longer be calculated in this competition so I can now risk, fly low, and do all sorts of crazy stuff which might help me to win because I know my rating can no longer be affected. This is definitely something we don't want.

The way it is at the moment is perfect. There is no need to do anything with what is counted and what isn't. If you want to count just valid days, that's fine. There is so few invalid days that this doesn't really matter. I will say it again: huge loss of rating as a result of a crash in a competition is punishing enough and there is absolutely no reason to make rating system not really work as intended just to punish crashes even more. By the way organisers of the competition can always give penalty points for crashes, but everyone moved away from that because people generally didn't like it, because one blue screen could give you penalty which eventually could make you lose the competition.

In the past, when people didn't know that in the future someone will calculate ratings, some pilots occasionally out of frustration just disconnected when they got low if they knew that there is enough competition days left. Disconnection was treated as a crash and because of that (if you implement your change) remaining flights will not count towards rating so again rating calculations will be flawed.

You have done an amazing job so far, just keep it as it is and don't make unnecessary changes. You could introduce sorting different columns, active and all time ratings, and things like that which do not actually affect key concepts of the rating calculation. By artificially selecting which days count and which don't rating will not really reflect true skill. I repeat my plea: please don't try to fix something that is not broken.

[BOD1]

Understood, you totally right! I didn't think about that behavior.. Anyway, I was not really convinced to implement this, you convinced me to not do it at all

[witor]

you convinced me to not do it at all

I am really happy to hear that, and I am happy that I was able to help

[witor]

Nice job with adding the sorting, but I just want to let you know that sorting of 'rank change' and 'rating change' columns does not fully work as it should. It does sort in the descending order correctly, but if you click sort again to sort it in the ascending order then sorting does not work.

I am wondering a little bit what 'points' on the rating tab actually mean? Are you going to use them for something? Do we even actually need them?

Looking at overall results of different competitions I thought that it would be nice to see an additional column on overall results page with the total rating change as a result of that competition. This would be the sum of all rating changes from all races in which a given pilot participated. It is not necessary, but if it is not to time consuming to be implemented please do consider it.

[witor]

I think rating should not be used for ranking. Rating distributes points for each race (the more competitors and the higher the Strength of Field, the more points)., this sum of points will be used to generate a yearly ranking. If a pilot selects his preferred tasks, he'll earn less points and will be low in this ranking.

Oh, now I see what you are going to do with the 'points' on rating page and I am sorry to say but I think it is a terrible idea as once again we end up with a system where the more you fly the more points you have and end up higher in the ranking list, which totally defeats the purpose of ELO based system that suppose to measure skill not participation (we already have pilot's ranking list, which measures participation and should stay as it was).

The way rating was presented yesterday was just fine. People on top of the list were really pilots who perform well in competitions (first 50 looked really impressive) and definitely deserved to be recognized as top condor pilots. Today, when points are used we have another meaningless list.

The initial idea was to have a rating list based on skill and you did a fabulous job implementing ELO system but by addition of 'points' you tried to fix something that was not broken and as a result you made it worse.

My suggestions:

1) Just forget about 'points' and go back to purely ELO based ranking.
2) instead of option to chose a specific year just have 2 options - all time and active (which will only show people who flew in the last year)

[alioth]

My suggestions:
1) Just forget about 'points' and go back to purely ELO based ranking.
2) instead of option to chose a specific year just have 2 options - all time and active (which will only show people who flew in the last year)

+1
I thought exactly the same when I saw the ranking list yesterday.

[Snapcase]

My suggestions:
1) Just forget about 'points' and go back to purely ELO based ranking.
2) instead of option to chose a specific year just have 2 options - all time and active (which will only show people who flew in the last year)

+1
I thought exactly the same when I saw the ranking list yesterday.

+2

[Fox9]

How is the case handled if a pilot does not upload his FTR (maybe he has a blue screen or he simply has a bad result)?

Fox9

[witor]

Competition organisers should use DShelper and upload logs of who joined the server to CC. If ftr is not sent it is treated as a crash.

Of course we need organisers to send logs from DShelper, which unfortunately it is not always the case, but I hope it will be in the future.

There is no other way to see if people are not trying to protect their rating by not sending ftr's of bad flights.

[BOD1]

I've to relaunch rating, the current system also notes competitions that do not have server logs. So about 30% of tasks won't be considered for rating, it can change a lot at the end!
@witor about column sorting, it works as expected: for example on "Rating (change)" column, one click sorts it on Rating values (descending), second click sorts it on Change values (descending).
About additional column on overall results with the total rating change, if I understand you, it is the same as displaying the difference between the last rating and the first? Or maybe you want to filter changes from parallel competitions?

About ranking and points: I don't want to rank pilots according to their rating, this is not recommended and is not representative of the pilots' current performance or active participation. Even if we only list active pilots, there is nothing to prevent a highly ranked pilot from participating only once a year to keep his ranking and limit the risk of going down in the ranking.
The ranking AND the rating can live together without any problem, their objective is not the same!
To make it easier to read the results, I can separate the ratings table (and put a ranking number in front of it if you like ) from the ranking table.

The FAI and iRacing (and other sports) do a similar ranking, based on points earned during the season.
I am however open to an adjustment on the points calculation to mitigate the gains brought by hyper-active participation (e.g. by averaging the total points earned, with a minimum participation in n races: score = total points / max(nb races, n) to encourage drivers to participate more.

[witor]

About additional column on overall results with the total rating change, if I understand you, it is the same as displaying the difference between the last rating and the first? Or maybe you want to filter changes from parallel competitions?

In a given competition lets say there are 3 races. Let's say someones rating changes by +20 after the first race, by -10 in the second and +30 in the third day. Then on the overall results page next to the final place of that pilot in a given competition we would see 'total rating change' of +40 to show that overall as a result of this competition this pilot has gained 40 rating points in total.

About ranking and points: I don't want to rank pilots according to their rating, this is not recommended and is not representative of the pilots' current performance or active participation. Even if we only list active pilots, there is nothing to prevent a highly ranked pilot from participating only once a year to keep his ranking and limit the risk of going down in the ranking.

If you don't want to rank pilots by they rating you could just not call it 'ranking' just keep the old ‘Pilots ranking list’ as it wa,s where people can submit flights, gain points and be on a ranking list the way is was before we started talking about ELO based 'rating'. From day one I didn't want to change the old ranking list. It can be as it always was. What I don't think is a good idea is to mix ELO based ratings with some points that are based on participation not skill. The most fundamental idea that made me start talking about skill (ELO) based ratings is the rating list in chess. I was hoping that we could have something similar in condor that would show not how much someone flew, but how good someone is in terms of winning with other condor pilots. Rating systems like this work on every single chess site where millions of people are playing an everyone has a rating, there is Irating, there is MMR (match making Rating) in starcraft and other rating systems usually based on elo or glicko in hundred's of games. That's why I don't think mixing participation based point ranking and skill based ELO ratings is a good idea as they are just two different things.

ELO rating system is used in chess and top players do not stop playing just because they want to keep their rating and I don't think it will happen in condor. I will definitely not stop flying just to keep my rating. https://ratings.fide.com/top.phtml?list=men

Anyway, participating only once a year would not have any negative effect. What if top pilot takes part in one flight just to keep his rating active and then crashes, and loses 200 rating points? Statistics and probability will do the job to keep ratings close to true skill level, which in condor doesn’t really change that quickly. If someone is really good in condor he will not become really bad in just few months. Anyway it doesn't have to be just one flight in the last year. This is something that we could think about. Maybe instead of a one flight in the last year we could say 10 flights in a year to keep rating active?

The FAI and iRacing (and other sports) do a similar ranking, based on points earned during the season.

We just need to be a bit careful. Iracing is similar in many ways to condor, but it also has some big differences. The main one is that I racing is trying to make money by selling tracks and cars. If you want to be on top of the standings for a given season you need to participate in many races on different tracks which means Iracing is making money. Standings in a given season are just designed to motivate people to buy stuff. It is in a way 'pay to win' strategy. We don't need that in condor. To be honest nobody serious in Iracing community really cares where they are in the standings in a given season as it doesn't really mean how good they are. Good drivers just care about their Irating or if they are the top of the world they care about winning prestigious race series with serious prize money. I had a feeling that the new points that you started calculating are based on Iracing season points and as I said before I think that it is unfortunately a terrible idea which we don't really need as we already have very good old ‘Pilots ranking system’ that rewards participation.

The ranking AND the rating can live together without any problem, their objective is not the same!

Exactly! I couldn't agree more! We can have old ranking system and new addition which is just ELO based RATING. That’s why I suggest to just get rid of the new ‘points’ column or rating page, don’t use word ‘ranking’ anywhere on the rating tab. Rating tab should just show what is happening with ELO ratings of participating pilots and it has to be clear that RATING on ‘rating’ page is calculated based on ELO, which is used in Chess and other games. As I mentioned before you already had it almost perfect but we bent back couple of steps by unnecessarily introducing ‘points’ to the ‘RATING’ tab.

I've to relaunch rating, the current system also notes competitions that do not have server logs. So about 30% of tasks won't be considered for rating, it can change a lot at the end!

I didn’t know that you can filter tasks and only use task results when you have server logs. I think it is a good idea. People should not be allowed to protect their rating by not sending the ftr files which would skew the validity of the ELO method and overall ratings. If server logs are not sent flight could (or should) be treated as not rated. That definitely does make sense.

[witor]

Well done!

After the most recent update I think we have something that is very, very close to what I was initially thinking about and what I think will work well. I personally greatly appreciate the amount of work you have put into this and I hope others will see it as a useful tool and a positive addition to Condor Club and will also appreciate your efforts.

Having competition ranking in a different tab and not mixing it with ccRating will work well in my opinion and yes, I totally agree that ccRating is a cool name .

If you are not too tired working on this project yet, I think there are some small (just cosmetic) final changes/improvements that you could implement.

1) Sorting of the double columns. I know you said it works fine but If I click on the arrows next to Rating (change); after first click column is sorted in decreasing ‘rating’ order, second click I get decreasing ‘change’ order. I can’t get the list to be in increasing ‘change’ or increasing ‘Rating’. Maybe split ‘Rating (change)’ column into 2 columns with separate arrows for sorting both ‘Rating’ and ‘change’? The same could be done with the first ‘Rank (change)’ column as it has the same problem.

2) If I understand correctly the ‘Nb Ratings’ column shows number of rated flights in a given year. I really hope that ccRating is not calculated just based on flights in the last year. I hope that all valid flights prior to the year selected on the top of the page are used in calculations as the more data is used the more reliable rating is. If that is the case, I think that ‘Nb Ratings’ column should show number of all flights used to calculate ccRating that we can see on ‘ccRating’ tab, not just flights in a selected year. For example, for me, if I select 2021 ‘Nb Ratings’ should show a much higher number (probably about 700’ if I select 2020 ‘Nb Ratings should be smaller (it should show my total flights minus the flights in 2021). This would allow everyone to see if someone’s ccRating is based on a small number of flights which means that statistically it might not yet reflect the true skill of that pilot or if it is based on large total number which would make it more reliable. Don’t you think that could be a better way of showing ‘Nb Ratings’?

3) I can see that some places (for example fifth place) is missing from ‘Rank (change)’ column. I assume that this is because whoever is in fith position on that list is not an active pilot anymore and (as we have previously agreed) is not being shown, which is great. My suggestion would be to just have ‘Rank(change)’ column without any missing numbers. This would show everyone where they are on an active list. As soon as whoever is in fifth position becomes active again everyone below would move one position lower on the list and person will jump into his deserved fifth position. This would prevent situation where pilot who is first on the list stops flying and nobody can be with number 1 on the list any more. The pilot who is currently in position that says ‘100’ on the list will struggle to see where he actually places on the active pilots list. I think it is more valuable to see where everybody is on the current list and if we want to see who is the person who was in fifth, we can always select different year on the top of the page, go back in time and everything works perfectly.

4) Lastly – it would be nice to see total number of ccRating points gained or lost in a particular competition on the ‘overall results’ page of a rated competition. This would be just the sum of CCRating changes from all flights in a given competition for a given pilot – Not a necessary feature but if we have ccRating change for each flight of the competition why not have the total on the overall results page… This however is not necessary so if you feel that you have better things to do I would totally understand.