Criticker ranking system has one enormous flaw

Ideas to improve Criticker and new feature requests, as well as announcements about new enhancements.
MTR
Posts: 12
242 Ratings
Your TCI: na
Joined: Fri Jul 12, 2019 7:13 am

Criticker ranking system has one enormous flaw

Post by MTR »

Imagine we have hypothetical user A who has only ever ranked 26 movies, each one with different rating from 75 to 100.

Then we have user B who has ranked those 26 movies and gave them exactly the same ratings as user A. But on top of that, user B also ranked additional 75 movies, each one with different rating from 0 to 74.

TCI of user A and user B is over 38 :!:

But quite obviously, it should be 0 :!:


So... I don't know how viable is this outside aforementioned example, but to get the TCI at 0, you would need to calculate percentiles only among shared rated titles. It seems it would also add a lot of new calculations to be made, so it might not be viable for not so big site, but leaving it as it is certainly a bad idea.

djross
Posts: 1152
10 Ratings
Your TCI: na
Joined: Sun Apr 16, 2006 12:56 am

Re: Criticker ranking system has one enormous flaw

Post by djross »

MTR wrote:But quite obviously, it should be 0 :!:


This is not obvious at all since it assumes that what matters is the numerical score rather than the overall distribution: if a user's lowest score is 75, then that's the worst movie they've ranked. If instead of 26 movies they ranked 2600 movies, the situation would still be the same, so if it's just that they haven't ranked any of the bad movies they've seen yet, then the best solution is to rank more movies. The system's fine.

PrestoBix
Posts: 71
0 Ratings
Your TCI: na
Joined: Thu Aug 20, 2015 6:48 am

Re: Criticker ranking system has one enormous flaw

Post by PrestoBix »

Yeah this totally isn't an issue at all.

livelove
Posts: 623
0 Ratings
Your TCI: na
Joined: Thu Sep 17, 2015 9:36 pm

Re: Criticker ranking system has one enormous flaw

Post by livelove »

@MTR: welcome to the forum.

The users who replied to you didn't get your point, which is a point well taken.
Be assured, you have raised a legitimate issue that is a regularly resurfacing topic of discussion here, see:

A question about percentiles
From Tiers to Percentiles: Introducing a Big New Change to Criticker
What happens to tiers if a user only rates movies he likes?
Disagree with the Ranking Scale
Do I need to see more bad movies?
Last edited by livelove on Fri Sep 27, 2019 4:57 am, edited 2 times in total.

amazedemon
Posts: 24
25 Ratings
Your TCI: na
Joined: Mon Aug 18, 2014 11:29 am

Re: Criticker ranking system has one enormous flaw

Post by amazedemon »

MTR wrote:TCI of user A and user B is over 38 :!:

Surely your assumption would mean the TCI is ~28, not 38 (or 37) as the 26 films that are rated identically contribute 26 zeros to the average.

That however is nit-picking. My main point, or rather question, is whether the TCI's (in general) are mutually commutative. That is to say would User A's TCI for User B be 0, and User B's TCI for User A be 28.

My TCIs for people who have commented are:
@MTR: not enough ratings
@djross: 3.5270
@PrestoBix: 2.9076
@PedroPT: 2.1738
@livelove: not enough ratings

Is your TCI with me the same?

NB. I'm still on the old tier system.

livelove
Posts: 623
0 Ratings
Your TCI: na
Joined: Thu Sep 17, 2015 9:36 pm

Re: Criticker ranking system has one enormous flaw

Post by livelove »

amazedemon wrote: My […] question, is whether the TCI's (in general) are mutually commutative.
My understanding is that this is the case.

amazedemon wrote:would User A's TCI for User B be 0, and User B's TCI for User A be 28.
following the above, no the TCI would be the same for both users (in both directions)

amazedemon wrote: @livelove: not enough ratings
Is your TCI with me the same?
your TCI with me currently is: 21.1429
(you would have to lower the "Films in Common Minimum" value to get a TCI for me)

livelove
Posts: 623
0 Ratings
Your TCI: na
Joined: Thu Sep 17, 2015 9:36 pm

Re: Criticker ranking system has one enormous flaw

Post by livelove »

MTR wrote: you would need to calculate percentiles only among shared rated titles
cross-reference:
the same proposal has also been made in this thread:

Take into account the fact that not all users have seen the same films only

uvlalid wrote:I propose that when calculating the TCI of a user, only the films that both users have seen are used in the equation. So instead of calculating the percentile scores of films based on all the films seen by a user, the percentiles should be based only on all the films both users have in common.

To give an example scenario, imagine there are two users:

User 1
Films watched - Score
Film A - 50
Film B - 0
Film C - 100

User 2
Films watched - Score
Film A - 100
Film B - 50
Film D - 0

Using my suggested system to calculate the TCI of User 2 for User 1, only Films A and B will be used which for both users will have percentile scores of 100 and 0 respectively in that case.

amazedemon
Posts: 24
25 Ratings
Your TCI: na
Joined: Mon Aug 18, 2014 11:29 am

Re: Criticker ranking system has one enormous flaw

Post by amazedemon »

Honestly I'm okay with the way my PSI appear at the moment as they are usually within a 2 point margin. Though there are many older films/ Japanese films with quite high PSI's that I have very little interest or motivation in watching. (Yes I know I'm a philistine).

Sometimes I just like playing around with data, and asking what if?
I'm currently playing around with an extended version of @uvlalid's example to see where that leads.

livelove
Posts: 623
0 Ratings
Your TCI: na
Joined: Thu Sep 17, 2015 9:36 pm

Re: Criticker ranking system has one enormous flaw

Post by livelove »

amazedemon wrote:I'm currently playing around with an extended version of @uvlalid's example to see where that leads.
great, please share if you come across something interesting

MTR
Posts: 12
242 Ratings
Your TCI: na
Joined: Fri Jul 12, 2019 7:13 am

Re: Criticker ranking system has one enormous flaw

Post by MTR »

[spoiler]
djross wrote:
MTR wrote:But quite obviously, it should be 0 :!:


This is not obvious at all since it assumes that what matters is the numerical score rather than the overall distribution: if a user's lowest score is 75, then that's the worst movie they've ranked. If instead of 26 movies they ranked 2600 movies, the situation would still be the same, so if it's just that they haven't ranked any of the bad movies they've seen yet, then the best solution is to rank more movies. The system's fine.


You haven't understood my post one a bit. :)

PrestoBix wrote:Yeah this totally isn't an issue at all.


Thank you for your "input". :)

amazedemon wrote:
MTR wrote:TCI of user A and user B is over 38 :!:

Surely your assumption would mean the TCI is ~28, not 38 (or 37) as the 26 films that are rated identically contribute 26 zeros to the average.


You either didn't understood my post one a bit or you simply don't understand TCI one a bit. :)

-------------------------------------------------------------------------------------------------------------------------------------------------------[/spoiler]

Anyway something that might interested livelove I guess. I made two simple examples and will post them in image form.

Example 1 (similar to my opening post) https://i.imgur.com/DB89dPA.png

Example 2 (showing that rating bad titles makes your TCI much worse than it should be with someone who only rates good titles) https://i.imgur.com/MKXF4bn.png

Example 3 (the best one yet, user A and user B have very similar tastes, but user A watches a lot of good movies that user B doesn't and user B watches a lot of bad movies, that user A doesn't) https://i.imgur.com/LYhMKF9.png

Post Reply