From Tiers to Percentiles: Introducing a Big New Change to Criticker

[fabiovisnadi]

I guess that, although the changes improved the system, a major problem involving the Tier/Percentiles method still remains: most of the users here tend to have more bigger ratings than lower ones and this difference between certain ratings system and one from another user doesn't involve necessarily more affability or less ridigity, but because it's natural, while we watch more and more films, to avoid some of them because they do not interest us or because we think they're bad, etc. And eventually, in the long run, 80% of our ratings will include films we enjoy and only the remaining 20% of them will be of films we consider bad. When this happens, it's possible that the tier of a film we consider good in our page is 2 or 3, while only the really bad ones are tier. In some cases, the films someone ranks 20 or the films someone ranks 65 are in the same tier, just because he avoids watching films that look like a "20" film. The tier system, although it helps on the PSI's and TCI's, depends on the idea of a proportional rating system, something like "this user rates from 1 to 10 and the other from 5 to 15, so his 2 is equal the other's 6" (in a more complex and sophisticated algorhytm, of course). I think that, even though this analogy is suitable for some of the users, a manual Tier system, like the colour system, would be more accurate for our specific tastes. I, personally, wouldn't want a film that I rated 70 in the same tier that a film I rated 20 just because I don't watch filmes that I think I would rate 20.

[BadCosmonaut]

I guess that, although the changes improved the system, a major problem involving the Tier/Percentiles method still remains: most of the users here tend to have more bigger ratings than lower ones and this difference between certain ratings system and one from another user doesn't involve necessarily more affability or less ridigity, but because it's natural, while we watch more and more films, to avoid some of them because they do not interest us or because we think they're bad, etc. And eventually, in the long run, 80% of our ratings will include films we enjoy and only the remaining 20% of them will be of films we consider bad. When this happens, it's possible that the tier of a film we consider good in our page is 2 or 3, while only the really bad ones are tier. In some cases, the films someone ranks 20 or the films someone ranks 65 are in the same tier, just because he avoids watching films that look like a "20" film. The tier system, although it helps on the PSI's and TCI's, depends on the idea of a proportional rating system, something like "this user rates from 1 to 10 and the other from 5 to 15, so his 2 is equal the other's 6" (in a more complex and sophisticated algorhytm, of course). I think that, even though this analogy is suitable for some of the users, a manual Tier system, like the colour system, would be more accurate for our specific tastes. I, personally, wouldn't want a film that I rated 70 in the same tier that a film I rated 20 just because I don't watch filmes that I think I would rate 20.

That problem certainly still exists for the tier system, but does it exist under the percentile system too?

I could have misunderstood, but I thought the percentile system was just the tier system with up to 100 'tiers' (depending on how many different ratings you use) instead of forcing 10 or less tiers (as it did under the old tier system). If there are effectively as many 'tiers' under the percentile system as different ratings you give out, then doesn't that solve the problem you're referring to?

For example, let's say you use a 10 point scale: 10, 20, 30, 40, 50, 60, 70, 80, 90, 100. Under the percentile system, each of those will be its own 'tier.' If you use a 100 point scale, then each of those 100 will be its own 'tier.'

And if that's true, then does the problem you're describing still exist? I'm pretty sure it doesn't, but I could be mistaken.

[AFlickering]

I guess that, although the changes improved the system, a major problem involving the Tier/Percentiles method still remains: most of the users here tend to have more bigger ratings than lower ones and this difference between certain ratings system and one from another user doesn't involve necessarily more affability or less ridigity, but because it's natural, while we watch more and more films, to avoid some of them because they do not interest us or because we think they're bad, etc. And eventually, in the long run, 80% of our ratings will include films we enjoy and only the remaining 20% of them will be of films we consider bad. When this happens, it's possible that the tier of a film we consider good in our page is 2 or 3, while only the really bad ones are tier. In some cases, the films someone ranks 20 or the films someone ranks 65 are in the same tier, just because he avoids watching films that look like a "20" film. The tier system, although it helps on the PSI's and TCI's, depends on the idea of a proportional rating system, something like "this user rates from 1 to 10 and the other from 5 to 15, so his 2 is equal the other's 6" (in a more complex and sophisticated algorhytm, of course). I think that, even though this analogy is suitable for some of the users, a manual Tier system, like the colour system, would be more accurate for our specific tastes. I, personally, wouldn't want a film that I rated 70 in the same tier that a film I rated 20 just because I don't watch filmes that I think I would rate 20.

i've expressed similar sentiments over the years, and it certainly remains the biggest problem with this system. when people like movies they've rated percentile 40 more than i like movies i've rated percentile 70 (or vice-versa), the TCI is gonna be off, and that's frequently and clearly the case. i think the TCI of many of my top TCIs is inflated (as in, improved) beyond our similarities in taste just because we watch a similar proportion of movies we enjoy versus movies we don't, and again vice-versa. i personally have given thought to rating a bunch of ultra-obscure films low just to balance my own rankings out, which isn't exactly ideal but is better than the alternative of going out of my way to watch a few hundred crappy movies. :p (note: i'm not actually going to do this as i feel it'd be kind of selfish at the expense of the site+movies in question, but i wouldn't be surprised if others did it)

that said, it isn't a dealbreaker or anything, and my PSIs are pretty accurate on the whole. they're putting in so much effort into improving the site lately that i don't want to complain too much, i'm grateful for the improvements that have been made.

edit: also, this problem is lessened somewhat by the fact that probably 90% of users will have more positive ratings than negative, and if we're mostly in that same boat it means our ratings will be more proportional. i do often come across people who very obviously have a different proportion than i do, though, and whose TCI becomes sort of meaningless as a result (which is categorically a bad thing for the site, as everything is based on the assumption that TCIs are meaningful)

[livelove]

I have posted an idea to the feature request section, which, I think, would get rid of both problems quoted below (the rounding errors and the missing 100% percentile) by storing a higher precision normalized score for internal calculations instead of rounding to percentiles (or tiers). No worries, the end-result could still be displayed to the user in the very same way it is currently done, as percentiles, but the internal accuracy would improve, and thus the prediction accuracy (for both TCI and PSI).

rounding errors:

Imagine two users see the same 100 films and have the exact same opinion of them.
[…]
The two users still have exactly the same opinions of the films. The only difference is the level of granularity of the two rating systems. How different would the two sets of ratings appear to Criticker?
Using the "midpoint" formula, there would be absolutely no difference if not for rounding. But rounding creates a small difference of 0.5 per film, for a total difference of 50.
Using the "lower bound" formula, the difference is 19 times greater than that: 950.

missing 100% percentile:

it's impossible to acheive a percentile rank of 100, because a score can never be lower than itself, and we use the floor as opposed to rounding the number. Percentiles can therefore range from 0 to 99.

The movies I gave a score of 100 to should be in the 100th percentile (≈99.67%), but it says 99th instead.

99th is the highest possible percentile because the site is calling the lowest percentile the '0th percentile'. to have percentile called 100th we'd need to get rid of the 0th (just like we had no 0th tier in the old system and went 1-10). might be a good idea, as the current way seems to be confusing people and 1-100 just generally makes more sense than 0-99 IMO, plus it's more in keeping with the previous system.

Yeah, as AFlickering says, this is all due to the fact that if you had both 0 and 100, you would end up with 101 percentiles. And if you rounded in the traditional way, the 0 and 100 "bins" would only be half as big as the other 99 bins (0 would represent the half a percent from 0.00 to 0.49, but 1 would represent the whole percent from 0.50 to 1.49, and so on).
So to get 100 bins of equal size, it seems like percentile systems usually choose to round all decimals either up or down.
A point in favor of the current 0-99 approach is that what's in front of the decimal stays the same, so 99.01% and 99.99% are both counted as 99%. I guess I slightly prefer 0-99 for that reason, but I can totally understand how 1-100 could seem more intuitive too. Either way works. And probably either way, there will still be questions from time to time...

[cke]

I guess that, although the changes improved the system, a major problem involving the Tier/Percentiles method still remains: most of the users here tend to have more bigger ratings than lower ones and this difference between certain ratings system and one from another user doesn't involve necessarily more affability or less ridigity, but because it's natural, while we watch more and more films, to avoid some of them because they do not interest us or because we think they're bad, etc. And eventually, in the long run, 80% of our ratings will include films we enjoy and only the remaining 20% of them will be of films we consider bad. When this happens, it's possible that the tier of a film we consider good in our page is 2 or 3, while only the really bad ones are tier. In some cases, the films someone ranks 20 or the films someone ranks 65 are in the same tier, just because he avoids watching films that look like a "20" film. The tier system, although it helps on the PSI's and TCI's, depends on the idea of a proportional rating system, something like "this user rates from 1 to 10 and the other from 5 to 15, so his 2 is equal the other's 6" (in a more complex and sophisticated algorhytm, of course). I think that, even though this analogy is suitable for some of the users, a manual Tier system, like the colour system, would be more accurate for our specific tastes. I, personally, wouldn't want a film that I rated 70 in the same tier that a film I rated 20 just because I don't watch filmes that I think I would rate 20.

i've expressed similar sentiments over the years, and it certainly remains the biggest problem with this system. when people like movies they've rated percentile 40 more than i like movies i've rated percentile 70 (or vice-versa), the TCI is gonna be off, and that's frequently and clearly the case. i think the TCI of many of my top TCIs is inflated (as in, improved) beyond our similarities in taste just because we watch a similar proportion of movies we enjoy versus movies we don't, and again vice-versa. i personally have given thought to rating a bunch of ultra-obscure films low just to balance my own rankings out, which isn't exactly ideal but is better than the alternative of going out of my way to watch a few hundred crappy movies. :p (note: i'm not actually going to do this as i feel it'd be kind of selfish at the expense of the site+movies in question, but i wouldn't be surprised if others did it)

that said, it isn't a dealbreaker or anything, and my PSIs are pretty accurate on the whole. they're putting in so much effort into improving the site lately that i don't want to complain too much, i'm grateful for the improvements that have been made.

edit: also, this problem is lessened somewhat by the fact that probably 90% of users will have more positive ratings than negative, and if we're mostly in that same boat it means our ratings will be more proportional. i do often come across people who very obviously have a different proportion than i do, though, and whose TCI becomes sort of meaningless as a result (which is categorically a bad thing for the site, as everything is based on the assumption that TCIs are meaningful)

Agree 100%. This is the only problem with the site to me and it's something I've raised with them before. But like you, I am hesitant to criticise too much because overall the site is amazing and I love it dearly.

With the system in place there is no avoiding that each user's ranking distribution has a fairly large effect on how the comparisons are made. The response I got back was it's the best system possible and without having put too much thought into possible solutions, who am I to disagree with that?

[BadCosmonaut]

I understand how it was a problem under the old 10 tier system, but I'm not sure I understand how it's a problem under the percentile system. Can someone ELI5 for me?

[livelove]

Agree 100%. This is the only problem with the site to me and it's something I've raised with them before. But like you, I am hesitant to criticise too much because overall the site is amazing and I love it dearly.
[…] The response I got back was it's the best system possible

Same here.
I think the takeaway of the thread: What happens to tiers if a user only rates movies he likes? and the thread: Disagree with the Ranking Scale is that we have to rate bad movies.

I am also wondering, whether there is another (better) way.
But I admit, that I haven't yet found a solution either.

[BadCosmonaut]

Same here.
I think the takeaway of the thread: What happens to tiers if a user only rates movies he likes? and the thread: Disagree with the Ranking Scale is that we have to rate bad movies.

I am also wondering, whether there is another (better) way.
But I admit, that I haven't yet found a solution either.

Under the new system, I don't think you have to rate a bunch of bad movies. You just have to rate 1 bad movie for it to register that tier. As long as a rating (and therefore a tier) exists for each different rating you can possibly give on your personal rating scale, then I don't see how the old problem being discussed could still be happening.

For example, let's say you've seen 1,000 movies and use a 10 point scale. Let's say it looks like this: 200 movies at 10, 200 at 9, 200 at 8, 200 at 7, 195 at 6, and 1 movie rated for each of 5, 4, 3, 2, 1. With this scale, the new system won't try to force any less than 10 tiers. It used to force tiers into 10% of your total amount rated (to the best of its ability). Now it doesn't need to do that, since you can have up to 100 tiers. And if that's true, then I can't see how the old problem is still a problem. There is no grouping of different ratings. There is no forced merging of what should be different tiers.

The example above would work just as well up to a 100 point scale. No more than that is needed, since the maximum scale you can have is a 100 point scale.

Do I misunderstand how the new system works? If not, then I don't see the issue.

[livelove]

I understand how it was a problem under the old 10 tier system, but I'm not sure I understand how it's a problem under the percentile system. Can someone ELI5 for me?

I'll try (can't do completely without maths though):
Let's assume a user ranks 10 films with the following scores: 91, 92, 93, 94, 95, 96 ,97, 98, 99, 100.
The film with the score of 91 is the Godfather.
==> In the (10) tiers system, the Godfather is in tier 1.
==> In the (100) percentiles system, the Godfather is in the 5% percentile.

So the granularity is higher (instead of jumping "tier-like" from 10% to 20%, you jump from 1% to 2% to 3% etc.), but in both cases the score 91 translates nowhere near tier 9 or the 91% percentile.

This is what the other users are talking about above.

In short: Regardless of how high (nominally) you rank your films, your 10% worst ranked films (formerly tier 1) are still in your bottom 10% percentile, that has not changed (what's new is the higher granularity, i.e. smaller steps in-between – 1% steps if you rank enough films).

[BadCosmonaut]

I understand how it was a problem under the old 10 tier system, but I'm not sure I understand how it's a problem under the percentile system. Can someone ELI5 for me?

I'll try (can't do completely without maths though):
Let's assume a user ranks 10 films with the following scores: 91, 92, 93, 94, 95, 96 ,97, 98, 99, 100.
The film with the score of 91 is the Godfather.
==> In the (10) tiers system, the Godfather is in tier 1.
==> In the (100) percentiles system, the Godfather is in the 5% percentile.

So the granularity is higher (instead of jumping "tier-like" from 10% to 20%, you jump from 1% to 2% to 3% etc.), but in both cases the score 91 translates nowhere near tier 9 or the 91% percentile.

This is what the other users are talking about above.

In short: Regardless of how high (nominally) you rank your films, your 10% worst ranked films (formerly tier 1) are still in your bottom 10% percentile, that has not changed (what's new is the higher granularity, i.e. smaller steps in-between – 1% steps if you rank enough films).

I addressed this though by saying you have to rate at least 1 movie for each point in your distribution scale. In your example, a score of a 91 is (I presume) intended by that user to be a good score. If those are the only movies that user has rated, then that means the user hasn't rated at least 1 movie for each possible point on their scale.