New user: A question about percentiles

[MarkBeronte]

Hi all, I'm new here and am so far am enjoying playing around with the site but I have a question about percentiles. But before I begin, I apologize if this is a common question, and if so just a link to the relevant thread would be much appreciated. I was unable to find one in my search.

Now I know percentiles are used to try to make comparisons between the different rating systems people use more valid, but isn't this system kind of sensitive to the number of movies people rate at the different levels of their system. For instance, considering how many bad movies there are, I usually do quite a bit of research into a movie before finding and watching it, and generally I end up liking most of the movies I watch, which tilts my scale quite a bit to the positive. Doesn't this mean that some highly rated movies will have a very low percentile simply because they are so few very lowly rated movies below them and doesn't this defeat the purpose of percentiles? Is there an easy fix? Thanks for info

Cheers

[BadCosmonaut]

This is discussed sometimes. Specifically, they recently introduced a new change, and the issue still exists under the new system (although I originally thought it didn't). We might be able to brainstorm possible solutions.

As far as an easy fix, I don't think there is one. One thing you can do that might help alleviate the issue a little (depending on your scores) is to rate as many movies as you can that you've seen.

[MarkBeronte]

The mistake was probably allowing people to use any arbitrary rating system they like, and then thinking you could normalize them all with clever math. A more reliable way to have some choices might be to have several different rating systems to choose from; 0-100, 0-10, 0-5 and you could then easily normalize between them. You could probably even still allow people to choose what values correspond to 5 or morel tiers; Greatest, Great Good, Ok etc, and use these to find people with similar tastes. I think this would be a much more reliable system. Anyway thanks for the reply, but sorry to hear it is actually an issue that just has to be tolerated. It seems a significant mar on what is otherwise a great idea for a site.

[BadCosmonaut]

I agree that establishing baseline values then letting people map their rating scale to those baseline values sounds like a good solution. Unfortunately, because there are so many abandoned users on the site who won't be around to map their scale to the baselines, I'm not sure the site could implement something like this without losing all those users from the TCI user pool. That said, if this would fix the issue going forward, I'd be all for it, and it would be better to do this sooner rather than later.

An idea I had to implement this would be to have 10 baseline values:

10: Masterpiece
9: Masterful/exceptional
8: Excellent
7: Great
6: Good
5: Decent
4: Not so hot
3: Bad
2: Notably bad
1: Terrible

(The exact wording used for each tier above could use some work. These are just my personal quips.)

Then let users pick any scale they like between 2 and 100 points.

For users with a 9 point scale or less, they can map each of their points to one of the baseline values above. For users with a 10 point scale, theirs would just be mapped exactly to the baseline values above. For users with a larger than 10 point scale, they can map which 10 points of their scale map to the 10 baseline values above. Then the criticker system can fill in the blanks for them between each baseline value when computing TCI/PSI.

I know the designers of the site want to allow people to use any scale they want up to 100 points. This system would allow users to do so. It will also work with users who skip numbers. Like if a user's scale is: 100, 99, 98, 97, 96, 95, 90, 85, 80, 75, 70, 60, 50, 40, 30, 20, 15, 10, 05, 04, 03, 02, 01. That's really just a 23 point scale, regardless of how spread out each point is. As long as that user maps the 10 baseline values above, the system should be able to guess the rest.

[MarkBeronte]

I think matching similar tastes using tiers like you suggest would be the best way to go. The only problem I see with it is the situation where two movies may be separated by only few points, but lie in different tiers. These movies should probably be a match but because they lie in different tiers they might not. Personally I wouldn't mind solving this problem by including movies 1 tier above or 1 tier below as a match to include such cases. After all, most users might put any particular movie 1 tier higher or lower anyway, based on their mood when they saw it, or any number of other factors, except perhaps a persons personal favorites.

[heythereyou]

The real issue is sample bias and the only way to fix sample bias is by having unbiased samples. The sample estimates should converge on the populations true values given that the sample is large enough and unbiased. In plain English: if additional movies rated were random then after awhile the biased sample should be overcome.

I think there could be a couple of solutions. First, there should be a mediocre/average category along with a bad category for recommendations. People could go here and rate films and for those who don’t watch 50 films a year this would be a practical fix. It would not be truly unbiased but it wouldn’t be hard to encourage normalization of scores that was not biased.

For those who watch 50 films a year this solution also work quite well. Possibly even better. I would also consider changing how films are rated in the first stage of use. It would be much better to ask for 10 good films, 10 average films and 10 terrible films. Then establish ranges you could compare the future ratings to and maybe offset the bias. That might be a little more complex but I am think it would be possible. Furthermore, you could have a confidence score in each estimated rating and encourage unbiased sampling by showing the confidence score improving by rating more average and bad films.

Another solution might be to introduce a ‘not interested’ score. This would essentially say the user believed the estimated score should be below registering worth watching.


As for me, I have a very specific scoring system. I have gone out of my way to create a lit of films I think are average or bad and will rate them until my scores normal distribution more accurately reflects my personal tastes.

[livelove]

considering how many bad movies there are, I usually do quite a bit of research into a movie before finding and watching it, and generally I end up liking most of the movies I watch, which tilts my scale quite a bit to the positive. Doesn't this mean that some highly rated movies will have a very low percentile simply because they are so few very lowly rated movies below them and doesn't this defeat the purpose of percentiles?

This is discussed sometimes. Specifically, they recently introduced a new change, and the issue still exists under the new system

sorry to hear it is actually an issue that just has to be tolerated. It seems a significant mar on what is otherwise a great idea for a site.

@Mark:
Yes, this is a regularly surfacing topic of discussion (which the site owners unfortunately have not yet commented on). See the quoted thread.

Criticker algorithm:
advantage: severe and lenient voters can be compared to each other
disadvantage: since we avoid bad movies, our lowest ranked movies (even with excellent scores nominally) are treated by Criticker as "terrible"

comparing scores at face value:
advantage: the scores you attribute as a user are respected
disadvantage: severe and lenient voters cannot be compared to each other

The upshot seems to be:
The ideal system would use an algorithm that respects users votes while authentically translating different voting systems to one another. I don't have an idea yet of how to accomplish this.

We might be able to brainstorm possible solutions.

Please, go ahead.

The mistake was probably allowing people to use any arbitrary rating system they like, and then thinking you could normalize them all with clever math. A more reliable way to have some choices might be to have several different rating systems to choose from; 0-100, 0-10, 0-5 and you could then easily normalize between them.

no, this has nothing to do with the problem at hand. For instance, a ranking scale of 0-5 can be easily translated to 0-100 by multiplying by 20. Different rating scales are not the problem. The problem we are talking about would still occur in the very same manner, if everyone used the 0-100 scale.

I agree that establishing baseline values then letting people map their rating scale to those baseline values sounds like a good solution. Unfortunately, because there are so many abandoned users on the site who won't be around to map their scale to the baselines, I'm not sure the site could implement something like this without losing all those users from the TCI user pool. That said, if this would fix the issue going forward, I'd be all for it, and it would be better to do this sooner rather than later. An idea I had to implement this would be to have 10 baseline values:

10: Masterpiece
9: Masterful/exceptional
8: Excellent
7: Great
6: Good
5: Decent
4: Not so hot
3: Bad
2: Notably bad
1: Terrible
(The exact wording used for each tier above could use some work. These are just my personal quips.)

Then let users pick any scale they like between 2 and 100 points.

Sorry, but I don't think this is a working solution. Not just due to the inactive users you mentioned. But even more importantly because you cannot force users to use the meaning of your scores if they have a different meaning to them. For example, there are users who rate a film 65 out of 100 and still say it's a terrible film. And this won't ever change, even if you tell them to use your quips. You just can't dictate that, if it's not the way other users perceive those numbers.

[clay4gem]

I can't see how to post anything other than replies? Is there something that needs to happen first? (Minimum no. of replies or something?) Sorry for posting it here - but couldn't see where else to put it.

[BadCosmonaut]

I can't see how to post anything other than replies? Is there something that needs to happen first? (Minimum no. of replies or something?) Sorry for posting it here - but couldn't see where else to put it.

They just updated the forums, so they may be having issues. You may want to email support at support@criticker.com

[clay4gem]

Thanks!