Since I am a subscribed member and have access to the feature that shows Prediction Accuracy,
I decided to play around with the "Films in common minimum". I now have rated 1100 movies, so I assumed I could begin to lower it.
Prediction Accuracy shows 4 parameters of which the three of them got better after the change (median difference shows 10 whatever I set it to):
The average difference (between my the probability scores and my actual ratings).
The amount of "spot on" (where the PSI matched my actual ratings exactly)
The amount of "way-of" PSI's (more than 34 from my actual rating).
It was at 25% and tried to reduce it to 20. All parameters showed better results. But I wanted to know what the optimal "Films in common minimum" was, so I kept lowering the value, waiting for a point where it would start to give worse predictions. But the predictions got better and better the more I reduced it. At last I was where it couldn't get lower: 3%. Here I get the most precise predictions.
During this journey, 25>20>15>10>8>6>5>3 I wondered if I had misunderstood something? People in this post have written about optimal percentages being 25 or 13, so can 3% really be my optimal "Films in common minimum"?
I do not understand tpbradbury's post, if I could I might see that I am way-off with the 3% (he seems to correct BadCosmonaut who uses a method similar to mine).
I can give you the numbers:
Average difference has come down from 11.91 to 9.47
Spot-on has risen from 154 to 162
Way-off has gone down from 36 to 11 (admittedly that is the only big change)
All this show that criticker predicts better now, right?
My best TCIs were in the 25.000 range before
Now they are in the 18.000 range.
(my former best TCI has now become my #78)
And yes, it's true that lots of them have not watched that many movies - my best TCI has only seen 80 movies - but I have seen 33 of them! And of these we totally agree on 13 movies! Only one we totally disagree on.
So am I right that 3% seems to be the best in my case? (maybe because my choice of movies is not so typical?)
Optimal 'Films In Common Minimum'
-
oplars
- Posts: 46
- 28 Ratings
- Your TCI: na
- Joined: Sun Jul 13, 2014 4:57 pm
-
geohawk
- Posts: 53
- 165 Ratings
- Your TCI: na
- Joined: Sat Apr 05, 2008 5:50 am
Re: Optimal 'Films In Common Minimum'
As many people like to say when looking at something like this: "It Depends!"
It appears that your experience is not uncommon - that as you lower the minimum % required, that you do create possibilities for users that more closely match your own ratings to enter your TCIs. But, because they tend to have fewer rated movies, the pool of films that can be used to calculate the prediction accuracy gets smaller. At a minimum, this means that there is greater variability in Prediction Accuracy. What tpbradbury is concerned is that this method of improving Prediction Accuracy may also be creating a bias. This is probably correct: as we increase the minimum, the pool of users must have seen a greater number of the movies that we have seen. So we sacrifice "accuracy" by increasing the breadth of movies for which we can get PSIs.
For instance: at 10% (my old norm), I can get enough ratings that I can get a prediction for which TV series from 1970 I am most probably will like ("Mary Tyler Moore Show"). If I drop down to 3%, Criticker is not able to calculate a PSI for any of the qualifying entries in its database.
I think there is also risk of bias on the other side: as you increase your minimum, your TCIs will more likely have to fall into a set of users with large amount of ratings. These include film buffs, aggregators ("users" that represent averages from other sites, such as imdb, Rotten Tomatoes, or Movielens), and others. The ones that are most similar to you will be favoured because they are most likely to be in the Top 10 ratings used to calculate your PSI. But from this pool, you might find that it is harder to find one that has a TCI that more closely reflects your own interests.
I often wish that sites like this would give us a mechanism to encourage "discovery" and take us to movies that we are less likely to consider just on our own. Criticker doesn't give us many tools for this, but this, and the "popularity" filter can do a bit of the work here.
It appears that your experience is not uncommon - that as you lower the minimum % required, that you do create possibilities for users that more closely match your own ratings to enter your TCIs. But, because they tend to have fewer rated movies, the pool of films that can be used to calculate the prediction accuracy gets smaller. At a minimum, this means that there is greater variability in Prediction Accuracy. What tpbradbury is concerned is that this method of improving Prediction Accuracy may also be creating a bias. This is probably correct: as we increase the minimum, the pool of users must have seen a greater number of the movies that we have seen. So we sacrifice "accuracy" by increasing the breadth of movies for which we can get PSIs.
For instance: at 10% (my old norm), I can get enough ratings that I can get a prediction for which TV series from 1970 I am most probably will like ("Mary Tyler Moore Show"). If I drop down to 3%, Criticker is not able to calculate a PSI for any of the qualifying entries in its database.
I think there is also risk of bias on the other side: as you increase your minimum, your TCIs will more likely have to fall into a set of users with large amount of ratings. These include film buffs, aggregators ("users" that represent averages from other sites, such as imdb, Rotten Tomatoes, or Movielens), and others. The ones that are most similar to you will be favoured because they are most likely to be in the Top 10 ratings used to calculate your PSI. But from this pool, you might find that it is harder to find one that has a TCI that more closely reflects your own interests.
I often wish that sites like this would give us a mechanism to encourage "discovery" and take us to movies that we are less likely to consider just on our own. Criticker doesn't give us many tools for this, but this, and the "popularity" filter can do a bit of the work here.