Quality and design of recommendations

[Alsweider]

When I look at my PSI list or the recommendations on the homepage, I find that the top films (PSI 100 to 80) have a lot of descriptions that I find of little interest, even if I filter them according to my preferences (genre, year, release type, popularity). If I force myself to watch a film with a high PSI whose description I'm not interested in, it often turns out that my assessment based on the description or trailer was more accurate than the PSI. Maybe this will improve over time when I have more ratings. My current data:

• 646 film ratings
  Films in Common Minimum: 10%

Predictive Accuracy:

• Average Difference: 12.34
• Median Difference: 8
• # Spot-On (<= 0.5 Diff): 39
• # Close (<= 5 Diff): 141
• # Way-Off (> 40 Diff): 25

The predictive accuracy is of course influenced by the fact that I pre-select films with a high PSI, but usually only watch them if the description also seems interesting. If I were to rely solely on the PSI, the deviation would certainly be greater.

When I browse through the current selection on streaming services, after a short time I find films that seem interesting to me and that often get a rating by me that roughly corresponds to my initial expectations, even if the streaming sites have little or no knowledge of my taste.

Streaming services present a relatively small selection in a section of the screen. Too much choice leads to overload and you can no longer decide. Criticker takes this aspect into account on the homepage. However, 8 recommendations seem a bit tight to me, so I often reload the homepage or go straight to the full PSI list. Clicking on "Show Another Film" is inconvenient because only one new film is suggested at a time and the link appears in a different position each time, depending on the length of the film description.
Streaming sites are often set up in such a way that you can use the cursor to jump through a list of covers, with a trailer or more detailed information about the content immediately displayed. This gives you a general impression very quickly. In the PSI list, you first have to go to the film page to find the cover, description or trailer. Perhaps it would speed up the process of finding accurate recommendations if there was a display mode that was more in line with the design of streaming services.

[oplars]

Sadly, the situation has not improved over time (with more ratings) for me.
I'm experiencing the same thing as you, but to an even greater extent, if you look at the numbers involved:

Average Difference: 10.08
Median Difference: 10
# Spot-On (<= 0.5 Diff): 185
# Close (<= 5 Diff): 326
# Way-Off (> 24 Diff): 103

I guess I continue taking PSI into account because according to my logic it SHOULD work. When (irl-) friends I share taste in movies with recommends a film to me, knowing my preferences, I always pay close attention and give it a shot, because I trust their judgment.
Criticker found my 10 closest taste matches out of millions - this MUST mean something significant.
I even scrutinize the top TCIs to determine if their ratings are consistently aligned or dispersed across red, yellow, and green, as a wide range of ratings, in my opinion, raises concerns about the reliability of PSI (if the movie have many ratings, I often take 20 or more TCIs into concideration).
Not only that, I also take a look at the stats, and if I can see that the better TCI, the better PSI, I consider it a strong indication that I will value that particular movie.
Yesterday I found 4 movies which all qualified using these criterias, one of them where even dark green by all top TCIs. - And I was bored to death by all of them (and yes, I only saw 15-30 minuttes of each - but I have learned over time to trust my impression more than the PSI, since as you can see, the PCI is only close 25 % of the time.
However, I comfort myself by recognizing that this approach delivers a more reliable guess compared to relying solely on IMBD.

User: IMDB
Rated 56881 titles
Your TCI: 34.75556986

- and my best TCIs are in the range: 22.000 - 23.500.

[Alsweider]

@oplars Thanks for the comparison data! At least your average difference is 2.26 points better than mine.
I think PSI is definitely a useful factor for generating recommendations. But to actually hit the mark, making sure the suggested films are entertaining, there's more to it. The system needs to make it easy to quickly spot the interesting films and skip the boring ones.
One thing the algorithm doesn't seem to consider (at least as far as I know) is how someone's taste evolves over time. Like, I might have been into Westerns a while back, but these days I'm more into thrillers. And yet, I get grouped with people who've gone the opposite way.
It could also help if the recommendations factored in more specific user details. Sure, this would be tricky, expensive, and raise privacy concerns, but it might really make a difference: information like age, background, origin, where you live, sex, the current weather, mood, or other biological data. For example, tracking skin conductance and heart rate while watching a film (like they do with lie detectors) could show how emotionally engaged you are. With that kind of info, you could add another layer to the algorithm that builds on top of PSI to create more refined recommendations.

[oplars]

I guess you know that you can filter genres in Full Database. And you can set all the other parameters, such as sort by PSI, range or years. So some things you suggest are already there. At some other sites you can choose mood. Some will show how complex, or how "dark" the movie is. They are not more precise than Criticker, usually quite contrary. But these functions might be something that could be taken into account at Criticker. But I guess it would only help a little.
I guess the problem is this: Some have a complex (or unusual) taste where many criteria have to me met - or which sticks in many directions.
I had a friend at high school, who said that his criteria for a girlfriend was blond hair and blue eyes. If he had a similar simple criteria for movies, the psi would be spot on every time. But again, one should think that with many rated movies the algorithm would work for everybody , even those with a very unique taste.

I cant grasp how my average difference can be smaller than yours. It doesn't make sense considering all the other data. A bug?

Your idea with psycho physiological measurement is pretty advanced

[Alsweider]

But these functions might be something that could be taken into account at Criticker. But I guess it would only help a little.

Usually, these filters are just plain filters and aren't used in the algorithm for personalised recommendations. Mood, for example, could be used by letting you set it on one or more scales when rating a film. The algorithm could then learn which moods make you rate certain films and genres more positively, or what kind of weather makes you enjoy watching comedies.

I guess the problem is this: Some have a complex (or unusual) taste where many criteria have to me met - or which sticks in many directions.

Yeah, absolutely perfect predictions probably won’t happen. One reason could be that as the quality of recommendations improves, your expectations also go up, and your taste is constantly changing.

I had a friend at high school, who said that his criteria for a girlfriend was blond hair and blue eyes. If he had a similar simple criteria for movies, the psi would be spot on every time.

Funny story, probably a question for evolutionary biologists why men often have such simple preferences in this regard, but not in others.

I cant grasp how my average difference can be smaller than yours. It doesn't make sense considering all the other data. A bug?

The lower Average Difference means that your predictions (PSI) were, on average, a bit closer to your actual ratings than mine were. Your average is pretty much in line with your median, which means the overall deviation tends to stay around 10%, without much influence from outliers. In my case, the average deviation is 4.34% higher than my median. That might have been caused by a few unusual outliers skewing the stats a bit, as they deviated more significantly from the usual prediction accuracy I get. For example, my difference could be 8 ninety-nine times, but on the 100th time, it’s 90. That would significantly change the average but wouldn’t have any impact on the median.

Your idea with psycho physiological measurement is pretty advanced

That’s true, it’s more of a theoretical thought experiment, but if the technical means were there, I’d definitely give it a try. The more data the algorithm has about the relationship between a film and its viewer, the more precise the predictions can be.

I’m currently reading the book "Why You Like It: The Science & Culture of Musical Taste" by Nolan Gasser. I think a lot of the insights from it could be applied here one-to-one, but for a small team with limited resources, that would obviously be challenging.

[Wait]

There seems to be a bias site-wise toward old drama movies. Whenever I select a movie that aligns more closely with my interests and rate it afterwards, I find that PSI was fairly accurate (both for movies I liked and disliked). However, my top PSI recommendations are filled with very old, often very long movies about war or other dramatic subjects. This is likely because people tend to rate those topics highly - rarely does one see a movie with a "heavy" subject rated poorly. This ruins the recommendations for me, as these subjects simply don't interest me, and when everyone on the site has those movies in their top percentiles, my recommendations become flooded with them. While you can filter by year or genre, you can only filter for positive values - many of these war/drama movies have multiple genre tags, so if I filter for suspense/thriller, I still get a bunch of war movies in my recommendations. Unfortunately, I haven't found a solution to this problem.

[Alsweider]

There seems to be a bias site-wise toward old drama movies. Whenever I select a movie that aligns more closely with my interests and rate it afterwards, I find that PSI was fairly accurate (both for movies I liked and disliked). However, my top PSI recommendations are filled with very old, often very long movies about war or other dramatic subjects. This is likely because people tend to rate those topics highly - rarely does one see a movie with a "heavy" subject rated poorly.

If the algorithm works as described, one can only assume that users with your current rating pattern tend to favour these old dramatic films. If your preferences differ, the algorithm needs to be taught somehow—either through more ratings from you or through additional users improving the accuracy of the recommendations. It is also possible to ignore uninteresting films. However, as a quick interim solution, better filtering options (with exclusion criteria etc.) would indeed be a very good idea.

[geohawk]

Possibly. One thing you can do is look and see which users are being used to generate the prediction score. If they are pretty low ranks (like not in your top 50), it is possible that the movie is being recommended because of general popularity.

I often like to create filters for subsets of movies or tv shows, the ranked list may be more useful in that context. Which comedy from the last five years am I most likely to appreciate, things like that.

One thing that Criticker will do is it will show you what your closest user matches are amongst those that have used the system recently. Unfortunately, it won't take that into account when making recommendations.

[Alsweider]

One thing you can do is look and see which users are being used to generate the prediction score. If they are pretty low ranks (like not in your top 50), it is possible that the movie is being recommended because of general popularity.

Interesting observation. I took a look at some of my top recommendations, and for many films, none of my top 100 TCIs are present. However, for Seven Samurai, I found ten at once.