From Tiers to Percentiles: Introducing a Big New Change to Criticker

[mpowell]

Not sure if anyone has mentioned this yet, but since the change to percentiles, the blue bar at the top of the page looks like this for me:

This should be fixed now!

[metalhank]

Hi there mpowell,
Not much of a talker here, with my thoughts after reading through most of this forum.
I think the site and tier or percentage system is great the way it is. I suggest you seriously think before changing the system for a handful of big talkers who I'm guessing do not represent the majority of users who may be (like myself) not talking much because they are happy with the product.
Cheers!

[mpowell]

Hi there mpowell,
Not much of a talker here, with my thoughts after reading through most of this forum.
I think the site and tier or percentage system is great the way it is. I suggest you seriously think before changing the system for a handful of big talkers who I'm guessing do not represent the majority of users who may be (like myself) not talking much because they are happy with the product.
Cheers!

Thanks for the vote of confidence! We'll definitely be considering every change carefully.

[zevel6]

I also have this same issue. The percentiles are not visible anywhere on the profile page. In fact, the only place the percentiles are visible for me is when clicking on the info box on a film under "All Your Rankings". Yesterday when I clicked into a specific film its percentile was visible but that seems to have disappeared today.

Alright, we've added percentiles to the scores found on the User Profile pages, much in the same way they are present when using Tiers. Also, we've made them much more visible on the Film Information page, when you've ranked a film.

Thanks! Personally I think this is a fantastic change and makes things much clearer.

[mpowell]

I agree that there does seem to be an issue with the TCI list after changing to percentiles.
For example, my top 12 TCIs currently show a TCI score of under 3 yet when i click the users the TCI shows 100x the number from the TCI list. It's as if the TCI list still calculates it per tier on some users.

This definitely sounds like a bug. It's possible there's some place in the code which is still using tiers instead of percentiles for TCIs, even for those accounts which have already switched. We'll take a look and see if we determine where it's happening.

This one should be fixed now! There was a utility that was configured incorrectly, which was causing these "tier-based" TCIs to re-enter the system.

[mpowell]

where cℓ is the count of all scores less than the score of interest, ƒi is the frequency of the score of interest, and N is the number of examinees in the sample.

In Criticker terms, that would be:

(# of films with a lower score + half the # of films with the score in question) / total # of films ranked
(multiplied by 100 and rounded down)

We have implemented this. The fact that we weren't accounting for other films of the same rating was a mistake. The percentiles now look better, from what we've seen.

Your percentiles will be recalculated the next time you rank any film.

Thanks so much for taking the time to write this out so clearly!

[Ramy]

Have you guys considered an extended color palette for the rankings to go along with this update? Maybe a different color for every 10% slice? or 20%?

0-19 (red)
20-39 (orange-yellow)
40-59 (yellowish-green)
60-79 (light green)
80-99 (dark green)

Would be nice.

[mpowell]

Have you guys considered an extended color palette for the rankings to go along with this update? Maybe a different color for every 10% slice? or 20%?

0-19 (red)
20-39 (orange-yellow)
40-59 (yellowish-green)
60-79 (light green)
80-99 (dark green)

Would be nice.

We have thought about this... and we totally agree. There's no reason we can't make more colors available, and this is high on our list. Originally we wanted to incorporate it with the percentile roll-out, but thought one mega-change at a time would be better.

[Ramy]

Awesome.

where cℓ is the count of all scores less than the score of interest, ƒi is the frequency of the score of interest, and N is the number of examinees in the sample.

In Criticker terms, that would be:

(# of films with a lower score + half the # of films with the score in question) / total # of films ranked
(multiplied by 100 and rounded down)

We have implemented this. The fact that we weren't accounting for other films of the same rating was a mistake. The percentiles now look better, from what we've seen.

Your percentiles will be recalculated the next time you rank any film.

Thanks so much for taking the time to write this out so clearly!

Was the average percentile for each movie updated as well?

[BadCosmonaut]

We have thought about this... and we totally agree. There's no reason we can't make more colors available, and this is high on our list. Originally we wanted to incorporate it with the percentile roll-out, but thought one mega-change at a time would be better.

Would it be possible to see the colors before the roll out so we can give feedback if needed? I remember seeing some pretty...unintuitive color recommendations on this forum a while back.