“The game is afoot!” [Sherlock Holmes in “The Adventure of Abbey Grange”.]
Recently, Stacker published their list of the 100 “best” mystery movies, which they came up with by combining scores from the Internet Movie Database (IMDB) and Metacritic. It’s definitely fun to look through and we’ll probably try watching some of the ones we haven’t seen. While I don’t really understand their scores (see the notes at the end of this post if you’re curious), it’s meant to be fun, so let’s have some!
One thing we can do is compare IMDB and Metacritic scores. (I’ll multiply IMDB scores by 10 so that they are comparable — they are on a scale of 0-10 while Metacritic is on a scale of 0-100.)
What we can see is that Metacritic is a lot more positive than IMDB: the dots are mostly above the gray line, which indicates the same score for both IMDB and Metacritic. The movie “The Souvenir” is particularly striking with its relatively low score on IMDB but relatively high score on Metacritic.
Those are the scores, but we’re also interested in the ranks. It could be that while the scores are different between IMDB and Metacritic, the ranks are roughly the same. However, that it is not the case. There is, in fact, not a lot of agreement about the relative merits of mystery movies across the two sites. In this chart, as above, the further a dot is from the diagonal gray line, the more different the rankings.
Since Stacker is combining these two disparate sources, what happens with those differences? Here are the top 10 movies according to the three sites, so you can see how they compare with the rankings on the others. There’s clearly something odd going on with Stacker, since it lists “Rear Window” as number 1, which neither IMDB nor Metacritic do. Hmmm… In any case, you can decide which site is closest to your own preferences.
Top 10 Mystery Movies on Stacker
| Stacker | IMDB | Metacritic | Title | Director(s) |
|---|---|---|---|---|
| 1 | 10 | 4 | Rear Window | Alfred Hitchcock |
| 2 | 15 | 3 | Vertigo | Alfred Hitchcock |
| 3 | 14 | 2 | Citizen Kane | Orson Welles |
| 4 | 4 | 8 | Psycho | Alfred Hitchcock |
| 5 | 2 | 11 | Spirited Away | Hayao Miyazaki |
| 6 | 28 | 1 | Three Colors: Red | Krzysztof Kieslowski |
| 7 | 13 | 5 | North by Northwest | Alfred Hitchcock |
| 8 | 20 | 6 | Rashomon | Akira Kurosawa |
| 9 | 27 | 7 | The Third Man | Carol Reed |
| 10 | 12 | 13 | Double Indemnity | Billy Wilder |
Top 10 Mystery Movies on IMDB
| Stacker | IMDB | Metacritic | Title | Director(s) |
|---|---|---|---|---|
| 18 | 1 | 59 | Harakiri | Masaki Kobayashi |
| 5 | 2 | 11 | Spirited Away | Hayao Miyazaki |
| 56 | 3 | 98 | The Usual Suspects | Bryan Singer |
| 4 | 4 | 8 | Psycho | Alfred Hitchcock |
| 62 | 5 | 99 | Oldboy | Park Chan-wook |
| 40 | 6 | 92 | Memento | Christopher Nolan |
| 37 | 7 | 89 | Coco | Lee Unkrich, Adrian Molina |
| 16 | 8 | 29 | The Lives of Others | Florian Henckel von Donnersmarck |
| 11 | 9 | 14 | Apocalypse Now | Francis Ford Coppola |
| 1 | 10 | 4 | Rear Window | Alfred Hitchcock |
Top 10 Mystery Movies on Metacritic
| Stacker | IMDB | Metacritic | Title | Director(s) |
|---|---|---|---|---|
| 6 | 28 | 1 | Three Colors: Red | Krzysztof Kieslowski |
| 3 | 14 | 2 | Citizen Kane | Orson Welles |
| 2 | 15 | 3 | Vertigo | Alfred Hitchcock |
| 1 | 10 | 4 | Rear Window | Alfred Hitchcock |
| 7 | 13 | 5 | North by Northwest | Alfred Hitchcock |
| 8 | 20 | 6 | Rashomon | Akira Kurosawa |
| 9 | 27 | 7 | The Third Man | Carol Reed |
| 4 | 4 | 8 | Psycho | Alfred Hitchcock |
| 21 | 96 | 9 | Don’t Look Now | Nicolas Roeg |
| 12 | 38 | 10 | The Maltese Falcon | John Huston |
Sticking just with Stacker, let’s take a look at the rank versus year — are older movies better than newer ones?
I’ve divided the chart into 4 quadrants. The patterns are pretty clear: the “better” movies are split pretty evenly between the top two halves, while the “lesser” movies are predominantly after 1979 (bottom right). On the other hand, the earlier period (left half) skews towards “better” movies, while the later period (right half) skews towards “lesser” movies. For those of you who like numbers, here are those same quadrants with the number of movies in each, along with the average ranks and average years for each quadrant.
| Ranks\Years | 1935-1977 | 1978-2020 |
|---|---|---|
| 1-50 | N: 23 Ave Year: 1956 Ave Rank: 18 |
N: 27 Ave Year: 2003 Ave Rank: 32.2 |
| 51-100 | N: 6 Ave Year: 1961 Ave Rank: 72.7 |
N: 44 Ave Year: 2008 Ave Rank: 75.9 |
Turning quickly to the movies themselves, we can compare the duration in minutes (runtime) to the year the movie was released. At least in these top 100, the later movies tend to be longer. The one extreme example is “Mysteries of Lisbon” which was originally a TV mini-series rather than a theater released movie.
Finally, we can also see which directors show up the most. The number one mystery director is no mystery… Alfred Hitchcock.
| Director | Number of movies |
|---|---|
| Alfred Hitchcock | 8 |
| Denis Villeneuve | 4 |
| Asghar Farhadi | 3 |
| Krzysztof Kieslowski | 3 |
| Bi Gan | 2 |
| Carol Reed | 2 |
| David Fincher | 2 |
| Francis Ford Coppola | 2 |
| Michael Haneke | 2 |
| Robert Altman | 2 |
Technical notes
I accessed the Stacker site, and a republished version of the story on SFGate on October 5, 2020. I collected the data (in the browser using javascript) from the latter version, filling in a missing entry (#7, “North by Northwest”) as well as interpolating a missing Stacker score of 93 for (#17, “LA Confidential”). All of the analyses and visualizations were done in R, using ggplot2.
Although Stacker says that they use “a weighted index split evenly between IMDb and Metacritic scores” I was not able to recreate their scores. In particular, their statement suggests that the Stacker Score is a linear combination of the IMDB and Metacritic scores:
Stacker = a * IMDB + b * Metacritic + c
However, I could not find any suitable values for (a, b, c) — the Stacker scores seem to be inconsistent with the linear combination model, so I don’t really know how they computed their scores. Another oddity is that even though the maximum Stacker score is 100 (as on Metacritic), over 3/4 (78) of the Stacker scores are higher than both of the other scores.
I recomputed a simple average of the IMDB (multiplied by 10 for comparability) and the Metacritic scores (i.e. a=5, b=0.5, c=0), and I found that those scores led to different rankings for about 1/3 of the movies (32). However, those differences were usually just 1 rank, and occasionally 2, so not a huge difference, and certainly not one that interferes with the fun here.