What's the deal with Cinemascore?

Cinemascore is a market research firm that surveys audience reactions to movies by polling moviegoers on opening night.  The lucky audience membership fill out ballots in the theatre and give the movie they have just watched a rating between A-plus and F, and Cinemascore aggregates the results into an average grade.

Cinemascore also collects other data, like demographic information, reasons for attending the movie and whether they would watch it again (or buy it) on DVD.  This additional information and the results of further data crunching are made available to subscribers, but is not publicly available.

Because Cinemascore does its polling on opening night, it is gathering data from the people who are the most motivated to see the movie.  The Cinemascore rating is frequently cited as an indicator of word of mouth - that difficult-to-measure concept where people choose to see (or not see) a movie based on what friends who have seen the movie have told them.

Collecting this information from the very people who are most psyched to see the movie implies that Cinemascore is surveying the most easily-pleased.  So to have good word of mouth, a movie should have a Cinemascore rating of at least A-minus. Anything B or lower is considered to be not that great, but it's generally expected that the better the Cinemascore rating, the better word of mouth will be, and the longer the movie's legs will be (so a B is still better than a D).

So, is it true?  Is a movie's Cinemascore rating a good indicator of how it will perform after opening weekend?

The following table summarises every Cinemascore rating I've been able to get my hands on - almost 400 movies covering most of the wide releases in the period 2010-2012 with a few from 2009 thrown in.

Cinemascore Rating	Number of Movies in Sample	Average Delist Multiplier	Average Delist/Adjust Multiplier
A+	12	3.08	1.17
A	48	2.67	0.99
A-	74	2.84	1.08
B+	72	2.66	1.01
B	72	2.54	0.96
B-	45	2.77	1.00
C+	36	2.28	0.86
C	11	2.17	0.83
C-	8	2.55	0.95
D+	4	1.94	0.79
D	1	2.20	0.81
D-	4	1.94	0.76
F	4	1.88	0.69

A quick look at the average delist multipliers seems to confirm the general logic - movies with higher Cinemascore ratings have longer legs than those which do not.

The table also introduces the delist/adjust multiplier, which this column will use as the metric for how well a movie does after opening weekend. The delist/adjust multiplier is obtained by dividing the moviestock's delist price on HSX by its adjust price.  A multiplier greater than 1 means that the moviestock delisted above its adjust price, while a multiplier below 1 means the moviestock was a post-adjust short.  The value of this multiplier over the normal delist multiplier is that it better compares moviestocks with irregular openings (like a mid-week opening, or a four day weekend) with regular weekends.

This is what the relationship between the Cinemascore rating and the delist/adjust multiplier looks like graphically.

This appears to confirm the general "better Cinemascore, better legs" rule of thumb.  There's a consistent downward trend, and R² is a very strong at 0.85.  (R² is the coefficient of determination, and it's a measure used in statistics to determine how well the trendline fits the data.  The closer R² is to 1, the better the fit.)  This suggests that the relationship between a Cinemascore rating and the average delist/adjust multiplier is very strong.

But looking closer, there are some odd blips.  Why would A-minus, and B-plus and even B-minus perform better over the long-term than movies with the treasured A rating?  Why would A movies on average delist below their adjust price?  There's obviously something happening underneath the averages, so we have to look at individual movies.

To do this, I replaced the Cinemascore ratings with a numeric grade from 0 to 10.  Because in the world of Cinemascore, A is supposed to be good, B satisfactory, C mediocre and anything less is terrible, I went with the following scale:

Cinemascore Rating	A+	A	A-	B+	B	B-	C+	C	C-	D+	D	D-	F
Numeric Cinemascore Rating	9	8	7	6	5	4	3	2	1	0	0	0	0

Here are the results:

The average "better Cinemascore, better legs" trend is still there, but the relationship between an individual movie's Cinemascore rating and its delist/adjust multiplier is weaker.  R² is just higher than 0.1, which suggests that any correlation between the two is just coincidence.

Look at the difference between the highest and the lowest dots for each Cinemascore rating.  There's a lot of variation.  The worst-performing movies for nearly every Cinemascore rating all had multipliers of around 0.7.   But plenty of movies with the same rating had multipliers of double that level.  The highest adjust/delist multiplier of the lot scored a B-minus.  Three A-plus movies even delisted lower than the multiplier.  If Cinemascore is supposed to be so reliable, how does it come up with results like this?

Let's look even closer at some individual movies that stray from the average.  Of the movies in the sample that achieved a Cinemascore rating of A, here are the 10 with the highest delist/adjust multipliers:

Movie	Release Date (wide)	Delist Multiplier	Delist/Adjust Multiplier
Hangover 1	05/06/2009	4.07	1.51
Lincoln	16/11/2012	4.55	1.50
How To Train Your Dragon	26/03/2010	3.62	1.34
Julie and Julia	07/08/2009	3.53	1.31
Secretariat	08/10/2010	3.52	1.30
Despicable Me	09/07/2010	3.37	1.25
Alvin & the Chipmunks 2	25/12/2009	3.47	1.24
Best Ever Marigold Hotel	25/05/2012	3.92	1.20
Hobbit 1 - Unexpected Journey	14/12/2012	3.12	1.15
Wreck-It Ralph	02/11/2012	3.04	1.13

Four of these ten movies are adult dramas - Lincoln, Julie and Julia, Secretariat and Best Ever Marigold Hotel.  Four more are animated children's movies - How to Train Your Dragon, Despicable Me, Wreck-It Raplh and (at least, partly) Alvin & the Chipmunks 2.  The Hangover was a breakout comedy with great word of mouth, and the Hobbit was a big movie released shortly before Christmas, so it's no real surprise to see either on the list.

Now, here are the 10 A-rated movies with the lowest adjust/delist multipliers.

Movie	Release Date (wide)	Delist Multiplier	Delist/Adjust Multiplier
Alex Cross	19/10/2012	2.16	0.80
Good Deeds	24/02/2012	2.10	0.78
Harry Potter 7 - Deathly Hallows 1	19/11/2010	2.06	0.76
Harry Potter 8 - Deathly Hallows 2	15/07/2011	2.03	0.75
Muppets	25/11/2011	1.41	0.74
Sparkle	17/08/2012	2.01	0.74
Glee 3D	12/08/2011	1.98	0.74
Why Did I Get Married 2	02/04/2010	1.96	0.73
Twilight 5 - Breaking Dawn 2	16/11/2012	1.90	0.71
For Colored Girls	05/11/2010	1.88	0.69

No fewer than four of these movies - Alex Cross, Good Deeds, Why Did I Get Married 2 and For Colored Girls - star Tyler Perry.  Three more -the seventh and eighth Harry Potters and the fifth Twilight - were the final chapters in hugely-hyped franchises with enthusiastic and relatively-young fan-bases.  For all these movies, the entire fan-base turned out on the first weekend and had a good time, but in general they didn't see it again and they didn't bring anyone else with them.  Glee is in a similar category - though obviously with a much smaller fanbase.  Muppets was released over Thanksgiving, when children's movies have terrible legs, and Sparkle drew people who wanted to see Whitney Houston.

Looking at the A-minus movies tells a similar story.  Movies released around Christmas - Avatar, Jack Reacher, Sherlock Holmes 2, Mission Impossible 4, Narnia 3, Invictus - were among the best performers, as were other movies targeted at adults (Lincoln Lawyer) and animated movies for children (Puss in Boots).  The bottom ten performances included movies that opened poorly like Won't Back Down and Extraordinary Measures, and more franchise movies like Underworld 4, Wimpy Kid 2, another Twilight (New Moon) and two more Tyler Perry movies (Madea's Big Happy Family and Witness Protection).

So, a high rating from Cinemascore can be an indicator of strong legs (and a delist above adjust) when it's combined with the right target audience - adults or children - or with the right release date - before Christmas.  Or preferably both.  That B-plus movie with better legs than any other movie in the survey?  The adult-targeted, Christmas-released True Grit.

A high Cinemascore rating is less valuable for franchise movies, movies aimed at teenagers and movies involving Tyler Perry.  The audience might turn out in droves on opening weekend and like the movie enough to rate it an A, but they won't convince anyone outside the target audience to actually go see it.

So what conclusions can we draw from this?

• Cinemascore ratings are a good predictor of how the average movie will do after opening weekend
• a lot of movies are not average, so Cinemascore ratings alone are not a good predictor of how an individual movie will do after opening weekend
• when predicting how an individual movie will do after opening weekend, you need to look at other factors, like who the target audience is, how hyped the audience was to see the movie on opening weekend, how big (or small) the opening weekend was and whether the movie has appeal beyond the target audience.