Why is tracking so awful? And why do traders still pay attention to it?

It's a constant lament on the HSX boards that tracking by organisations like RS and MTC affects moviestock prices. Why should this happen when tracking numbers are often wildly inaccurate?

Let's go back a step first. RS and MTC are two predictors of movie box office.  RS is ReelSource, which is a company that sells box office tracking data to whoever wants to buy it.  MTC stands for “Major Theatre Chain”, a certain theater chain that we won't name here.

Both RS and MTC develop their predictions through surveying moviegoers about which movies coming out soon they have heard of, and which ones they want to see. The data that comes out of these surveys is known as 'tracking', and they use the tracking data to make predictions about a movie's opening weekend.

RS and MTC's predictions for the next weekend are frequently posted on the HSX boards on the Sunday and Monday before opening.  Both (and particularly RS) often come under fire for being inaccurate.  As of writing, for 2013 releases RS' accuracy is at about 62% and MTC's is about 72%.  There's no clear pattern to their inaccuracy either.  Some weeks, one or both will massively under-estimate a movie's opening weekend. Other weekends, one or both will over-estimate it. So, the argument goes, why bother with tracking?

Let's look at some of the reasons why tracking often doesn't come up with the goods:

Tracking comes out at the beginning of the week.

We normally see tracking for movies opening on Friday come out on Sunday or Monday.  So tracking misses the whole big week-before-release marketing push.  This is when marketing and publicity is at its heaviest, particularly for advertising aimed at general audiences, like television commercials. So the data from tracking doesn't even include the final marketing push. Strange as it may seem, many casual movie-goers don't plan to see a movie weeks in advance, they decide based on what looks good that week.  So tracking data doesn't factor in a chunk of the audience that hasn't decided whether to see the movie yet.

Tracking focusses on frequent movie-goers

The people most likely to be surveyed are frequent movie-goers - people who go see more than one movie a month. According to the MPAA's Theatrical Marketing Statistics, frequent movie-goers make up only 13% of the population in North America, but they buy 57% of all the movie tickets.  They are also more likely to be aged between 12 and 40, with people aged 12-24 being especially over-represented.

Demographics aside, frequent movie-goers are far more likely than the general audience to know what movies are coming out and whether or not they want to see them.  But interest among frequent movie-goers (or lack of it) doesn't always translate to whether the general audience - or specific demographic segments - will turn up to see a movie.

There's a long list of movies where tracking has not factored in demand from one or more specific demographic segments.  Probably the best known examples are the Passion of the Christ, Tyler Perry movies, and the Alvin and the Chipmunks movies. But these aren't the only ones - examples from 2012 and 2013 include Fast and Furious 6, Ted, Magic Mike, The Devil Inside, The Lorax, The Purge and The Great Gatsby.

And there are more than a few cases where high interest and hype among frequent movie-goers has not lured in the general audience to see the movie.  Snakes on a Plane is probably the classic example of hype being way ahead of audience interest, but the movies that are most consistently over-estimated by tracking are sequels.  Compared with MTC tracking, seven of the ten biggest under-performers since 2012 were sequels (and an eighth was a re-release).  So far in 2013, the three greatest under-performers are Hangover 3, Star Trek 2 and Die Hard 5. This may be because sequels have high audience awareness, and this can get confused with actual interest. For example, people may tell the survey that they want to see a new Die Hard movie (because who wouldn't, Die Hard movies are awesome) but they later decide against it after they see how bad the movie looks.

Coming up with an opening weekend prediction isn't the point of tracking

Tracking is really about measuring whether the marketing and publicity for a movie is succeeding.  It asks respondents if they are aware of the movie, if they are aware of the movie if they are given a hint (such as the movie's name), whether they want to see the movie and whether it would be their first choice.  This is the information the movie studios and distributors are paying for, to help with business decisions such as "should I spend more on marketing this film?", "should I put this film in the big theatre or the small one?" and "should I clean out my office before the weekend?".

Of course, a lot of people don't like loads of data, and prefer simple one-word answers they understand, so predictors take the data from tracking and try to translate it into a simple number for the weekend.  But this is probably more art than science, and maybe they don't put a lot of work into it. But that's the number we see.

Unfortunately, we don't get to see the data, or interpret it for ourselves.  There have been a few sites, such as the now-defunct It's On The Grid column, that used to make some information on tracking, broken down by age groups and gender, publicly available.  But sadly there aren't any good data sources like that any more, unless you pay for them.

Those are the basic reasons why tracking numbers are not always accurate.  So, if tracking is at best frequently inaccurate and not great at telling the whole story, who are traders on HSX so interested in it?

There are a bunch of reasons for this.  While it's not the optimal tool to work with, it's the best one we have.  RS and MTC are the earliest indicator we have of whether a movie is going to perform on opening weekend.  MTC is about as accurate as other prediction sites, whether they use tracking or some other data source. The real value of tracking is not in its numerical accuracy, but rather in whether it points us in the right direction of the eventual adjust - tracking can be wrong, but it doesn't lie. It doesn't need to be 100% accurate to tell you that a movie is going to over- or under-perform. And aside from all of these factors, tracking numbers are still going to affect the market, so it's better to know what the number is and play accordingly than not know and lose your shirt.

It's also important to remember that when it comes to movies and opening weekend, nobody knows anything.  There is always the potential for movies to surprise and to disappoint, and you never really know what is going to happen before it actually happens.  Pointing out that tracking is often inaccurate is an easy answer that provides no real solution to better predicting the box office.