The list of baseball projection systems seems to grow each year as more people think they can predict the future. It's important to know which of those projections are the best because you can win or lose your league depending on what projection system you use in fantasy baseball. This website gives you cheatsheets and you are given you the choose which projection system you want to use and that becomes a heavy decision. To help understand which projections are best, I'll look at the 2013 hitter projections and see which projections would have helped you most for your fantasy leagues last year.
The MethodIf you're using one projection system for your fantasy baseball draft analysis, you're actually not totally focused on whether a prediction of 40 HR, for instance, ends up being correct. Because, what if that entire projection system had predicted fifty guys to have 40+ HR? In that case, you may not have paid as much attention the guy that was accurately predicted from that group. What you want to be correct is how far above or below average they are projecting a player to be in each stat.
So, with that being said, I standardize all of the projections for each statistic so that I look at the predicted z-score in that stat for each player (z-score being how many standard deviations above/below the mean that projection was).
At that point, there are a variety of statistical methods to use to analyze whether the projections were accurate in projecting a player to be above or below average. I chose to use Mean Absolute Error (MAE) for this analysis. Compared to Root Mean Squared Error (RMSE), it doesn't penalize large mistakes quite as much which I find to be a good thing because you have the option to bench a player in fantasy baseball that is performing vastly different than expected. Mean Absolute Error, in this case, is the difference between the projected z-score and the actual z-score from the actual 2013 results averaged out among the whole pool of players we are comparing.
I only included players in this analysis that were showing up in drafts last pre-season and were shared among all of these analyzed projection systems. Though some players did well that weren't drafted, this is about who helped you the most on draft day. I also removed players who ended up not playing last season or played in an extremely limited capacity. This left 259 hitters in my pool of payers from last season.
The 2013 RankingsFor a full rundown of the competitors, see my post that introduces the baseball projections.
After performing the analysis as stated above, here are the rankings for how well each projection system performed:
Trends or Anomalies
CAIRO also had a very good performance across the board with the exception of the AVG and SB categories. Somehow the marriage of the Fangraphs Fans and Steamer projection did not go as well as expected for the hitting projections. We'll have to see if that trend continues for the pitching projections when we look at that next.
CAIRO deserves the silver medal for a strong but inconsistent performance. Despite being a bit all over the map, they still did a great job at predicting the overall WERTH value for each player and that's certainly a good thing.
In addition, the non-scientific method of just letting the Fans do the picking has shown some value, especially when it comes to projecting HR totals. It's a bit hit-or-miss beyond that but we have to start to acknowledge the power of the fan HR projection.