Aug
22
Photo Credit: Library of Congress, Prints & Photographs Division, National Child Labor Committee Collection, [reproduction number LC-DIG-nclc-05276]
When you read a summary of the results of a new study, your first temptation is often to consider implementing some of the approaches in your own training (assuming the study generated positive results). However, yielding to such temptations on a regular basis leads, at best, to an inconsistent approach as you jerk from one technique to another, or “hop on the latest bandwagon”. Therefore, a better approach is to step back and take some time to assess the potential value of any such study, and then consider whether it fits with your goals and realities.
First, wherever possible, try to look at a more direct summary of the study (or read the study itself) instead of relying on a reporter’s (or blogger’s) interpretation. This eliminates one major source of misunderstanding and bias. When you aren’t able to do so (because the study is expensive or inaccessible), look for alternative reviews or viewpoints on the study results, to make sure you are getting a more balanced perspective. Alex Hutchinson’s book, Which Comes First, Cardio or Weights? is a broadly helpful resource in looking at a wide range of physiological studies and you may find that he has discussed the specific topic (if not the study) you are evaluating.
You should also maintain a general skepticism towards “quick fixes”. Your own experience should illuminate the fact that making gains in your performance or losing weight takes time, and any study that claims to make such processes seem effortless is likely to have a lot of limitations.
Once you are able to find more details on the study, here are some of the key elements to look for in assessing its value:
Funding for the study – was the study sponsored by a corporation with a vested interest in the results? If so, even if the work was performed by a nominally independent researcher, it is likely that the results or interpretation were influenced by the desire to promote the sponsor’s product or to get more funding for future studies. Such studies must be viewed with an extra degree of skepticism as you review the details.
Flaws in the sampling plan – as reviewed in the previous post in this series, there are many ways in which the sampling can introduce bias, such as:
- Introducing survivor bias (i.e., the only results reported are from those who “survived” the training)
- Utilizing a small sample size, which makes statistically significant results difficult to achieve (and, conversely, makes the lack of such results even less indicative of the “no correlation” argument)
- Studying a non-representative population – the results for elite athletes may differ from “everyday Joe’s”, or men may see different results than women, etc.
If in any way the sample is not representative of your situation, proceed with caution at best, and think through reasons why you may see different results than the subjects of the study did. For example, if the study subjects were less fit, gains may come easier for them than they do for you, if you already log 50 miles per week (as in the case of the 10/20/30 workout discussion).
Changing multiple inputs at once – while this wasn’t covered specifically in the previous post, often research involving training plans includes more than one change made at the same time. While the benefit may be attributed to a specific change (like, the addition of the intervals in the 10-20-30 study), it may actually be related to another (more rest through reduced overall mileage, or weight loss from adding intervals, etc.). Thus it makes sense to look for other potential contributors to gains the study participants made, to see if perhaps the attribution to the variable studied is incorrect.
The placebo effect – while most good studies try to design this out, it can be impossible to do so at times. Again referring to the 10-20-30 study, the participants knew their training approach was different and may well have run a more aggressive 5K.
Once you have done as much review of the study details as you can manage, you may still find that there is merit in the results and want to add the new approaches to your training or diet. To the extent possible, you should use the same practices that a good researcher would use:
- Implement one change at a time
- Allow sufficient time for results to occur
- Gather sufficient data – one good race or workout doesn’t provide enough evidence that the change is working
- Take a critical eye to the results – are there other circumstances out of your control (like cooler weather) that may also be contributing to your improvements
In summary, any runner committed to performance gains would be well served by studying the sport and seeking improvements to training approaches. Reading about new study results can be a source of new ideas, but you must always consider the quality of the research and it’s applicability to your own situation before using such results to feed your training plan designs (or the plans that a coach would design for others). Otherwise, you risk being pulled all over the place in chasing the latest trends and thus losing the key to long-term success – consistency.
Pingback: Stupidity About Studies – How Writers Get It Wrong | Predawn Runner