Specific Threats to Validity
ShapeTracker Intervention Stakeholders
- Commissioner Middleton County (MC)
- Middleton County (MC) Health Department (HD) ShapeTracker (ST) team
- MC HD Information Technology staff
- MC HD marketing/communications staff
- Other MC HD staff
- CDC
- Group that developed ST for CDC
- MC community members
- Wellness advocates in MC
- Obesity prevention programs in MC
- Healthy eating related resources in MC
- Exercise resources in MC
- Smartphone providers in MC
- California communities that implemented ST
- Smartphone companies
- MC government
Logic Model
| Inputs | Activities | Outputs | Short-term Outcomes | Intermediate Outcomes | Long-term Outcomes |
|---|---|---|---|---|---|
| ShapeTracker App | Mobilize local resources and customize ST for MC | # of times ST app downloaded in MC | Decrease perceived barriers to healthy eating and physical activity | Increase healthy eating in MC | Reduce obesity in MC |
| Funding from HD and CDC | Market ST in MC | # of MC residents actively using ST | Increase self-monitoring of healthy eating and physical activity | Increase physical activity in MC | Reduce obesity related health problems in MC |
| Team of experienced staff | Keep app updated and maintain content | Amount of use for different ST functions | Increase self-efficacy for healthy eating and physical activity | ||
| High smart phone use in MC community | Provide technical assistance to users | User satisfaction with ST | Increase social supprt for healthy eating and physical activity | ||
| Marketing & communication resources in HD | # of friends referred to ST | ||||
| IT resources in HD | # of MC residents actively using ST long-term | ||||
| Local obesity prevention resources in MC |
ShapeTracker Intervention Evaluation Questions
Process Questions
- Was a broad range of local resources successfully engaged in the project?
- Was ST successfully disseminated to a broad segment of MC community?
- What proportion of MC residents have:
- downloaded the app?
- became active users of the app?
- became active long-term users of the app?
- For those who stopped using ST, why did they discontinue?
- How much are ST users using different functions in ST?
- How satisfied are MC residents with the ST app and its specific features?
Outcome Questions
- Does use of ST increase:
- self-monitoring of healthy eating and physical activity?
- self-efficacy for healthy eating and physical activity?
- social support for healthy eating and physical activity?
- Does ST decrease perceived barriers to healthy eating and physical activity?
- Does use of ST increase healthy eating and physical activity among county residents?
- Does use of ST lead to lower BMI among those who are obese or overweight?
- Does use of ST help those with normal BMI prevent overweight?
The evaluation questions we have chosen to address in our evaluation plan.
Now let’s review some of the potential threats to validity for our ShapeTracker intervention.
Threat to Validity:
Internal Validity: Selection
ShapeTracker Example:
“Our evaluation design is subject to selection bias. There is no randomized control group. Differences between the Middleton and Upperton county samples, other than exposure to ShapeTracker, may explain observed differences in outcomes.”
Threat to Validity:
Internal Validity: History
ShapeTracker Example:
“It is possible that outside events, such as another local obesity prevention program launched during the study period, can affect the outcomes and bias the results.”
Threat to Validity:
External Validity
ShapeTracker Example:
“The findings of this evaluation should not be generalized to outside of Middleton county without caution. The evaluation has some generalizability to communities with similar demographic composition and rates of smartphone use.”
Threat to Validity:
Statistical Conclusion Validity: Low Statistical Power
ShapeTracker Example:
“A power analysis was conducted to verify that the sample sizes of the representative household surveys are adequate for this evaluation.”
Threat to Validity:
Statistical Conclusion Validity: Unreliability of Treatment Implementation
ShapeTracker Example:
“We acknowledge that our evaluation design is dependent on adequate uptake of the ShapeTracker app among Middleton county residents. If the dose of the intervention in the community is inadequate, we may not be able to detect impact.”
Threat to Validity:
Construct Validity: Inadequate Explication of Constructs
ShapeTracker Example:
“We have identified previously validated measures for each of our three outcomes.”
Question 1:
Generalizability to other groups or settings refers to:
The correct answer is B:
Generalizability of findings to groups or setting outside of the evaluation study is called external validity.
Question 2:
History and maturation are especially serious threats to internal validity in:
The correct answer is D:
History and maturation are threats to validity particularly in pre-post designs that study only one group over time. If there is change in the outcome for that group, the change could be attributed to events or natural changes that occurred during the intervention period.