Once you have collected your data, you need to look at what it really shows you. Sometimes (but very rarely) the conclusion is obvious. If every single member of a reading club made a huge gain in reading age and no member of the control group made any such gain, then you can be reasonably confident that the reading club benefited the pupils’ reading ability! In the real world, however, data is rarely that clear-cut. There will usually be variation between individuals and the degree of this may shed doubt on your conclusions. If an improvement is shown, you need to judge if this improvement is significant.
The reliability of any conclusions drawn will increase if the actual data is reliable and accurate. Conclusions will be strengthened when trends are very pronounced. Some small differences may not be significant, although they may be if they show real consistency. A simple way to judge the significance of any difference is to analyse the results statistically. Ensure that any conclusions drawn do not go beyond what the data actually indicates.
This data collection and conclusion is critical to an initiative and its future success, and has a number of advantages:
The data can show whether there was any significant change in the dependent variable(s) you hoped to influence. Collecting and analysing data helps you see whether your intervention brought about the desired results.
The term “significance” has a specific meaning when you’re discussing statistics. The level of significance of a statistical result is the level of confidence you can have in the answer you get. Generally, researchers don’t consider a result significant unless it shows at least a 95% certainty that it’s correct (called the .05 level of significance, since there’s a 5% chance that it’s wrong). The level of significance is built into the statistical formulas: once you get a mathematical result, a table (or the software you’re using) will tell you the level of significance.
Thus, if data analysis finds that the independent variable (the intervention) influenced the dependent variable at the .05 level of significance, it means there’s a 95% probability or likelihood that your program or intervention had the desired effect. The .05 level is generally considered a reasonable result, and the .01 level (99% probability) is considered about as close to certainty as you are likely to get.
A 95% level of certainty doesn’t mean that the program works on 95% of participants, or that it will work 95% of the time. It means that there’s only a 5% possibility that it isn’t actually what’s influencing the dependent variable(s) and causing the changes that it seems to be associated with.
They can uncover factors that may be associated with changes in the dependent variable(s). Data analyses may help discover unexpected influences; for instance, the effort was twice as large for those participants who also were a part of a support group. This can be used to identify key aspects of implementation.
They can show connections between or among various factors that may have an effect on the results of your evaluation. Some types of statistical procedures look for connections (“correlations” is the research term) among variables. Certain dependent variables may change when others do. These changes may be similar – i.e., both variables increase or decrease (e.g., as children’s proficiency at reading increases, the amount of reading they do also increases). Or the opposite may be observed – i.e. the two variables change in opposite directions (as the amount of exercise they engage in increases, peoples’ weight decreases). Correlations don’t mean that one variable causes another, or that they both have the same cause, but they can provide valuable information about associations to expect in an evaluation.
They can help shed light on the reasons that your work was effective or, perhaps, less effective than you’d hoped. By combining quantitative and qualitative analysis, you can often determine not only what worked or didn’t, but why. The effect of cultural issues, how well methods are used, and the appropriateness of your approach for the population – these as well as other factors that influence success can be highlighted by careful data collection and analysis. This knowledge gives you a basis for adapting and changing what you do to make it more likely you’ll achieve the desired outcomes in the future.
They can provide you with credible evidence to show stakeholders that your program is successful, or that you’ve uncovered, and are addressing limitations. Stakeholders, such as funders and community boards, want to know their investments are well spent. Showing evidence of intermediate outcomes (e.g. new programs and policies) and longer-term outcomes (e.g., improvements in education or health indicators) is becoming increasingly important to receiving – and retaining – funding.
Their use shows that you’re serious about evaluation and about improving your work. Being a good trustee or steward of community investment includes regular review of data regarding progress and improvement.
They can show the field what you’re learning, and thus pave the way for others to implement successful methods and approaches. In that way, you’ll be helping to improve community efforts and, ultimately, the quality of life for people who benefit.