The use of statistics in assessment

The main criterion for choosing particular statistical indicators is the level of statistical measurement. In 1946, Stevens identified four such levels :

1. rated (or discrete);
2. ordinal;
3. interval;
4. attitude.

In the future, it was these levels that were used to describe empirical data.

The nominal level of statistical measurement includes setting numbers and data into correspondence for their further distribution into groups . The difference between ordinal level variables and nominal variables is that the former have an ordered relationship between their categories . For example, participants in the retraining program at the end of this program can be divided into “successful” (those who completed the courses and were able to find work within two weeks), “partially successful” (completed the courses, but could not find work within two weeks) and "unsuccessful" (could not finish the courses). Ordinal variables in this case are characterized by consistency , while nominal variables serve solely to distinguish categories. Ordinal variables play a key role in evaluation , since ordinal norms of estimation are usually used to evaluate the opinions of program participants. Interval and relationship variables reflect the underlying numerical range .

An important detail of the Stevens classification is the need to compare the level of statistical measurement with the analytical method. Once the level of statistical measurement of key variables in the analysis has been selected, in most cases the choice of a suitable analytical method is, in essence, a simple observance of formality. Thus, comparing the analytical method with the level of statistical measurement is an important task for evaluators.

When a process or program is being calculated, numbers can be grouped in various ways. If the resulting statistics, such as averages, are used to describe a group of elements, then the digital data is called descriptive statistics .

In many cases, the number of those who receive services from the program is so large that a full examination will lead to excessive costs . In this case, a representative group is selected from the total number, i.e. output statistics are used. When choosing a representative group, the evaluator should be guided by the following principles :

1. The group being evaluated must be known and identifiable.
2. The ability to select one or another category of the evaluated group must be calculated.
3. A representative group should be of the appropriate size relative to the size of the entire evaluated group.
4. It is necessary to evaluate the adequacy of the representative group.

The confidence level reflects the amount of data needed by the evaluator in order to assert that the program being tested has the desired effect. In the social sciences, a 95% confidence level is traditionally used. However, for most community programs, a level of 95% is redundant. A confidence level in the range of 80-90% is sufficient for an adequate evaluation of the program . Thus, it is possible to reduce the size of the representative group, thereby reducing the cost of the assessment.

In the process of statistical evaluation, the null hypothesis is checked, which consists in the fact that the program did not have the desired effect. If the results obtained differ significantly from the initial assumptions about the correctness of the null hypothesis, then the latter is rejected.

To select the most appropriate statistics in a given situation, evaluators need to use various criteria . The determination of the main issues during the assessment, the choice of the method of collecting information, and the type of the group being evaluated to a large extent influence the choice of statistical methods.

The size of the representative group is important. If you select a too small group, this may lead to an incorrect evaluation of the program , too large a group involves unnecessary costs for the evaluation. Also, for the choice of a statistical method, the number of observations of units of measure is important. For example, if two or more observations are made for the same unit of measurement , the change in time can be examined, which entails the selection of a statistical method suitable for such an analysis .

Before applying statistics, the evaluator needs to consider the distribution of units of measure across various variables . Such an analysis allows you to determine the extent to which units differ from each other. For example, if for the evaluation it is of interest to the race of participants in continuing education courses, and if only two out of fifty-six participants differ from the rest by race, then it will not be possible to use the race as a variable in the evaluation process.

Statistical indicators are important in assessing the effectiveness of the program and its social effect. A well-organized system of statistical indicators provides evaluators with summary information that allows a better assessment of the effect produced by the program. It is important that the evaluation process uses analytical processing of the collected numerical data. Using statistics in the assessment process allows you to:

1. Compare programs (or their individual components) at different time periods;
2. Compare the effectiveness of the programs with the goals set at the beginning of the program. This allows you to adjust the process of applying the program in practice.
3. Compare the effectiveness of such programs in different places. A similar approach can be used in centralized systems to identify ineffective application of the program in practice.

Statistics are widely used in meta-analysis. Meta-analysis is a special type of information synthesis used in the evaluation process. The statistics for this method of analysis are used to summarize, synthesize and interpret data obtained during empirical studies. Combining the results of various studies, a meta-analysis allows you to increase the positive effect of using statistics and get more accurate results.

• Joseph S. Wholey, Harry P. Hatry, Kathryn E. Newcomer Handbook of Practical Program Evaluation. Wiley_Default, 2004.
• John M. Owen, Patricia J. Rogers Program Evaluation: Forms and Approaches. SAGE, 1999.
• Valerie J. Caracelli, Methodology: Building Bridges to Knowledge, in: Evaluationsforschung: Grundlagen und ausgewaehlte Forschungsfelder / ed. by R. Stockmann, Leske + Budrich, Opladen. 2000, S. 165-191.