Program evaluation : an investigation of methods for measuring the effectiveness of the interviewing institute

The investigation of worth or value to be found in a training program or other development activity is termed program evaluation. The reasonableness of this activity is highly regarded by many authorities and practitioners, yet few organizations have been actively involved in a systematic effort to determine the specific impact in participant learning, behaviors (beliefs) or organizational results attributable to training program or other developmental activities. Many have felt comfortable in "validating" the worth of programs through use of participant testimonials and participant reported (self-reported) questionnaires. This report investigated various methods for collecting self-report program evaluation data and applied these methods to an existing training program. The objectives were to document the relative effectiveness of different self-report methods in representing a participant's actual success (change) in the program and also to measure the internal validity of the program. Study was made of selection interviewers attending three Interviewing Institutes conducted by the University of Houston's Personnel. Psychology Services Center (participant N = 22, 21, 21). Of interest was the skills enhancement of interviewers participating in the Institute training and specifically how this change (Pre to post) could be measured. Several types of measurement techniques were used to document the degree participant interviewing behaviors changed during the course of the week-long training program. These included self-reported program ratings, expert judgment and the use of a behavioral scoring system. Ratings or scores gathered prior to (pre) and following the training program (post) were collected in the following interviewer performance areas: effective questioning technique, structured interviewing format, interpersonnel supportiveness, initial interview rapport building, active listening and collection of relevant material. Actual interviewer behavior was recorded on videotape and this record was observed and scored by trained judges. These individuals produced indices of the participants' interviewing behavior and also made expert ratings of participants1 behavior in each of the six performance dimensions plus providing on overall rating. Pre- and post-training observations were taken. [...]