OTHER INDIRECT METHODS

Definition:

An indirect method is any instrument in which a student does not demonstrate his or her learning and abilities. A focus group is a group discussion conducted by a moderator with typically 7-12 individuals who share certain characteristics. Careful and systematic analysis of the discussions provides information that can be used for improvement. Archival records are biographical, academic, or other data available from the college or other agencies. Exit interviews ask individuals to share their perceptions of their own attitudes and/or behaviors in a face-to-face dialogue.

Examples:

Indirect methods (other than written surveys) may include focus groups, archival records, and exit interviews.

Costs:

Moderator compensation and training
Data compilation and processing
Storage
Time to review results and make improvement decisions
Clerical support
Supplies (paper, scantron, etc.)
Time and space for interviews and/or focus groups
Programming/technical support for archive searches/reports

Advantages:

Focus Groups

Useful to gather ideas, details, new insights and to improve question design
Helpful in the design of surveys
Can be used to get more in-depth information on issues identified by a survey
Can clarify issues not completely understood from another instrument.
Unlike a survey, a moderator can ask follow up questions when necessary.
Can be used in conjunction with a quantitative study to confirm or broaden one's understanding of an issue.

Exit Interviews

Provide immediate feedback to the interviewer
Frequently yield benefits beyond data collection through interaction with students
Can include a greater variety of items than is usually possible on questionnaires or surveys.
If done by a third-party, many of the same externality and validity advantages of a survey apply.

Archival Records

Tend to be accessible
Built upon data collection efforts that have already occurred
Can be cost efficient
Nonintrusive measurement
Useful for longitudinal studies
Good way to establish baseline data

Disadvantages:

Focus Groups

Not suitable for generalizations--biased sample
Moderators require training
Differences in the responses between groups can be troublesome
Groups can be difficult to assemble
Moderator has less control than in individual interviews
The data are complex and thus difficult to analyze

Exit Interviews

Requires direct contact, which may be difficult to arrange
May be intimidating to interviewees, thus biasing results
As with surveys, results tend to be highly dependent on the wording of the items
Time consuming for large populations
If done by a third-party, logistics can be difficult to arrange
Confidentiality issues

Archival Records

May be hard to determine which data are relevant and available.
Datasets may need to be combined and transferred from multiple sources.
Confidentiality issues.
It is difficult to identify the cause of a problem.
Availability of data may discourage the development of other more appropriate measures.
May encourage attempts to "find ways to use the data" rather than assessment related to specific goals and objectives.

Implementation Suggestions:

Offer incentives to focus group participants.
Anticipate low turn-out and therefore over-recruit.
Train moderators to use open-ended questions, pauses, and probes.
Train moderators to find opportunities to move into new topic areas.
Plan exit interviews carefully, being careful with logistics and question design (as in surveys).
Train interviewers and moderators to put students at ease.
Interview purposeful samples of students when it is not possible to include all.
Consider telephone interviews as well as face-to-face. Encourage dialogue.
Have a time limit for focus groups and interviews.
Give very careful, explicit directions.
Obtain informed consent when necessary.
Be wary of FERPA regulations when using archival records.
Only use archival records that are relevant to specific goals and objectives of learning and development.

Recommendation:

Focus groups should be used to increase validity and clarity of surveys. Exit interviews may be a good idea, but seem logistically difficult for our institution. Exactly when does a student "exit?" Archival records should be used to examine general trends. These records are already in place. It would be quite easy to have this information as a piece of our "assessment plan." The difficulty will be to harness the relevant information.

Bibliography/Resources:

Dobson, Ann, Conducting Effective Interviews: How to Find Out What You Need to Know and Achieve the Right Results, Trans-Atlantic Publishers, 1996

Morgan, D., Focus Groups as Qualitative Research, University Paper series on Quantitative Applications in Social Sciences, Sage

Krueger, R., Developing Questions for Focus Groups, University Paper series on Quantitative Applications in Social Sciences, Sage

Stewart, D. and Shamdasani, P., Focus Groups: Theory and Practice, University Paper series on Quantitative Applications in Social Sciences, Sage

Van Kollenburg, Susan E., ed. A Collection of Papers on Self-Study and Institutional Improvement: Proceedings of the 106th Annual Meeting of the North Central Association: Serving the Common Good: New Dimensions in Higher Education. Chicago, The Higher Learning Commission. 2001.