Article: Data collection by Coda Messerle

Coda Messerle, M. (1973). Data collection: A cybernetic aspect of a learning assistance center. In G. Kerstiens (editor). Technological Alternatives in Learning, Sixth Annual Proceedings of the Western College Reading Association. Albuquerque, NM. 51-58.

THE LEARNING ASSISTANCE CENTER

Using as a model the Learning Assistance Center at California State University, Long Beach, we can describe a Learning Assistance Center as a support, delivery, and referral service for the entire campus community: students, faculty, staff, and administrators. There are three major aspects to its services—personal learning skills, academic aids, and support of faculty instruction. Assistance is given in these three areas through a systems approach. Briefly and concisely, the CSULB Learning Assistance Center is totally learner-centered with a diagnostic/prescriptive rationale that considers learning to be individualistic, mathemagenic, personalized, and cybernetic (9;10).

It is the latter aspect of a LAC—the cybernetic aspect within a systems approach to instruction—which will be the focus of this paper. More specifically, we will discuss data collection and analysis as a cybernetic aspect of a Learning Assistance Center.

CYBERNETICS AND ACCOUNTABILITY

In a recent ERIC paper that overviews systems literature, Twelker and others define a systems approach to instruction as “...a systematic way of identifying, developing and evaluating a set of materials and strategies aimed at accomplishing a particular educational goal.” (27:1)

Banathy, a major proponent of a systems approach to instruction, points out that cybernetic data or “continuous feedback of performance data into the system” is critical “for the purpose of making adequate adjustments in the system.” (4:82)

He further states that

The self-adjusting characteristics of systems development prescribe change as a perpetual process in the development, operation, and maintenance of systems. We can safely say that the only valid means of maintaining a system is by purposely changing it. (4:82)

For quality control (4:90) a Learning Assistance Center must have incorporated within its system a mechanism for continuous feedback, ongoing analysis of that -feedback, and means for adjusting and changing the system -where appropriate. In fact, all the three primary actions of a system—problem definition and organization, system analysis and development, and system evaluation—are interrelated by feedback built in the system. (27)

In terms of accountability being concerned with effectiveness and efficiency, a Learning Assistance Center is held accountable through this cybernetics approach to systems. This allows the LAC to be accountable not only to itself as a system and a program, but also to be accountable to its learners and learning facilitators, the campus it services, its sponsoring agency, and to other Learning Assistance Centers.

Improved data collection, analysis, and subsequent changes within the system are required for improved accountability. Agencies at all levels— federal, state, local, and campus—are demanding increased efficiency in the use of resources for improved accountability. An excellent reference on accountability for LAC personnel is Metfessel’s and Hammond’s paper entitled “Everything You’ve Always Wanted to Know About Behavioral Objectives But Were Afraid to Ask, or How to Develop Accountability Programs in the Affective Domain.” (18)

Further, Nita Myers Earnheart, a learning practitioner involved with UCLA’s Learning Center, in a recent paper warns us that “The word accountability is increasingly in the campus air, hanging particularly heavy over the heads of those student services concerned with learning problems and techniques (13 34)

LITERATURE AND RESEARCH

Although the literature contains many surveys and descriptions of Learning Centers and their programs (3; 5; 6; 11; 15; 20; 26; 29), little can be found that discusses data collection and analysis to derive accountability. Bleismer documents this in his most recent available review of the literature, “1971 Review of Research on College-Adult Reading.” He states that “Purposes for presenting a number of programs at this conference [National Reading Conference] last year were mainly ones of description (Cartwright, 1970; Christ, 1970; Ewing, 1970); but some also included reports of evaluations to some extent.” (8:299) Bleismer further states that “The contents [of the Proceedings of the Western College Reading Association’s Fourth Annual Conference] appeared to be mainly descriptions of specific programs in the WCRA area; but several included evaluation reports or other research aspects . . .“ (8:298) Bleismer alludes here to the papers by Adams (1), Biggs, and others (7), Hagstrom (14), and Reid (24).

A further search of the literature indicates that there were six articles that treated accountability with reference to learning assistance programs.

Among these were two by Martha Maxwell (16; 17), one by Stafford North (21), one by Chester Tifiman (25), one by Deborah K. Osen (23), and one by Drexler and Pepper (12).

In this last paper, Drexier and Pepper caution “that the most commonly used criterion of success of a program can often be misleading.” (12:193) This warning refers to the current practice of evaluating and discussing program effectiveness as it centers around assessment of the total program. We must remember that a program is comprised of several components. We must also remember that each of these components should be assessed separately with the understanding that they interface with one another. An overall program evaluation leads only to the evaluation of the program, not to the evaluation of its component parts. An evaluation of separate programs and activities in a Learning Assistance Center as parts affecting the total system can lead us then to accountability of each specific component.

GATHERING SENSOR DATA

Let’s turn now to a discussion of some practical means for gathering sensor data that assesses Learning Assistance Center programs and activities. The suggestions given below have been successfully employed at CSULB’s Learning Assistance Center. Only three aspects of the Center’s total assessment procedure will be considered: 1) usage of the Center, 2) usage of the materials in the Center, and 3) usefulness of the material to the learners in the Center.

Evaluation Aspect #1: Usage of the Center

To determine usage of the Center, we are presently using a form which we have labeled the “Sign-in Sheet.” Every entrant to the LAC is asked to sign in. In the appropriate spaces provided, the entrant prints his name, enters his student ID number, checks his campus status (freshman, sophomore, junior, senior, graduate, EOP, faculty, staff, or visitor), and further, if a student, gives the course number and faculty member from whose class he was referred, or if a faculty, staff, or visitor, gives his school, department, and position.

Several graphs of usage of the Center are kept from the data provided by the “Sign-in Sheet.” First, a day-by-day graph, and at least once a semester, an hour interval graph, is kept of the number of persons entering the Center. In this manner the Center’s personnel determine the peak days and hours of operation and the Supervisor schedules her staff’s work hours accordingly.

Second, a month-by-month tally of users, broken down into the categories of campus status, illustrates the increase or decrease in overall and categorical usage of the Center. Of course, this is sensor data which becomes optimally useful only when the LAC staff asks itself the reasons for the appearance of the data.

Evaluation Aspect #2: Usage of the Materials in the Center

We have employed at our LAC another form which provides usage data on materials—the "Check-out Slip.” All materials within our Center are checked out on this slip. A month-by-month tally of the check-out slips provides the LAC staff with an accurate count of the number of times any piece of software was used in the Center, and also an accurate count of the number of times -each specific item of software was used in the Center. This sensor data is analyzed by the LAC staff on a bimonthly basis. During these assessment meetings the - question is asked: “Why is Wordcraft/1 being used more often than Wordcraft/2?” This process of assessment of materials usage in conjunction with an assessment of material usefulness (see “Evaluation Aspect #3: Usefulness of the Material to the Learners in the Center,” this paper) leads to many effects. For example, in some cases, it culminates in the buying or production of more programs of the type that are proving themselves cost-effective, and in other cases it -leads to the steering away from those programs which are not proving themselves cost-effective. Another result of the assessment procedure is the encouragement on the part of the LAC staff of greater utilization of present programs (encouragement, on one hand, might simply mean making the program more accessible, or, on the other hand, publicizing the program). Another effect which often arises out of the analysis of usage of a specific program leads to the, adaptation of that program for greater usefulness.

Evaluation Aspect #3: Usefulness of the Material to the Learners in the Center

Further data is collected from the “Check-out Slip;” this data is in terms of usefulness of the material to the learner. The learner, upon returning his checked-out materials to the LAC aide, is requested by the aide to check both the content and format usefulness of the material on a three-point scale—very helpful, somewhat helpful, and not at all helpful. At the bottom of this slip a space is provided for further comments.

This information is also tallied once a month and the results are reviewed on a bimonthly basis. (Mention should be made here that at the present time we are in the process of setting up a computer program which will tally this sensor data). If, in the process of scrutinizing the results from this data collection, it is noted that a particular program’s content and/or format is judged by many learners as not at all useful, the LAC staff attempts to evaluate the “Why?” The answer to this question is often found within the learner’s file folder on the “Activities Log and Response Sheet.” Every regular user of the Center is provided with this sheet for logging his activities in the Center and making responses to them. Also, the LAC staff member/counselor who interfaces with the learner makes written responses on this same sheet to the learner’s reactions and comments. This is in addition to the regularly weekly scheduled appointment between the learner and his LA Counselor. From this response column the Center derives more data on the usefulness of its material.

CONCLUSIONS AND ONE STEP BEYOND

We have only examined a few of the methods employed at CSULB’s LAC to measure -the effectiveness of some of the Center’s component parts. For a more comprehensive and detailed account of assessing learner programs refer to Nicholas J. Anstasiow’s article “Measuring Change—A Time Dimensional Problem” (2) and Metfessel’s and Michael’s paper “Multiple Criterion Measures for Evaluation of School Programs” (19).

It is admittedly true that the assessment methods described above require a considerable amount of time and effort on the part of the LAC staff and its learners, but the reward of accountability and subsequently the ability to upgrade an operation from the data gathered is certainly well worth it. In fact, being aware of this cybernetic aspect of assessment --

of a system is a must for any systems approach. Twelker and others remind us that: “If evaluation techniques are not a prominent part of the proposed procedures [of a systems approach], either the approach is weak or it is not a systems approach. Evaluation techniques along with the careful statement of objectives are critical parts of any ‘useful’ systematic approach.” (27:10) It is also admittedly true that the record-keeping activities which we have discussed are not psychometrically sophisticated. However, they are a movement toward more accountability within Learning Assistance Centers. They are certainly better than the informal evaluations based on opinion and intuition which unfortunately seem to be more representative of evaluations in this area. O’Hare and Lasser, in a recently published monograph: Evaluating Pupil Personnel Programs (22:20-21), conceptualize evaluation and evaluative research on a continuum (Fig. 1).

The least sophisticated level of program evaluation would be informal evaluations based on opinion and intuition. This level is succeeded by a level represented by data collecting, administrative reporting, and reporting environmental characteristics such as physical facilities or counseling techniques used. The gathering of sensor data which we outlined above falls at this level. The other extreme of the continuum is represented by evaluative research—designs which attempt to relate environment, pupil, and behavior change. Philosophically and theoretically many persons involved with learning have already reached this most sophisticated level of evaluation. John A. Wood and Anne Marie Bernazz Haaze, in an article published in the Twenty-First Yearbook of the National Reading Conference, broached this subject when they stated that:

Instructional methods as well as educational environment differ and these differences interact with personal variables of the learner to either facilitate or inhibit learning - . . Insufficient attention has been paid to specific learner characteristics and their effect in specific learning situations. Without this knowledge, attempts at individualized instruction are limited to broad concepts which do little to adjust the educational climate so that it can accommodate specific learner characteristics and maximize learning. Instead of predicting main efforts between treatments, we must begin predicting interactions based on our knowledge of sociological, personalogical and methodological factors that affect learning. (30:161-162)

Thus, there is a felt need for practical suggestions to be made regarding assessment and evaluation of what Leland Kaiser labels “the ecosystem.” (28) We need, as Learning Assistance practitioners and directors, to go one step beyond implementation of just a systems approach and to begin thinking and acting in terms of an ecological systems approach.

Figure 1

Informal Evaluations Based on Opinion, Intuition Data Counting, Administrative Reporting Environmental Characteristics
Statement of Observable Behavioral Outcomes--Subjective Evaluation of Outcome Attainment
Statement of Observable Behavioral Outcomes—Criterion Referenced Assessment of Outcome Attainment
Longitudinal Studies— Criterion-Referenced Assessment of Outcome

Attainment
Evaluative Research (e.g., National study of Guidance and Studies Relating Environment, Pupil and Behavior Change)

REFERENCES

1. Adams, Royce W. ‘~The Use of Tutors in the Santa Barbara City College Reading Lab,” in Frank L. Christ (ed.) Interdisciplinary Aspects of Reading Instruction. Volume IV, Proceedings of the Fourth Annual Conference of the Western College Reading Association. Los Angeles, 1971, pp. 3-7.

2. Anastasiow, Nicholas J. “Measuring Change—A Time Dimensional Problem,” in Frank P Greene, Investigations Relating to Mature Reading, Twenty-First Yearbook of the National Reading Conference, Volume I, Tampa, 1972, pp. 128-133.

3. Anderson, Clarence A. ~A Description of the Flint Community Junior College Reading Program,” an unpublished paper presented at the Annual Conference of the North Central Reading Association, Flint, Michigan, October-November, 1969. ERIC ED 036395.

4. Banathy, Bela H. Instructional Systems. Palo Alto, California: Fearon Publishers, 1968.

5. Beitler, Lorraine and Martin, Irene. “The Design and Implementation of a Learning Center for Career Oriented Students,” an unpublished paper presented at the Twenty-First Annual Conference of the National Reading Association, Tampa, December, 1971. ERIC ED o56 850.

6. Berg, Jerry; ElIison, Jim; and Rose, Doris. “Learning Center Concept in- Action,” in Frank L. Christ (ed.) Interdisciplinary Aspects of Reading Instruction.Volume IV, Proceedings of -the Fourth Annual Conference of the Western CollegeReading Association. Los Angeles, 1971,pp. 20-23.

7. Biggs, Barbara et al. “The Student-Development Center: A Ten-Week Experiencein Re-Education,” in Frank L. Christ fed.-) interdisciplinary Aspects of Reading -Instruction. Volume IV, Proceedings of the Fourth Annual Conference of theWestern College Reading Association. Los Angeles, 1971, pp. 24-3 1.

8. Bleismer, Emery P. “1971 Review of Research on College-Adult Reading,” in Frank P. Greene led.) investigations Relating to Mature Reading. Twenty-First Yearbook of the National Reading Conference, Volume I, Tampa, 1972, pp. 298-330.

9. Christ, Frank L. “-Learning Assistance Systems and Programs . . . An Outreach Service of the University Counseling Center,” an unpublished paper from California State University, Long Beach, 1972.

10. - Christ, Frank L. “Systems for Learning Assistance: Learners, Learning Facilitators, and Learning Centers,” in Frank L. Christ (ed.) Interdisciplinary Aspects of Reading Instruction. Volume IV, Proceedings of the Fourth -Annual Conference of the Western College Reading Association. Los Angeles, 1971, pp. 32-41.

11. Davidson, Donna. “Reading/Study Skills in a Two-Year College,” in Frank L. Christ led.) Interdisciplinary Aspects of Reading Instruction, Volume IV, Proceedings of the Fourth Annual Conference of the Western College Reading Association. Los Angeles, 1971,pp. 42-51.

12. Drexler, John A., Jr., and Roger S. Pepper. “Methods of Evaluating Reading Programs for Marginal Admission Students,” in Frank P. Greene led.) College Reading: Problems and Programs of Junior and Senior Colleges. Twenty-First Yearbook of the National Reading Association Volume II, Tampa, 1972, pp. 189-197.

13. Earnhart, Nita M. “Evaluation: Toward Responsible Accountability,” in Gene Kerstiens led.) Reading: Putting All the Cards on the Table. Volume V, Proceedings of the Fifth Annual Conference of the Western College Reading Association. Reno, 1972, pp. 34-41.

14. Hagstrom, Jon. “A Comparison of the Reading Abilities of a Junior College Population and the Readability Levels of Their Texts,” in Frank L. Christ (ed.) Interdisciplinary Aspects of Reading Instruction. Volume IV, Proceedings of the Fourth Annual Conference of the Western College Reading Association. Los Angeles, 1971, pp. 89-95.

15. Levinson. Betty and Carol Pfeffer. “A University Learning Center,” in Frank L. Christ led.) Interdisciplinary Aspects of Reading Instruction, Volume IV, Proceedings of the Fourth Annual Conference of the Western College Reading Association. Los Angeles, 1971, pp. 89-95.

16. Maxwell, Martha J. and Thomas M. Magoon. “Self-Directed Educational Skills Programs, A Descriptive and Comparative Evaluation,” The Journal of College Student Personnel, December, 1962, pp. 86-89.

17. Maxwell, Martha J. ~‘Evaluating an Individualized College Reading Service: Alternative Criteria and Statistical Models to the Test-Retest Paradigm,’~ Journal of Reading Behavior, Volume IV. Summer, 1972, pp. 1-8.

18. Metfessel, Newton S. and M.D. Hammond. “Everything You’ve Always Wanted to Know about Behavioral Objectives but Were Afraid to Ask or How to Develop Accountability Programs in the Affective Domain,” an unpublished paper from the Iniversity of Southern California, 1971.

19. Metfessel, Newton S. and William B. Michael. “Multiple Criterion Measures for Evaluation of School Programs,” an unpublished paper from the University of Southern California, 1971.

20. Newman, Loretta M. Community College Reading Center Facilities, Los Angeles: ERICICJC, Topical Paper-21, 1971, ED o51 792, pp. 1-19. - North, Stafford. “Evaluating Learning Center Programs” in Marshall Gunselman led.) What Are We Learning About Learning Centers? Oklahoma City: - Eagle Media, 1971, pp. 55-73.

22. O’Hare, Robert W. and Barbara Lesser. Evaluating Pupil Personnel Programs. Monograph—No. 2 o~f the California Personnel and ‘Guidance Association, Fullerton, California, 1971.

23. Osen, Deborah K. “A Systems Approach to Establishing Accountability in the College Reading Program,” in Gene -Kerstiens (ed.) Reading: Putting All the Cards on the —Table. Volume V~ Proceedings of the Fifth -Annual Conference of the Western College Readmg Association. Reno, 1972, pp. 113-118.

24. Reid, Bruce. “A Behavior Management Program for Teaching Introduction to Composition,” in Frank L. Christ fed.) Jnterdisciplinary Aspects of Reading Instruction. Volume IV, Proceedings of the Fourth Annual Conference of the Western College Reading Association. Los Angeles, 1971, pp. 113-120.

25. Tiliman, Chester E. “Measuring Outcomes in College Reading Programs,’ in Frank P. Greene fed.) college Reading: Problems and Programs of Junior and Senior Colleges. Twenty-First Yearbook of the National Reading Association, Volume ii, Tampa, 1972, pp. 205-212.

26. Touhey, John- F. “The Learning Center: its Role and Future Implications for the Junior College,” an unpublished paper presented at the International Reading Association Annual Conference, Anaheim, California, May, 1970, ERIC ED o42 586.

27. Twelker, Paul,; Urbach, Floyd D.; and Buck, James E. The Systematic Development of Instruction: An Overview and Basic Guide to the Literature, Stanford: ERIC/CMT, 1972.

28. Western Interstate Commission on Higher Education. Report of the Epidemiology, Campus Ecology, and Program Evaluation Task Force. The Ecosystem Model: Designing Campus Environments. Dr. Daniel Wilner, Chairman. Boulder, Colorado, 1973.

29. Williams, Gil; Jacobson, Pat; and Arnold, James. “Prescriptive Teaching Linked to a Learning and Tutorial Center,” in Frank L. Christ fed.) Interdisciplinary Aspects of Reading Instruction. Volume IV, Proceedings of the Fourth Annual Conference of the Western College Reading Association. Los Angeles, 1971, pp. 147-1 55.

30. Wood, John A. and Haaze, Anne M.B. “Aptitude Interaction Analysis: -A Necessity in Reading Research,” in Frank P. Greene fed.) Investigations Relating to Mature Reading. Twenty-First Yearbook of the National Reading Conference, Tampa, 1972, pp. 161-169.