|
Main Entry: psy·cho·me·tri·cian
Pronunciation: -m&-'tri-sh&n
Function: noun
Date: circa 1939
1 : a person (as a clinical psychologist) who is skilled in
the administration and interpretation of objective psychological tests
2 : a psychologist who devises, constructs, and standardizes
psychometric tests
Read how one person prepared
for the exam in the article "Encouragement
from a New Member" from the March/April 1999 News Highlights.
Learn how you can hold the
archival certification examination in your home town.
The Academy's psychometrician is Holly A. Traver,
Research Associate Professor with the Department of Cognitive Science at
Rensselaer Polytechnic Institute in Troy, New York. |
|
Anne
Diffendal
Regent for Examination Development and
Chair, Examination Development Committee, 2000-2004
How well does the archival
certification examination reflect, with a high degree of certainty,
that a person passing the test has command of a considerable range of
archival principles and techniques and has the ability to apply them
in context? Anne Diffendal offers her perspective. |
Any single ACA exam is the
result of a complex series of events that began nearly twenty years ago. Some
of the following details are less immediately relevant to the question of
efficacy than others, but it is important to have context for the process of
exam development.
It is also important to
remember that the archival certification examination is a certification
exam. It is not a graduate school entrance exam, nor a comprehensive for a
master’s or doctor’s degree, nor the final exam for a course (though it
contains questions similar in form to those on other kinds of tests).
Because most archivists have little or no experience with testing for
certification or licensure, they may question the ACA exam from lack of
understanding.
The Academy undertakes to
present a good certification exam (in which “evaluative efficacy” is one
among many elements) by:
-
following an accepted
process developed from practical experience and legal challenge;
-
applying the best
information available from the educational testing and measurement
profession; and
-
involving a large number
of “job experts” (archivists) in a variety of ways at all stages of
the process.
1. the process
The accepted process of
developing and maintaining an exam of this type is the result of 50-60 years
of experience involving thousands of organizations that certify or license
practitioners. In establishing the certification program, the Society of
American Archivists hired one of the most respected national test
development firms to guide archivists in the process. We have faithfully
followed the recommended process and procedures.
2. the practice
We have used the best information available from the discipline
of psychometrics in developing and maintaining the item bank and exam. One
issue, considered at the outset, was what kinds of questions to use on the
exam. The choices were among what are known as “limited response”
questions, which include multiple choice, matching, and true-false
questions; and “constructed response” questions, which include essay,
short-answer, and oral questions. Nearly all certification/licensure exams
include multiple-choice questions; and it is very common that they contain
only this type of question.
Multiple-choice questions are widely used for a number of reasons. They can
be constructed to test many different levels of knowledge, including higher
cognitive skills and problem solving. They offer an effective way achieve
one of the goals of a certification exam, that is, to test a broad range of
knowledge and skills. There is by far the most research on this type of
question, making conclusions about the validity and the reliability of any
given test quite firm, as well as permitting development of guidelines for
writing good questions and avoiding poor ones to aid in question-writing.
And finally, they are efficient and economical to score.
After each exam is given,
statistical tests are applied to the test as a whole and to each question.
Results are transmitted to the Examination Development Committee, as are any
questions or comments by test takers that are reported by proctors, Academy
officers and board members, etc. Exams are revised and questions are revised
or eliminated based upon the statistical tests and reported questions and
comments.
The ACA exam has continually tested well according to standard statistical
measures. It generally contains only one or two individual questions that
have tested badly; an exceptionally low number for a test of one hundred
questions, according to my understanding.
3. the archivists
At every step of the process,
archival practitioners have made the decisions, prepared the role
delineation statement, developed the questions, and constructed the exam.
Under the direction of expert facilitators, panels of archivists wrote the
role delineation statement; other panels reviewed it; it was “validated”
(a test developers’ term) through a mailing to a random sample of SAA
members. The same process using a different set of archivists was followed
to assign weight to the various domains on the test. Panels of archivists
have subsequently revised the preservation and management domains. Groups of
archivists were convened to write questions to comprise the original item
bank.
Thereafter, a standing committee of Certified Archivists has written new
questions. This same committee has revised existing ones based on test
results, changes in best practice, or new professional literature. Each
question in the item bank is linked to a domain and to a task statement and
a knowledge statement within that domain. The questions for each test are
chosen to reflect the entire range of the role delineation statement.
Examination Development Committee chairs have sought to ensure that the
committee represents a wide variety of employing institutions, subject
expertise, length of experience in the profession, and geography.
In the early-nineties, a panel of experienced archivists, including some
skeptics, took the exam and participated in a panel discussion at an SAA
Annual Meeting. Once, the entire ACA Board took the current exam. A few
Certified Archivists from the initial class took the exam rather than submit
a petition for certification maintenance. These folks were pleased, if not actually
pleasantly surprised, with the test.
The
process of exam development includes continuing evaluation and revision by
archival practitioners for the purpose of producing an effective test. |
