Skip to content

Word Attack: “Objective”

March 9, 2011
by

objective (adj): not influenced by personal feelings, interpretations, or prejudice; based on facts; unbiased: an objective opinion.

Source: Random House Dictionary

As reformers of all stripes rethink teacher evaluation, and as states try to reduce budget shortfalls by cutting education spending, there have been many discussions about the best way(s) to ensure that we keep the best teachers in their classrooms, and eliminate the worst. Though there is much to be said about the various proposals people have put forth, and though I have strong opinions about most of them, there’s one oversight in the conversation that has been bothering me for months.

It appears that many people don’t quite understand what “objective” means, or are confused about when it’s appropriate to use the term.

For instance, I recently had a conversation with a political staffer, in which we debated the merits (or lack thereof) of using student test scores to evaluate teachers. While arguing in favor the idea, he defaulted to an argument I’ve heard too many times to count: “Well, don’t you agree it would be nice to have an objective measure of a teacher’s performance?”

“Perhaps,” I responded. “But standardized test scores are not objective.”

Test developers use their judgment (which will be subject to their personal, cultural, and institutional biases) to create test items they think embody the learning standards their test is supposed to measure. And aside from certain types of math items that consist solely of computation and rely on a machine to compare a student’s answer to the correct one (assuming, of course, that the company doesn’t screw up and that there are no mechanical failures), the majority of test items require someone to interpret whether, and to what extent, a student’s answers reflect an understanding of whatever the test item is supposed to measure. Students’ scores can change greatly depending on the instructions given to scorers, people’s choices about what constitutes “proficiency,” or numerous other factors that can’t be considered objective.

Even scoring machines are programmed to mimic the kinds of judgments human scorers give, by analyzing certain features of test-takers’ answers (word length, response length, etc.) that are associated with the kinds of scores generated by humans’ subjective scoring practices. (If that process still smells a bit objective to you…OK…but then we need to talk about whether it’s adequate to the task at hand! My totally biased opinion: Displays of truly intelligent performance can’t be fairly or adequately assessed by looking at how many words were put on a page, or how long those words tend to be.)

Lesson number one: Just because a process results in a number, that doesn’t mean that the process or the number are objective.

The “objective” line also comes up when comparing the current last-in-first-out (LIFO) system for layoffs to merit (another word to be explored at length!) -based layoff schemes. Though there are good arguments to be made for and against the seniority system (see this post by teacher Heather Wolpert-Gawron for an excellent exploration of the pros and cons of LIFO, and other ways to approach evaluation), it’s frustrating to see how often people pan it because they believe the alternatives are more “objective.”

For instance, New York City-based Educators 4 Excellence proposed the following system to determine which teachers would go first when layoffs were ordered. Their system would lay off teachers who:

  1. Are chronically absent as defined by missing more than eight percent of a given school year, without a documented medical reason.
  2. Received an unsatisfactory rating in the past year.
  3. Have not been able to find a full-time position in the past six months (the Absent Teacher Reserve (ATR) pool).

Before going on to explain their rationale behind each choice, they state that “The policy team chose these three criteria because they are objective and proven measures that influence teacher and student performance.”

Now, the first criterion is objective: assuming the necessary data have been entered correctly, it should be a simple thing to figure out which teachers have been absent eight percent of the school year or more. The second two…not so much. Unsatisfactory ratings are based primarily on an administrator’s judgment, which are definitely influenced by that person’s feelings, interpretations, or prejudice. (They’re also based in part on student test scores, which cannot be considered objective– or fair, or accurate, in some cases.) And though determining whether or not a teacher is in the ATR pool should be a simple fact-finding exercise, the decision to excess a teacher is subject to the same subjective issues as the evaluation process.

(I find it funny, too, that they’re promoting their protocol over the seniority system because it’s “objective” when it isn’t…and the seniority system is. You can say whatever else you want about it, but LIFO is objective. Assuming that hire dates have been entered correctly, ranking teachers based on when they started working for the district is a process based on facts.)

Lesson number two: Just because you like a certain process, that does not make it objective. And just because you don’t like its alternative, that doesn’t mean the alternative isn’t objective.

As we continue to look for better ways to make difficult educational decisions, we should make an effort to look at the pros and cons of all proposals on the table, and discuss them honestly and openly so we can create something we can all agree is fair. Regardless of where we stand on any given issue, we should all concede that most of these dueling proposals are heavily subjective– and that’s not necessarily a bad thing. What’s really at issue here is whose judgment we decide to trust, whether their judgment is well-informed, and whether we’ve done enough to minimize the effects of a strong bias in one area or another.

8 Comments leave one →
  1. Matt Di Carlo permalink
    March 9, 2011 4:16 pm

    Right, and even if you consider test scores “objective,” the use of estimates from value-added and other growth models relies on subjective human judgments at almost every level. (http://shankerblog.org/?p=706)

  2. March 9, 2011 4:43 pm

    No kidding the test scoring isn’t objective. It’s not even coherent. No one should be allowed to have an opinion without reading something by Todd Farley and/or Dan DiMaggio, two insiders who tell what goes on inside the test scoring centers. Farley, who worked in the industry for more than a decade, including at the Princeton campus of the prestigious ETS. Scoring is all bullshit. It’s done by temps who are what Farley calls “Drunks, Dingbats, and Dilettantes.”

    From Dan DiMaggio (http://www.monthlyreview.org/101201dimaggio.php):

    “Scorers often emerge from training more confused than when they started. Usually, within a day or two, when the scores we are giving are inevitably too low (as we attempt to follow the standards laid out in training), we are told to start giving higher scores, or, in the enigmatic language of scoring directors, to “learn to see more papers as a 4.” For some mysterious reason, unbeknownst to test scorers, the scores we are giving are supposed to closely match those given in previous years. So if 40 percent of papers received 3s the previous year (on a scale of 1 to 6), then a similar percentage should receive 3s this year. Lest you think this is an isolated experience, Farley cites similar stories from his fourteen-year test-scoring career in his book, reporting instances where project managers announced that scoring would have to be changed because “our numbers don’t match up with what the psychometricians [the stats people] predicted.” Farley reports the disbelief of one employee that the stats people “know what the scores will be without reading the essays.”

    Get Farley’s book called “Making the Grades.” Read it. It’s short, very funny, and completely horrifying. Standardized testing is a complete scam making billions for private for-profit companies. No way, you say? Remember Halliburton, Blackwater, Enron?

  3. Cal permalink
    March 9, 2011 4:53 pm

    Excellent analysis and comments. One thing missed. The questions are designed as much to “spread the field” as they are to assess knowledge, skill, or ability. Perhaps more so.

  4. Ajay Srikanth permalink
    April 2, 2014 12:11 pm

    Awesome post. Also, NONE of the criteria used by E4E are objective. How did they come up with the 8% figure for absences? They used their judgment and felt it was appropriate. Cut scores for proficiency are determined in the same way

Trackbacks

  1. A people-(em)powered evaluation system :: Reclaiming Reform | Education policy of, by & for We the People.
  2. Some Ideas on Testing « educationclearinghouse
  3. The Best Resources For Understanding How To Interpret Education Research | Larry Ferlazzo’s Websites of the Day…
  4. This Week’s “Round-Up” Of Good Posts & Articles On Education Policy | Larry Ferlazzo’s Websites of the Day…

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 155 other followers

%d bloggers like this: