Mathemagical Voodoo and protection charms

As we move toward the experimental side of research, the limitations of quantitative designs are what I find most interesting. Researchers’ number fetishes can produce grotesque misrepresentations and allow them to overlook obvious details and mechanisms. I really like the tease that L&A give on page 99, when they point out that “one reason why variance is not accounted for by certain measures is that they themselves are not highly reliable.” Coding is central to quantifying results, but meaningful reliability with some variable seems impossible. Diedrich’s study on 300 first month college papers included a judgment on the “flavor” of the writing. I could hardly believe what I was reading. Chocolate, caramel, vanilla—what? Beach reminds us that “there is no widely agreed-upon taxonomy regarding different writing types” (221), and I wonder if we wouldn’t be well served by a widely used handbook for coding compositions. Of course, many assumptions about writing would underpin such a work, but the alternatives suck.

Pianco’s study struck me for its limitations. She did tons of work to find a few statistically significant relationships, and then she basically used qualitative data on those significant categories to conclude that remedial writers “glanced around the room during their pauses, sometimes as a diversion, at other times as a way of finding the correct word, or something to say next,” among other broad and intuitive findings. Even comparing remedial and traditional students is as sketchy as comparing flavors of writing. Focusing on remedial students ignores their unique backgrounds, and analysis of flavors could prejudice backgrounds that produce particular flavors of writing. A “small part” of Pamela Eckert’s study used what I see as a possibly unethical pigeonholing of students into groups like “jocks” and “burnouts” when she used them categorically for empirical analysis (Beach 220). “Hey kid, you’re a burnout, right?” Or “Hey, would you say your buddy is more of a burnout or a jock?” However she identified kids, it rubs me the wrong way.

I also want to give some props to Beach for his “Limitations...” section. I too share Beaugrande’s general doubt about “controls” that “make the context dissimilar to ordinary language activity” and about the assumption that “artificial contexts” can generalize to “natural contexts” (235). The maturation and drop-out effects were given in nice, simple reminders (233), and the limitations of holistic ratings (236) were especially insightful—basically we should be careful to work quickly to see relationships between variables, and features make often make better variables than classifications.

In addition to the interesting example cases in both texts, the guidelines for designing research were pretty insightful. Lauer and Asher readily admit that the “distinction between [independent and dependent variables] is rather imprecise in descriptive research” (86), and Beach explains that his own journaling study did not consider “the effects of using the dialogue-journal writing” on any out-come variables” (225). Clearly journaling is a/effecting “the level of partners’ interpersonal involvement,” but researchers can claim to give neither variable priority as they please (this is sort of a question too).?

The Lauer and Asher section on analysis of variance was cool too. Of course, the main idea is to make sure that variables are not interacting so that we can make predictions and generalize.
However, I don’t exactly understand the limitations of ANOVA. The illustrations suggest that specific interactions can be identified. How does this differ from correlation? I get sci-fi chills to think of mapping a multivariate study of thousands of individuals and using computers to account for all the interactions taking place, then using an individual or group’s known characteristics to make predictions. THIS is why so many have a penchant for numbers. Reading Foundation by Asimov and imagining a science of psychohistory or similar mathemagical sciences is just plain cool!

Oh...I didn't mention the overlap with ethnographies and case studies. Well, someone else can handle those sides of the triangle. See you Thursday!