meta: takin' it to the next level, baby!

I immediately wondered, "How do you pick your studies?" and in no time, the task seemed as daunting as any experimental study. Hillocks reviewed about 500 and chose only 60. Wow. Even if five to ten studies can be worthwhile for meta-analysis, a meta-analysis's literature review puts our measly annotated bib to shame. It's interesting that the effect size for a study, the impact on the original study's dependent variable, now becomes the criterion variable. The actual methods used in the original studies disappear after the choice to use the studies is made. The criteria for selecting studies seem rather arbitrary, and avoiding bias in this selection might be the biggest challenge in meta-analysis. Once criteria (favored methods) for choosing studies are established, that bias is furthered by then seeking homogeneity. Although I can't argue with the logic behind homogeneity--that correctly performed studies should yield similar results--I cannot help but feel sorry for studies designed with new ways to test similar variables. Meta-analysis seems to dismiss any research that goes against the grain. Only meta-analyses can compete with meta-analyses, and the results of meta-analysis may be given too much clout.

Maybe I'm just paranoid. If the results usually say one thing, let's see how strongly they say it using meta-analysis!

mostly off-topic title: elephant talk

L&A: Experimenting on students sounds kind of odd and vaguely wrong, but as a new teacher (2nd year), I feel like I am eager to try lots of new strategies in my classroom with only descriptive evidence and some outside research to guide my decisions. The idea of teaching a whole course with only one experimental variable different between control and experimental classes sounds absurd to someone whose curriculum and strategies are in constant revision. Of course, not all experiments are like the one I describe or the sentence combining example from L&A. I know that they used two instructors each teaching a control and a variable group, but didn’t these instructors see the effects of sentence combining and feel tempted to change other, controlled aspects of the experimental section’s course? It seems like a long-term experiment could face more problems from interactions. Another thought about O’Hare’s study: What are the 84 observation units mentioned at the top of page 157? Finally, I hope others were as relieved as I when L&A brushed aside most of the threats to validity by citing randomization. What about when randomization just isn’t possible? I’m looking forward to quasi-experiments; will that tell us?

We learned from Brodkey the difficulty of interpreting and analyzing conversation; the problems that arise in interpreting talk are much like those in writing. Like my classmates, I was intrigued by the "triplet" nature of those "tacit rules" in turn-taking that govern teacher-student talk. I would argue that I often ask students to ask a question about a specific subject, so the process is a little different. In any case, teachers create systems of conversation that students generally choose or learn to follow. I also sometimes feel like my conversations with students in my speech class follow the concern raised by Freedman and Katz. When I know a topic well, the conversation is different than if I am helping a student write an informative speech about which I know relatively little (I'm happy to say though that my kids usually are surprised to find that I know a lot more about a lot more topics than they do....hehe). Page 114 brings up another good point: Sometimes the best conversations end in disagreement or tension. Strong students thrive on this stuff when they feel safe. On the same page, the "functional taxonomy" of utterances reminded me of the challenge of coding questioning types in my observation for this class. To conclude, I want to ask, did I miss something? Wouldn't we benefit from a bit about identifying the characteristics of talk that produce successful outcomes? That would be really interesting!

Mathemagical Voodoo and protection charms

As we move toward the experimental side of research, the limitations of quantitative designs are what I find most interesting. Researchers’ number fetishes can produce grotesque misrepresentations and allow them to overlook obvious details and mechanisms. I really like the tease that L&A give on page 99, when they point out that “one reason why variance is not accounted for by certain measures is that they themselves are not highly reliable.” Coding is central to quantifying results, but meaningful reliability with some variable seems impossible. Diedrich’s study on 300 first month college papers included a judgment on the “flavor” of the writing. I could hardly believe what I was reading. Chocolate, caramel, vanilla—what? Beach reminds us that “there is no widely agreed-upon taxonomy regarding different writing types” (221), and I wonder if we wouldn’t be well served by a widely used handbook for coding compositions. Of course, many assumptions about writing would underpin such a work, but the alternatives suck.

Pianco’s study struck me for its limitations. She did tons of work to find a few statistically significant relationships, and then she basically used qualitative data on those significant categories to conclude that remedial writers “glanced around the room during their pauses, sometimes as a diversion, at other times as a way of finding the correct word, or something to say next,” among other broad and intuitive findings. Even comparing remedial and traditional students is as sketchy as comparing flavors of writing. Focusing on remedial students ignores their unique backgrounds, and analysis of flavors could prejudice backgrounds that produce particular flavors of writing. A “small part” of Pamela Eckert’s study used what I see as a possibly unethical pigeonholing of students into groups like “jocks” and “burnouts” when she used them categorically for empirical analysis (Beach 220). “Hey kid, you’re a burnout, right?” Or “Hey, would you say your buddy is more of a burnout or a jock?” However she identified kids, it rubs me the wrong way.

I also want to give some props to Beach for his “Limitations...” section. I too share Beaugrande’s general doubt about “controls” that “make the context dissimilar to ordinary language activity” and about the assumption that “artificial contexts” can generalize to “natural contexts” (235). The maturation and drop-out effects were given in nice, simple reminders (233), and the limitations of holistic ratings (236) were especially insightful—basically we should be careful to work quickly to see relationships between variables, and features make often make better variables than classifications.

In addition to the interesting example cases in both texts, the guidelines for designing research were pretty insightful. Lauer and Asher readily admit that the “distinction between [independent and dependent variables] is rather imprecise in descriptive research” (86), and Beach explains that his own journaling study did not consider “the effects of using the dialogue-journal writing” on any out-come variables” (225). Clearly journaling is a/effecting “the level of partners’ interpersonal involvement,” but researchers can claim to give neither variable priority as they please (this is sort of a question too).?

The Lauer and Asher section on analysis of variance was cool too. Of course, the main idea is to make sure that variables are not interacting so that we can make predictions and generalize.
However, I don’t exactly understand the limitations of ANOVA. The illustrations suggest that specific interactions can be identified. How does this differ from correlation? I get sci-fi chills to think of mapping a multivariate study of thousands of individuals and using computers to account for all the interactions taking place, then using an individual or group’s known characteristics to make predictions. THIS is why so many have a penchant for numbers. Reading Foundation by Asimov and imagining a science of psychohistory or similar mathemagical sciences is just plain cool!

Oh...I didn't mention the overlap with ethnographies and case studies. Well, someone else can handle those sides of the triangle. See you Thursday!

no title--surveys and pluralism

The Lauer and Asher text didn't say much that an intro. to statistics course doesn't say, but it was a nice concise way to organize the statistical side of surveying. I would definitely refer back to this chapter before going into any survey research, but I'd hit up a stats textbook and a knowledgeable colleague too. I had one question about table 4-1 on page 58. For the Texas researchers' example on page 59, wouldn't we be able to plug in 42% and 48% as p and q to find the confidence interval? Should we always use the "worst case" scenario of a 50% p value? If we would plug in the actual p and q values, does this eliminate the need to use the corrections in table 4-3? I am feeling sick right now, and have been taking day/nyquil for two days, but this chapter wasn't the easiest to figure out. When L&A say that "for the Texas research, the use of the correction Table 4-2 was unnecessary, because the sample size did not approach the population," I wonder how big the population of writing directors actually is compared to the 127 who completed the survey. I'm guess there are over 2,540 college writing program directors in the country, so they are under the 5% required for a correction, but couldn't L&A comment on this for clarity?

I was disappointed to not read much about creating unbiased questions. L&A basically referred us to the references and gave us a list of problems that can befall writers of questionaires. I should interject here that I personally hate questionaires. I always try to cheat at least a little bit on surveys to skew the researchers' findings. At West Perry, where I teach, we have to survey our students each semester to comply with a technology grant we received as a school. The kids ask me, "What can I put down so we get better computers?" Of course, I don't know exactly what criteria the state uses, but my point is that almost everyone is savvy and skeptical about surveying.

Kirsch's text is dense! I wish I were feeling more physically up to the challenge today **cough cough, puppy dog face. I'll just dig into a couple of the things that caught my eye.
Kirsch questions Berieiter and Scardamalia's statement "that 'a holistic ideology ... poses an actual threat to writing research [because of its] opposition to any research (or instruction) that deals with less than the full act of writing carried out under natural conditions'" (250). I have to agree with Kirsch here. Experimental research is limited in the types of questions it can answer and could potentially produce a type of writing founded in and limited by experimental research. Holistic approaches to composition are not necessarily less valid just because they are difficult to experiment with.

After Kirsch discusses the arguments for and against a single methodology in composition studies, she states, "The question, then, is whether scholars are willing to break from a relatively rigid adherence to their disciplinary orientation in order to entertain alternative methodologies" (256). This quote made me think of the common secondary-ed imperative that "all teachers teach writing." The parallel is far from perfect, but an important point is revealed in the comparison. The writing demands of each discipline are different, and all disciplines can contribute to composition studies. As far as whether we should have a pluralist methodology for composition studies, Kirsch's arguments for cultural equality (251) and Newkirk's reminder of the reality of research processes used in the field today make a pretty strong case for pluralism. To me the "question is" whether researchers should or must take a pluralistic route and become jacks of all trades, or whether researchers can share the effort--playing nice--as a community.

I really liked the Kirsch chapter's change in direction to an example of a forward thinking piece of research that involved the input of subjects as participants with clearly articulated roles in the research. The purpose of composition studies is to produce better writers, right? How can anyone oppose including the subjects? What Kirsch's chapter highlights most for me is that methodological pluralism is the only way to seek a body of knowledge about composition that is fair to diverse subjects (I mean "people," but subjects as disciplines applies as well).

I'm pooped. Ttfn.

Triangulation? Try strangulation... : - )

Great quote: "Hymes (1980) states that the validity of an ethnography depends on an 'accurate knowledge of the meanings and behaviors of those who participate in them" (quoted in LA 41). Maybe if Hymes removed “accurate,” the focus would shift to the types of knowledge on which validity depends, but my first thought was, “Wow. What a great circular explanation.” The phenomenological focus is probably what Hymes meant to describe, but the quote still made me laugh. The catalogue of challenges faced by ethnographers who want “accurate knowledge” was probably the most interesting part of the chapter for me. I was shocked to see that many uneducated people see causal relationships in single co-occurences, and the warning about our tendency to focus on novel data was another great reminder. For me, this was a novel way to look at an obvious truth: Redundant data (though it is often overlooked) is the key to induction.
The concept of triangulation is the most important characteristic of ethnography, so I guess I should comment on that. Composing an ethnography is a daunting task to consider because of the multiplicity of data sources and the biases and relationships that must be discussed by the researchers. This would take a lot of writing! I noticed at least one reference to the scarcity of publication real-estate available to thorough qualitative studies (43).On page 41, L&A mention that Lemke and Bridwell turned a program evaluation into an ethnography, so non-academic audiences may be a way to support cost of publishing thorough ethnographic research. Ethnography seems to be really useful for a report to present to a school board. A school board and school administration would be interested in an ethnographic form of research because it is personalized yet thorough, in contrast to scientifically generalizable research methods or case studies. Chris, I see you're thinking the same thing by pointing out the problem of 90-90-90 schools being treated as the perfectly generic population.
Moss’s narrative gave good practical examples of Lauer and Asher’s prescriptions for good ethnography. In her post, Chris mentioned the catch-22 of insider knowledge, and I wanted to put my take on it in different words. Moss titles the section “Conducting and Ethnography: Making the Familiar Strange,” and it seems odd to me that the ethnographer who is native to the community under study must make the familiar strange to allow new perspectives, but he or she must also not make the familiar appear strange, less the informants feel uncomfortable working with a stranger.
One aspect of ethnography research that L&A did not cover was Moss’s position that the researcher must be “responsible to the community,” (169) but this statement raised my eyebrows and I feel deserved a little more discussion. Moss “did not want anyone in the community to be dissatisfied with what [she] had written,” but this seems like an impossibility. How do we decide when tough conclusions benefit the community more than those that avoid controversy? In a school district for example, would pointing to widespread parental distrust of the educational establishment be so distasteful to the community that it would put a (triangular) wedge between the community, the researchers, and the those who would utilize the research? If a charismatic and popular community leader seems to be able to smooth out tensions, in say a church community, would that give an ethnographer more license to reveal faults or “hard truths.” I know that data collection is controlled by an IRB, but the ethics of analysis and interpretation are another matter, right? Fortunately, ethnographers generally undertake their research to raise understanding and awareness about the communities they describe—or so we hope.

a case of the blues

Jeez. Lauer and Asher made the writing of acceptable case studies seem hopelessly complicated to perform so as to achieve some validity, and for me, Newkirk's narrative was a major downer. I mostly side with Newkirk's "tell stories anyway!" attitude, in part because I see a response to his "lame" (130) justification of focusing on one, possibly exceptional case.

Lauer and Asher identify case studies as a preparatory activity not attempting "to establish cause and effect relationships," but to help "the researcher to identify new variables and questions." They seem to discount their usefulness, but at the same time call for an almost wasted rigor through coding and statistical analysis of coding consistency. Newkirk points out that in his experience, positivist researchers use more experimentally designed research—not case studies—to choose hypothesis and questions. I got to wondering why we should go to all the trouble of improving the case study if it never attains credibility.

Although he doesn't spend too much time theorizing it, Newkirk does point out a light at the end of the tunnel. The superiority of experimental or more empirical research is somewhat refuted by Newkirk when he reminds us to make our subjects human and to remember that variables are "transacted" (131). I think that references to differentiation in pedagogy would be helpful to his case. If we use large-sample data about writers in general, we still must teach individual students. If a student seems to fit a case, we can possibly use that case study to help an individual student. The farther students deviate from the norm, the more we can maybe make a case for case studies. Researchers could try to create subgroups to research experimentally and then match students to their subgroups, but no group will ever represent an individual, and case studies are an efficient way to "read" our students.

I also agreed with less enthusiasm about Newkirk's position on the potential for case studies to find new approaches. Newkirk reasons that because case studies "draw their authority from … enduring narratives that writers borrow to shape their accounts," they are "profoundly conservative and conventional" (136). We will see things as we want to see them, and only as they have been "archetypally" seen before. To structure their narratives and suggest directions for practice, even reform-minded case study authors use deeply rooted cultural myths (true or untrue)--for example, the victim-victimizer relationship in the education of minorities (146-147). Unfortunately, his analysis of case study writers does little to validate the rhetorical claims they make. Newkirk reminds us to question the validity and bias of case studies while applauding the storytelling form for having its own form of realism.

interesting and thorough

Broad ideas:
This article is so descriptive about the different approaches to observation that it left me spinning a little bit, not knowing which direction I should go. All I really want to say is wow, but I’ll elaborate. Observation is so familiar that it is essentially how we all learn “the kind of ‘common sense’ or ‘cultural knowledge’ that Johnson (1975, p. 21) has argued lies at the base of all human knowledge” (Adler 377). It is very “natural” for us to observe people, to describe what we see, and to generalize; however, I don’t really like the term “naturalistic” for qualitative research. Quantitative research seems to support a law-based view of reality equally well.

Methods Issues:
And while I’m quibbling over vocabulary, what of these terms, dramaturgical and ethnographical (not to mention the usual redundant ‘al’ endings)? Does anyone really focus only on people as individual actors outside of the systems in which they act, or would that be a misrepresentation of dramaturgical observation? I know when I have observed classrooms in the past that I have scrambled to evaluate individuals as they do things that I deem significant, and in the downtime, I try to describe the setting and structures that govern the environment. Goffman seems to only care about social interactions, and this gives me the idea that dramaturgical perspective seems like it may be valuable to answering only certain questions in composition—maybe those about writing as a performance or peer responses.

The spectrum ranging from detached observer to interactive or full-participant observers was worth reading as well. What interested me here was the comment about “practitioners’ attitudes shifting toward greater involvement ... in their settings” (379). I commented, “WOW,” in my PDF document when I read that the Adler kiddos posed as druggies—what a trip. I felt like I was watching a crime drama or documentary on TV with gang infiltration. Too bad it’s often unethical, or even impossible because of authenticity, to become a full member of the group a researcher observes.

I really liked the comments on the various forms that observation artifacts take. Videos, audio recordings, notes on structured sheets with graphic depictions of the setting, and Denzin’s mandated features (380) all need to be considered when entering observations. Regarding the advice on p. 381, observers would be a little naive to think that beginning with a descriptive approach and moving to more “focused observations” will greatly improve objectivity, but at least it will lend some credibility to the researcher’s methods. A more challenging idea, “theoretical saturation ... when the generic features of their new findings consistently replicate earlier ones” (381). This seems like a difficult proof to establish if one is seeking “verification.”

So....Verisimilitude? vraisemblance? Writing skill is crucial here, but here’s a tangential question: Doesn't our reading of fiction affect our expectations for verisimilitude in writing? We read “realistic,” or “believable,” or “catalogued” details in fiction and expect (and meet expectations for) similar details in observations.

Problems:
Of course we should try to have diversity in our teams of observers, but it needs to be said. I wonder if they privileged gender diversity’s importance over diversity in race or age because somehow it’s more politically correct to say that gender difference is natural, rather than socially constructed. This brings me to another type of diversity—that of membership. Could we add authentic meta-observation or “auto-observation” as a late stage of observation so that we move from the outside with description down to an focused, insider style of observation?

Last comment:
I liked the idea that observers can be discoverers of new social situations, often naming their discoveries as they identify new roles. I don’t call plan to call anyone the “inserter,” “voyeur,” or “waiter” in my own observations, but I like naming stuff.

Two Narratives and a Bore

Alvarez: Despite the intrusive transcription errors, I did enjoy this article. The narrative presentation was refreshing, and the charts made note-taking a breeze. One of the most unreadable yet interesting moments in our version of the text was an idea that Alvarez teased and dropped: “Yet, while we consider the effects of our students’ literacy habits and preparation have upon their learning and ability we teach them, we still have to examine the values and literacy habits which guide our selection, organization and assessment of what we are teaching ...”. (68? .pdf-1). I do see that the focus of the article is “how to begin teacher research,” but the pre-colonic title phrase is maybe a little too personal to Alvarez and has too little to do with the “rhetorical” purpose of her article. Maybe it’s too much to ask that she include discussion of the specific habits of her students, but I think we all would have savored even a small taste of how students’ specific literacy habits led to the their appreciation of a new model for teaching ELA—a question she no doubt has considered.

Another key to Alvarez’s article is that “the first research project a teacher engages in will change the research habits he/she teaches to the students” (70? .pdf-3). She later rejects “the outline, note card, static, positivistic format we are all so familiar with,” and her phrase led me to do a little light reading about positivism. I would be grateful and much impressed if someone can explain how this philosophy/epistemology fits in with the traditional high school process of looking up existing information, organizing it, and presenting it. What happened to “empirical designs”? Did anyone else take from Alvarez’s proposal of a “dynamic, fluid” research method that she also supports an equally fluid and unstable understanding of the results of research in general? Sort of a “Go research. Then do what works for you” attitude?
One last critique of the charts: What’s up with “Step # 4” falling into both “phases” of the research process? Isn’t Phase 2 the implementation phase?

Lauer & Asher: The first few pages of the introduction didn’t really blow my mind. So rhetorical inquiry is deductive and usually turns to other disciplines for analogy before reasoning out a theory; ok. As usual, the language caught my attention. I noticed the differences in the use of rhetorical from Lauer’s argumentative, or convincing along a continuum to persuasive or inspiring (in Alvarez: ”...the teacher research project becomes a rhetorical act, requiring us to do more than research” [72? .pdf-5]). Oddly enough, this continuum for which I have provided a tiny, illustrative taxonomy, are oriented with the beginning and end of a purposeful composition. :-) COOL.

Here’s a question for page 7 of Lauer: Is simply utilizing prior rhetorical inquiry to support empirical inquiry still considered to be multimodal? Let’s not feel pressured to create new rhetoric when empirical studies are in high demand by virtue of induction’s fallibility (haha at least it keeps everyone busy)! It seems to me that repeated empirical tests and reports on the outcomes of a previously rhetorically proposed practice should be a constant task in composition studies or any similar field. We deal with lots of continuums in this chapter: conceptual to operational, descriptive to experimental, even the balance between pure rhetoric and pure empirical inquiry.

Here’s a definition question: “Constructs” seem to be different than the 400 or so “dimensions of human behavior” mentioned in the “General Principals...” section? What are examples of these 400? Also, prewriting as they define it seems to be a pretty narrow “construct,” and I can’t wait to start defining my terms before researching—what a task! I know that the basic essay writing model we use ignores Lauer’s example researchers’ definitions and treats planning as the prewriting itself.

Ede: Lisa Ede has an interesting life story in the literary world. She fell into composition studies and inquired blindly as she unknowingly spread her New Critical herpes into the new discipline—how awful. The article finally got interesting when she took on North and explains that lore is an untenable replacement for inquiry (320-321), yet practitioners are threatened by theory (321). That daily teaching practices may need to govern theory (322) is “action research” in a nutshell. I think one of the best examples of the importance of methodological openness is the controversial case of collaborative writing where deep-rooted biases and distrusts about authorship bring methodology more into the open. This section made me wonder if as practitioners we should be interesting our students (they are already involved) in the theoretical issues at stake in their writing. Should I open up a can of worms with ninth graders? In example: “Johnny, do you see how this method of prewriting and planning limits you to biases?” Would even a twelfth grader step up to bat when Nolan Barthes is throwing heat high and inside with “How is it possible to write?” Ede ends by giving one last shout out to her fem-idol, Flax, and hesitatingly accepts all methodologies of inquiry. I end by suggesting we call composition studies less a discipline than a community and sing the chorus of War’s “Why Can’t We be Friends?”