ENG 602 (01): Quantitative Research in Writing

Meetings: W4-630

Instructor: Professor Stephen A. Bernhardt
Office: 222 English
Office Phone: 646-2027
Email sbernhar@nmsu.edu
Office Hours: Monday and Wednesday, 130-230 and by appointment

Textbooks required:

Babbie, Earl. The Practice of Social Research, 7th Edition. Belmont, CA: Wadsworth, 1995.

Hayes, John R., Richard Young, et al. Reading Empirical Research Studies: The Rhetoric of Research. Mahwah, NJ: Lawrence Erlbaum Associates, 1992.

Lauer, Janice M., and J. William Asher. Composition Research: Empirical Designs. New York: Oxford Press, 1988.

Phillips, John L., Jr. How to Think about Statistics, 5th Edition. NY: Freeman, 1996.

TCQ Special Issue on Research Methodology in Technical Communication, Patricia Goubil-Gambrell, Editor, Winter 1998. (Bernhardt has copies).

Recommended Readings on Issues with Quantitative Methods in Technical Communciation and Composition

Course Description: This course concentrates on issues in designing and conducting empirical research, particularly studies driven by data. It is intended for students contemplating dissertation work. The emphasis is first on coming to terms with what topics or issues are worth investigating and then how to frame well defined research questions. Questions of appropriate method are seen to derive from the topic and questions identified, as well as emerging in the conduct of inquiry. This course can usefully complement a series of courses that includes English 548 (Research in Written Composition) and 601 (Qualitative Research). The course satisfies the PhD program requirement for a course in statistics and data analysis.

The course will cover standard research designs, terms, and concepts. It will help students develop statistical literacy: the ability to read and critique quantitative studies. The course should be useful whether or not a student decides to pursue quantitative studies in dissertation research. Models we will investigate include surveys, true and quasi experiments, program evaluations, and usability testing.

I envision a course that involves students in reviewing the literature on various topics, mapping studies of their own, and reading and critiquing classic studies in composition and professional communication.

Our books include chapters on case studies, ethnographies, field research, and various forms of naturalistic and phenomenological research, all of which is very important to gain a full picture. The Special Issue of TCQ is in fact mostly taken up with cultural, historical approaches, as opposed to quantitative, experimental approaches, but I wanted you to have this issue so you would have a good sense of where the field is right now. In this course, we will concentrate on the quantitative issues; however, it is impossible to consider research methods in isolation, especially since the dominant paradigms in the field tend to be qualitative or blended approaches. I urge you to read the chapters that are not assigned to fill out your background in qualitative methods.

Required Assignments:

• Comparative study review and critique: The purpose of this assignment is to take a first pass at reading and understanding study designs, while using the language of research design and data analysis in applied ways. Read and review two or three studies of how readers read online texts or work in online environments. The studies can be by the same researcher if you like. Compare the kinds of questions the researcher asks and the kinds of methods used to address the questions. Assess the experimental designs—the definition of constructs, the selection and grouping of subjects, the nature of the treatments or tasks performed by the subject, the kinds of data gathered and/or the measures of performance, the handling of the data. Assess the validity, reliability, and usefulness of the results. (Due September 8).
• Draft research proposal (nutshell): The purpose of this assignment is to take a first pass at mapping out a quantitative study of written composition, working at a very general conceptual level. In no more than 2 pages, pose a research question, a statement of significance, definitions for key constructs, a research design, a description of the data to be gathered, a description of the data analysis, and a list of unresolved questions, issues, or difficulties in your proposal. (Due September 22).
• Literature review: The purpose of this assignment is to gain familiarity with relevant published works related to your research proposal, and in doing so, to see whether there has been a coherent program of research that can help you further conceptualize and rationalize your study. The literature reviewed should establish the theoretical significance of the problem and critically review related studies for both findings and methods. It should end with an assessment of what the current state of knowledge is and why the study you propose is well justified and needed. 5-10 pages, 10-20 sources expected, depending on the topic and the extant literature. (Draft Due October 13; revisions October 27).
• Usability study critique: The purpose of this assignment is to allow us to compare different approaches to evaluating the usability of various paper and online documents, working from the assumption that quantitative methods are particularly useful for establishing how well documents meet readers’ purposes. Critique a single, quantitative, usability study—a test of how well a document, help system, manual, tutorial, website, or other text works for a defined group of readers or users. Pay particular attention to how the researcher selects the text for evaluation, designs the usability study, and reports the quantitative data. (Due Nov 3)
• Full research proposal: The purpose of this assignment is to push you to define in detail a research project you could carry out that would lead to publishable scholarship. Expand the earlier draft proposal, incorporate an abbreviated literature review (as would be appropriate to a journal publication), and give full details of methods, including a description of the data to be gathered and your approach to its analysis. Consider the need for a pilot study or some method of testing your instruments before running the study. (Due Nov. 10)
• Program evaluation or writing assessment critique: The purpose of this assignment is to consider the issues involved in attempting to assess the quality or effectiveness of a writing program or course (academic or workplace). Choose a single study that uses quantitative assessment to assess writing improvement (primary trait, portfolio evaluation, holistic scoring) or to assess a program (outcomes assessment, program evaluation study). Evaluate in particular the validity and reliability issues of writing assessment. (Due December 1).

Reading Schedule

I’d like to consider this list open to change as we go along. It is very hard to predict a coherent sequence of readings at the beginning. In a course like this, where I am asking you to develop your own study design, you almost need to know everything right at the start. So it might be best to peruse all the readings first and spot helpful material and studies that suggest ideas to you for your own design. I am also asking you in your assignments to read various studies and critique them or use them in your lit review, so there is additional reading and some of what you find might be worth sharing with everyone. I don’t expect you to read everything here with the same level of care and attention—be alert as a reader and move quickly through some of this (like the longer passages of Babbie) and more slowly where you are working analytically and critically to master certain concepts or understand and appreciate an elegant design.

Sept 1

Hayes, Ch 14, Redefining Revision for Freshmen, pp 349-370,
Hayes, Ch 15, Cooperative Writing Direct Effects and Transfer, pp. 371-384.
Schriver, in pack, Connecting Cognition and Context in Composition

Sept 8

Babbie, Part 1, An Introduction to Inquiry, pp. 14-79
Lauer, Introduction, pp. 3-23

Charney, in TCQ special issue, From Logocentrism to Ethnocentrism: Historicizing Critiques of Writing Research

Charney, coursepack, Empiricism Is Not a Four Letter Word

Sept 15

Hayes, Introduction and Reading Research Reports, pp. 1-42
Hayes, Ch 3, What Happens When Our Students Read and What Can We Do about It?, pp. 43-74.
Hayes, Ch 17, The Effects of Observational Activities on Student Writing, pp 409-432

Sept 22

Babbie, Part 2, The Structuring of Inquiry, pp. 80-229

Hayes, Ch 9, The Frequency and Placement of Topic Sentences in Expository Prose, pp. 211-228

Hayes, Ch 10, The Frequency and Placement of Main Ideas in Children’s Social Studies Textbooks: A Modified Replication of Braddock’s Research on Topic Sentences, pp 229-246.

Sept 29

Bernhardt at Conference of the Americas in Puebla

Phillips, Introduction, Chapters 1, 2, 3, 4: Meet as a class, talk over, render a list of unresolved issues and questions for Bernhardt to take up.

Oct 6

Hayes, Ch 5, This Was an Easy Assignment: Examining How Students Interpret Academic Writing Tasks, pp. 135-154.

Lauer, Ch 7, Measurement

Lauer, Ch 8, True Experiments

Lauer, Ch 9, Quasi Experiments

Babbie, Ch 9, Experiments

Oct 13

Lauer, Ch 4, Sampling and Surveys, pp 54-81
Lauer, Ch 5, Quantitative Descriptive Studies, pp. 82-108
Babbie, Ch 10, Survey Research, pp. 255-278.
Miller and Faigley, coursepack, Writing on the Job

Oct 20

Lauer, Ch 6, Prediction and Classification Studies
Phillips, Ch 5, Interpreting Individual Measures
Phillips, Ch 6, Correlation
Hayes, Ch 11, Expertise in News Writing
Hayes, Ch 18, Investigating the Control of Writing Skills, pp 443 ff.

Oct 27

Hayes, Ch 12, The Importance of Teacher Knowledge in College Composition Placement Testing, pp. 287-316.
Phillips, Ch 7 Description to Inference
Phillips, Ch 8, Precision of Inference
Phillips, Ch 9, Significance of a Difference Between Two Means

Nov 3

Hayes, Ch 7, Designing for the Real World: Using Research to Turn a "Target Audience" into Real People, pp. 155-176
Phillips, 10, More on the Testing of Hypotheses

Phillips, Ch 11, Correlation, Causation, and Effect Size

Nov 10

Babbie, Ch 14, Quantifying Data
Babbie, Ch 15, Elementary Analyses
Bernhardt, coursepack, Teaching Composition with Computers, Program Evaluation Study and Timed Observation Study

Bernhardt, coursepack, Text Revisions by Basic Writers

Nov 17

Babbie, Ch 16, The Elaboration Model
Babbie, Ch 13, Evaluation Research
Hayes, Ch 13, How Characteristics of Student Essays Influence Teachers’ Evaluations and Why Do Teachers Give the Grades They Do?, pp. 317-348

Dec 1

Babbie, Unobtrusive Research, pp 305-336.
Babbie, Ch 17, Social Statistics
Hayes, Ch 19, Discourse Synthesis: Creating Texts from Texts, 467-512

Dec 8

Blyler, in TCQ special issue, Taking a Political Turn: The Critical Perspective and Research in Professional Communication.
Business arising . . . and leftovers!

Dec 13-17 Finals week