Hi Meg,

Thanks for writing.

We start by thinking about what a teacher's understanding of her class would be if she were to give a score (0-2) for any area (claims, reasoning, etc), then consider what experiences and information students would need to get in order to support or scaffold their understanding of that particular dimension. We then do two things: We go to the literature and find out what, if anything, is out there that would help us to forma direction to take in providing support and secondly we go to our own, robust curricular work from at least a decade of working on and refining lessons about argumentation and explanation.   We then work to create lessons with this combination of information. 

Megan

Meg, I forgot to answer your second question! We are currently piloting both the app and the RMLs in 5 classrooms and look forward to hearing what the teachers think about each aspect.  We also held two focus groups with teachers in February, where we showed them both the tool and the RMLs and asked for feedback. At the time, teachers had never actually taught them so their initial ideas were cursory but in general they all liked the idea that there were many options for teachers to choose from and the fact that the RMLs were so pointed and focused on particular aspects of oral argumentation. 

Hi Rachel, that's a great question. Our current focus for the RMLs is supporting NGSS practices through content-embedded learning experiences--but we've been thinking about it in terms of what is foregrounded/backgrounded versus what is present/absent. In developing the tool and RMLs, we're trying to strike a balance between being contextualized (so that the practices aren't divorced from the science content that makes engagement in the practice meaningful) and being somewhat content-agnostic (so that teachers can use the RMLs in conjunction with whatever content they are teaching). The nice thing about this project is that we get to spend the first phase of the study piloting and revising the tool and the RMLs to get this balance right. Our expectation is that the curriculum used in conjunction with this tool and RMLs will include additional supports for content learning. For example, for the RCT phase of the study, we are using the tool in conjunction with Amplify Science, which includes  formative assessments, and suggestions for instructional adjustments, that foreground the DCIs as students engage in oral argumentation. 

Megan and Eric, that does make sense. We also started our project in a content-agnostic way by first focusing on the SEPs. But as we focused more on argumentation in our formative assessment work we found that we saw some content misconceptions emerging and students providing evidence for an incorrect claim. That's where it becomes difficult to get the balance right. I am also wondering what your work with teachers on this was like. Did you do anything with calibrating among your group of teachers? We have found that "consensus discussions" among teachers are very revealing, and I would think that as you scale this up you might want to get convergence on what the levels mean. Or maybe that does not matter? Anyway, I would love to have your take on our project video as there is some intersection in what we are looking at. 

Thanks, Rachel--the "consensus discussions" you describe are very much in line with efforts Bryan Henderson at ASU engaged in with the tool's initial development, and we're currently engaging pilot teachers in something similar (both as part of the PD to use the tool, and as part of the practitioner review/feedback cycles we'll be engaged in as we iterate the tool for the RCTs in the next phase. Along those lines, and coming from a formative assessment background, the learning you describe (for teachers, and for us as researchers) really resonates--I'm reminded of Beverly Falk and Suzanna Ort's, Sitting Down to Score (1997). 

And I will definitely take a look at your project video!

Great! I look forward to your feedback. Thanks for the connections, too.

Thanks, Steven! Development of the tool itself was guided by a body of literature similar to those you cite, in particular, we drew on the work of Michaels, O’Connor, and Resnick (2008) on Accountable Talk. The Michaels et al. (2008) notions of Accountability to the Standards of Reasoning and Accountability to Knowledge guided the assessment of argument products. That is, to evaluate the substantive content of oral arguments, we created items measuring the degree to which students were accountable for both the logical requirements of a valid argument, in addition to the scientific accuracy and relevance of their utterances. As for assessment of argumentation processes, items were guided by the Michaels et al. notion of Accountability to the Learning Community, which emphasizes respect for, and critical attention to, the contributions of others so that ideas can be built upon one another.

In thinking about how the tool will advance PCK in particular, we drew heavily on the formative assessment literature, recognizing that supporting formative assessment of oral argumentation provides particular leverage for improving science teaching and learning. The effort to design a teacher tool that facilitates the use and usefulness of formative assessment is motivated by the well-documented positive impacts of formative assessment on student learning (Black, Harrison, Lee, Marshall, & Wiliam, 2004; Black & Wiliam, 1998; Fuchs, Fuchs, Hamlett, & Stecker, 1991; Hattie, 2009; Marzano, 2004; McTighe & Brown, 2005; and many others!). In particular, we're proposing that providing teachers a means of formatively assessing science discourse can provide a shift from an authoritative classroom discourse (Nystrand, et al., 1997), where the teacher presents science from a place of authority, to a more dialogic (Chinn, O’Donnell, & Jinks, 2000; Smart & Marshall, 2012) discourse that allows students to play a more active role in the direction and goals of the discussion. This expectation is consistent with the pedagogical shifts associated with formative assessment, including a shift toward a more constructivist learning model and one in which students play a more active role in their own learning (Black & Wiliam, 1998; Perrenoud, 1991; Shepard, 2000; Torrance & Pryor, 2001; Turnstall & Gipps, 1996; Whiting, Van Burgh, & Render, 1995, April). Such shifts suggest that tools to support formative assessment are likely to impact teacher practice and professional knowledge as well. In fact, improvements in teacher PCK and Pedagogical Learner Knowledge (PLK) (Grimmett & MacKinnon, 1992) associated with formative assessment may be core to the mechanism through which student effects are realized (here, we're building on work such as: Darling-Hammond, Ancess, & Falk, 1995; Falk & Ort, 1998; Franke, Carpenter, Levi & Fennema, 2001; Goldberg & Roswell, 2000; Kornhaber & Gardner, 1993; Puttick & Rosebery, 1998; Whiting, Van Burgh, & Render, 1995). 

Hi Jennifer, building on Megan's comment, a big part of the current phase of the study is to get teacher feedback and suggestions for how to make the tool as user-friendly as possible. While the iPad version is the current prototype, we definitely intend to create the tool so that it is platform-agnostic (likely, web-based and operable through any browser), once we've had a chance to iterate a bit on the design. 

oh, now that's a fun idea!

one thing we've struggled with for the current research effort is the technical difficulty of accurately transcribing (or even recording with fidelity) the audio of class-level student discourse. We have a nice set of classroom videos that we use to train scorers (and some that we use as part of the PD with teachers who are using the tool), but we have found audio files alone difficult to work with. I'd certainly love any advice you might have on capturing multi-voiced audio!

Yes to scaling up! We will be researching the tool in relation to its use as part of the Amplify Science middle school curriculum, which includes argumentation-rich "Science Seminars" (modeled after the Socratic seminar) in each unit. The ultimate goal is to develop the tool for wide-scale use, and a big part of our initial work is working with teachers to make it as useful and useable as possible. 

Our current study will feature an RCT covering a few months of instructional use of the tool, for which we will be examining things like changes in teacher PCK in relation to using the tool (and the Responsive Mini-Lessons meant to accompany the tool), but a more longitudinal study is outside of the current research plan. That would be super interesting, though, and I'd love to think about ways to make that happen with the next study of DiALoG!

that looks like a super cool tool--thanks for the tip!