Quality Criteria: Example 1

Exploring the quality criteria used by Camtree through examples from the library. 
This report, by Rachel West, discusses the role of dialogic practices in promoting teacher learning. As you read the report, you can see how Camtree’s quality criteria are addressed. You can read the full, published, report (which includes the appendices and references) in the Camtree Digital Library, here: https://hdl.handle.net/20.500.14069/41.

First of all, let’s look at the title and abstract that Rachel wrote to accompany the paper and the keywords she selected to describe it. These are important because they allow readers to assess the relevance of the report to their own learning or practice, and allow the report to be ‘indexed’ in library catalogues.

How might explicitly increasing the use of dialogic approaches affect whole-group dialogue in virtual teacher professional development sessions run for an international examination board?

A clear title summarising the main question that Rachel set out to answer.

Abstract

Background and purpose: This inquiry aims to explore what happens when trainers running virtual professional development (PD) sessions for teachers use a more dialogic approach to whole-group debrief sessions by setting up ground rules and introducing a repertoire of “talk moves”. The continuing professional development sessions focused on a specific curriculum subject and were designed for teachers new to teaching it. The PD sessions covered the curriculum framework, learning objectives and how to use the resources to teach the subject effectively.

Clear statement of purpose and context.

Aims: The goal was to identify opportunities in the existing PD programme where dialogic approaches could be used and to put this into practice, then to measure how the dialogue had changed using coding of whole-group dialogue based on the T-SEDA analysis framework (T-SEDA Collective 2021) as well as reflection sessions with the trainer and feedback from teachers.

Overall goals are outlined, which ground the study focus and design.

Study design or methodology: The intervention involved one trainer, who was running a virtual PD event for teachers in the USA. The PD session covered teaching and assessing a syllabus for teachers who were new to it. The baseline data was gathered from a past virtual event run by the same trainer with different teachers in the USA.

Design is briefly introduced, including who led the study and who participated.

Findings: The initial findings indicate that instances of the focus talk categories (build/elaborate, reasoning and reflect on dialogue) increased by using the tools and strategies mentioned, and the average proportion of teacher talk time, as well as average turn length and percentage of the whole group who contributed to each debrief all increased from the first iteration to the second iteration.

Findings clearly related to the stated focus (talk categories), design (two iterations) and findings.

Conclusions, originality, value and implications: This could have had a positive impact on learning for teachers in this event.

A small-scale study, so value and  implications are not overstated.

Keywords: dialogue; debriefing; teacher professional development; ground rules; talk moves; adult learners

Keywords indicate content, analytic interest, field, and context.

Now, we’ll look at each of the main sections of the report in turn to see how they relate to the Camtree quality criteria. You can navigate through the main sections of the report using the ‘tabs’. Where a number appears in the comments, a specific criterion has been addressed.

A reminder of the criteria:
  1. FOCUS OR CHALLENGE: Is there a clear outline or analysis of the focus, issue or challenge that the inquiry addresses
  2. CONTEXT OR BACKGROUND: Is some appropriate prior research evidence, data or literature used to ground the inquiry and/or interpret the outcomes?
  3. AIMS OR QUESTIONS: Are there clear aims or inquiry questions that relate to the challenge faced?
  4. APPROACH OR METHODS: Is it clear what was done to address the challenge? This would include details of any intervention or change to practice, data collection activities, who participated, and how they were chosen?
  5. GOOD RESEARCH PRACTICE: Was good research practice followed throughout the inquiry, addressing any ethical issues, safeguarding concerns, gaining informed consent for participation and use of data, and anonymisation?
  6. RESEARCH FINDINGS: Are findings reported clearly and related to the challenge, aims and inquiry questions?
  7. CLAIMS: Does the evidence presented match the strength of claims made?
  8. IMPLICATIONS: Are implications of findings suggested, even if tentative or small scale? Does the report identify what other education practitioners might find useful?
Context & Questions
Plan and Methods
Ethical Considerations
Research Findings
Evaluation & Reflection

Context

I work in the professional development (PD) team in an international education organisation which provides curricula, exams, and education programmes. We provide PD to support teachers around the world to deliver our syllabuses and curricula. I am responsible for the professional development and management of a group of about 700 trainers. For this inquiry, I worked with one experienced trainer to study a virtual training event for 16 teachers (all female) from state-run secondary schools in the USA, teaching the lower secondary age group 11-14. The teachers attending the PD sessions had a range of experience, some were new to the curriculum and some had been teaching it for some years.

1
Note on personal and organisational context, which feeds into the specific study context.

Motivation, focus and questions

I will start by outlining my motivation for this inquiry, and then go on to outline relevant aspects of dialogic theory on a classroom and then training room level.



Motivation

Our curricula are supported by the latest research findings and often include elements that may be new for teachers across the world. Some curricula include specific learning objectives for speaking and listening, which teachers say is a new challenge for them to incorporate into their lessons. One motivation for this study is our need to support teachers to deliver these and other curricula by incorporating dialogic approaches into our teacher professional development. During the pandemic, my organisation began to offer “virtual” PD on Zoom to replace cancelled face-to-face PD. This new mode offered many challenges to trainers of how to offer a comparable experience to face-to-face PD. It is common to find that participants are less willing to “unmute” and speak in front of the group and interaction can be low. We will continue to offer virtual training from now on so it is important for us to improve engagement and dialogue between participants.

1
Introducing the challenge and why the specific intervention was needed.

As my study focuses on a dialogic approach in teacher PD, I will first look at theory from the classroom and then to relate it to an adult learning environment. Whole-group discussion in training often mirrors whole class teaching, and the same patterns of talk are found. Sinclair and Coulthard’s 1975 work identified 3 common talk “moves” in the classroom: initiation, response and follow up (IRF). O’Connor & Michaels (2012) acknowledge that while IRF is helpful to review or check what students remember, it is not effective to build a culture where “students take each other seriously, take risks and build complex arguments together”.

2
Identifying how the study design is informed by research evidence …

In contrast to the IRF approach, a dialogic approach to teaching was developed by Alexander (2004, 2017) and Mercer (2000). It involves the teacher orchestrating classroom discussions to ensure that all learners are engaged and that knowledge is jointly constructed, by encouraging students to build on their own and each others’ contributions, contrasting different students’ ideas and prompting students to evaluate each others’ ideas instead of the teacher being the one to give feedback immediately (Mercer, Hennessy and Warwick, 2017; Resnick et al., 2018; Berry, 2006; Chin, 2006, quoted by Cook et al., 2018) As well as whole-class discussions led by a teacher, learners also take part in Exploratory Talk (Mercer, 2000) where they have discussions in small groups without a teacher, building on, explaining, discussing and justifying their ideas and inviting each other to do the same. A dialogic approach has been found to have a strong impact on students’ learning (Howe et al. 2019; Alexander, 2018).

2
… and showing the author understands what the research evidence means for their context.

Focus and rationale for this study

I have decided to focus on whole-group dialogue for this inquiry because in whole-group talk, the trainer has a chance to “model’ useful ways of using language” (Littleton & Mercer, 2007) for teachers who may be new to it. From our past observations of trainers it was noted that approaches were often mainly authoritative, (Scott, 2006) i.e.: “closed to the points of view of others, with its direction having been set in advance by the teacher” (Mortimer, Scott & Aguiar 2006). They suggest that a balance of dialogic and authoritative approaches should be used at appropriate parts of a session.

3
Funnelling from the literature to a specific focus and inquiry question.

I have decided to focus on whole-group dialogue for this inquiry because in whole-group talk, the trainer has a chance to “model’ useful ways of using language” (Littleton & Mercer, 2007) for teachers who may be new to it. From our past observations of trainers it was noted that approaches were often mainly authoritative, (Scott, 2006) i.e.: “closed to the points of view of others, with its direction having been set in advance by the teacher” (Mortimer, Scott & Aguiar 2006). They suggest that a balance of dialogic and authoritative approaches should be used at appropriate parts of a session.

3
Funnelling from the literature to a specific focus and inquiry question.

My reading, coupled with my initial reflections with the trainer enabled me to formulate my inquiry question: In what ways did the trainer taking an explicitly dialogic approach influence teachers’ use of reasoning (R), building/elaboration (B) and balance of teacher and trainer contributions?

The first part of the inquiry question refers to talk quality. The mechanisms that were proposed to try for his were: negotiating ground rules (Mercer, 1996, quoted in Littleton & Mercer, 2007), using wait time of at least 3 seconds (Budd-Rowe, 1986), “revoicing” teachers’ contributions, inviting teachers to build on their initial comments, probing for reasoning, and giving talk roles in group work (Resnick et al. 2018).

The second part of my inquiry question is underpinned by the findings that in high achieving classrooms: “students’ contributions in…lessons occupied a far more substantial role (indeed, 84% more speech)” (Hennessy et al., 2021). My goal was to investigate the formation of dialogue created by a more equal balance of trainer and teacher contributions in virtual whole-group debrief sessions.

3
A clear statement of the central inquiry question, a rationale for it, and the goal of the inquiry.

Ethical Considerations

As this project involved teachers who were paying for PD sessions, I needed to apply the 2019 BERA ethical guidelines very carefully. The trainer involved and all participants (both past and present) were given information about my inquiry’s aims and content, GDPR complying data collection and storage, as well as their rights, such as the right to access their information and to withdraw from the project. They were then asked to sign an online consent form (see text in Appendix 5). My senior management and the country senior manager (the gatekeeper) were consulted on a suitable event to study, and gave their consent for this inquiry to take place.

In my role, I manage the quality assurance of trainers. To do this project I needed to take on a researcher role, which was different because I was studying the event from an analytical stance rather than giving feedback. I assured the trainer that whatever they did I would appreciate their input and that there would be no work-related consequences for them.

5
Clear statement of the ethical guidance used to ground the work, specific ethical considerations, and actions needed to ensure ethical conduct of the inquiry.

Research Findings

I will explore the two parts of my research questions separately with reference to relevant data from my baseline and iteration 1 and 2 of my focus event:

Quality of Talk

In my baseline observations, which took place over 3 episodes of virtual training, I noted relatively high incidences of reasoning per minute (0.45/min) and build/elaborate contributions (0.28/min) from teachers, but no instances of reflecting on dialogue (see Table 1 and Figure 1).

Table 1: Observed behaviours per minute – focus talk categories

Behaviour Baseline Iteration1 Iteration 2
Build 0.28 0.69 0.70
Reasoning 0.45 0.30 0.47
Reflect on Dialogue 0 0.15 0.38

Figure 1: Observed behaviours per minute – focus talk categories

6
Summary table used to highlight key findings, while the chart illustrates trends.

In the 1st iteration, Build turns were higher than in the baseline (0.69/min), and this was due I think to the trainer focusing on encouraging teachers who had made an initial brief comment to build on their initial answers. The trainer issued a total of 8 “invitations to build” (e.g.: Teacher 2, can I ask what strategies you came up with for your own classrooms?) and there were four instances of teachers building on others’ contributions in the zoom chat. The trainer purposefully used wait time (Budd- Rowe, 1986), sometimes pausing for up to 10 seconds. This seemed effective in a virtual environment where it takes longer to unmute to talk.

6
Examples of specific teacher practices illustrate the categories.

I was surprised to find that Reasoning turns in the baseline session (0.45/min) were higher than in the 1st iteration (0.3/min), as I had not done any intervention at that point, but as I will indicate further into the report, this was carried out by fewer teachers than in iteration 1. I think the reason for this might be that the baseline course was a pedagogy-focussed event, where the main emphasis was on improvements in classroom practice whereas the focus event for the 1st and 2nd iterations was a curriculum-focussed event. This made me realise that the amount of content to be covered might always be in tension with the amount of dialogue possible.

6
Interpretations with indication of how these relate to inquiry questions.

The baseline training materials had been designed to include questions where teachers were explicitly asked for their reasoning about a big pedagogical question (e.g.: Why is talk important?) and each group was invited to report back after a breakout discussion. Conversely, in the curriculum-focussed event questioning was more focussed on task-based questions (e.g. “what feedback would you give this learner?”) with less explicit emphasis on reasoning in the questions on the slides. This gave me a compelling insight into how the wording of questions on the slides in the training materials can influence a trainer’s practice. Reasoning may also have been lower in the 1st iteration because the teachers who participated on zoom chat tended to write short comments and not elaborate. In the 2nd intervention session after iteration 1 we reflected on whether the trainer could have invited reasoning more and they decided to focus on inviting reasoning in iteration 2.

7
More tentative interpretations are also provided of some weaker data

Because of this issue, I decided to from then on to focus on the changes between iterations 1 and 2 but to leave baseline data in graphs as a reference to show how dialogue might be enhanced on a more content-heavy course, in comparison to a pedagogy-focussed event that lends itself more readily to dialogue. The number of RD (reflect on dialogue) turns was fairly low (0.15/min) in iteration 1. There were just two instances of RD over the 14 mins of dialogue, which were prompted by the trainer “revoicing” teacher comments and probing for more reasoning. These turns were longer and included deep reflection and reasoning from the two teachers involved.
In the 2nd iteration, incidence of RD turns doubled to 0.37/min (Table 1 and Figure 1). One episode in particular saw 9 instances of RD turns. This again seemed directly linked to the trainer’s actions. This was the debrief of the Poem Talking Points activity . Teachers shared extensive reflections on how they had challenged each other and how this had impacted their discussion and ideas. The teachers also reflected on how they could implement the strategy in their classrooms which is an important objective of teacher PD sessions. The trainer used revoicing several times and invitation to reason once and this led to more Build and RD turns by the teachers as they were encouraged to add more to their first comment. The number of B turns stayed the same (see fig. 1) and R turns increased by more than 50% from iteration 1 to 2, which suggests deeper teacher reflection.

6 and 7
Interpretations and possible cause-and-effect are grounded in observational data.

Ground rules for talk

No explicit focus on ground rules was apparent in the baseline sessions, and implicit norms of behaviour were relied on. Ground rules for talk were negotiated at the start of session 1 in the first iteration, by putting the teachers into breakout groups and then debriefing and summarising all the agreed rules onto one frame on the Miro board (see fig. 2). At this point, use of ground rules was trainer-led with some teacher involvement

Figure 2: Ground rules for talk: screenshot from course Miro board

6
Use of participants’ work as a data source.

By iteration 2, the trainer had become more confident in referring to the ground rules before starting activities. The Poem activity started by the trainer explaining the purpose and learning gains associated with learners challenging each other and then explicitly asked teachers to challenge each other and reflect on how this felt. They reported back in some depth about this, and this directly led to 0.37/min RD turns and 0.48/min Reasoning turns, a large increase on iteration 2 (see fig.1). This took the use of ground rules to the second level of the T-SEDA Ground Rules Rating Scale.
At the end of the course, the delegates were asked how these talk rules had affected their talk and they shared several ideas including: “Explicitly talking about talk helped it improve.” They showed enthusiasm for the strategies modelled (including introducing ground rules) and the exploratory talk activities in which they took part and several mentioned that they were going to use them in their own classrooms.

6
Participants’ voices strengthen the claims made.

Rachel's research findings continue, discussing 'talk time' and the division of time between trainer and teacher talk.  As in the previous sections, data tables are used to draw attention to significant patterns and charts are used to illustrate trends.  Claims are supported by the evidence presented. 

You can read the paper in full here: https://hdl.handle.net/20.500.14069/41.

Changes to Practice

This inquiry has been an enriching experience both for me and the trainer involved. The trainer emailed after the event and said how effective they felt these virtual sessions had been and that they would continue to set up talk ground rules and keep an awareness of trainer vs. teacher talk in future virtual courses. 

As I have responsibility for trainer PD, there are a number of themes that I have already included and am going include in our trainer PD offerings:

  • We have just run a session “How to support high quality dialogue and argumentation in science – from oracy to writing”) for 26 science trainers. Next week we will run a session “Dialogic approaches in training and teaching” for languages trainers.
  • I will explore further ideas for future trainer PD sessions to unpack what successful educational dialogue in training might look like with our team of trainer network “conveners” who coordinate the content of these sessions. Themes may include discussing how to set up ground rules at the start of training.
  • I would like to invite the trainer involved in this study (if willing) to share their experience with other trainers and to share tips on enhancing dialogues in training.
  • I am also going to work with colleagues who have also completed this module to share experience and resources with our team and wider organisation.

8
Discussion of the impact on participant’s and the authors’ practice, and next steps.

I am also going to work with colleagues who have also completed this module to share experience and resources with our team and wider organisation.

This study also gave me an insight into of the power of questions on the training materials to prompt dialogic approaches. While this is not my direct responsibility, I would like to explore enhancing questions such as “what feedback would you give this learner?” by adding the words “…and why?” to increase reasoning in teacher breakout room discussions. I would also like to standardise the practice of setting up ground rules for talk at the start of all virtual training events by adding a slide to the PowerPoint presentation.

8
Potential wider implications and further inquiries.

Reflective evaluation on the process

The interventions seem to have been somewhat effective to encourage teachers to take a more active part in the virtual PD sessions, as well as give them ideas about dialogic practice to take back to their classrooms. I was very fortunate to be working with a very reflective trainer who was open to trying new strategies and was able to quickly adapt their practice, and this contributed hugely to the success of this project. The teachers in the training were enthusiastic and open to engaging with the strategies used, and this also really helped the inquiry to be successful.

There were various challenges that I faced during this study. The biggest was finding an appropriate baseline course to study as there was no comparable virtual course that the trainer involved had run in the past. I only got consent from 9 of the teachers who attended that course which somewhat skewed my data. If I were to do an inquiry like this again, I would ensure that I had more time to conduct several comparable iterations with the same trainer. I would also ask teachers to complete an evaluation form, as I relied on our standard feedback form which wasn’t focussed on dialogue and so I didn’t get any useful data from this.

7
Recognition of the limitations of the study.

Next Steps

As I complete this inquiry, I feel that this project, despite its restricted size and other limitations, has acted as a useful experiment to see how common classroom dialogic approaches can be adapted for a teacher PD environment. It was also satisfying to see how activities could be specifically designed to prompt specific dialogue moves.

With the trainer’s agreement, I will study the next iteration of this training event to find out whether we can encourage a more equal balance of trainer and teacher talk and to encourage teachers to challenge each other through specially-designed activities. During this inquiry, I noticed that the trainer spoke after almost every teacher contribution and I would like to study this further in future. It would be interesting to find out how this reliance on the trainer might be reduced and teachers might be encouraged to interject to respond to each other rather than waiting to be prompted by the trainer. This will be challenging given the relatively short timespans of virtual courses, but I would like to work with the trainer involved to try to develop strategies to encourage teachers talk to each other in whole group debrief sessions. This may involve asking teachers to nominate another teacher to speak after they have spoken or assigning roles to group members (Resnick et al., 2018).

To sum up, I am convinced that dialogic approaches work to create a better learning environment in virtual PD and this inquiry has motivated me embed these practices more deeply in our training for teachers.

1 and 8
Returning to the initial challenge, and identifying opportunities for further inquiry.

Inquiry Plan and Methods

Two trainers originally took part in this study, but I made the decision to just focus on analysing one trainer’s event due to the tight time frame. I completed one inquiry cycle containing three repeats. I analysed recordings of the main room of these virtual sessions:

  • 1 baseline observation of an event from (9 months ago)
  • The first iteration of the focus event, (sessions 1 and 2)
  • The second iteration of the focus event, (sessions 3 and 4)

Due to limited data available, the baseline event involved the same trainer training teachers in the same country, but these were different teachers to the focus event and also a different type of PD, focussed on general pedagogy rather than a specific subject. This was a limitation of my study and I will talk in more detail about this in the Findings section below.

4
Identification of

who took part,

and the overall design of the inquiry.

78 mins of recordings were transcribed and coded using template 2A from the T-SEDA scheme. I focussed on the talk categories of build on/elaborate (B), reasoning (R), and reflect on dialogue (RD). The first two as they are signifiers of deeper thinking, and RD because teachers need to think metacognitively when experiencing new ideas to reflect on how they will apply it in their own teaching practice. Challenge (CH) initially a focus, but as there were no instances of CH observed in the baseline observations, and the trainer felt that teachers would be highly unlikely to challenge each other in whole-group sessions, CH was rejected as an explicit focus for the study. However, it was kept in mind as a focus for talk inside breakout rooms. Another focus was the use of ground rules for talk, and I used template 2F to measure this – Student Participation and Talk Rules Rating Scale from the T-SEDA pack.

4
Clear statement of 

what data was collected, what data was analysed, and why.



Before iteration 1, I undertook an intervention session with the trainer, introducing dialogic theory and reviewing the data from the baseline sessions using the resource described below. Between the first and second iterations I had another intervention session with the trainer, reviewing the analysis of sessions 1 and 2 from the first iteration. We noticed quite a high level of trainer talk (72% vs.28% teacher talk), and decided to try to further reduce the proportional trainer talk time. The development of the Poem activity (Appendix 2) came about during this session. We realised that the teachers were not challenging each other’s ideas in breakout groups, as they were focussed on finding common ground due to the barriers created by the virtual environment. This was understandable, but the knowledge that challenging each other would help them to deepen their learning (Howe et al. 2019) drove us to try to develop an activity that would “force” Exploratory Talk amongst the teachers by making “the difference or agreement between perspectives more salient” (Hennessy, 2011) and to encourage Reflection on Dialogue based on the experience of challenging each other. After the 2nd iteration was complete, the trainer and I had an evaluation session, reviewing the analysis of sessions 3 and 4 and evaluating the whole course from a dialogic perspective.

4
Discussion of changes in practice and focus in the course of the inquiry.