In the previous post I looked closely at the data on teacher effectiveness for six different high schools. In this follow-up post we look more closely at a range of data from two successful schools and one school identified in the Cuomo Report on Failing Schools. Finally, I draw some important conclusions and recommendations from the lessons of the data.
A closer look at two successful schools and one unsuccessful school in NYC. It appears that teacher effectiveness ratings are not valid measures in light of other available data, as the data showed in the previous post. Some of the better schools in NY (as measured by graduation rates and exam scores) have lower teacher effectiveness scores than some of the most struggling high schools in the state – sometimes greatly so.
In this post I want to dig deeper via additional data that is available for high schools in New York City to show that parent, student, and teacher survey data: School Quality Reports; and narratives from site visits further support the notion that the teacher effectiveness ratings are likely not accurate, especially in struggling schools.
Data on a successful high school. The first results that made this clear were presented in the previous post. School #3 had internal ratings of teacher effectiveness that were much lower than those in most other schools in NY – and especially School #5, which is a struggling school but rated all its teacher higher than most other high schools in NYS. Yet, 100% of School #3 students pass the Regents English Exam and there is a graduation rate of 91%, with a population that is 51% Black and Hispanic. Nor is it one of the well-known highly-selective schools.
I happen to know School #3. I have worked there, given feedback to staff and many teachers have been to our UbD workshops. I have seen excellent teaching at this school: challenging work, lots of interaction, and respect for kids. I know the people doing the supervising and the ratings: they have high standards about the right things academically and pedagogically.
My impressions can be more objectively supported by other available NYC data.
The bulk of the data below comes from the New York City School Survey Report for 2013-14. Data exists there from surveys of parents, teachers, and students. (In the data, blue-colored data comes from parents, orange-colored data comes from teachers, and maroon-colored data comes from students.) I chose the same 9 pieces of data for each school and list it below. The data for each school is listed in the same order:

1. Overall Survey Results
2. Parent Satisfaction
3. Student Satisfaction
4. Teacher Views on Improvement
5. Student Views on Improvement
6. Teacher views of school culture
7. Student views of school culture
8. Student account of instructional approaches
9. Selected quotes from the school Quality Review Report, based on site visits

A Successful high school (School #3 from the last blog post).

1. Overall Survey Results

2. Parent Satisfaction

3. Student Satisfaction

4. Teacher Views on Improvement

5. Student Views on Improvement

6. Teacher views of school culture

7. Student views of school culture

8. Student account of instructional approaches

9. Selected quotes from the school Quality Review Report, based on site visits

Impact

The school’s curricula is coherent and engages all students in higher-level thinking that promotes college and career readiness for all learners.

Supporting Evidence

The curriculum supports metacognition and student empowerment. Literacy is taught across the curriculum with claim-based writing throughout all units of study. Grade wide argument writing yearlong plans delineate research based argument units throughout science, social studies and English classes. They include on demand tasks and performance based assessments.

All lessons have a clear aim that names the purpose, product and technique to be modeled and practiced. Charts and tools are cited in each unit to support independent thinking and practice. Tasks are engaging and include questions that emphasize accountable-talk discussions. All units begin with a launch that includes an engaging debatable question or an interesting problem or conflict. Tasks require students to make connections, explore and experience. In the molecular genetics unit, students solve crimes by looking at evidence. Students compare the STR length between the crime scene DNA sample and the suspects to identify the assassin in the scenario.

All units are structured and include an objectives, essential questions, enduring understandings, skills criteria, vocabulary development, discussion, practice and assessment. The thematic lens of the Imperialism/Colonization in India unit emphasizes student’s deep understanding of the age of imperialism and industrialism and focuses on independence and non-violent resistance.

Impact

Classes were marked by high-level questioning, meaningful student work products and student to student discussion for most learners to accelerate student learning.

Supporting Evidence

 Teachers’ goal set around the Danielson Framework, identifying goals that are grounded in school wide practices noted in the school teaching toolkit. These include techniques for asking questions that promote student thinking and give time to observe student understanding. Across classrooms visited, teachers used noted techniques such as turn and talk, think time, stop and jot, pass off, conferencing and questioning to promote metacognition and student empowerment.

 Focus questions were asked in all classrooms which supported student’s conceptual understanding. Students were also involved in structured discussions. Students were prepared with notes, texts from their research. Students developed their own questions, made connections, used evidence to support their claims and analyzed information.

 Flexible grouping and partnerships supported differentiation. Students were placed in guided groups to reteach skills or to provide extension activities to higher level students. Groups received different assignment sheets with tiered questions based on assessments. In the global history class, students examine primary and secondary sources to point out the author’s argument on imperialism.

Findings

The school leader communicates high expectations to the entire staff, students and parents and has created a culture of mutual accountability for these expectations with clear messaging and appropriate supports.

Impact

There is a coherent and thoughtful vision of school improvement replete with achievable goals and clear interim checkpoints to ensure the success of all students at the college and/or career level.

Supporting Evidence

 The principal sets clear expectations to teachers around professional responsibilities and implementing Danielson’s Framework for Teaching. There is a clear culture of high learning expectations for all students to ensure that they own their educational experience and are prepared for the next level. Students participate in advisory goal-setting conferences.

 During teacher team meetings, staff communicated a unified set of high expectations for all students and provides clear and effective feedback to students. Teachers also provide support to parents in understanding the expectations which include requiring four years of math and science for graduation. Additional requirements include “exhibitions” in 11th and 12th grade, work internships, college classes in 11th and 12th grade and attendance at college trips and Family college nights.

Findings

All teachers are involved in focused team sessions that collaboratively analyze classroom practice, assessment data, student work, and curricular products for the purpose of making thoughtful adjustments to teacher practice to increase student achievement and ensure Common Core alignment.

Impact

Teacher teams have a clear instructional focus supported by professional structures and protocols. This collaboration results in shared improvements in teacher practice and increased student learning. The inquiry based teacher teams’ model distributive leadership and builds a sense of communal ownership for student success.

Supporting Evidence

 The school has solid infrastructures of distributive leadership that promotes effective collaboration and collective monitoring of student progress. During weekly grade level team meetings, teachers’ action plan around students of concern and review student achievement data . Grade level inquiry team data meetings are also held three times a year based on school wide assessment. School-wide data teams comprised of grade team leaders cull and prepare data for inquiry teams. Monthly department-head led meetings are held to share investigations and to plan out school-wide initiatives. Weekly youth development team meetings include college guidance discussions.

A struggling high school. By contrast, a struggling NY City high school that I also know (having worked there twice), is on the Cuomo list of “failing” schools. Here are its survey and quality report data:

1. Overall Survey Data (and comparative graduation data)
   

2. Parent Satisfaction

3. Student Satisfaction

4. Teacher Views on Improvement

5. Student Views on Improvement

6. Teacher views of school culture

7. Student views of school culture

8. Student account of instructional approaches

 
It is noteworthy that the teachers in this school say that these instructional approaches happen “Often” – a student/teacher disconnect that does not occur at the other two schools discussed here.

9. Selected quotes from the school Quality Review Report, based on site visits

What the school does well

• The principal organizes resources and time in order to support instructional goals and increase student outcomes from a social-emotional and academic perspective. (1.3)

• The school uses various assessment practices to analyze student performance, target instruction, and provide students with feedback in order to increase student achievement and academic progress over time. (2.2)
  • The principal expects that all teachers will utilize the school-wide grading policy, more frequent formative assessment strategies, including exit slips, written reflection, and the use of rubrics.
  • Teacher teams meet weekly in each small learning community (SLC) to analyze and discuss student data, construct item analysis and determine where students have gaps in instruction.

What the school needs to improve

• Increase the alignment of curricula across grades and content areas to Common Core Learning Standards, and refine units of study in order to increase rigor in tasks to advance the post-secondary readiness of all learners. (1.1)
  • With the support of a curriculum director/assistant principal, the school continues to work with teacher teams on aligning curricula to Common Core Learning Standards and to further develop unit plans and lesson plan templates to effectively support students, yet this process of curricula refinement is inconsistently documented and only beginning to emerge in the math and social studies curricula. Additionally, the progress made in curriculum development is not being accurately communicated to the principal in a timely manner.

• Deepen academic rigor by consistently designing challenging tasks and utilizing effective questioning that elicits higher-order thinking and extends learning for students on all levels. (1.2)

• The principal believes that students learn best when they are given the opportunity to delve deeper into rigorous content, engage in student-centered instruction, collaborate and discuss evidence and viewpoints with their peers, and reflect on the process and learning. However, these practices as evidenced in classroom observations, are not being consistently implemented, as the majority of instruction observed was teacher-directed with many tasks not appropriately challenging, with an absence of text-based discussion and lack of conceptual understanding.For example, in an English class, the students’ response to the teacher when asked to analyze the influence of X in the text Y, was met with disengagement from most. One student stated, “We have been reading this same story over and over again and we need a new story”, while other students ignored the teacher and began talking about things unrelated to class.

• In addition to the breakdown of classroom management, the lesson lacked rigor, directions for students to follow, and an ultimate objective, leaving the teacher scrambling to gain control and students not engaged in any meaningful learning.

• In a history class, teachers asked students to annotate, but did not hold them accountable for what they annotated. Students were asked leading and lower-level questions which resulted in the teacher answering the questions rather than allowing students to engage in productive struggle.

• In another class, the teacher designed a lesson using the lesson plan template the school devised, yet none of what was written in the plan was executed preventing students from interacting at high levels and from multiple entry points, which eventually caused the lesson to fall apart.

• Out of 11 classes observed, there were only three where students were frequently asked to explain their answers. Furthermore, differentiation and multiple entry points for a variety of learners were not observed anywhere, with the majority of lessons requiring all of the students to do the same work.

• The result is that across classrooms, not all students are consistently provided with the opportunity to engage in higher-level thinking or reflection, which is evident from low-level discussions and quality of student work products.

• Enhance the monitoring of curriculum development and teacher team practices to ensure that teachers are effectively meeting the learning needs of all students as they work to meet the expectations of the Common Core. (5.1)

• The principal monitors student progress and assesses teacher instructional practices across grades and content areas to ensure coherence and to ensure teachers participate in ongoing inter-visitations to learn best practices from each other. However, while there is student work posted with tasks and rubrics that align to Common Core expectations, there is insufficient evidence that academies regularly revise and modify curriculum plans to ensure that the learning needs of all students are being planned for, resulting in only some students being prepared to meet the expectations of the Common Core.

• While there is a system in place for teacher teams to meet weekly, there is no evidence that the school has an accountability structure in place to regularly evaluate and adjust the SLC’s inquiry team practices and monitor the connection between the work they engage in during team meetings and the alignment to school goals.

What are the teacher effectiveness ratings for this school? Not one teacher is deemed Ineffective and only 5 are Developing., 90% are Effective and 5% are highly Effective.
Yet, note, above (under #9) that the only positive comments in the site report referred not to teaching but to actions taken by leadership: there are no positive comments about “what the school does well” related to teaching per se. (In fact, despite the positive site review of the school leadership, the teachers strongly dislike the work of the Principal and other administrators, as visible in other teacher survey data.)
A third comparison. Finally, I went looking for another New York City high school that had the same demographic profile as the struggling high school, above, but had nonetheless earned good ratings for making adequate improvement on their graduation rates and test scores. (Recall that school accountability in NY is based on improvement, not absolute scores and graduation rates, so schools with modest absolute scores can still be top-rated in New York. Put differently, the struggling school above failed to make adequate progress in the benchmark data of graduation rates and exam scores for 10 years.)
This “effective” school has the same demographics as the struggling school I cited above (more than 90% Black and Hispanic) and is located in Harlem.

1. Overall Survey Results

2. Parent Satisfaction

3. Student Satisfaction

4. Teacher Views on Improvement

5. Student Views on Improvement

6. Teacher views of school culture

7. Student views of school culture

8. Student account of instructional approaches

9. Selected data and quotes from the school Quality Review Report, based on site visits

What the school does well

• School wide instructional practices are coherent and reflect a common belief to support all learners, thus resulting in greater student participation and ownership. (1.2)

• Across a vast majority of classrooms, instruction is driven by the shared belief that students learn best by organizing for effort, having clear expectations in order to master core concepts, and using knowledge actively. Moreover, in the vast majority of classrooms, students were able to articulate the purpose behind each lesson, unit and the intended learning outcomes.

• Administrators and teachers strategically use a variety of measures of student learning data and common assessments to adjust instruction and ensure increased academic achievement. (2.2)

• All staff members review student work against agreed-upon and established proficiency rubrics. During the interview, students claimed that the use of common rubrics across classrooms has provided them with feedback that leads to clear next steps which has led to improved outcomes and an increase in successfully completing required coursework as evidenced by the increase in percentage of students completing the necessary credits and coursework to graduate, meet college eligibility requirements and/or enter work force. Departmental teams meet to further refine student tasks, including rubrics, and modify unit plans to ensure Common Core Learning Standards (CCLS) alignment, which are accessible to all students.

• Across the school, teacher teams engage in well-structured inquiry-based professional collaborations to analyze data in order to strengthen instructional capacity resulting in improved student outcomes. (4.2)

• Teacher teams work together to support a “three week,” collaborative professional development cycle which involves unit planning, working with outside staff developers and a lead teacher, participating in a “lesson study,” and looking at student work. For example, all teacher teams begin by identifying gaps in student understanding from the previous unit in order to address and plan. The members of the team follow up by observing each other teach a planned lesson from the planned unit, and then provide verbal and written feedback on the observed team member’s instruction. In addition, teachers analyze student work from the observed lessons during both content area and inter-departmental teacher collaborative inquiry.

What the school needs to improve:

• Strengthen school-wide curricula to include rigorous and engaging tasks, and strategically align the instructional shifts to support college and career readiness for diverse learners. (1.1)
  • The school has integrated a “backwards” unit-planning template that is used across all subject areas. A review of lesson plans reflects daily aims to unpack the purpose of each lesson and the daily learning objective for students. Each lesson fits within a unit of study towards a culminating task and includes opportunity for peer-to-peer discussion.
  • Furthermore, lessons included close readings and text analysis and/or problem-based learning activity. However, written prompts and assignments embedded within the lesson plan reflected mostly “yes” or “no” questions.
  • While the school ensures that the units of study and lesson plans in social studies and science as well as ELA include the use of primary resources and a variety of content area texts such as magazine covers, advertisements and current newspaper articles to engage students and promote high levels of thinking, the strategic integration of planned student activities and tasks to include increased Depth of Knowledge (DOK) questions to raise student levels of thinking from content knowledge to analysis across all classrooms is limited. As a result, the goal to promote all students’ facility with an assortment of complex texts, academic vocabulary and speaking, listening and language needed for college and career readiness is limited and impacts increasing student achievement.

By the way, here are the teacher effectiveness ratings for this school: almost identical to that of the struggling high school:
 
[I think the data make pretty clear that the struggling school is indeed struggling and that the teacher effectiveness ratings are simply not credible.]
My tentative conclusions, from all the data of both blog posts.

1. Local and state teacher effectiveness ratings are flawed if, as my sample suggests, teachers in less effective high schools generally get high ratings, and often get much higher ratings than those in more successful high schools (especially if they have been struggling to improve for up to 10 years).

2. The combination of survey data and Quality Review site visits presents a fairly consistent and credible picture of the relative strengths and weaknesses of schools – but these rarely align with teacher effectiveness ratings in either successful or unsuccessful schools.
3. This disconnect of Ratings vs. Review and Achievement data can only be reduced when there are exemplars of teaching to anchor the ratings, supported by calibration meetings across schools in a district. Without such exemplars, the teacher rating system will be based only on building-level norms (and politics) – and thus as flawed and non-standards-based as letter grades given to students. Like grades, in other words, it appears that teacher effectiveness in schools is being scored on a curve (at best), rather than against any credible standard. (There is credible data to show that the typical % of employees in all walks of life whose work is judged to be ineffective is around 8%; in all NY schools the average is 1%, as the data show).NYSED (and/or each district) needs to rectify the lack of official models and calibration protocols.At the very least, the state should say that teacher ratings of 100% effective in ANY school are of questionable validity on their face, and that extra justification for such ratings should be provided). Beyond that audit indicator, NYSED might require schools to provide 2-3 samples of video excerpts for effective teachers to be sure that there is calibration across the state (as is typically done in the AP and IB programs with scored student work, and on writing assessments in the state). The Teaching Channel videos could easily jumpstart such a process as could videos that exist in other states such as California.

4. For “rating inflation” to be lessened, there must be far better incentives for all administrators to be more accurate and honest in rating teachers (and teachers, in proposing credible value-added SLO measures). I have heard from Principal and Supervisor friends of mine that the politics of school climate and the possible counter-weight of exam scores entices everyone to rate teachers higher than they truly believe is warranted.Perhaps, for example, schools should only be required to give aggregate “standards-based” ratings separate from the more delicate teacher-by-teacher “local norm” ratings. At least there might be an independent check on local rating standards by NYSED or via regional peer meetings chaired by BOCES staff, in the same way that IB teachers have their student grades “moderated” by the external reviewers of exams.

5. It is high time we made a critical examination of the wisdom of ranking each of the four dimensions in the Danielson Framework equally.

As it stands now, it is quite possible for a teacher in any school to do well on the three non-teaching dimensions of the four, do poorly on the teaching dimension, but still get a good “teacher effectiveness” score. That seems like a highly misleading way to rate “teacher effectiveness” (even as we should, of course, value planning, professionalism, and community relations).

6. Good schools improve and improvement is possible in all kinds of schools serving all kinds of students. Despite what many anti-reformers endlessly argue, from this data it is incorrect (and very fatalistic) to say that poverty plus ethnicity means ongoing adequate school and teacher improvement is impossible, as I have long argued that the data show. Furthermore, there are outlier schools in this and every other state that cause successful absolute levels of student achievement in spite of the SES-related obstacles. (See, for example). Alas, the most successful such Charter schools in New York are not represented in the NYSED accountability data resources cited in this post.

A final thought. Do I think that Governor Cuomo is up to some crass politics in his report? I do.
Do I think that this is a hard time to be an educator? I do – so much harder than when I was a full-time teacher (where we pretty much left alone, for good and for ill).
Yet, I also believe that until we get an honest and credible accounting of teacher effectiveness in all schools (and especially in struggling ones) we will perform a great disservice to kids in need – and, yes, to their teachers who deserve more accurate feedback than many now receive.
POSTSCRIPT: Numerous people tweeted back after the previous post that many NYS struggling schools are under-funded and that far too much was thus being asked of them. While I agree that many schools are under-funded and that teachers in these schools face very difficult barriers, it seems odd that everyone who responded with this argument failed to engage with the teacher effectiveness data I provided. Many also said that without better funding and other kinds of state support, improvement in such schools is “impossible.” The data simply do not support this fatalistic conclusion, nor does my work in New York City and Toledo where we, too, have seen solid gains through UbD training and implementation in previously struggling schools.
Let’s get on with improving the schools our students currently attend, doing what is in our control to improve them.