Hard to believe I have written 200 posts – the equivalent of 2 books, for no pay, and with surprisingly little pain! Thanks for reading and commenting: the thoughtful back and forth in the comments makes this all worthwhile.
[Oops: I updated the post to provide the link to the graphics site.]
As part of my continuing writing on literacy, I offer a short post – some graphics with a few questions.
The graph below comes from a site that specializes in education data. Here we see a critical data set in the argument over schools and poverty – New York City 3rd grade scores by school vs. SES levels:
The graph is actually interactive: by mousing over a dot you can see the name of each school. Here are a few of the important outliers to the somewhat general trend (the red dots equal charter schools):
(You can see just NYC elementary schools here)
My questions are:
- How tight is the trend of links to SES and reading comprehension, really? Whatever your biases, just look at the graph: what, precisely, does the graph tell us in detail? Describe the data, don’t just conclude glibly. Describe the variance, especially.
- If you are convinced that poverty is everything in school achievement, then what do you make of the dozens of outliers – in both directions?
- Why aren’t district and state officials constantly sharing what the good outliers do?
- Why aren’t you visiting those schools to learn from them if your results are weaker than those of the outliers?
In my next post, I will look more closely at the do’s and don’ts of using the comprehension strategies. As the previous posts on literacy have shown, many researchers are critical of the way the strategies are taught (or NOT taught), and few studies show solid transfer of learning into independent reading. There is no doubt that comprehension can be improved through teaching cognitive and comprehension strategies; the most common implementation of these strategies is often doomed to fail. Every teacher of reading, ELA, and English needs to understand how and why.
33 Responses
This is a really important point. While I do believe that poverty is often the catalyst for the lack of student achievement in reading, it is clear that there are ways to overcome that hardship. We, as the educational community, regardless of the populations we work with, need to be working hard to research those successful methods and figure out ways to successfully implement them with these students who are at risk for reading failure.
Thanks for the posts. I learn from you with every line.
The saddest thing of all is that, by inspection, the overall average % at proficient or more is around 35. A plot of %proficient against %Eng as Sec Lang could tell an interesting story.
But this is old news, as NAEP scores have revealed for decades (and I have noted before): overall performance is poor. Many wish it blame the new test and Commissioner, but the disconnect between older cut scores in the states and NAEP results forecasted this low % years ago.
I like your 2nd point – I’ll try to find the data.
Hi,
Could you post the link to the graph? I would love to use the interactive “mouse-over” feature.
Thanks!
Will do – I thought I had…
Saying “look to the outliers” to see what works is like saying anecdotal stories are informative for causation. This comparison of SES and % of free and reduced is simply that. It’s just a comparison. It’s dependent on a lot of factors. And I know you know this but your statement seems to imply that we can learn a lot from the outliers. We do not know if the outliers are randomized error or a combination of some other factors. It could be that it is as simple as an error in recording of the free or reduced lunch applications for that school. Or it could be that there is an amazing reading teacher there who stays all day- throughout the night and weekends teaching kids. It could be that outlier has a benefactor who pays for all the kids to have private lessons Or that the school helps parents to apply for healthcare and benefits.
I’m not saying not to look at the outlier. But, looking at the outlier does not change the pattern. And it doesn’t mean the pattern is anything more than a relationship of variables.
You can see it that way. I prefer to say: since results are terrible across the board nationally, we have little to lose and much to gain from a look at the successful outliers (and variance) – if only to see that there is no hammerlock against literacy provided by poverty. The relationship just isn’t that tight, and far too many people say it is causal. At the very least the variance in this graph argues against glib causal arguments. And we might learn that the outliers are doing something of note. of course, we might not. But by not looking and throwing up our hands we give up on kids.
I can look at it that way? No- this is how this works. It’s not causal. We can pull out factors to analyze across all schools to see if some impact this relationship. And looking at the outliers can help you to figure out some factors to try to analyze. But you can’t know by just looking at the outliers what makes them different from all the others in achievement despite low SES. Just like this pattern is not causal, outlier differences in instruction cannot be determined to be causal.
Just to be clear- I did not say and do not believe there is a “hammerlock” for kids in poverty trying to learn. I’m the last person who would say that. I do say that we are not doing anything at the moment (or very little) to truly try to improve the problem of poverty in this country. Each year, there are more and more of our nation’s children who fall into poverty. It’s a complex issue. Trying to glean an answer by looking to one outlier, testing out what they are doing (replicable or not), and waiting for those results is a long, long road to take that may not be fruitful. We have tried that with charter schools. It has not worked out well because it’s not a good approach. There are other, better ways to reduce poverty (SES) across the board. As educators, we need to get behind all efforts to reduce poverty.
And I do look forward to further studies which do more analysis into those factors that influence success in low SES schools. 🙂
Look at all the outlier red dots in the low SES end: most of them are Success Academy Charter Schools in NYC. I plan to visit them and a few other outlier schools in the City in the spring. (People I respect who visited are impressed with their pedagogical standards).
What are some of the pedagogical standards? Are these schools simply narrowing instruction and teaching to the test?
It’s pretty hard to ‘game’ literacy tests – one reason why the scores are so low nationally. Think about it: kids have to respond to novel passages and transfer their learning of decoding, comprehension strategies, etc.
But my aim is to find out what some of the NYC outlier schools are doing to better answer your question. I’m just surprised more people don’t do this.
Have you checked out what Nancie Atwell’s school in Maine? What about her recommended literacy strategies- increasing students’ trading volume and giving them more choice. I look forward to reading your last post about effective reading strategies and how to best instruct our students so they will apply to reading. Please post what u witness at these outlier schools!
I have not, and would like to. Volume of reading plus choice makes a difference in basic skill and attitude but does not significantly impact comprehension ability of non-fiction. The research is clear: there has to be an ongoing direct teaching of comprehension strategies, ongoing assessment of comprehension, and student self-assessment of comprehension and process.
As a high school special education teacher who has taught in public schools for over 20 years and closely followed the paths of research and education policy throughout that time (including attending an unconvincing research presentation by Karin Chenowith about the outliers she has studied), I thoroughly agree with Jupiter Mom. JM, do you have a blog or write articles? I would love to follow. It’s time the education policy world stopped fully believing every data point on every chart. Obviously, there will be outliers in any data set and many will be due to manipulation of statistics, error, or unrecorded causes that can’t be replicated. How many times we do we have to say that correlation is not causation? For example, Success Academy requires that all students and families agree to follow their philosophy. It’s likely that they push out (either overtly or covertly) anyone who doesn’t toe the line, sending them back to public schools for acceptance whether the students want to be there or not, agree with policies or not. This would likely explain at least part of the difference.
Has anyone looked into the use of “commissioned”/artificial texts (produced by companies) as opposed to the use of trade books – books written by real world authors??? Are we assuming that schools are teaching reading “strategies” with real books (which are more linguistically complex than commissioned texts) or are teachers using commissioned texts? If the tests, NAEP, PARCC, etc. are testing with “real” texts, teachers should stop teaching with artificial texts. Truthfully, regardless of what the test makers use, teachers should teach with real texts. If districts have $$ to buy “programs” that on the surface demonstrate that students are “progressing” as they read “commissioned texts,” in reality, they still haven’t learned to make their way through a linguistically more complex text. Take the money for the “program” and buy well-written books.
Here is one small example of a real text positioned next to a commissioned text:
Real: Level T: Bridge to Terabithia Katherine Paterson
“The western sun danced on the windshield dazzling his eyes…He watched the car go out of sight and then turned and ran with all his might to the house, the joy jiggling inside of him so hard that he wouldn’t have been surprised if his feet had just taken off from the ground the way they sometimes did in dreams and floated him right over the roof.” (note verb choice, use of modifier phrases, complex sentence structure and complex ideas…)
Commissioned Reading A to Z, Level T, Caribou Man, Retold by William Harryman
“Just after sundown, he found the igloo in which the two rabbits lived. When he entered the igloo, the rabbits asked why he had followed them. When he finished speaking the male rabbit spoke. “Our lives look carefree, but at any moment an eagle or a hawk could take us for dinner. Or a fox or wolf might chase us for a snack. Even smaller animals take our children.” (note simple verbs, simple short sentence patterns)
Lorraine, i think this is critical. All the research I have read in the past month complains about the simple texts being used to teach and assess comprehension (be it trade or textbook text). Comprehension is not even an issue if the text is simple – that’s going to be one of my final recommendations – text has to be intellectually challenging, with lots of inferences to be teased out explicitly by readers. The other key factor is that the key research of interest to me is the research that embeds clearly-nonsensencial or clearly inappropriate text into texts to see if readers catch it – many do not, including highly fluent readers.
All of this says to me that a critical stance has to redeveloped early, with carefully selected text that demands genuine comprehension strategy. More soon.
OK agree completely. Interesting note on the nonsensical text….I continue to follow you. Many thanks to you.
I can’t wait for your next post. I don’t believe we can just blame poverty for poor achievement. A fixed mindset will have us on a train to… NOWHERE. You offer something to think about – the outliers. Chenoweth offers perspectives of the outliers in 2 of her books. High achievement CAN be attained in low SES schools. We can no longer use low SES = low achievement as an excuse. Kids deserve better!
I fully agree. SES surely matters, but to use it as an excuse is unprofessional and wrong.
I am currently researching the effects of our summer school program (and many others in operation) because as we know, up to 66% of the achievement gap between low and high SES students, by the time students reach high school, has been blamed on “summer slide.” I’m hopeful that by implementing such programs, we can make a difference. Though I do believe we have a poverty issue in the US, the schools can’t fix that alone. To wait for something to change, is like you said, unprofessional and wrong. We have a moral obligation as leaders!
Great idea. Equally important would be to re-design hs, based on pre-assessments of incoming students and their abilities, especially in literacy. Right now, HS is dysfunctional in that it assumes everyone is ready to do HS level work and has no Plan B when it doesn’t work out. Why, for example, can’t there be an intensive – and engaging! – program in literacy development for demonstrably unready kids? Why can’t we permit kids to take electives in their interests, for example, and put in a heavy dose on reading instruction and challenge in such a course?
We are an elementary school district, so we are looking for ways to reduce/eliminate the gap early. Yes – what a novel idea – meeting kids where THEY are at, not where we believe they should be based on biased opinions and assessments. Make it relevant and engaging? What a concept! I was introduced, by my Politics in Leadership professor, to the short film, Precious Knowledge. It documents the life of the Mexican American/Raza studies program at Tucson HS in the mid 2000’s. Graduation rates increased by nearly 100%. Why? Learning was fun, engaging, challenging, students related to and had a personal connection to what they were learning, they were motivated, and the teachers believed in the students. Unfortunately politics got in the way, and the program was discontinued in 2011, much to the demise of the students. Sound familiar? Thanks for engaging in conversation and sharing your thoughts. I look forward to future posts!
I’ve never heard anyone ever say, “low SES kids can’t achieve” nor “give up on them” nor “we shouldn’t bother trying to do something until poverty is gone”.
We have this divide in education- those who accuse, “they say we ignore poverty” and those who accuse “they say we can’t do anything until poverty is gone”. Both are wrong and we have to stop. Of course, like everything else, the truth is someplace in the middle.
The thing to note about low SES is that we may need to change how we educate children (as Grant notes here about hs). I watched it myself. We had 2 or 3 kids in each of my children’s classes each year who were not only low SES but were the only minorities. The teachers in the classroom did their best and really were wonderful teachers. However, they couldn’t bridge the divide. The perception was that they needed a lot of extra help and there was only so much that 1 teacher could do. I think this is common. The numbers of low SES students was low so this school was “high achieving”. So, everyone would say this was a “successful” school. This wouldn’t be an outlier but despite the view that it’s successful, there was still a divide. Most of these low SES students followed the predictable path.
So, how can we work to reduce the impact of low SES on education? Change how we teach. Meet the kids where they are at that moment. Have them be in charge of their learning. Reduce the amount of “power point teaching” and lectures. Make the curriculum more culturally and ethnically diverse and relevant. Bring some of Grant’s teaching methods into the classroom and make sure the students know what to expect. Stop pulling kids out of the classroom for tutoring. Stop cutting recess if they didn’t bring in their homework. Stop having homework be part of a grade. There are many changes that could be more responsive to home difficulties. Make it so a child from any background can succeed. And success is not necessarily high scores on a standardized test.
School should be a safe, welcoming community center- not something one must survive. Until we address these issues, we will have great difficulty changing the situation. I hope both sides of this can agree with that.
Our high schools currently use the Fast Forward computer-based reading intervention program. I’ve heard through the grapevine that may change…please elaborate on your idea regarding electives of interest with a heavy dose of reading…sounds like you’ve given it some thought!
I have given it some thought only because I remained always more focused the HS and its problems than any other part of schooling – just the old HS teacher in me.
I will address solutions in the last post in this series on literacy. Meanwhile, the next one is on effective vs. ineffective teaching of comprehension/cognitive strategies, a discussion of assessment of comprehension, and motivation for older readers, especially boys who are falling further and further behind for reasons related to all these issues. Then, some recommendations – including different kinds of courses at the HS.
I would be interested on hearing more about your school’s experience with Fast Forward, however!
Is the data correct for PS106 in Queens? Doesn’t seem to match the NYSED data and the school has received some unfavorable publicity.
Surely, almost no one believes “poverty is everything”. As you have shown, it doesn’t take “outliers” to find many, many examples of low income schools beating high income schools. These non-outliers may provide more useful information, actually, since outliers often reflect advantages that are not available in other settings.
In this post as well as the one on high schools the lines between good schools and good student achievement seemed to be blurred. Do the outliers reflect unusually good schools, unusually good low income students, or both? How would you know?
It looks like there might be a dozen high poverty schools that might be called outliers. The students at several of these schools are unusual in ways other than achievement. Three are almost all Asian, one is a magnet school, and four are charter schools (meaning the students actively sought out a school with stricter discipline and academic standards). These differences may or may not be relevant, but they do raise the question of whether these schools would produce similar results with different students.
This graph brings forth the “Visible Learning” question – how much influence can we attribute to the student, the teacher, the school, the home, the principal, or peers. John Hattie has statistically estimated the influence of each (e.g., http://www.decd.sa.gov.au/limestonecoast/files/pages/new%20page/PLC/teachers_make_a_difference.pdf ). The biggest factor is student ability, and we can’t always choose which students to which school. However, the next biggest factor that we can address, accounting for 30% of the variance, is the quality of teaching.
[revised]
Thank you Grant for the focused and relevant discussion of reading comprehension and transfer of learning. It it the focus of much of my work with teachers and schools, and one basic thing that schools and teachers overlook is that students, especially middle level and high school students, are simply not spending sufficient time actually reading. Like any skill that we hope to gain proficiency in, practice is essential. Giving students time to read text that is sometimes self-selected and at an appropriate level, in a quiet classroom where they can concentrate and read silently, is often missing from our schools. The amount of real reading that students do in school is something schools should audit. When we assign reading for homework, there is no guarantee that it happens.
Penny Kittle’s work on reading with high school students offers honest discussion on the volume of reading that our students actually do, and this may be part of the reason our students aren’t growing in reading. They don’t read much. https://www.youtube.com/watch?v=gokm9RUr4ME.
There is a great deal of instruction taking place in the name of reading that often takes the form of the teacher talking. Finding time in the school day for students to actually read independently is critical because we don’t get good that things we don’t do often.
Teachers’ College at Columbia University offers work on reading curriculum and the continuum of reading to make the invisible visible in regards to comprehension, strategy instruction, and thinking about reading. It is complex and takes time for schools to implement well, but it is based on the gradual release model and transfer of skills and strategies toward independence in reading from basic comprehension to work on analyses and interpretation of complex text. They have new work coming out this summer, with goals of moving students from basic proficiency in reading to sophisticated levels of analysis and interpretation.
One premise of their work is that students themselves make choices as readers and are engaged in the thinking work of reading, …thus owning the learning.
Reblogged this on that MADDENing teacher.
Thanks for the series, Grant.
Love the focus on outliers with strong performance. This is the same approach shown in the book Influencer by Patterson, Grenny, Maxfield, McMillan and Switzler.
They studied how to change signifcant and intractable problems such as the Guinea Worm in Africa. The Carter Center and Center for Disease Control went into sub-Saharan Africa to study villages that should be having Guinea Worms but didn’t (sound familiar). They then analzyed what these villages did differently in order to share with neighboring villages that were making no progress. The results were instantaneous.
This idea of studying positive deviance should encourage all of us to look closely at our neighboring districts, schools, and teachers who are getting outlier results to determine the practices that are making a difference.
Thanks for the forum to openly discuss these big issues. I especially appreciate the debate about Hirsch, Pondisco, and Willingham as I have appreciated their contributions. It seems to me that a comprehensive reading comprehension program needs to include volume of reading with choice, and the direct teaching of metacognitive strategies.
As for the background knowledge question, is it not true that students will still need to have clear understanding of the domain knowledge of their non-fiction reading in order to wrestle with the text?
Thanks again for your investment in this discussion. When you turn this into a book, I’m buying!!!
Thanks for writing 200 posts! I enjoy reading them and am grateful to you for sharing your expertise and wisdom.
Thanks, Monica; very kind! Without the thoughtful comments and lively back and forth I doubt that I would have gone this far!
Hey comments , I was enlightened by the analysis , Does someone know where my business would be able to obtain a fillable a form form to complete ?