Data Doesn't Create Meaning. We Do.

I found this TED Talk by Susan Etlinger to be interesting in and of itself, so I think it's worth 12 minutes of your time. Several of the things she said really resonated with me, so I'll discuss them briefly after you watch.


At just past the one-minute mark, she says:

We have to ask questions, and hard questions, to move past counting things to understanding them.

This is reminiscent of the oft-used quote (usually attributed to Einstein, but he probably didn't say it), 

Not everything that can be counted counts, and not everything that counts can be counted

but I sorta like this one better. Because counting things is often a good thing but we can't stop there, we have to provide the context, the understanding, the wisdom to do something good with what we've counted.

At about the 6:30 mark, she says,

this is what happens when assessments and analytics overvalue one metric — in this case, verbal communication — and undervalue others, such as creative problem-solving

This sums up my main objection to PISA/PARCC/CMAS/fill in your own state test. We're so proud of ourselves for coming up with the metric that we've stopped asking ourselves whether it's an important metric in the first place. (I just finished Yong Zhao's new book where he goes into great detail discussing the history of education in China, and why the PISA results - and especially the conclusions assigned to those results - are almost meaningless.) We are overvaluing a metric that may (or may not) show how well you will do in school, but has very little worth in determining how well you will do in life.

At about 8:20, she brings it home,

And at this point, you might be thinking, "Okay, Susan, we get it, you can take data, you can make it mean anything." And this is true, it's absolutely true, but the challenge is that we have this opportunity to try to make meaning out of it ourselves, because frankly, data doesn't create meaning. We do. So as businesspeople, as consumers, as patients, as citizens, we have a responsibility, I think, to spend more time focusing on our critical thinking skills. Why? Because at this point in our history, as we've heard many times over, we can process exabytes of data at lightning speed, and we have the potential to make bad decisions far more quickly, efficiently, and with far greater impact than we did in the past. Great, right? And so what we need to do instead is spend a little bit more time on things like the humanities and sociology, and the social sciences, rhetoric, philosophy, ethics, because they give us context that is so important for big data, and because they help us become better critical thinkers. (emphasis mine)

At various time in my life I've taught students mathematics, so in some ways I'm a big fan of data. But the mistake we've made (and are currently doubling-down on with our new state tests) is confusing data with meaning. Data is only as good as the questions you ask, the way you ask them, the way you collect it, and - critically - how you then interpret the data.

Or, as Susan says at about 10:40,

if I don't know what steps you took, I don't know what steps you didn't take, and if I don't know what questions you asked, I don't know what questions you didn't ask

In education we currently have a love affair with data, without bothering to ask whether the questions we're asking are the right ones, or the only ones.

Data doesn't create meaning. We do.

Data doesn't define learning. We do. Or at least we should.

Data-Driven Schools: Homework

In my school's student handbook we state,

Homework is an expectation . . . Achieving students do homework at least 5 out of every 7 days . . . Do homework Sunday through Thursday, take Friday and Saturday off! . . . Average nearly two hours of homework each night.

Since we're increasingly encouraged to be "data-driven", I have a few questions.

Let's start with the "two hours of homework Sunday through Thursday." This has been an expectation since I started at Arapahoe . . . in 1991. I wonder what kind of "data" we based the two hours on. Why not 1.5 hours? Or 2.5 hours? Or for that matter, why not 111 minutes instead of 120? (We have an overly fond appreciation for numbers that end in 5 or 0.)

What kind of research did we do to determine that 120 minutes was the appropriate and most effective amount of homework each night? I'm one of only about 4 or 5 staff members who've been here since 1991, we've never done any research on this since then that I know of, and I don't know of any research that was done before then, so I suspect there is none. So if we just made up this number, how is that "data-driven"? Perhaps we need to sit down and rethink this and decide if that's truly the best number.

Of course if we did that, then we'd probably also want to look at the research on the effectiveness of homework in general. Alfie Kohn has been a longtime skeptic on the value of homework, so much so that he wrote a book called The Homework Myth. In that book he argues that the research shows no support for homework at all at the elementary level, and at the high school level there is only a weak correlation between homework and increased test scores (and, of course, that then leads into the debate about whether those test scores are meaningful or worthwhile). It's fair to say that he advocates for no homework at all, other than reading or self-assigned homework.

He recently wrote an article in the Washington Post about a new study that looked at homework and its effect on test scores and grades. In terms of test scores,

Was there a correlation between the amount of homework that high school students reported doing and their scores on standardized math and science tests? Yes, and it was statistically significant but “very modest”: Even assuming the existence of a causal relationship, which is by no means clear, one or two hours’ worth of homework every day buys you two or three points on a test. Is that really worth the frustration, exhaustion, family conflict, loss of time for other activities, and potential diminution of interest in learning? And how meaningful a measure were those tests in the first place, since, as the authors concede, they’re timed measures of mostly mechanical skills? (Thus, a headline that reads “Study finds homework boosts achievement” can be translated as “A relentless regimen of after-school drill-and-skill can raise scores a wee bit on tests of rote learning.”)

And the effect on grades?

There was no relationship whatsoever between time spent on homework and course grade, and “no substantive difference in grades between students who complete homework and those who do not.” This result clearly caught the researchers off-guard. Frankly, it surprised me, too. When you measure “achievement” in terms of grades, you expect to see a positive result — not because homework is academically beneficial but because the same teacher who gives the assignments evaluates the students who complete them, and the final grade is often based at least partly on whether, and to what extent, students did the homework. Even if homework were a complete waste of time, how could it not be positively related to course grades?

And yet it wasn’t. Again. Even in high school. Even in math. The study zeroed in on specific course grades, which represents a methodological improvement, and the moral may be: The better the research, the less likely one is to find any benefits from homework. 

It's important to note that not everyone agrees with Kohn's interpretation of the data, but even most of what I've read in support of homework tends to show it having a relatively small effect on student "achievement" (I prefer the word learning, myself), and often ignores the question of whether this work should be done at home or could be done at school.

I find it interesting, however, that we haven't looked at any of the research, or any of the dialogue between folks like Kohn and Willingham, we've just decided it's good, and that two hours five days a week is the optimal amount. So why do we assign homework?

In general, I think there are three main reasons that I've heard teachers use (and have used myself).

1. Students need the practice.
2. I can't cover the curriculum unless I give homework.
3. It teaches responsibility.

The research provides little or no support for number one. What little support it does give could be accomplished by giving them time in class to practice. At what point did we decide that school was so important that we decided to assign students a "second shift" of work at home after school was purportedly over?

Which leads to number two: there's not enough time to cover the curriculum. I agree with the diagnosis 100%, but not the treatment. Instead of assigning homework (and assigning students a "second shift") in order to cover the curriculum, we should change the curriculum.

I struggle with the increasing emphasis on covering more, and more advanced topics, earlier and earlier, and the emphasis on curriculum over learning. For example, we are now teaching topics in Algebra I (typically a freshman course) that we used to teach in Algebra II (typically a junior course). Why? And does it matter if you learn Algebra by age 15, or would it be okay if you mastered it at 16? (Or 25 for that matter?) We say we want to create lifelong learners, yet our policy is that they must learn things at certain ages that we determine (and standardize for all students). It's as if we think there's an expiration date on learning.

As far as the third reason, I have yet to see any research that shows that assigning homework teaches responsibility. In fact, anecdotally, I would say that it does not. How many high school teachers have you heard complain about students not doing homework? Yet we've been assigning them homework for years, shouldn't that have taught them responsibility by now? But, even if it did, would that be the best way to teach them responsibility? I would suggest that giving them meaningful and important things to do might teach them responsibility better than assigning homework of dubious value.

So, where does that leave us? If we truly believe that "data-driven" is the way to go, then the data is telling us that we need to step back and reexamine both our assumptions and our practices. I've previously suggested with textbooks that the default should be to not get a textbook, and then we have to justify why we need one. I would propose something similar for homework, the default should be no homework, any homework we assign should be justified. And that justification has to be well thought out and can't rely on any of the three reasons above, and has to also take into consideration the social and emotional health of our students.

And what about "average two hours of homework each night Sunday through Thursday"? Show me the data.

Data-Driven Schools: Sleep

Over the last few years there have been many articles regarding the research surrounding sleep. These articles not only focus on health, but frequently focus on the importance of adequate sleep for learning, and often focus on the need for teenage brains to get enough sleep (most of the articles seem to indicate that, for most teens, 9 hours is the minimum they need). The most recent, of course, was the recommendation from the American Academy of Pediatrics:

Pediatricians have a new prescription for schools: later start times for teens. Delaying the start of the school day until at least 8:30 a.m. would help curb their lack of sleep, which has been linked with poor health, bad grades, car crashes and other problems, the American Academy of Pediatrics says in a new policy.

The influential group says teens are especially at risk. For them, "chronic sleep loss has increasingly become the norm."

The policy, aimed at middle schools and high schools, was published online Monday in the journal Pediatrics.

Studies have found that most U.S. students in middle school and high school don't get the recommended amount of sleep — 8½ to 9½ hours on school nights — and that most high school seniors get an average of less than seven hours.

This is a topic I've brought up frequently over the last few years, but it has gained very little traction. It's not that folks disagree with the research or the recommendation, it's mainly the problems it causes in three areas: after school daycare (older students watching their younger siblings), after school sports (practices and games), and after school employment. While I agree that those are real issues we should consider and tried to help mitigate, I don't think they should take precedence over our students' health and learning.

(I'm not sure I agree with the Executive Director of the National State Boards of Education who is quoted in that article suggesting that it's costs related to busing that's the problem. All school start times could simply be shifted later, or secondary schools could be shifted to start after elementaries - neither would affect the cost of busing.) 

Our student newspaper staff just did a survey where they asked a variety of questions and, interestingly, one of them was about sleep. Let me be clear, this is not a scientifically valid study, but given the sample size (323 students out of roughly 2150) and the distribution method (all students received a link in their student email accounts, so decently random), I think the data is going to be reasonably accurate.

The newspaper staff polled upperclassmen (11th and 12th graders) separate from underclassmen (9th and 10th), although the data for sleep was fairly similar. The choices students had were: less than 5, 5-6, 6-7, 7-8, and more than 8 hours of sleep. For both underclassmen and upperclassmen the median response was 6-7 hours, with the distribution of both groups skewing toward the left (fewer hours of sleep), with upperclassmen a bit more skewed than underclassmen.

So now we have some reasonably actionable data about our students. The American Academy of Pediatrics recommends 8½ to 9½ hours (with some studies recommending slightly more), and our students are reporting they get between 6-7 hours a night, with a significant number getting even less than that. (I should probably also mention that first period for us starts at 7:21 a.m.)

So I find it interesting in this age where schools are increasingly "encouraged" to be data-driven (at least when we're talking about test scores), that this set of data doesn't appear to be driving anything (except decreased health, increased accidents, and decreased learning for our students). While I frequently question data-driven decision making related to test scores (because I question the quality and meaningfulness of the data itself), in this case I think the data is pretty clear-cut: our students are not getting enough sleep, and it's adversely affecting their well being.

I wonder if we're willing to take on the challenge of making the right decision for our students' health and learning, even if it means inconveniencing adults?

Sometimes This Stuff Still Amazes Me

Just a quick post to note that I still find my own personal networked world to be pretty fascinating and amazing. Yesterday I was talking with my wife about a homework assignment Abby had in math where she needed to gather some data. So I threw together a quick Google Form, posted on my blog, and tweeted it out.

Very quickly the responses started coming in, mostly from Twitter I suspect because I doubt that many folks had seen the post at that point. (Next time I may add a question about where they found out about the survey just to confirm that.) About a day later we now have 299 responses (as of this writing) from 43 states and 18 countries (counting the U.S.). (You can see the results embedded in that post.)

Now this particular survey and this particular post are nothing earth-shattering, but it again reminds me of how different the world is from when I was growing up; how easy it is to connect with others around the world, and certainly how easy it is to gather data via Google Forms, a blog and Twitter. While I certainly still need to do a lot of thinking about how best to utilize this capability in meaningful ways, I think we all as educators need to be constantly asking ourselves the question, "What can we do now (that is relevant and meaningful for students) that we couldn't do before?"

Abby playing guitar for Grandma via Skype.

How Many People in Your Family?

This post is more of just an extended tweet than it is a blog post, but I thought it might be interesting to share.

My daughter had a math assignment where she was supposed to gather data on how many people were in people's families, then graph it and determine things like median, mode, range, maximum and minimum (interestingly, not the mean, although they've done that for other problems - which is probably good since median makes much more sense for this problem). Just for fun I created a quick Google Form and tweeted it out.



The only required question was "How Many People in Your Family?" with some directions on how to define that for this problem, but then I also asked (just because I was curious) two optional questions: your location and your age.

Well, that quick tweet generated 95 responses (so far). Since Abby is at the point where it's still really hands-on with the data and you generate graphs by hand (to get a better understanding of the concept), that was a little more than she needed, so she decided to just use the first thirty-two. Here are her results:










In case you're curious, here are the results for all 95 data points:

The median and the mode were both 4 and the standard deviation was 1.54.

For those of you curious about the ages of the respondents, the mean was 40.37, the median and mode were both 39, and the standard deviation was 10.96. (It might be interesting for someone to do a more randomized survey of educational tweeters to see if anything could be deduced from the results - both age and family size data.) We had responses from all over the U.S., Canada, the United Kingdom, Australia and Thailand.

So, nothing profound here, but I just thought it was interesting in several respects. First, how many folks responded to a tweet that obviously was not going to have much benefit for/impact on them. Second, how easy it was to generate data via a tweet and a Google Form (not randomized, I realize, but still interesting). Finally, I found the age and family size of the folks who responded interesting, even if I can't draw any major conclusions from it. (Perhaps: The mostly educational twitterers who follow me and responded to this tweet are typically between the ages of 30 and 50 and have two to five people in their immediate family - not a huge surprise.)

Thanks to everyone that helped Abby with this assignment and, if you have anything more profound you can generate from this, feel free to leave it in the comments.

Google Maps, Google Earth and the Summer Games

Just a quick post to point out this story from the LA Times:

From where do the most Beijing Summer Olympic medal winners hail? Despite Michael Phelps being well on his way to breaking Mark Spitz’s record of seven gold medals at a single Games, it’s not North America that has the most gold medals so far. According to the Google Maps’ Summer Games 2008, as of Aug. 12, 2008, China has won the most Olympic gold. The United States has won the most overall medals.

The Stadium page is my favorite of all the maps. Click on a stadium and watch a video as Google Earth zooms in. They also offer a photo and info about the venue.

Nothing major here, and probably no real educational relevance (other than curiosity, which isn't such a bad thing), just thought it was interesting. I bet there are more creative uses of Google Maps/Earth and the Olympics. If you know of any, please leave a URL in the comments.

I'm way behind in my aggregator (with no hope of catching up now that school is officially upon us), but it sure seems like somebody, perhaps dy/dan, should have a post coming soon with some great problems using current Olympic data. (I'm pretty sure he posted a while back doing some predictions with Olympic data, but I can't find it at the moment - somebody leave that URL in the comments as well and I'll update the post.)