Hollylynne:  So, I'm here with our experts, Chris, Webster, and Susan, and I want to start off thinking about that with the statistical investigation, we often think of it as happening in a cycle or through phases, and we, in literature you can see it being referred to sometimes as five different phases, sometimes there's four different phases, in this course we're thinking about it as happening in four different phases; posing a question, collecting your data, analyzing your data, and interpreting your results.  So if we think about that as a framework of how one engages in a statistical investigation, how could it be helpful for teachers and students to use that frame of reference in doing statistics?  Anybody like to start, Susan?

Susan:  When we did the Teach Stat project which was when I first came to North Carolina, working with the statisticians at Appalachian State, they put me in touch with Neil Graham's materials from England and he is the one that trained me in the four phases.  I learned about that and so, I used those four phases as a framework for the Teach Stat project, and I now use it as a way to help teachers think about things.  But I helped them first by looking at it and saying, "What you see in textbooks and what you do often is in the analyze phase".  You go right to doing the mean, you go right to doing the graph, you go right to doing whatever it is that the stat procedural stuff, I said, "But it's really irrelevant unless you have a question you wanna ask, that's what separates stat from other kinds of mathematical ideas is the questions".  You pose a question, and then all of a sudden you have to think about, 'Why am I asking it, what am I trying to do, do I wanna compare data sets, do I wanna describe a data set, do I want to look at a trend in a data set and then that influences how you collect the data, that's the kind of data...what data do you collect?  You might have to go back and forth, and then all of that comes into the analyze phase in which there's a lot of components to it, and once the analysis phase is finished, then you come back and say, "So what!  So what's the answer to my question, how does interpretation happen?" And I find teachers, like many of the ones I work with, and probably me when I was first teaching elementary school missed the first two phases, you go right to that analyze phase, and because you don't have the first two phases you can't go to the fourth phase so you're just doing exercises, and I think that that's what we want to try to avoid.

Chris:  I think I can follow up with that Susan it's interesting, one of the things that really brought to light for me, the importance of writing this document, the GAISE framework document is we were beginning to see more statistics come in to K-12 through the NCTM standards here in the United States, was my children at that time were in elementary school or middle school, and I would go in to my children's' schools, and especially at the elementary school, I would see all of these beautiful posters on the wall, lining the hallway of where data had been graphed.  And so I would go to the teachers and ask, "Well, what question were you trying to answer when you collected this data?"  Like shoes, favorite color of shoes or you know, how many times a day do you brush your teeth, things of this sort, and they would just kind of look at me like, "What kind of question were we trying to answer?", and I said, "And why were you collecting this data, how did you plan to use this data?"  And the typical response would be "Well, such and such standards says that we need to teach our children how to create a bar graph, that's what we did."

Hollylynne:  Right, collect the data to create a bar graph.

Chris:  Right, that's what we did, and we created a bar graph, that's it.  And it was like, they were doing the best they could, they were interpreting the standards the best they could and that is when it was like a light bulb moment for me.  We as statisticians have to help our teachers understand that this is all about an investigative process, there's some type of research question that you're trying to answer or there's some type of information that you're needing about your classroom or about your school.  Maybe the local shoe dealer has contacted the school and said, "I can only order a certain number of shoes, what style shoe do I need to order that's popular with the kids?"  To try to help the teachers understand that there needs to be a motivation...

Hollylynne:  And a real context.

A real context for why you are collecting this data.  And I think that that was a real eye opener when we wrote the GAISE Framework that the most important thing that all teachers need to begin with is that four step process or five step depending...you know, depending on what country they're...on how they've set it up.

Hollylynne:  Good, good.  Web, do you have anything to add to that?

Webster:  Yeah, I would say, you know, anytime that you just analyze data for the sake of analyzing data you trivialize statistics.  You're not illustrating the true power of...you're not really talking about statistics in some sense, you're losing sort of the bigger picture.

Holylynne:  You're graphing some data.

Webster:  Yeah, you're just graphing some data, and so that's...that's, I think,  a really bad practice to have.  One other thing I would say is one of my pet peeves is I believe Susan's right, we provide nice, clean data sets to our students in inter-stat courses and they start at that point analyzing data, and that's something I think, a bad habit that we have, even at the college level, because it's just not the way the world works, and then you have to provide that context, it's not as natural as seeing it from beginning to end so all four of the phases are very important, and if you make a mistake anywhere along the way then you've compromised the entire study and, you know, they have to all work correctly before you get anything of value.  One of the things, and I was talking with one of the Vice Presidents at SAS earlier this week, and he said, "You know, if you think about the effort in reaching decisions with data, 10% of it is in the analysis phase, 90% of it is in the earlier stages".  You gotta make sure the data is right, and having those sorts...validating data, subsetting, grouping data the right way so that you can actually talk about the questions of interest, that's 90% of the effort, and that's something I hope that we in Intrastat start doing more of that, and talking about the skills that are required to do that.

Chris:  Well and as a follow up, one of the phases I think that many years ago traditionally was never taught in the Intrastat course was how to collect the data, you know, are you going to use a survey, are you going to use an experimental design, is it going to be an observational study? And, of course, we all know that the types of conclusions that we're going to be able to make depend upon the way the study is designed, and I think we're still at fault for maybe not necessarily spending enough time in our Intrastat courses on that second phase which is the type of design that we should be using.

Holylynne:  My next question, I think you all have started hinting at this, but you know, when you think about the practice of doing statistics, what are some of the key practices or if you will, habits of mind that you think about while you're engaging doing statistics.  And how can we help our students and our teachers kind of engage in that in classrooms?

Susan:  Well, why don't I start because I only have one that I rely on?

Holylynne:  Okay, good, a rule of thumb.

Susan:  I was doing a session at the National Council of Teachers of Mathematics annual meeting one year on some statistics work with teachers, and in the audience was Dick Shafer, who is a statistician who has done a lot of work in education and stats, and at the end of it, one of the teachers raised her hand and said, "Well, I have to tell my students to help my students know when they choose to use the median, or when they choose to use the mean for a set of data."  And that actually wasn't what my session had been about in terms of rules for this kind of thing, and so, I thought, "Well, Dick Shafer's here, I said to Dick, I said, "Do you want to try to fill that one?" and he stood up and he said, "It all depends."  And that's exactly what you have to have with statistics that doesn't happen in mathematics, it all depends on your question, it all depends on your data, it all depends on what you're trying to say, and so any choice is made.  I knew what she wanted to know, I can tell you what the rules are in terms of picking the median versus the mean based on data set and how it's queued, but that isn't what he said, he said, "It all depends on what the purpose of your investigation is, what your data is, what's going on".  So when people ask me about that I say, "It all depends, that's what statistics is all about".

Hollylynne:  I like to kind of think about that as embracing uncertainty, and being comfortable with that.

Susan:  Right, and I love to say, "Well, it all depends". 

Hollylynne:  Webster, you want to add to that?

Webster:  Yeah, so I look at lots and lots of data sets each week so just in the analysis phase, which I, you know, I don't like focusing purely on that, but just when you get a set of data, too often people start trying to produce graphs and things of that nature immediately, that's a very bad habit.  Because you first of all, have to consider the context of the data, you have to validate the data relative to that context, does the data make sense, you know, you really have to think very carefully about the data and its appropriateness, is it appropriate for certain types of analyses before you do anything else.  And then once you've passed that data validation stage, and you understand your data and the values that you have in some reasonable way, and you want to take it a step further and summarize it, then you have to think very carefully about, you know, what parts of the data do you want to use, oftentimes, you do want to filter the data and look at certain cases.  In terms of understanding variation in data, the ability to filter and group the data in reasonable ways is something that is very, very important.  Those are things that I use daily whenever I look at data sets.

Holylynne:  Yeah, yeah, I think so.

Chris:  So I'm going to put in a plug in for a forthcoming document, hopefully, it will be available in April for the National Conference for Teachers of Mathematics annual conference.  I've had the good fortune of chairing a writing team for a new document commissioned by the American Statistical Association called The Statistical Education of Teachers, and this is a companion document for the book, The Mathematical Education of Teachers, Number 2, and one of the things that was developed with the Common Core standards here in the United States was something that we call the mathematical practices, and this is basically certain practices of, oftentimes, they're called the process standards, you know, it's sort of our habits of mind, and how we deliver the curriculum.  And one of the things that we did in this document is we took the Mathematical Practices, there are seven of them, and we actually put them through a statistical lens.  So we wrote descriptions of the mathematical practices through a statistical lens.  I thought I might just mention two of them.  The first one is Making Sense of Problems and Persevere in Solving Them, and I think, Webster actually, has just rewritten what we wrote.

Hollylynne:  Great minds think alike.     

Chris:  Great minds think alike.  Basically, what Webster just said about his habits of mind and how he thinks about working with statistics is exactly what we try to describe with that first mathematical practice.  In terms of, you know, once you have your research question that you're then trying to follow up on, you must constantly persevere, and you must constantly re-evaluate each of those four steps, and you also have to realize that they're inter-related.  It's like you just don't finish with one and move to the next one.  The other one that I thought I would mention is the practice called Attend to Precision, and I think that, as far as a mathematician is concerned, the first thing that you think about with Attend to Precision is computational precision, but in statistics it's really more than computational precision, and I think the best way that I can summarize this long paragraph that we wrote is we wrote the statement, "In statistics you are precise about ambiguity and variability."

Holylynne:  Yeah, so it's not just that...you have to embrace that.

Chris:  You have to embrace it.

Holylynne:  You have to embrace that when you describe something as a typical value is this, you have to embrace that; that doesn't always mean that that will always happen, but that you need to expect that variability around that number.

Chris:  That's correct.

Holylynne:  Yeah, yeah, that's nice, so I think that's a great lead in to my next question.  So, as math teachers, statistics is often taught, at least at the K-12 level within by math teachers, and so I think it's really important to consider how mathematics and statistics are different.  And you were starting to hint at that.  What else could we add to that, how are mathematics and statistics different?

Webster:  Start with that one?  I think, you know, it stops being math and starts becoming statistics when data comes into the equation, and that data is the differentiator between the two.  And I would say statistics does not equal probability, and so, probability in my mind falls on the math, it's out of the equation, so that's taught on the fourth floor of SAS, not on the 5th but the statisticians are.  You know, statisticians use probability as a tool to answer questions, but we want to be very careful to not equate statistics and probability.  Those are very different ideas, and when it's taught by math folks, I think, sometimes there's too much of an emphasis on probability and the data portion, the statistics portion, gets minimized and so that's the way I would see it, the data is the differentiating factor.

Chris:  In the GAISE document we tried to make the point that as far as the role of probability is concerned, we want teachers to understand that it's our way of quantifying randomly and helping...we use probability to help us understand the role of randomness in terms of how we use our statistical [inaudible 14:33].

Hollylynne:  Right, right.

Susan:  And I often, with...particularly with elementary and middle school teachers, talk about the fact that you teach science, you teach social studies, and the reality is the statistics can...it does get used here all the time, and whatever you do with math it's simply a tool in those disciplines too, and I think math is a tool of statistics when you're talking about statistics, and it's not, and statistics is not mathematics.

Chris:  I think we think of one differentiation we try to give is that with mathematics it's more a determining a listing.  I think one of the hardest things for students when they're first really learning those statistical reasoning skills is understanding that there's not necessarily just one right answer, and that there's a lot of different approaches that you might use to analyzing the data.

Hollylynne:  Right, different graphs might give you different insights into the data.

Chris:  Exactly, exactly, and that leads to the point that with statistics context is critical.

Hollylynne:  Absolutely.

Chris:  Whereas, I like to tell my students that my mathematical friends over in the math department will tell me, you know, context gets in the way of the beauty of the mathematics.  It's not that they're opposed to context, but it just gets in the way of seeing the beauty of the mathematics and they're exactly correct.

Susan:  They are.

Chris:  But as a statistician we thrive on context.  

Hollylynne:  Right.

Chris:  A set of numbers, data means nothing...

Hollylynne:  It's not interesting...it's not interesting.

Chris:  Is meaningless if it doesn't have context, and of course, with statisticians we're concerned with quantifying variability.  I like to tell my students that mathematicians are really just statisticians where the standard deviation is zero.  It takes them a while to appreciate that, but they eventually understand what I'm saying.  So it's a different way of thinking, and I think that that's what's very challenging for our students initially, as well as, our teachers that are really trying to get a grasp of how to teach the curriculum, that we're not...

Hollylynne:  Right, it's a different way of thinking.

Chris:  It's a very different way of thinking.

Hollylynne:  Yeah, absolutely, good, thank you.