Below are my notes from the AERA 2014 session Rigor and Realism: Doing Educational Science in the Real World. The ideas are significant for learning technologies researchers focused on impacting practice and developing real-world innovations/interventions. While I share many of these values and discussed a number of them in past work (e.g, here and here), what appears below are Catherine Snow’s ideas.
Rigor and Realism: Doing Educational Science in the Real World
The Wallace Foundation Distinguished Lecture: Catherine Snow, Henry Lee Shattuck Professor of Education, Harvard University
Some educational researchers are adopting new models for doing educational research, models that start from problems of practice, prioritize the challenge of utility to educators, and presuppose partnership relationships between researchers and practitioners. In attempting to implement such approaches, we often find that attention to the conditions of real-world practice may compete with attention to the constraints of rigorous design. That familiar problem can be exacerbated by the conflicting epistemologies of real-world decision-making vs. rigorous scientific knowledge-building. This conflict, in its multiple forms only some of which will be discussed, is a dilemma rather than a problem; it demands careful consideration of approaches to balancing the desirable features of rigor and of realism when they conflict.
Notes from the presentation and twitter feed
A conflict exists between rigorous research and the demands of educational systems. Educators and researcher must work together as partners.
Researchers have to acknowledge the realities of practice.
Researchers should start with urgent problems of practice – not simply gaps that exist in the literature
Education research/science can be highly rigorous, but it needs to be relevant. If it’s not impacting students in schools, it doesn’t matter. Focus of ed research is often quality, but we have to make the case for utility. It should be about utility and relevance.
Education research should move from questions of “whether” to questions of “how” ( GV side note: especially relevant to learning technologies, as noted by past research – e.g., don’t ask whether online learning works, but ask how does it work, how do we make it better, under what conditions, for whom, etc).
Recruiting the next generation of scholars into this work may endanger their potential to publish and get grants
We need to modify practitioner preparation to provide guidance about the challenges of collaboration with researchers (e.g., teach value/limitations of research)
Numerous colleagues are invested in this model (e.g., WT Grant, iterative design work, design-based implementation)
Progress is slow. Are there really exciting initiatives that would make a huge difference that might change both edu research and practice?
- Build the partnership model into the preparation of doctoral students by institutionalizing and acknowledging relationships and acknowledging relationships. Schools of education need to embrace this approach and faculty members need to be supported even if it doesn’t have immediate payoff in journal articles. A way for schools of education to become relevant and keeping us from the constant danger of becoming second rate departments of Arts and Sciences rather than 1st grade institutions of social change
- Promote accountability by developing reliable and feasible measures of classroom practice that might eventually take the place of student outcome measures
- Take the wisdom of practice seriously and develop a mechanism for systematizing and curating it. Journal review process is elegant and effective in maintaining standards for a certain kind of knowledge, but why do we dismiss anecdotes that teachers tell us? Because there is no epistemological structure for evaluating and curating that knowledge. What does that structure look like? Even though practitioners generate knowledge, it disappears. Developing a mechanism to capture and curate knowledge would be a hugely powerful mechanism. We acknowledge wisdom of practice but don’t take it seriously. We need to raise the level of the wisdom of practice to a more respectable level, by making it indeed more respectable. Urgent agenda because of internet sharing of uncurated practices. There currently isn’t a way to curate teacher knowledge ( GV side note: Curation of internet resources is prevalent, and a number of individuals are arguing for a system of publish then filter – lots of connections to openness, digital participation, and digital/media literacies re: curation)
As a field we need an exciting agenda of research that is cumulative, rigorous and realistic
I visited Educause 2013 this year, largely after an invitation by Tanya Joosten and Amy Collier to participate on a panel exploring what makes technology pilots successful. The panel was entitled Prepare for Lift-Off: Becoming a Successful IT Pilot Site. Laura Pasquini took some notes on a google doc and Tanya posted the slidedeck here. The session was described as follows:
“Your campus is an innovator in many ways, and you’ve been approached to be a pilot site for a new campus IT product. You’d like to say yes to the idea, but you’re not sure you have the infrastructure to make it work. Join a panel of your university colleagues to learn the ropes and discover what it takes to successfully deliver and host technology pilots on your campus. The panelists will offer a dynamic conversation on the importance of stakeholder involvement, faculty engagement and selection, faculty development and support, technical infrastructure, student support, research and evaluation, and critical steps your institution needs to take to ensure your pilot not only flies but soars.”
Photo by Jason Jones
This was my first time at the conference. My goal throughout the conference was to explore this group’s horizon, or what this group is currently seeing as being promising initiatives for higher education. In summary, the focus was on: competency-based learning, learning analytics, and MOOCs. Openness was relatively absent. Research was largely absent. Vendor-driven solutions were pervasive, and I left yearning to know more about innovations created and implemented by learning designers and/or by institutions themselves.
There were two innovations that I have been thinking about since the event:
- I had a lovely chat with Rob Farrow who shared with me the work that the Open University is doing with the OER Research Hub. The project aims to collect evidence in relation to the claims surrounding openness, and more specifically to answer the question ‘What is the impact of OER on learning and teaching practices?’ Given my beliefs about the inordinate value that research brings to educational technology, you can see why I was exited about the topic.
- The second innovation that I learned about was Class Mob, which is a prototype developed through the Breakthough Models Academy. There are some interesting projects in that link, but I thought that Class Mobs represented a truly novel idea centering around the development of an alternative educational system that supported learners, accounted for what we know about teaching/learning, encouraged corporations to extend traditional higher education, and empowered individuals to have a say in their education.
I have just returned from the University of New Hampshire where I gave a keynote talk at the 12th annual Faculty Instructional Technology Summer Institute. My hosts (Terri, Stephanie, Marshall, Dan, Ken, Shane) organized an excellent event and were so welcoming and gracious that it was difficult to leave!
Photo courtesy of UNH Information Technology
This year’s faculty member participants represented departments that have launched or were exploring the launch of an online program. Professional development events like this one have a number of goals including helping participants understand online education, gain technological and pedagogical skills, alleviate anxiety, share, foster community, and create a sense of shared purpose.
My talk focused on exploring the opportunities, challenges, truths, myths, and realities of online education. I argued that our goal as educators and designers is to create and foster learning experiences and opportunities that are effective, fulfilling, inspiring, meaningful, caring, empowering, and democratic. Using this goal as the starting point, my fellow faculty members and I explored the online learning landscape and discussed a variety of topics that included the “no significant difference phenomenon” as it pertains to online vs. face-to-face education, competency-based models, disaggregation and unbundling, online program management services, the role of the faculty member, the quest for efficiency and automation, and openness.
I am including my presentation below. This is the first talk in which I included practical advice and simple strategies that a faculty member new to online learning may find helpful in their teaching. If you are interested in that aspect of online education make sure to explore the last few slides of my talk.
I just returned from the 2013 Sloan-C Emerging Technologies for Online Learning conference in Las Vegas. What a fantastic gathering! The value of the conference to me was the numerous great conversations with new friends (Jen Ross, Christopher Brooks, Amy Collier, David Wicks) and old friends (Tanya Joosten, Laura Pasquini). And, as always, I finally met friends and colleagues who I have interacted with online for a while (Mark Lee, Rolin Moe).
* * *
Amy notes that the unconference was fantastic. She is spot on!
I’ve been trying to make sense of the conference and my experiences since I left. My friend and colleague Joel Donna (of 3ring) came to Austin to spend some time with me on Saturday-Monday and the conversations I had at the conference continued with him as well. Here’s what has been on my mind:
1. Three years ago, I used to have conversations with colleagues wherein I was desperately trying to make the case that technology-enhanced pedagogy was a powerful approach to have in our “how to improve education” toolkit. I wouldn’t be surprised if at times I was called a technology evangelist (any of you that follow my work know that I am not). Nowadays, I am finding myself on the other end of the spectrum – cautioning colleagues about the narrative that education is broken, educational technology is the fix, and for-profit corporations are here to save the day. If Gardner Campbell was here, he would have said, “That is not what I meant at all; That is not it, at all.” What is education for? Who is it for? What does it mean to learn? If education really “is broken,” what exactly is broken? Is the funding structure broken? Are the pedagogies that we use broken? Is instructor preparation broken? Is our understanding of how people learn broken? Is the notion of academic freedom broken? What is broken?
In the world that I inhabit, “broken” refers to educational systems that employ unjust practices, disregard unequal access, promote exploitation, and embrace pedagogies of hopelessness and marginalization. Unfortunately, I suspect that the notion of “broken” that I perceive may be unlike the notion of “broken” that popular narratives embrace.
2. I can try to convince individuals that this contemporary fable of education being broken is a story told and retold by powerful individuals/entities who have something to gain by creating alternative systems (…and just to clarify, I am not arguing that education is perfect – see above). Do we stop there? Ideally, no. What educators and researchers need to do is to become involved in the design and development of educational systems and educational technology. If we don’t, someone else will design our future for us. Do we really want that? Do we really want future educational systems designed without input from educators and researchers? I hope not. I am working on a project related to this and I hope to be able to share it with you within the next two weeks.
3. I met a a lot of colleagues at the conference that are thinking about similar issues. This makes me quite happy. And I am very glad and fortunate to be able to spend time with all of you!
* * *
I had a great time participating in the Career Forum roundtables, giving advice to PhD students about academia and sharing my own experiences. I value this. I value having conversations with students and spending time together answering difficult questions. The question that keeps coming up here is: What is your passion? Is it teaching? Is it service? Is it a particular research method, a particular pedagogy, or worldview? How does that relate to the world at present? How can you pursue your passion? And to close the circle, unstructured time with colleagues is important and can be very productive for these types of conversations.
* * *
I was originally invited to the conference to give a plenary talk on emerging technologies. Huge thanks to David and Jen for all their help in making this a success. My presentation was recorded and I am really hoping that it will be made available online for free (hint, hint). My slides are below, and a storify of my talk, courtesy of Laura Pasquini, is here.
While yesterday I mentioned that there was not much conversation about research, today was a little bit different. The value that educational research can contribute was highlighted by Alan November (November Learning) and Richard Culatta (US Department of Education). Richard in particular suggested that edtech startups work with educational researchers and teachers in designing their products. Such a simple idea (and one that we teach in our MA and PhD degrees), but one that is rarely taken into consideration it seems. Working within disciplinary silos (whether that’s teachers developing educational technology alone, or engineers developing educational technology alone) is not how complex problem (like education) are solved. On the one hand, the absence of educators from a design team could lead to development of tools/products that don’t solve any sort of educational problem. On the other hand, the absence of educators might lead design teams to think outside of the constraints of current systems. Each time I think about this, I return to the design process used by IDEO, which highlights diversity and interdisciplinary thinking. The video below is a good example of this type of thinking:
. . . . .
The acknowledgment that educational technology startups will benefit from partnering with educational researchers is important. It is the same acknowledgement that was made by the NSF in the computing education for the 21st century program (disclosure: I am co-PI on one of those grants). Actually, the NSF went beyond simply acknowledging the value of partnerships and made CS-Education partnerships a requirement in applying for funding.
. . . . .
Richard mentioned that the Obama administration is developing initiatives that value teachers (e.g., Project Respect). I look forward to learning more about this, but to also understand how such policies align with the high stakes climate that the administration has continued pursuing. While individual policies may be worthwhile, they exist within a larger ecosystem, and I would love to know how the administration sees its high stakes approach aligning with these initiatives.
. . . . .
Towards the end of the day I managed to position myself right in the middle of conversations related to venture capital, business models, and investment in startups. And the statement that follows has been in my mind since. A panelist said: Do we care about learning outcomes? Absolutely. Once we first make money for our investors. This statement was followed by another statement noting that a sizable return on investment is the “mandate.”
. . . . .
I’m not that naive to believe that everyone is in this space for noble causes. I also don’t think that everyone is in this space for monetary gain. When building the educational systems of the future, all of us (educators, researchers, investors, designers, and developers alike) should make sure to ask: For whom is this future being built? Who benefits? And who is left out?
This week I am spending time at the SXSWedu conference. It’s described as a conference that “features four days of compelling presentations and informative sessions from education professionals, industry leaders, and policy practitioners committed to engaging all learners.”
. . . . .
These are not my usual stomping grounds. Indeed, AECT, AERA, E-learn, SITE, EdMedia, ALT-C and all the other conferences I’ve been to feature groups of individuals committed to education and learning as well, but none of them feature the entrepreneurial atmosphere and the “disruption is imminent” aura that this conference is epitomizing.
. . . . .
I had a great lunch with George Siemens today. I came across George’s work in 2007 and have chatted on and off with him over the past 5 years on various social media platforms but we hadn’t actually ever met face-to-face until today. I joked that I will be wearing my ethnographic hat during my time at the conference, trying to make sense of a culture different than my own. While my research aims to ultimately make a difference in education and people’s lives, and, a number of edtech startups and I are (seemingly) operating in the same area, I am not so sure I am in the “edtech space” (as it is affectionately called by the numerous entrepreneurs I met at the opening reception). And I don’t fully understand the different rounds of venture capital funding. But, that’s the language that’s dominating the conversations so far. But, I do believe this is something that more education researchers need to know about. After all, when individuals propose solutions for the problems of education, we need to listen. And to question. For more on this you should read this piece from Audrey Watters, (who is also at the conference and we got to spend some time chatting together today).
. . . . .
Language is a strange thing. It’s strange because the same words might have different meaning to different people. Take the words “democratizing education” for example. What do you think of when you hear those words? I think of Paulo Freire, equity, education as a public good, the freedom from pedagogies of oppression. I wasn’t sure what these words meant when I heard them today. I believe they meant “freeing education from the control that educational institutions exert on it.” And even though it sounded good (who doesn’t want to “democratize education” anyway?!) I’m not sure that progressive educators’ visions of democratic and equitable educational systems align with the visions of democratic educational systems that were discussed today. And that’s another reason why more educators and researchers need to be here, and need to be in these conversations.
. . . . .
Research was absent. Perhaps I was at the wrong sessions. Even at the sessions that I went to, any mention of evaluation (let alone research) was problematic. For example: “15% of our students told us that they [insert survey response here].” There was no mention of how many individuals were surveyed, what the return rate was, or whether the evaluation questions were created and validated by an independent party. I understand that educational research might not be on the radar of commercial entities and investors. But, it’s important. And, if we are truly dedicated to making change in education, however small or large it is, then we should be investigating whether the tools we create work, how they work, and in what contexts they work.