Below are my notes from the AERA 2014 session Rigor and Realism: Doing Educational Science in the Real World. The ideas are significant for learning technologies researchers focused on impacting practice and developing real-world innovations/interventions. While I share many of these values and discussed a number of them in past work (e.g, here and here), what appears below are Catherine Snow’s ideas.
Rigor and Realism: Doing Educational Science in the Real World
The Wallace Foundation Distinguished Lecture: Catherine Snow, Henry Lee Shattuck Professor of Education, Harvard University
Some educational researchers are adopting new models for doing educational research, models that start from problems of practice, prioritize the challenge of utility to educators, and presuppose partnership relationships between researchers and practitioners. In attempting to implement such approaches, we often find that attention to the conditions of real-world practice may compete with attention to the constraints of rigorous design. That familiar problem can be exacerbated by the conflicting epistemologies of real-world decision-making vs. rigorous scientific knowledge-building. This conflict, in its multiple forms only some of which will be discussed, is a dilemma rather than a problem; it demands careful consideration of approaches to balancing the desirable features of rigor and of realism when they conflict.
Notes from the presentation and twitter feed
A conflict exists between rigorous research and the demands of educational systems. Educators and researcher must work together as partners.
Researchers have to acknowledge the realities of practice.
Researchers should start with urgent problems of practice – not simply gaps that exist in the literature
Education research/science can be highly rigorous, but it needs to be relevant. If it’s not impacting students in schools, it doesn’t matter. Focus of ed research is often quality, but we have to make the case for utility. It should be about utility and relevance.
Education research should move from questions of “whether” to questions of “how” ( GV side note: especially relevant to learning technologies, as noted by past research – e.g., don’t ask whether online learning works, but ask how does it work, how do we make it better, under what conditions, for whom, etc).
Recruiting the next generation of scholars into this work may endanger their potential to publish and get grants
We need to modify practitioner preparation to provide guidance about the challenges of collaboration with researchers (e.g., teach value/limitations of research)
Numerous colleagues are invested in this model (e.g., WT Grant, iterative design work, design-based implementation)
Progress is slow. Are there really exciting initiatives that would make a huge difference that might change both edu research and practice?
- Build the partnership model into the preparation of doctoral students by institutionalizing and acknowledging relationships and acknowledging relationships. Schools of education need to embrace this approach and faculty members need to be supported even if it doesn’t have immediate payoff in journal articles. A way for schools of education to become relevant and keeping us from the constant danger of becoming second rate departments of Arts and Sciences rather than 1st grade institutions of social change
- Promote accountability by developing reliable and feasible measures of classroom practice that might eventually take the place of student outcome measures
- Take the wisdom of practice seriously and develop a mechanism for systematizing and curating it. Journal review process is elegant and effective in maintaining standards for a certain kind of knowledge, but why do we dismiss anecdotes that teachers tell us? Because there is no epistemological structure for evaluating and curating that knowledge. What does that structure look like? Even though practitioners generate knowledge, it disappears. Developing a mechanism to capture and curate knowledge would be a hugely powerful mechanism. We acknowledge wisdom of practice but don’t take it seriously. We need to raise the level of the wisdom of practice to a more respectable level, by making it indeed more respectable. Urgent agenda because of internet sharing of uncurated practices. There currently isn’t a way to curate teacher knowledge ( GV side note: Curation of internet resources is prevalent, and a number of individuals are arguing for a system of publish then filter – lots of connections to openness, digital participation, and digital/media literacies re: curation)
As a field we need an exciting agenda of research that is cumulative, rigorous and realistic
I recently wrote a paper which examined the activities and practices arise when researchers and educators use social media and online networks. This is part of my ongoing work to understand open scholarship, networked participatory scholarship, and scholars’ practices online. In this paper, I used ethnographic data and my own experiences to try to make sense of scholars’ online participation. Some of the interesting findings are the following:
- numerous scholarly practices are occurring in the open (e.g., sharing drafts of manuscripts, sharing syllabi, supporting doctoral students)
- participation in online social networks is not limited professional endeavors; For example, they share “their vulnerabilities and struggles (e.g, with a divorce) and [seek] help with personal issues and causes that they are passionate about (e.g., equal rights legislation).” Importantly, engagement with and and sharing about issues unrelated to the profession is a value that is celebrated by this community. It is not uncommon, for example, to encounter blog entries discussing the positive outcomes of social sharing and Twitter profiles proudly declaring that updates are composed of a mix of personal and professional tweets.
- scholars have appropriated social technologies and utilized peer-to-peer networks to access and share research papers that they do not have access to (e.g., PirateUniversity.org, ThePaperBay.com, the Scholar subreddit, the #ICanHazPdf hashtag)
The last point is particularly significant. In the paper, I argue that the use of social networks and peer-networks to share knowledge that is often behind journal paywalls suggests that individuals are willing and able to circumvent and defy restrictions to the sharing of knowledge and research. In fact, open scholarship is a value that is close to the hearts and minds of numerous scholars who use the Internet for professional purposes. Kroll (2011) described the sharing of copyrighted manuscripts as “an act civil disobedience toward the scientific publishing enterprise” and since I published this paper, Elsevier has started taking action against this activity by sending takedown notices to academia.edu. How does one respond to actions that are in direct conflict to strongly-held values and ideals? That is a question that every academic needs to consider.
A copy of the paper, along with its citation and abstract appear below:
Veletsianos, G. (2013). Open Practices and Identity: Evidence from Researchers and Educators’ Social Media Participation. British Journal of Educational Technology, 44(3), 639-651.
The ways that emerging technologies and social media are used and experienced by researchers and educators are poorly understood and inadequately researched. The goal of this study is to examine the online practices of individual scholars in order to explore and understand the activities and practices that they enact when they use social media for scholarship. Using ethnographic data collection methods and basic interpretive analysis techniques, I describe two emergent phenomena evident in scholars’ social
media participation: scholarly practices enacted openly in digital spaces and the self that is disentangled from academic matters. These phenomena raise issues related to “sharing,” scholar identity, participation and social media as a place of gathering.
Note: As with any qualitative and ethnographic work, the results should be seen in context and shouldn’t be generalized to “all scholars participating online do X.”
I was at the Educause Learning Initiative conference last week (#ELI2014), where I had some interesting conversations and discussions around online learning, MOOCs, research methods, and the future of higher education.
Amy Collier and I presented early results from our qualitative studies looking at learners’ MOOC experiences (if you have not yet responded to our call to share your lived experiences with us, please consider this invitation). Our talk was entitled “Messy Realities: Investigating Learners’ Experiences in MOOCs.” Our thinking is guided by the notion that even though surveys and big data yield insights into general behavioral patterns, these methods are detached and can distance us rather than help us understand the human condition. As a result, the phenomenon of “learning in a MOOC” is understudied and undiscovered. During the session, we shared what we have been finding in our studies, highlighting the messiness of learning and teaching in the open.
Karen Vignare and Amy Collier were also very kind to extend an invitation to a number of us to share our work with individuals participating in the leadership seminar they organized. It was fantastic to hear Katie Vale (Harvard), Matt Meyer (The Pennsylvania State University), Rebecca Petersen (edX, MIT), and D. Christopher Brooks (EDUCAUSE) discuss their work, and once again, I felt grateful that we are having these conversations more openly, more frequently, and with greater intent.
Below are my rough notes from my 5-7 minute presentation. I appreciate parsimony (who doesn’t?), and in the words of D. Christopher Brooks, this is the litany of things I think:
I am a designer and researcher of education and learning. I study emerging technologies and emerging learning environments. I’m also a faculty member , and I have been teaching in higher education settings both face-to-face and online since 2005.
To contextualize my comments on MOOCs, first I want to describe my experiences with them:
– I have facilitated one week of the #change11 MOOC was organized by George Siemens and Stephen Downes in 2011. This MOOC had a distinctively connectivist flavor with each week being facilitated by 1 person.
– I have enrolled in a number of MOOCs, and have even completed a small number of them.
– I have repurposed MOOCs in my own courses. For example, I have asked students to enroll in MOOCs and write about them.
– I have published an e-book with my students, sharing stories of student experiences with MOOCs.
– Finally, I am actively involved in studying learners’ experiences in MOOCs in order to understand the human element in these emerging learning environments.
I have recently come to the realization that I have an ambivalent relationship with MOOCs. My relationship with MOOCs is one of the most ambivalent relationships I have had with anyone or anything. This relationship is more ambivalent than the love-ignore-hate relationship that my cat has with me!
On the one hand, I appreciate the opportunities for open learning that MOOCs provide. I also appreciate how MOOCs have brought us together to discuss issues around technology, teaching, and learning. At the same time, I cringe at the narratives around big data, I cringe at the hype, at the ignorance around what education is and should be about.
I want to talk about two topics today: MOOC research and the MOOC phenomenon.
On MOOC Research
– We don’t know much about MOOCs
– The things that we know about MOOCs are mostly the result of surveys, learning analytics, and big data research
– The existing research and the existing methods that we use are informative, BUT they simply paint an incomplete picture of MOOCs. We should be asking more in-depth questions about learner and instructor experiences in MOOCs
– Qualitative and interpretive research methods can and will help us better understand MOOCs, open learning, and open scholarship
– Descriptions of learner behaviors are helpful, but these descriptions only provide a glimpse and superficial summary of what students experience and what they do in digital learning environments. To give you an example, emerging research suggests that students may be “sampling” courses; a behavior that we don’t frequently see in traditional online courses or traditional face-to-face courses. Nonetheless, “sampling” is not how participants would describe their experiences or the ways they participate MOOCs. To illustrate, consider family-style Mediterranean meals that consist of numerous dishes, where participants sample a wide array of food. If you ask a person to describe this meal, to explain it to someone else, or to simply tell you about the meal, they will likely describe the meal as a feast, they might describe the tahini as lemony, the variety of flavors as intriguing, the whole meal as satisfying. Different people will also describe the meal differently: Tourists might describe the meal as fulfilling, heavy, or even extravagant; locals might describe the same meal as appropriate, or better than or worst than meals that they have had at other restaurants. “Sampling” may be an appropriate descriptor of the act of eating a family-style meal, or exploring a MOOC, but the descriptor does not fully capture the experience of sampling.
On the MOOC as a Phenomenon
MOOCs. The acronym stands for massive, open, online courses. That is not what MOOCs are though. MOOCs are a phenomenon. They represent something larger than a course and should be seen in conjunction to the rebirth and revival of educational technology. They represent symptoms, responses, and failures facing Higher Education. For instance, MOOCs are a response to the increasing costs of Higher Education; represent the belief that the purpose of education is to prepare students for the workforce; represent the belief that technology is the solution to the problems that education is facing; are indicative of scholarly failures; seem to represent the belief that education is a product that can be packaged, automated, and delivered; and, are a response to failures by researchers, designers, administrators, and institutions to develop effective and inspiring solutions to the problems of education (alternatively, they might also represent the failure of existing systems to support creative individuals in enacting change)*.
The MOOC is an acronym that elicits strong feelings: excitement, fear, defiance, uncertainty, hope, contempt…. To address these feelings we have to address the failures of higher education and the underlying causes that have given rise to MOOCs. For this reason, instead of talking about MOOCs at my own institution, I discuss innovations and approaches that I value, including networked scholarship, openness, flexibility, social learning, and the design and development of new technologies.
* NOTE: Rolin Moe and I are working on a paper refining and delineating these. If you have thoughts, concerns, or input on any of these issues, we’d love to hear form you!
We have a new paper available that continues our research on networked participation/learning and networked participatory scholarship. This one arises out of Royce Kimmons’ dissertation, which I had the joy, honor, and good fortune of chairing.
The media is filled with stories pertaining to educators’ and researchers’ participation in online social networks. For example, a debate erupted in Kansas in December 2013 regarding faculty members’ use of social media and teachers have found themselves in trouble for their social media updates. Yet, little research has been done to understand the relationship between educator identity and participation in Social Networking Sites (SNS) or to examine the implications that institutional regulation of such media may have upon educator identity.
In our latest research study, we developed a framework to understand how a group of teacher education students viewed their developing identities within social networking sites as they began the life transition to becoming educators. We found that educator identity consists of a constellation of interconnected acceptable identity fragments (AIF)*. These acceptable identity fragments are intentional, authentic, transitional, necessarily incomplete, and socially-constructed and socially-responsive.
We arrived at the term “acceptable identity fragment,” because study participants:
- shaped their participation in social networking sites in a manner that they believed to be “acceptable” to their audiences,
- viewed this participation to be a direct expression of “identity” or their sense of self, and
- felt this expression to only represent a small “fragment” of their complete identities.
The AIF suggests that participants in a given social context may limit their participation or expression of identity in a way that is appropriate to that specific context or is acceptable to the specific relationships they have with others in that context. The existence of the AIF means that educator identities within SNS are contextual and intentionally limited and structured. Participants believe that, when participating in SNS, they are expressing their identities in a limited, though authentic, manner. In their view, such expression represents a genuine fragment of their identities.
This view of educator identity contrasts sharply with previous views of identity by highlighting the complicated, negotiated, and recursive relationship that exists between educator participation in SNS and educator identity.
First, existing literature assumes that individuals have an authentic identity and suggests that they attempt to express these identities in varying degrees via social media. Our research finds that human beings may not ever find themselves in social contexts wherein they will choose to (or are even able to) express their full authentic identities and, instead, express a different AIF depending upon the situation.
Second, in Goffman’s view (1959), identity is adaptable and constantly emergent as we “act” in contexts. In the AIF view, there is no “acting” occurring, but rather we see a guarded revelation of fragments of the self. Thus, identity was not an emergent phenomenon of the scene; it was controlled and revealed partially.
Finally, Turkle (1995) suggests that the online self lacks coherence and is fluid. However, participants in our study were operating from what they believed to be a coherent sense of self and judged their SNS participation based upon alignment with that sense. Participation did not lack coherence – it was merely a partial manifestation.
What does this mean for educators, educational administrators, and educational researchers?
First, if the AIF is intentional and authentic, then it seems important for educators to retain control of their SNS participation. If institutions seek to prescribe appropriate and inappropriate uses of the medium, then it seems that this will prevent educators from being able to make meaningful choices regarding authentic self-expression and self-representation
Second, if the AIF is transitional, social media technologies must accommodate individuals’ transition into new life phases. At present, social media spaces do not support this (e.g., Facebook’s Timeline and the difficulty of deleting participation history en masse). If technologies doe not support the transition into new life phases, they risk being abandoned.
Third, educators should seek to recognize the assumptions that SNS platform developers are making about human nature, meaningful social participation, relationships, and so forth and consider the impact that such assumptions may have on their participation and identity.
Fourth, judgments made about educators based upon their participation in SNS should consider life transitions, time-based contexts (e.g., behavior as a college freshman vs. behavior as a student teacher), and the embedded values of the media.
Finally, if the AIF is a necessarily incomplete component of a larger identity constellation, any judgments of educators based on SNS participation must recognize that the relationship of the AIF to overall identity is subject to interpretation and may not reflect an individual’s perception of how the AIF represents authentic identity. Fragmentation of identity, then, should be seen as a valuable response to complex social situations. SNS platforms should account for this, and as we make judgments about others based upon their fragmented identities, we should be cognizant of the complex relationship existing between the AIF and one’s larger identity and dispel the myth of a simple authentic vs. inauthentic binary.
You can download a pre-print copy of the study from the link below:
Kimmons, R., & Veletsianos, G. (2014). The Fragmented Educator 2.0: Social Networking Sites. Acceptable Identity Fragments, and the Identity Constellation. Computers & Education, 72, 292-301. Journal link.
* The usual grounded theory and interpretive research caveats apply.
Are you interested in a post-doctoral fellowship in any of the following topics?
- open online learning
- emerging forms of online participation
- digital and open scholarship
- online social networks
- learner, instructor, and scholar experiences in any of the above
If so, I would love to see an application from you to our call for Banting post-doctoral fellows! The call is open to Canadians and non-Canadians alike.
On the call listed above, you will see that we are seeking applicants for multiple positions. The section relevant to my interests is the following:
Working with Dr. George Veletsianos, Canada Research Chair (CRC) in Innovative Learning and Technology, the post-doc will focus on emerging technologies and innovations in online education, and in particular open education, open/digital scholarship, and social media/networks. The experiences and practices of learners, instructors, and scholars with emerging forms of online participation (e.g., MOOCs, social media) are ill-understood and ill-researched. The objective of a Banting post-doc within this research program will be to make sense of participants’ experiences and practices with open online education and social media/networks in higher education and to understand why individuals use these emerging innovations in the ways that they do. Research questions may include, but are not limited to: What is the nature of open online learning, teaching, and participation? What does the experience of open online learning/teaching and/or social network learning consist of? What is the lived experience of open scholars? How is technology changing scholarship? How do scholars perceive and construct their identity using social media/networks? How do individuals use social media/networks to cope with the expectations of their academic roles (e.g., being a doctoral student, being a newly-hired faculty member, etc)?
What is the value of a critique?
The value of critique is to help us see a phenomenon through a different lens, to help us make sense of something in a different way, and to spark a conversation. This is the purpose, and value, of a paper we recently published with IRRODL on the topic of open scholarship.
The paper identifies the assumptions and challenges of openness and open scholarship and attempts to put forward suggestions for addressing those. A summary of our paper, appears below:
Many scholars hope and anticipate that open practices will broaden access to education and knowledge, reduce costs, enhance the impact and reach of scholarship and education, and foster the development of more equitable, effective, efficient, and transparent scholarly and educational processes. Wiley and Green (2012, pp. 88) note that “only time will tell” whether practices of open scholarship will transform education or whether the movement “will go down in the history books as just another fad that couldn’t live up to its press.” Given the emerging nature of such practices, educators are finding themselves in a position in which they can shape and/or be shaped by openness (Veletsianos, 2010). The intention of this paper is (a) to identify the assumptions of the open scholarship movement and (b) to highlight challenges associated with the movement’s aspirations of broadening access to education and knowledge. The goal of this paper is not to frame open scholarship as a problematic alternative to the status quo. Instead, as we see individuals, institutions, and organizations embrace openness, we have observed a parallel lack of critique of open educational practices. We find that such critiques are largely absent from the educational technology field, as members of the field tend to focus on the promises of educational technologies, rarely pausing to critique its assumptions. Selwyn (2011b, pp. 713) even charges that our field’s inherent positivity “limits the validity and credibility of the field as a site of serious academic endeavour.” Our intention is to spark a conversation with the hopes of creating a more equitable and effective future for digital education and scholarship. To this end, this paper is divided into three major sections. First, we review related literature to introduce the reader to the notion of open scholarship. Next, we discuss the assumptions of openness and open scholarship. We then identify the challenges of open scholarship and discuss how these may limit or problematize its outcomes.
Common assumptions and challenges are summarized as follows:
|Common themes and assumptions||Challenges|
|Open scholarship has a strong ideological basis rooted in an ethical pursuit for democratization, fundamental human rights, equality, and justice.||Are these ideals essential components of the open scholarship movement or are merely incidental to those who are pioneering the field?|
|Open scholarship emphasizes the importance of digital participation for enhanced scholarly outcomes||Scholars need to develop an understanding of participatory cultures and social/digital literacies in order to take full advantage of open scholarship.Need to redesign university curricula to prepare future scholars to account for the changing nature of scholarship.
|Open scholarship is treated as an emergent scholarly phenomenon that is co-evolutionary with technological advancements in the larger culture||Technology both shapes and is shaped by practice.Technology is not neutral, and its embedded values may advance tensions and compromises (e.g., flat relationships, homophily, filter bubbles).|
|Open scholarship is seen as a practical and effective means for achieving scholarly aims that are socially valuable||Open scholarship introduces new dilemmas and needs (e.g., personal information management challenges; Social stratification and exclusion).|
Given the topic, the best home for this paper was the International Review Of Research In Open And Distance Learning, through which you can download the paper for free in an open access manner:
I am in Cyprus to meet with a number of colleagues and give an invited talk at ICEM 2012.
Talk title: What does the future of design for online learning look like? Emerging technologies, Openness, MOOCs, and Digital Scholarship
Abstract: What will we observe if we take a long pause and examine the practice of online education today? What do emerging technologies, openness, Massive Open Online Courses, and digital scholarship tell us about the future that we are creating for learners, faculty members, and learning institutions? And what does entrepreneurial activity worldwide surrounding online education mean for the future of education and design? In this talk, I will discuss a number of emerging practices relating to online learning and online participation in a rapidly changing world and explain their implications for design practice. Emerging practices (e.g., open courses, researchers who blog, students who use social media to self-organize) can shape our teaching/learning practice and teaching/learning practice can shape these innovations. By examining, critiquing, and understanding these practices we will be able to understand potential futures for online learning and be better informed on how we can design effective and engaging online learning experiences. This talk will draw from my experiences and research on online learning, openness, and digital scholarship, and will present recent evidence detailing how researchers, learners, educators are creating, sharing, and negotiating knowledge and education online.
In their paper “Intentional Web Presence: 10 SEO Strategies Every Academic Needs to Know” Patrick Lowenthal and Joanna Dunlap offer excellent advice to academics mindful of their web presence and cognizant of the potential impact that the Internet may have on their scholarship. I’ve come to use most of these strategies over the years, but I am excited to see these collected at one location.
I’ll add an 11th strategy: Use an RSS aggregator to (e.g., Google Reader) to gather resources of interest effortlessly and consistently. For example, I receive alerts of the latest journal issues at my aggregator (you can also have these emailed to you). I also follow a number of colleagues’ blogs through there, so I don’t have to visit individual sites. My RSS aggregator also serves as an archiving mechanism.
As academics and scholars engage in the emerging practice of using “participatory technologies and online social networks to share, reflect upon, critique, improve, validate, and further their scholarship” (which is an argument that we made in this paper), these strategies are important to keep in mind.
What other strategies do you use?