Category: online learning
A number of literature reviews have been published on MOOCs. None has focused exclusively on the empirical literature. In a recent paper, we analyzed the empirical literature published on MOOCs in 2013-2015 to make greater sense of who studies what and how. We found that:
- more than 80% of this literature is published by individuals whose home institutions are in North America and Europe,
- a select few papers are widely cited while nearly half of the papers are cited zero times,
- researchers have favored a quantitative if not positivist approach to the conduct of MOOC research,
- researchers have preferred the collection of data via surveys and automated methods
- some interpretive research was conducted on MOOCs in this time period, but it was often basic and it was the minority of studies that were informed by methods traditionally associated with qualitative research (e.g., interviews, observations, and focus groups)
- there is limited research reported on instructor-related topics, and
- even though researchers have attempted to identify and classify learners into various groupings, very little research examines the experiences of learner subpopulations (e.g., those who succeed vs those who don’t; men vs women).
We believe that the implications arising from this study are important for research on educational technology in general and not jut MOOC research. For instance, given the interest on big data and automated collection/analysis of the data trails that learners leave behind on digital learning environments, a broader methodological toolkit is imperative in the study of emerging digital learning environments.
Here’s a copy of the paper:
Veletsianos, G. & Shepherdson, P. (2016). A Systematic Analysis And Synthesis of the Empirical MOOC Literature Published in 2013-2015. The International Review of Research in Open and Distributed Learning, 17(2).
The attention that education and educational technology are receiving are significant, and I feel fortunate to be able to participate in efforts to improve education. Nonetheless, for the past couple of years has been bombarded with announcement after announcement of what the latest and greatest technology can do for education. These announcements are almost always filled with claims about their potential impact. Here’s a clipping of a recent email i received:
Let’s pause. Research from ISTE “found that the use of education technology (EdTech) resulted in 35 percent of students showing higher scores on class assessments and 32 percent increased engagement?” The link pointed to this post, written by the amazing Wendy Drexel. Let’s hone in on what Wendy actually wrote:
“Nationwide, we are seeing powerful results from the effective use of technology in classrooms. For example, results of research by ISTE and the Verizon Foundation earlier this year into the use of education technology had teachers reporting that 35 percent of their students showed higher scores on classroom assessments; 32 percent showed increased engagement; and 62 percent demonstrated increased proficiency with mobile devices. In fact, 60 percent of participating teachers also reported that by using their mobile devices, they provided more one-on-one help to students, and 47 percent said they spent less time on lectures to the entire class.”
These are interesting and worthwhile results. But they are mischaracterized in the email above. From the summary of the research posted on the ISTE site and a more detailed report of the research (pdf), we can begin to see how some edtech companies purport that there is evidence of impact and use that to further their cause. Here’s a summary of two issues ignored by the ad/email:
- The email claims that use of edtech resulted in 35% and 32% more students scoring higher in classroom assessment and engagement. What the research actually reported about this particular area is the following: teachers reported that edtech use led to increased scores/engagement. In more plain language, the teachers said that their students did better. We don’t know if they did or did not.
- The email paints a direct and unequivocal relationship between edtech use and outcomes.: “use of educational technology resulted in.” That’s not actually the case. Why?
- The actual research showed that even though there were differences in math and science outcomes between schools that participated and schools that did not, the results were not statistically significant. In different words: similar results could be expected without the use of this particular technology
- How is the technology used? The research report notes: “During site visits, observers noted that edtech-using teachers used technology to efficiently
facilitate drill and practice test preparation activities.” In other words: Edtech helps with teaching to the test and that seems to work. Put differently: We have powerful technologies that empower people to be creative and allow global collaboration, but have created systems that put teachers in situations in which they have to use these tools in simplistic ways.
The British Journal of Educational Technology and BERA approached us to create an infographic for the article we (Amy Collier, Emily Schneider, and myself) published last year: Digging Deeper into Learners’ Experiences in MOOCs: Participation in social networks outside of MOOCs, Notetaking, and contexts surrounding content consumption
Below is the outcome (and a pdf version is here):
“Personalized learning” is that one area of research and practice that brings to the forefront many of the debates and issues that the field is engaging with right now. If one wanted to walk people through the field, and wanted to do so through *one* specific topic, that topic would be personalized learning.
Personalized cans? (CC-licensed image from Flickr)
Here’s are some of the questions that personalized learning raises:
- We have a problem with labels and meaning in this field. Heck, we have a problem with what to call ourselves: Learning Technologies or Educational Technology? Or perhaps instructional design? Learning Design? Learning, Design, and Technology? Or is it Learning Science? Reiser asks: What field did you say you were in? The same is true for personalized learning. Audrey Watters and Mike Caulfield ask what does “personalized learning” mean and what is the term’s history? Does it mean different pathways for each learner, one pathway with varied pacing for each learner, or something else?
- The Chan-Zuckerberg initiative and the Bill and Melinda Gates Foundation endorse personalized learning. What is the role of philanthropy in education in general and educational technology in particular? Should educators and researchers “beware of big donors” or should they enthusiastically welcome the support in the current climate of declining public monies?
- Where is the locus of control? Is personalization controlled by the learner? Is the control left to the software? What of shared control? Obsolete views of personalization and adaptive learning focus on how the system can control both the content and the learning process ignoring, for the most part, the learner, even though learner control appears to be an important determinant of success in e-learning (see Singhanayok & Hooper, 1998). The important question in my mind is the following: How do we balance system and learner control? Such shared control should empower students and enable technology to support and enhance the process. Downes distinguishes between personalized learning and personal learning. I think that locus of control is the distinguishing aspect, and that the role of shared control remains an open conceptual and empirical question. Debates about xMOOCx vs cMOOCs fall in here as well as the debate regarding the value of guided vs discovery learning.
- How do big data and learning analytics improve learning and participation? What are the limitations of depending on trace data? Personalized learning often appears to depend on the creation of learner profiles. For example, if you fit a particular profile you might receive a particular worked-out example or semi-completed problem, and problems might vary as one progresses through a pathway. Or, you might get an email from Coursera about “recommended courses” (see my point above regarding definitions and meanings). Either way, the role that large datasets, analytics, and educational data science – as well as the limitations and assumptions of these approaches, as we show in our research – is central to personalization and new approaches to education.
- What assumptions do authors of personalized learning algorithms make? We can’t answer this question unless we look at the algorithms. Such algorithms are rarely transparent. They often come in “black box” form, which means that what we have no insight into the processes of how inputs are transformed to outputs. We don’t know the inner workings of the algorithms that Facebook, Twitter, and Google Scholar use, and we likely won’t know how the algorithms that EdTechCompany uses work to deliver particular content to particular groups of students. If independent researchers can’t evaluate the inner workings of personalized learning software, how can we be sure that such algorithms so what they are supposed to do without being prejudicial? Perhaps the authors of education technology algorithms need a code of conduct, and a course on social justice?
- Knewton touts its personalization engine. Does it actually work? Connecting this to broader conversations in the field: What evidence do we have about the claims made by the EdTech industry? Is there empirical evidence to support these claims? See for example, this analysis by Phil Hill on the relationship between LMS use and retention/performance and this paper by Royce Kimmons on the impact of LMS adoption on outcomes. If you’ve been in the position of making a technology purchasing in K-12/HigherEd, you have likely experienced the unending claims regarding the positive impact of technology on outcomes and retention.
- And speaking of data and outcomes, what of student privacy in this context? How long should software companies keep student data? Who has access to the data? Should the data follow students from one system (e.g., K-12) to another (e.g., Higher Ed)? Is there uniformity in place (e.g., consistent learner profiles) for this to happen? How does local legislation relate to educational technology companies’ use of student data? For example, see this analysis by BCCampus describing how British Columbia’s Freedom of Information and Protection of Privacy Act (FIPPA) impacts the use of US-based cloud services. The more one looks into personalization and its dependence on student data, the more one has to explore questions pertaining to privacy, surveillance and ethics.
- Finally, what is the role of openness is personalized learning? Advocates for open frequently argue that openness and open practices enable democratization, transparency, and empowerment. For instance, open textbooks allow instructors to revise them. But, what happens when the product that publishing companies sell isn’t content? What happens, when the product is personalized learning software that uses OER? Are the goals of the open movement met when publishers use OER bundles with personalized learning software that restricts the freedoms associated with OER? What becomes of the open agenda to empower instructors, students, and institutions?
There’s lots to contemplate here, but the point is this: Personalized learning is ground zero for the field and its debates.
I’m at the Open Education 2015 conference, and I am struck by the continuing focus on costs, and the absence of theorizing openness, (and by extension OER and open textbooks). Is this a problem? Reducing costs is of course important. There’s no question about it. But whether the absence of theory is a problem depends what we believe theory does. After hearing many talks start with statements akin to “we asked faculty to use open textbooks, but…” or “we hoped the institution would embrace openness because it reduced costs, but…”, I thought that it might be worthwhile to ask more why questions:
- Why do some faculty do and others do not adopt open textbooks?
- Why do some faculty revise OER?
- Why do some faculty choose to publish their work in closed journals?
Theorizing openness can help us answer many of these questions. Because openness does not exist in a vacuum. I think that a sociocultural theoretical framing of openness can help practitioners and researchers make better sense and use of openness. Here’s a quote from a recent paper that argues for and clarifies this framing:
“A sociocultural perspective on openness, open practices and open scholarship views these practices as being socially shaped, and the technologies used to enact openness as necessarily, if not always intentionally, embedding their developers’ worldviews, values, beliefs, and assumptions into their design and the activities they support and encourage. By recognizing that open practices are shaped by social, cultural, economic, and political factors, this perspective rejects the notion that such practices are deterministic and holds that, with adequate information and evidence, learners, instructors, and researchers have the agency to accept or reject any particular technology or practice or to find alternative uses for it that will better serve their needs.” (p. 202)
Veletsianos, G. (2015). A Case Study of Scholars’ Open and Sharing practices. Open Praxis 7(3), 199-209. http://openpraxis.org/index.php/OpenPraxis/article/view/206/168
Related: See this presentation by Royce Kimmons which argues the following:
“[O]penness is more than economy. The freedoms afforded by open practices have great promise for improving the pedagogy and professionalism in our educational institutions as educators are empowered to differentiate, collaborate, and innovate in ways that were impossible under non-open paradigms.”
We recently published a special issue for Educational Media International by asking authors to submit papers focusing on the following question: What is it like to learn and participate in MOOCs? This has now been published.
We developed this special issue to enhance our collective understanding of learner experiences and participation in MOOCs because the scholarly community still has an incomplete mosaic of students’ learning experiences with open online learning.
The following papers are included:
Editorial: Contributions to the mosaic describing learners’ experiences with open online learning (pdf)
George Veletsianos and Vrasidas Charalambos
Learning from MOOCs: a qualitative case study from the learners’ perspectives
Yeonjeong Park, Insung Jung and Thomas C. Reeves
A classroom at home: children and the lived world of MOOCs
Yin Yin, Catherine Adams, Erika Goble and Luis Francisco Vargas Madriz
What makes a cMOOC community endure? Multiple participant perspectives from diverse cMOOCs
Maha Bali, Maureen Crawford, Rhonda Jessen, Paul Signorelli and Mia Zamora
Fulfilling the promise: do MOOCs reach the educationally underserved?
Lorrie Schmid, Kim Manturuk, Ian Simpkins, Molly Goldwasser and Keith E. Whitfield
Examining learners’ perspective of taking a MOOC: reasons, excitement, and perception of usefulness
M. Liu, J. Kang and E. McKelroy
- Note: While the journal is not open access, a number of the authors above have self-archived copies of their paper, like I am doing above.
We are hosting a symposium on Openness, Digital Learning, and Networked Scholarship.
Please consider joining us (for free) by visiting the livestream page (http://livestream.com/royalroads/events/4446545)
November Tuesday 17th 2015, (10am- 3pm Pacific)
Organized by the School of Education and Technology & the Canada Research Chair in Innovative Learning & Technology
This symposium is intended to raise awareness about open educational resources, open pedagogy, and emerging approaches to digital learning. It provides a showcase for the work being done at Royal Roads University (RRU) and convenes open education practitioners and researchers.
In keeping with the RRU strategic mandate, this symposium builds on the work currently being done at RRU by our Canada Research Chair (CRC) in Innovative Learning and Technology and the School of Education and Technology (SET) to investigate promising innovations in educational technology. It is an opportunity to contribute back to the open community and further the developments in this area.
|10:00 – 10:05||First Nations Welcome||Asma-na-hi (Asma) Antoine|
|10:05 – 10:15||What makes RRU unique and a hotspot for innovation?||Steve Grundy|
|10:15 – 10:25||Introductions and context||George Veletsianos|
|What can Open be: Advances at the Provincial, National & International level||Mary Burgess|
|11:05 – 11:40
|For whom, for what? Not-yetness and challenging the “stuff” of open education||Amy Collier
|11:45 – 12:55||Break|
|1:00 – 1:30
|Creative Commons: Where are we now?||Paul Stacey|
|1:35 – 2:05
|Expansive Openness: Why Reducing Cost is Not Enough for Realizing the Full Benefits of OER||Royce Kimmons|
|2:15 – 2:45
|Panel Discussion: What can Open do?
* Each panelist to weigh in on panel topic and then open to the floor for questions
|Amy Collier; Jen Ross; Royce Kimmons; Center for Teaching and Learning; RRU Library; George Veletsianos|
|2:45 – 3:00||Wrap Up||Elizabeth Childs|
* Each session, excluding the panel will consist of a 20 minute presentation followed by a 10 minute Q&A
Mary Burgess is the executive director of BCCampus which supports the work of the B.C. post-secondary system in the areas of teaching, learning and educational technology. Prior to joining BCcampus in 2012, Mary Burgess was the Director of the Centre for Teaching and Educational Technologies at Royal Roads University where she started the University’s first open educational resources project. She is a career instructional designer and longtime advocate of OER.
Dr. Elizabeth Childs is an Associate Professor at Royal Roads University and the Program Head of the MA in Learning and Technology program. Her research interests include the design and implementation of flexible learning; online networked communities and, the professional development and support for learners and faculty in these emerging online learning environments.
Dr. Amy Collier is Associate Provost for Digital Learning at Middlebury College. She provides leadership in creating and sustaining a global learning community at Middlebury through the effective use of digital pedagogies and technologies. Amy studies how digital environments can foster emergence in teaching and learning.
Dr. Steve Grundy is vice-president academic and provost at Royal Roads University. He is responsible for the overall academic direction and quality of the university’s academic programs. He is particularly interested in the directions of post-secondary education, the evolution and development of online learning and new models of university governance and leadership.
Dr. Royce Kimmons is an Assistant Professor of Instructional Psychology and Technology at Brigham Young University where he studies technology integration in K-12/higher education, emergent technologies, open education, and social networks. He received his PhD from The University of Texas at Austin and formerly served as the Director of the Doceo Center for Innovation + Learning at the University of Idaho.
Dr Jen Ross is co-director of the Digital Education group at the University of Edinburgh, teacher and former programme director on the MSc in Digital Education, and co-creator of the E-learning and Digital Cultures MOOC and the manifesto for teaching online. Her research interests include online distance education, MOOCs, digital futures, reflective practices, and museum and gallery learning and engagement.
Paul Stacey is Associate Director of Global Learning at Creative Commons. Paul’s core expertise is in adult learning, educational technology, and open education. Prior to joining Creative Commons, Paul led Open Educational Resource (OER) and professional development initiatives across all the colleges and universities in British Columbia Canada
Dr. George Veletsianos holds a Canada Research Chair in Innovative Learning and Technology and is an Associate Professor at Royal Roads University. He has dedicated his research to understanding the practices and experiences of learners, educators, and scholars in emerging online settings such as online social networks and digital environments.
I was at a small gathering last week, called the Digital Learning Research Network. It was hosted at Stanford and it aimed to explore the messiness of digital learning. This was not representative of Silicon Valley’s uncritical love affair with technology. Many colleagues wrote reflections about it: Catherine Cronin, Kristen Eshleman, Josh Kim, Jonathan Rees, Tim Klapdor, Alyson Indrunas, Adam Croom, Whitney Kilgore, Matt Crosslin, Laura Gogia, Patrice Torcivia, and Lee Skallerup Bessette (to name a few). When was the last time you were at a small conference, other than the ones focusing on blogging, and this many people took time after the event to blog about it?
The messiness of digital learning isn’t a new development. It is something that educational technology evangelists ignore, but as a researcher who has an affinity for qualitative data, and as one who is increasingly using data mining techniques on open social data, I can tell you that mess is the norm and not the exception. I’m not the only one.
For me, the conference questioned educational technology but looked to it for empowerment. It critiqued universities but saw them as places to create a more just and equitable society. It brought attention to the US-centric conversations happening in this space, but recognized that we can learn from one another. It sought research, but did not seek to emulate research-focused conferences. It allowed Dave to share his thoughts but called him out on it when it was time to stop. ;)
I see the conference as the start of a longer and larger conversation. Many of us are doing research in this space and many were missing. Let’s expand the conversation.