Over the last year or so, we’ve interviewed more than 200 individuals who have participated in a number of open courses. We are working on a project in which we are using learner narratives and vignettes from these interviews to help administrators, faculty, researchers, and learning designers understand learners and improve their learning experience. Though there are many ways that are used to understand learners (e.g., dashboards) we believe that in-depth vignettes of typical experiences may allow for greater sensitivity of the learners’ lifeworld and realities. We will be using these stories to problematize various aspects of digital learning. Each story will be followed by a longer analysis of the issues raised in the story. For now, below is one such (DRAFT!) story. What do you think? Is there anything else that you’d like to see in this narrative? Is it interesting? If you are an administrator, faculty, researcher, or learning designer, does this story add anything valuable?
Title: Why not?
Theme: Open learning opportunities are oftentimes costless and relatively risk-free.
Mary and her demanding Pomeranian, Kylie, live deep in the heart of Texas. “I have a passion for the law!” the thirty-year-old exclaimed when we called her on her landline. She had seriously considered going to law school and had even passed her LSATS, the law school entrance exams used for US Universities. But having just finished four intense years of a bachelor’s degree, she decided to wait a bit. “Law school just didn’t seem like a good choice at the time,” she reflected. Five years later, Mary has settled into her work as a business consultant. Her interest in the law is still keen, and she’s never completely given up the dream of law school, but it’s been tempered with a bit of realism. “I don’t know if I can afford to spend another three years in the classroom,” she confided to us, “I don’t know if I still have the same passion for the legal industry as I did five years ago.”
During an afternoon enjoying frozen mango margaritas with a friend, trying to cope with the scorching sun, Mary learned about MOOCs. Shortly thereafter, she signed up for a number of courses, dabbling in some and promptly forgetting about others. One day, ContractsX, a course on contract law taught by a Harvard professor, popped up on her screen and she decided to “give it a shot”. What had she got to lose? “It’s a free class, taught at one of the more well-respected institutions. Why not?!” she laughed.
The course was flexible and fit into her busy life. On Saturday mornings she would sit in her office, with Kylie by her side and a warm cup of dark roast coffee in her hand, and use her trusted iPad to watch Harvard Law lectures. These weren’t just any lectures. Professor Fried was a masterful storyteller, a king of his trade. It was through these short, interesting, and memorable stories that Professor Fried taught concepts relating to contract law. “I can’t believe that I’m sitting here, I’m learning this material from Harvard law!” The fast pace and cramped content made the course challenging, Mary acknowledged, and she didn’t always do as well as she would have liked on the course tests. But, as she was able to go back to review the answers and re-watch the videos, this didn’t stress her too much, and she ended up passing the course with flying colours. Proud of her certificate of accomplishment, Mary enthused, “It makes me want to keep coming back for more!”
Even though it was a personal interest in the law that led her to sign up for this course, Mary has found what she learned in ContractsX helpful when she has to deal with contracts in her own job. She has enthusiastically recommended the course to co-workers and friends. She’s currently taking a number of other open courses and is anxiously awaiting the second version of the Contracts course. While Mary’s dream of attending law school, may not have changed, her confidence in herself has: “I never thought of applying to Harvard. There was no way I would be getting in. But then, five years later, I’m taking a course from Harvard. I wouldn’t say that I’m a Harvard law student, but at least now I could sit across from a Harvard law student and have a clear conversation with them. It’s very rewarding to know that.”
A number of literature reviews have been published on MOOCs. None has focused exclusively on the empirical literature. In a recent paper, we analyzed the empirical literature published on MOOCs in 2013-2015 to make greater sense of who studies what and how. We found that:
- more than 80% of this literature is published by individuals whose home institutions are in North America and Europe,
- a select few papers are widely cited while nearly half of the papers are cited zero times,
- researchers have favored a quantitative if not positivist approach to the conduct of MOOC research,
- researchers have preferred the collection of data via surveys and automated methods
- some interpretive research was conducted on MOOCs in this time period, but it was often basic and it was the minority of studies that were informed by methods traditionally associated with qualitative research (e.g., interviews, observations, and focus groups)
- there is limited research reported on instructor-related topics, and
- even though researchers have attempted to identify and classify learners into various groupings, very little research examines the experiences of learner subpopulations (e.g., those who succeed vs those who don’t; men vs women).
We believe that the implications arising from this study are important for research on educational technology in general and not jut MOOC research. For instance, given the interest on big data and automated collection/analysis of the data trails that learners leave behind on digital learning environments, a broader methodological toolkit is imperative in the study of emerging digital learning environments.
Here’s a copy of the paper:
Veletsianos, G. & Shepherdson, P. (2016). A Systematic Analysis And Synthesis of the Empirical MOOC Literature Published in 2013-2015. The International Review of Research in Open and Distributed Learning, 17(2).
The British Journal of Educational Technology and BERA approached us to create an infographic for the article we (Amy Collier, Emily Schneider, and myself) published last year: Digging Deeper into Learners’ Experiences in MOOCs: Participation in social networks outside of MOOCs, Notetaking, and contexts surrounding content consumption
Below is the outcome (and a pdf version is here):
The New York Times published an article on an edX course (Introduction to Mao Zedong Thought) offered by Tsinghua University. Inside Higher Ed (IHE) wrote about it, too. The following quote from IHE articles summarizes the articles:
“That course is raising eyebrows because, despite hours of video lectures and supplemental material in the course, students would still have to tab over to Wikipedia to learn about the millions who died as a result of Mao’s land reforms or that his economic initiatives led to what may have been the greatest famine in human history, which killed tens of millions. Introduction to Mao Zedong Thought references those events glancingly in passing as “mistakes,” and generally heaps praise on Mao and his philosophies.”
I was asked to provide commentary for the New York Times article, and since it wasn’t included in the writeup, I thought it would be a good idea to share it publicly rather than leave it hidden away in my email inbox. Here is what I said:
Open courses are transparent, and that’s one of their positive aspects. They allow anyone to examine the ways that course creators think about a topic. The instructional materials from the Mao course are available to anyone to examine and study. One can look at the materials and ask: How do these materials position Mao Zedong? What are the elements of Mao’s thought that the creators of this course want to highlight? What elements of Mao’s thoughts are left behind and what are the elements that are being highlighted? What is the story that is being told here, and who stands to benefit from this story?
Stephen Downes made a similar argument in the IHE article: ““courses that might have been offered behind closed doors are offered for everyone to see.”
Now, that’s parsimonious :)
We recently published a special issue for Educational Media International by asking authors to submit papers focusing on the following question: What is it like to learn and participate in MOOCs? This has now been published.
We developed this special issue to enhance our collective understanding of learner experiences and participation in MOOCs because the scholarly community still has an incomplete mosaic of students’ learning experiences with open online learning.
The following papers are included:
Editorial: Contributions to the mosaic describing learners’ experiences with open online learning (pdf)
George Veletsianos and Vrasidas Charalambos
Learning from MOOCs: a qualitative case study from the learners’ perspectives
Yeonjeong Park, Insung Jung and Thomas C. Reeves
A classroom at home: children and the lived world of MOOCs
Yin Yin, Catherine Adams, Erika Goble and Luis Francisco Vargas Madriz
What makes a cMOOC community endure? Multiple participant perspectives from diverse cMOOCs
Maha Bali, Maureen Crawford, Rhonda Jessen, Paul Signorelli and Mia Zamora
Fulfilling the promise: do MOOCs reach the educationally underserved?
Lorrie Schmid, Kim Manturuk, Ian Simpkins, Molly Goldwasser and Keith E. Whitfield
Examining learners’ perspective of taking a MOOC: reasons, excitement, and perception of usefulness
M. Liu, J. Kang and E. McKelroy
- Note: While the journal is not open access, a number of the authors above have self-archived copies of their paper, like I am doing above.
Spoiler: We’ve been toying with automating the collection of literature on MOOCs (and other topics). Interested? Read further.
Researchers use different ways to keep updated with the literature on a topic. On a daily basis for example, I use Table of Content (TOC) alerts, RSS feeds, and Google Scholar alerts. Many colleagues have sought to keep track of literature on a topic and share it. For example, danah boyd maintained this list of papers on Twitter and microblogging; Tony Bates shared a copy of the MOOC literature he collected on his blog; Katy Jordan also kept a collection of MOOC literature.
A Google Scholar Alert
The problem with maintaining an updated list of relevant literature on a topic is that it quickly becomes a daunting and time-consuming task, especially for popular topics (like MOOCs or social media or teacher training).
In an attempt to automate the collection and sharing of literature, my research team and I created a python script that goes through the Google Scholar alert emails that I receive (see above), parses the content of the emails, and places it in an html page on my server, from where others can access it. The script runs daily and any new literature is added to the page.
We aren’t there just yet, but here is the output for the MOOC literature going back to November 2012. All 400 pages. I placed it in a Google Document because the html file is 2.5mb (and its easier for people to just download it in a format that they prefer)
In theory this is supposed to work quite well, but there’s a couple of problems with it:
- The output is as good as the input. Google Scholar (and its associated alerts) are a black box – meaning there’s no transparency of what is and isn’t indexed.
- It’s automated – which means it’s not clean and some “mooc literature” may not really be mooc literature because Google Scholar alerts work on keywords in the body of papers/text rather than keywords describing the papers/text.
We plan on to make the source code available and describe the process to install this so that others can use it for their own literature needs. My question is: How can the output be more helpful to you? Is there anything else that we can do to improve this?
The op-ed below first appeared on Inside Higher Ed. This is a first glimpse of a partnership with HarvardX intended to examine learners’ experiences in open courses.
The Invisible Learners Taking MOOCs
“Anyone, anywhere, at any point in time will be able to take advantage of high quality education.”
That could be a tagline from just about any enthusiast or provider of open online courses (often called MOOCs). The intention certainly seems laudable and, if not transformational, at least desirable.
What are the caveats?
Recent research suggests that the majority of people enrolled in these open online courses are highly educated. As far as US participants are concerned, a large percentage also live in high-income neighborhoods.
And yet, despite the extensive research and data on open online courses, we really do not know much about these millions of learners engaged in everything from courses on computer science to poetry to physiotherapy to gender studies to bioinformatics.
In fact, apart from a few anecdotes of extraordinary individuals who overcome insurmountable struggles to succeed (e.g., the exceptional Nigerian man who completed 250 courses) or abstract descriptions of learners and their activity (e.g., “less than 10% complete courses,” “auditors,” or “latecomers”) these learners might as well be invisible.
And thus, my fellow researchers and I are asking more questions. We want to better understand open courses and their learners (and their successes and their failures). How do these people experience open courses? Why they do they things that they do in these courses?
We are currently in the midst of conducting the largest series of interview studies in open courses, and we have just released our first study. Our research is motivated by the fact that very few commentators and researchers to date have paused to talk to learners and to listen to them describe their experiences and activities.
In fact, what researchers know about MOOCs is largely the result of analyzing the data trails that learners leave behind as they navigate digital learning environments.
So far we have interviewed more than 70 individuals who have completed a range of MOOCs. Three of our initial findings question the initial excitement that surrounded MOOCs and contradict the initial hope that these types of courses can help anyone, anywhere, at any point in time to succeed.
Successful online learners have sophisticated study skills. For example, nearly every individual that we have interviewed described his or her notetaking strategies. Learners described how they combine notes across multiple courses and how they arrange notes in order to use them in their exams or future studies.
Learners also described an array of strategies to deal with unfamiliar content, such as using resources external to MOOCs to clarify their understanding of what they learned.
Bjorn, one of the learners we interviewed, reported watching all lecture videos twice. He said: “I read an article about how priming really helps the mind cement content.” And then he applied that insight to his studies: “Instead of watching the videos and taking notes and pausing constantly,” he “watched the video in fast speed first, just really concentrating on the content, and then afterwards, watched it through while taking notes.” This strategy was aimed at improving the processing of new information and demonstrates the sophistication with which some learners approach studying.
Such complex approaches to studying are neither innate nor universal, and throw a cast of doubt over the claim that “anyone” can equally participate in and benefit from these courses.
Flexibility and a flexible life are often essential for engaged participation. A significant proportion of the learners we interviewed either live flexible lives that enable them to participate or appear to be exceptional in their abilities to create time to participate in these courses.
Individuals that live flexible lives are often retirees who frequently tell us that they have time available to explore topics that interest them. Numerous others shared with us that they create time to participate.
For example, a British engineer goes to work an hour earlier every morning in order to work on MOOCs, and an American mother watches MOOC videos when she is not busy caring for her newborn.
Personal and professional circumstances structure the ways that people participate in MOOCs. And here is the conundrum: While online learning experiences can generally be more flexible than face-to-face ones, time is a limited resource, and the individuals who have the privilege of time and flexibility are not necessarily the ones that the quest for the democratization of education via MOOCs aspired to serve.
Online learning is an emotional experience. Somewhere between enrollment numbers, statistics describing completion rates, and the fascination with big data, we forgot that learning experiences are deeply emotional.
Anxiety, appreciation, embarrassment, and pleasure are some of the emotions that learners used to describe their experience in these courses to us.
One of our interviewees, Maria, lives in Greece and works for the public sector. She was “pleasantly surprised” with her experience, especially because she “never thought [she] would be able to study the subject.” She continued: “The most important thing for me is that I can actually learn about the things I have wanted to learn about ever since I was a child. It’s really a dream come true. I will never be able to use it for work or I will never be able to change my profession under the circumstances right now – but I really like and I really want to learn about astronomy and cosmology just for the – just for the joy of it. And that’s why I am going to keep on taking classes.”
Understanding why learners had these emotions is significant in improving digital learning initiatives. More importantly, innovation that lacks care and appreciation for the human condition is not an aspirational strategy to get behind for a bright future.
Our research is providing a better understanding of open online learning and the learners that participate in such endeavors. We are finding that the democratization of education and knowledge are noble goals, but free access to content can only go so far in eliminating societal and global inequities.
What’s the value of a course that features high completion rates but perpetuates gender stereotypes?
What’s the value of a course that is freely available but cannot be accessed by people in remote areas of East Texas or remote areas of British Columbia because of language or technological barriers?
Alternatively, isn’t a course that helps people explore their passions desirable, even if only a small minority participate for the duration of it?
We’ve interviewed learners in Australia, Canada, El Salvador, France, Greece, India, Ireland, the Netherlands, Puerto Rico, the United Kingdom, and the United States. These individuals are not mere statistics to which phrases like “any” “always” and “anywhere” can apply.
Ultimately, our research calls into question whether open courses, in their current form, are the democratizing forces they are sometimes depicted to be—and even whether “educating a billion” with MOOCs is a laudable goal.
By getting to know these invisible learners, we think we can build a better foundation for online learning, the design of digital learning experiences, and the use of technology in education. It is already clear from our initial interviews that in order to create more egalitarian structures for education, we need to start peeling away the multitude of barriers that prevent the most vulnerable populations from participating. And that’s a good goal for all of us who care about learning, teaching, and education.
Acknowledgements: Numerous colleagues, research associates, and students contributed to the research reported here, including: Amy Collier (Stanford), Emily Schneider (Stanford), Peter Shepherdson (University of Zurich), Laura Pasquini (Royal Roads University), and Rich McCue (University of Victoria & Royal Roads University). Special thanks to Justin Reich and Rebecca Petersen (Harvard University) and the rest of the HarvardX research team.
Multidisciplinary, interdisciplinary, and crossdisciplinary research represent promising approaches for studying digital learning. Prior research however, discovered that research efforts directed at digital learning via MOOCs were dominated by individuals affiliated with education (Gašević, Kovanović, Joksimović, and Siemens, 2014). In their assessment of proposals submitted for funding under the MOOC research initiative (MRI), Gašević and colleagues show that more than 50% of the authors in all phases of the MRI grants were from the field of education. This result was interesting because a common perception in the field is that the MOOC phenomenon is “driven by computer scientists” (p. 166).
We were curious to understand whether this was the case with research conducted on MOOCs (as opposed to grant proposals) and used a dataset of author affiliations publishing MOOC research in 2013-2015 to examine the following questions:
RQ 1: What are the disciplinary backgrounds of the authors who published empirical MOOC research in 2013-2015?
RQ 2: How does the disciplinary distribution of the authors who published MOOC research in 2013-2015 compare to that of the submissions to the MRI reported by Gašević et al. (2014)?
RQ 3: Is the 2013-2015 empirical research on MOOCs more or less interdisciplinary than was previously the case?
Results from our paper (published in IRRODL last week) show the following:
– In 2013-2015, Education and Computer Science (CS) were by far the most common affiliations for researchers writing about MOOCs to possess
– During this time period, the field appears to be far from monolithic, as more than 40% of papers written on MOOCs are from authors not affiliated with Education/CS.
– The corpus of papers that we examined (empirical MOOC papers published in 2013-2015) was less dominated by authors from the field of education than were the submissions to the MOOC Research Initiative.
– A comparison of affiliations with past published papers shows that recent MOOC research appears to be more interdisciplinary than was the case in research published in 2008–2012.
We draw 2 implications from these results:
1. Current research on MOOCs appears to be more interdisciplinary than in the past, suggesting that the scientific complexity of the field is being tackled by a greater diversity of researchers. This suggests that even though xMOOCs are often disparaged for their teacher-centric and cognitivist-behaviorist approach, empirical research on xMOOCs may be more interdisciplinary than research on cMOOCs.
2. These results however, also lead us to wonder whether the trend toward greater interdisciplinarity of recent research might reflect (a) the structure and pedagogical model used in xMOOCs, (b) the greater interest in the field of online learning, and (c) the hype and popularity of MOOCs. Could it be that academics’ familiarity with the xMOOC pedagogical model make it a more accessible venue in which researchers from varying disciplines can conduct studies? Or, is increased interdisciplinary attention to digital education the result of media attention, popularity, and funding afforded to the MOOC phenomenon?
We conclude by arguing that “The burgeoning interest in digital learning, learning at scale, online learning, and other associated innovations presents researchers with the exceptional opportunity to convene scholars from a variety of disciplines to improve the scholarly understanding and practice of digital learning broadly understood. To do so however, researchers need to engage in collaborations that value their respective expertise and recognize the lessons learned from past efforts at technology-enhanced learning. Education and digital learning researchers may need to (a) take on a more active role in educating colleagues from other disciplines about what education researchers do and do not know about digital learning from the research that exists in the field and, (b) remain open to the perspectives that academic “immigrants” can bring to this field (cf. Nissani, 1997).”
For more on this, here’s our paper.