Canada Research Chair in Innovative Learning and Technology & Associate Professor at Royal Roads University

Category: online learning

Multidisciplinary, interdisciplinary, and cross-disciplinary research on MOOCs and digital learning

Posted on July 2nd, by George Veletsianos in moocs, my research, online learning, scholarship. No Comments

Multidisciplinary, interdisciplinary, and crossdisciplinary research represent promising approaches for studying digital learning. Prior research however, discovered that research efforts directed at digital learning via MOOCs were dominated by individuals affiliated with education (Gašević, Kovanović, Joksimović, and Siemens, 2014). In their assessment of proposals submitted for funding under the MOOC research initiative (MRI), Gašević and colleagues show that more than 50% of the authors in all phases of the MRI grants were from the field of education. This result was interesting because a common perception in the field is that the MOOC phenomenon is “driven by computer scientists” (p. 166).

We were curious to understand whether this was the case with research conducted on MOOCs (as opposed to grant proposals) and used a dataset of author affiliations publishing MOOC research in 2013-2015 to examine the following questions:

RQ 1: What are the disciplinary backgrounds of the authors who published empirical MOOC research in 2013-2015?

RQ 2: How does the disciplinary distribution of the authors who published MOOC research in 2013-2015 compare to that of the submissions to the MRI reported by Gašević et al. (2014)?

RQ 3: Is the 2013-2015 empirical research on MOOCs more or less interdisciplinary than was previously the case?

Results from our paper (published in IRRODL last week) show the following:

- In 2013-2015, Education and Computer Science (CS) were by far the most common affiliations for researchers writing about MOOCs to possess
- During this time period, the field appears to be far from monolithic, as more than 40% of papers written on MOOCs are from authors not affiliated with Education/CS.
- The corpus of papers that we examined (empirical MOOC papers published in 2013-2015) was less dominated by authors from the field of education than were the submissions to the MOOC Research Initiative.
- A comparison of affiliations with past published papers shows that recent MOOC research appears to be more interdisciplinary than was the case in research published in 2008–2012.

We draw 2 implications from these results:

1. Current research on MOOCs appears to be more interdisciplinary than in the past, suggesting that the scientific complexity of the field is being tackled by a greater diversity of researchers. This suggests that even though xMOOCs are often disparaged for their teacher-centric and cognitivist-behaviorist approach, empirical research on xMOOCs may be more interdisciplinary than research on cMOOCs.

2. These results however, also lead us to wonder whether the trend toward greater interdisciplinarity of recent research might reflect (a) the structure and pedagogical model used in xMOOCs, (b) the greater interest in the field of online learning, and (c) the hype and popularity of MOOCs. Could it be that academics’ familiarity with the xMOOC pedagogical model make it a more accessible venue in which researchers from varying disciplines can conduct studies? Or, is increased interdisciplinary attention to digital education the result of media attention, popularity, and funding afforded to the MOOC phenomenon?

We conclude by arguing that “The burgeoning interest in digital learning, learning at scale, online learning, and other associated innovations presents researchers with the exceptional opportunity to convene scholars from a variety of disciplines to improve the scholarly understanding and practice of digital learning broadly understood. To do so however, researchers need to engage in collaborations that value their respective expertise and recognize the lessons learned from past efforts at technology-enhanced learning. Education and digital learning researchers may need to (a) take on a more active role in educating colleagues from other disciplines about what education researchers do and do not know about digital learning from the research that exists in the field and, (b) remain open to the perspectives that academic “immigrants” can bring to this field (cf. Nissani, 1997).”

For more on this, here’s our paper.

Learners’ experiences in MOOCs: Notetaking, social networks, and content consumption

Posted on May 26th, by George Veletsianos in learner experience, moocs, my research, online learning, papers. 26 comments

What do learning experiences in MOOCs look like? Amy Collier, Emily Schneider and I have just published a paper that provides some in-depth answers to this question. Here is a copy of the paper in pdf. The paper is part of a special issue published by the British Journal of Educational Technology which can be found here (there are many excellent pieces in that issue, so be sure to read them).

In addition to trying to understand learner experiences, in the paper we describe that we did this study because ”ease of access to large data sets from xMOOCs offered through an increasing number of centralized platforms has shifted the focus of MOOC research primarily to data science and computational methodologies, giving rise to a discourse suggesting that teaching and learning can be fully analyzed, understood and designed for by examining clickstream data”

Our abstract reads:

Researchers describe with increasing confidence what they observe participants doing in massive open online courses (MOOCs). However, our understanding of learner activities in open courses is limited by researchers’ extensive dependence on log file analyses and clickstream data to make inferences about learner behaviors. Further, the field lacks an empirical understanding of how people experience MOOCs andwhy they engage in particular activities in the ways that they do. In this paper, we report three findings derived by interviewing 13 individuals about their experiences in MOOCs. We report on learner interactions in social networks outside of MOOC platforms, notetaking, and the contexts that surround content consumption. The examination and analysis of these practices contribute to a greater understanding of the MOOC phenomenon and to the limitations of clickstream-based research methods. Based on these findings, we conclude by making pragmatic suggestions for pedagogical and technological refinements to enhance open teaching and learning.

We reported 3 main findings:

1. Interactions in social networks outside of the MOOC platform

A number of learners alluded to interactions they have had with individuals who are part of their social networks. These include digital connections with other participants in a MOOC, face-toface interactions with friends and family, and face-to-face interactions with new connections in a MOOC.

2. Notetaking

Despite the fact that none of the popular MOOC platforms support integrated notetaking at the time of writing this paper, nearly all interviewees reported taking notes while watching lecture videos. Only one interviewee never took notes. However, the tools used to take notes and the subsequent use of notes varied substantially by learner.

3. Consuming content

All individuals participating in this study discussed factors that shaped the ways they consumed MOOC content, shedding light on the context surrounding their participation. Scholars in the learning sciences have long highlighted the critical role of the environment, arguing that learning must be understood as a sociocultural phenomenon situated in context and culture (Brown, Collins & Duguid, 1989). Patterns of MOOC content consumption can be examined by clickstream data, but these contextual factors help explain why learners exhibit particular patterns of participation.

 

 

Veletsianos, G., Collier, A., & Schneider, E. (2015). Digging Deeper into Learners’ Experiences in MOOCs: Participation in social networks outside of MOOCs, Notetaking, and contexts surrounding content consumption. British Journal of Educational Technology 46(3), 570-587.

Who studies MOOCs? (or, thinking with Siemens & Downes)

Posted on May 2nd, by George Veletsianos in my research, online learning, open, papers. 28 comments

If it wasn’t abundantly clear by now, George Siemens and Stephen Downes are two individuals that are making significant contributions to the field. I respect them both and I enjoy engaging with their work. They have been having a conversation  regarding the research and academic diversity in MOOCs, (here and here and here) as a result of a report that George and colleagues released on the history and current state of blended, distance, and online education.

I am writing to add to that conversation because my colleagues and I analyzed some parts of the literature published on MOOCs, and have some results that are relevant and interesting. The paper is under review but the editor gave me permission to share our findings.

We studied the disciplinary distribution of the authors who published MOOC research in 2013-2015 and compared it to the submissions to the MOOC Research Initiative (MRI) reported in Gašević et al., (2014). Our tests showed that the MOOC literature published in 2013-2015 differed significantly from the MRI submissions: our corpus had a greater representation of authors from Computer Science and the Gašević et al., corpus had a greater representation of authors from Education and Industry. In other words, our corpus was less dominated by authors from the field of education than were the MRI submissions. One of Downes criticisms is the following: “the studies are conducted by people without a background in education.” This finding lends some support to his claim, though a lot of the research on MOOCs is from people affiliated with education, but to support that claim further one could examine the content of this papers and identify whether an educational theory is guiding their investigations. 

We also compared author affiliation information in our papers with the papers used in Liyanagunawardena et al.’s (2013) review of the 2008-2012 MOOC literature. We found that the two samples differed significantly. For example, the Liyanagunawardena et al. corpus was relatively over-represented in the Independent Researcher category. This result suggests that the bodies of literature published in 2008-2012 and 2013-2015 differ in significant ways. This may or may not hold true for the writing that has occurred in blogs, unpublished reports etc. We don’t know and we haven’t studied that. 

Finally, and most significantly, we found that the disciplinary makeup of the literature is changing over time: there’s greater interdisciplinary activity in MOOC research now than in the past. This result is very interesting and its implications are worth examining. Suffice to say that this provides opportunities (can we capitalize on the expertise of one another to improve education?) and challenges (are newcomers to the field capitalizing on what we now about the science of learning?). The move to greater inter- and cross- disciplinarity in the field is evident in other initiatives. See for example, the Digital Learning Research Network.

Keep in mind that this research faces some of the same limitations raised by Downes (i.e. like Siemens, our inclusion criteria mean that some research is included while other is excluded). However, it also addresses some of those criticisms. For example, we tried to verify whether some of our results could have arisen by chance by running 10,000 computer simulations on the samples. The computer is confident that they could not have arisen by chance.

I’m hoping this paper will be out of peer-review soon so that I can share, but I’m thankful to the editor that allowed us to share our findings.

The Defining Characteristics of Emerging Technologies and Emerging Practices in Online Education

Posted on April 14th, by George Veletsianos in my research, online learning, scholarship. 10 comments

My amazing colleagues Amy Collier and Jen Ross have been blogging about the chapter they wrote for the second edition of my Emerging Technologies in Distanced Education book, and have thus encouraged me to blog about it :)

First, the good news: Athabasca University Press will be publishing Online Learning: Emerging Technologies and Emerging Practices in an open access format.

One of the interesting aspects of editing a book is interacting with chapter authors (15 in this case) and helping to create a coherent narrative. In this case, we were starting with the first edition of the book as a foundation, but more importantly, we were building on the first chapter of the first book in which I defined “emerging technologies.” I’ve clarified in the second book that this collection focuses as much on the emerging practices of online learning/participation (e.g., networked scholarship, open education, learning analytics) as much as it focuses on the technologies that underpin them. Hence the new title.

In the first edition of the book, I argued that emerging technologies are “not yet fully researched” and “not yet fully understood.” Thus, you can quickly begin to see that emerging technologies and emerging practices are intertwined, as, open scholarship for example, or learners’ experiences with automated assessment, are not yet fully research and not yet fully understood.

In the second edition of the book, I revisit this work, and argue that what makes technologies and practices emerging are not specific technologies or practices, but the environments in which a particular technology or practice operate. This definition recognizes that learning, teaching, and scholarship are sociocultural phenomena situated in specific contexts and influenced by the cultures in which they take place. Emerging technologies and practices appear to share four characteristics:

  • Emerging Technologies and Emerging Practices are Not Defined by Newness
  • Emerging Technologies and Emerging Practices Are Evolving Organisms that Exist in a State of “Coming into Being”
  • Emerging Technologies and Emerging Practices Are Not Yet Fully Understood or Researched (what Amy and Jen branded as not-yetness)
  • Emerging Technologies and Emerging Practices Have Promising But as Yet Unfulfilled Potential

There’s 8 more great chapters in the book, and I’m looking forward to sending them off to the publisher. More in the coming months!

Mediated learning experiences and activities in MOOCs and open courses

Posted on April 13th, by George Veletsianos in emerging technologies, moocs, my research, online learning, open. 17 comments

At AERA this week, Amy Collier, Emily Schneider, and I will be presenting a paper that makes a series of arguments regarding learner activities and experiences in MOOCs in relation to clickstream-based MOOC research. One of the implications of our work is the following: learners’ participation and experiences in these courses resist binary and monolithic interpretations as they appear to be mediated by a digital-analog continuum as well as a social-individual continuum. In other words, learning and participation in MOOCs are both distributed and individually-socially negotiated. The following visual (which provides some hints on our results) makes this point clearer:

digital-analog-social-individual

* and since the work of peer reviewers often goes unrecognized, let it be known, that this insight was prompted by a comment from one anonymous reviewer. So, whoever you are, thank you for your input.

 

Emerging Practices in Open Online Learning Environments

Posted on February 23rd, by George Veletsianos in emerging technologies, moocs, my research, NPS, online learning, open, scholarship. 33 comments

I joined Audrey Watters, Philipp Schmidt, Stephen Downes, and Jeremy Friedberg in Toronto last week, to give a talk at Digital Learning Reimagined, an event hosted and organized by Ryerson University’s Chang School. I presented some of our latest research, and tried to highlight research findings and big ideas in 15 minutes. Below are my slides and a draft of my talk.

Welcome everyone! It’s a pleasure and an honor to be here. Even though I’m the person giving this talk, I’d like to acknowledge my collaborators. A lot of the work that I am going to present is collaborative and it  wouldn’t have been possible without such amazing colleagues. These are: Royce Kimmons from the University of Idaho, Amy Collier and Emily Schneider from Stanford University, and Peter Shepherdson from the University of Zurich. The Canada Research Chairs program, the National Science Foundation and Royal Roads University have funded this work.

I want to start my talk by telling a story.

This castle that you see here is one of the most recognizable parts of Royal Roads University (RRU). But, don’t let the castle fool you. RRU was created in 1985. It’s purpose was to serve the needs of a changing society by serving working professionals through graduate digital education and multidisciplinary degrees. It has grown since 1985. It has matured, developed a social learning model that is now infused in all courses, developed new areas of focus, forged global partnerships, and continues to explore how to improve what it does through pedagogical and technological approaches.

Why am I sharing this short story about RRU?

Because this story, minus the specific details, is a common story. It’s also a Ryerson story, a story that is played out at the University of Southern New Hampshire, a story that Open Universities around that world have gone through. It is a story that repeats itself over and over for years and years.

What is the essence of the story?

It is often assumed that universities have been static, unchanging since the dawn of time. The short story I shared illustrates that universities are, and have always been, part of the society that houses them, and as societies change, universities change to reflect those societies. The economic, sociocultural, and technological pressures that universities are facing are sizable, and for better or for worse, usually for both, there’s a continuous re-imagination of education throughout time. Throughout time. Universities have always been changing.

As universities are changing and exploring different ways to offer education, faculty, researchers, and administrators engage in a number of practices that I like to describe as emerging. Emerging practices & emerging technologies are those that are not necessarily new, not yet fully researched, but appear promising.

Online learning and openness are example of emerging practices.

Online learning has a long history. But it also has a new history, with the development of multimedia platforms, media that can be embedded across platforms, syndication technologies that enable learners to use their own platforms for learning and so on. So, even though some of the problems that online learners are facing in contenmporary situations are not new (eg dropout), learners abilities’ to congregate in online communities is expanded through newer technologies and that poses different sorts of challenges and opportunities.

Another emerging practice is openness. Openness refers to liberal policies for the use, re-use, adaptation, and redistribution of content. Openness is also a value: It refers to adopting an ethos of transparency with regards to access to information. And this ethos ranges from academics publishing their work in open formats, to teaching open courses, to creating open textbooks. And it doesn’t stop at individual academics or institutions. In 2014 the Premiers of Alberta, British Columbia, and Saskatchewan signed a Memorandum of Understanding to facilitate creation, sharing, and use of Open Educational Resources. In the same year, SSHRC, NSERC, and CIHR have drafted a tri-agency open access policy to improve access to and dissemination of research results (NSERC, 2014);

There is a growing interest in and exploration of online learning and openness, practices which are still emerging. Next, I will share four recent results from our research into these practices that I believe are interesting to consider because they reveal the tensions that exist when dealing with emerging topics.

First, research into online learning is becoming more interdisciplinary

Interdisciplinary research into online learning means that individuals from a diverse range of disciplines, not just education, are interested in making sense of online learning. It is hoped that more research into online learning and more research from multidisciplinary groups will help us learn more about online learning and about learning in general.

We have evidence to show that research into online learning is becoming more interdisciplinary. I won’t bore you with the statistics, but we measure diversity in published research using a nifty measure and found that the period 2013-2014 can be described as more interdisciplinary than the period 2008-2012.

This is a positive trend, but before I explain its significance, let me explain to you how I view technology.

My perspective on online learning centers around the idea that technology is socially shaped . That means that technology always embeds its developers’ worldviews, beliefs, and assumptions into its design and the activities it supports and encourages.

What does this mean for interdisciplinarity? This means that we have both an opportunity and a challenge.

Our opportunity: to use our respective expertise to improve education.

Our challenge: to actually do interdisciplinary thinking and to go into the study and design of future educational systems with an open mind and the realization that our own personal experiences of education may not be generalizable. A lot of educational technology is produced by people of privilege and to develop educational technology that matters and makes societal difference, we need diversity in thinking and experience.

Our second finding refers to the increasing desire to collect, mine, and analyze data trails to make inferences about human behavior and learning. This practice is often referred to as learning analytics and educational data mining. This practice is a reflection of a larger societal trend toward big data analytics. The idea is that by looking at what people do online one can understand how to improve education.

A couple of things that researchers discovered for example are:

-Students generally stop watching online videos after 4-5 minutes. This then encourages the creation of 4-5 minute lecture videos
-Students fall in discrete categories when they are in MOOCs. For example students who are just sampling content, students who are disengaged,  or they are on track for completing. Once you identify categories you can identify and support learner needs

Data trails. Nearly everything that learners do online is tracked. Can we understand learners and improve learning by analyzing their data trails?

While these approaches can help us explain what people do, they often don’t tell us why they do they things they do nor how they actually experience online education.

My colleagues and I are interviewing MOOC students to learn about their experiences in MOOCs.

I am now going to tell you about our third result. We find that learners schedule their learning, use of resources, and participation to fit their daily life. This is in stark contrast to the idea of undergraduate education situated at a university and happening at particular time periods.

One retired individual in Panama that we interviewed works on his class early in the morning every day. Why does he do that? He does that because at that time his daughter is asleep. She is homeschooled and once she wakes up she needs access to the 1 computer that they have in the household to do her own schoolwork. In this case a lack of resources necessitates this scheduling.

One individual that we interviewed moved from the UK to the USA to be with her partner. She is currently waiting for her work permit, driver’s license, and so on, and she was enrolled in multiple MOOCs at the same time. She would work on her courses on Monday because she just “wanted them out of the way,” and so she would work on these courses straight throughout the day.

The fourth and final finding that I have for you today, is that MOOC platforms to date have not offered learners the ability to keep notes, so that particular activity, by virtue of being unsupported by the platform goes undetected when researchers only look at data trails.

Unsurprisingly, learners keep notes. A number of students that we talked to described that they keep notes on paper, frequently keeping a notebook for particular courses and returning to them during exams or during times that they needed them. Learners of course also keep notes in digital format. Usually in word documents, but again documents are dedicated to particular courses, but sometimes they are dedicated to particular topics across courses.

To give you an example, of how we believe this activity could be supported in the future and how we believe innovations  can contribute to learning, we recommend designers support this practice by pedagogical innovations such as scaffolding notetaking, but also by technological innovations, by developing online systems for notetaking. What is important here is that such systems should support learning by being interoperable, by allow learners full and unrestricted access to their notes, supporting them to be able to import & export their notes between platforms. Such a design is in line with emerging ideas in the field which call for learners to own their data.

To summarize:

1. Online learning is becoming more interdisciplinary, but we need to work together and address our assumptions
2. There is excitement about learning analytics, but we also need to understand why people do the things that they do
3. For example, we see that online education needs to accommodate lives as opposed to the other way round
4. And we see that by interviewing people we can get a better sense of the things that they do that don’t get captured by the digital trails they leave behind.

Thank you for being a great audience. I am really excited to hear the speakers that follow me, as I am sure you are!

visual

A visualization of my talk, created by Giulia Forsythe

Experiences from the trenches: An add-on to the MOOC special issue CFP

Posted on December 8th, by George Veletsianos in cfp, moocs, my research, online learning, open, scholarship. 16 comments

My colleague Charalambos Vrasidas and I are editing a special issue for Educational Media International focusing on learner experiences in massive open online courses. We are interested in empirical and theoretical manuscripts as well as systematic reviews/analyses/syntheses of the literature. Preliminary abstracts are due by December 19th. We have planned for the process to be prompt and aim for the issue to be published within 8 months or so.

As part of the special issue, and prompted by a note by Al Filreis, we have decided to include a section that enables individual learners to tell their own stories about their experiences with MOOCs. If you have taken an open course and would like to write a short piece about an aspect of your experience, this section of the special issue would be relevant to you. Like all other submissions, these will be peer-reviewed as well.

Individuals interested in this route can submit a 200-word abstract summarizing their intended submission and a 200-word bio by the 19th of December to moocs@cardet.org.

Invitations to submit full papers will be send on or before January 9, 2014. Manuscripts should be formatted using APA style and should be 1,200 words long, including references. The process to be followed thereafter is as follows:

  • March 1, 2015: Full-length papers due via email at moocs@cardet.org
  • May 1, 2015:  Notification of acceptance/rejections
  • June 30, 2015: Final papers with revisions due
  • 2015: Special issue is published

 

Ethics and transparency in the Coursera Learner Outcomes Survey

Posted on December 6th, by George Veletsianos in moocs, online learning, open, scholarship. 24 comments

On December 5th, Coursera sent an email inviting individuals to participate in a survey intended to investigate whether participation on Coursera “has had any career, educational, or social impact in [their] life.” The email also stated: “Your survey response will be used as part of a research study conducted by the University of Pennsylvania, the University of Washington, and Coursera, examining the impact of MOOCs.”

Research studies examining the impact of MOOCs outside of individual courses and studies that use methods other than clickstream data, are worthwhile and needed. I applaud Coursera and its partners for the effort to address this research gap.

However, the lack of information pertaining to the research is concerning.

By clicking on the email invitation, potential participants land on a page that describes the research study as follows:

Screenshot 2014-12-06 07.40.12

Both the University of Pennsylvania and the University of Washington have offices in place to support researchers in conducting research in  ways that protect human participants (the UPenn IRB is here and the U of Washington site here). Importantly, these offices are not just regulatory: they provide help and support. The UPenn site for example states that the mission of the IRB includes providing “professional guidance and support to the research community.”

At the very minimum, potential participants should be informed about the study and should provide their agreement to participate in the study. This process is called informed consent. The University of Washington IRB website describes it as follows:

Researchers are required to obtain the informed consent of all participants in human subjects research prior to enrolling those individuals in a study. The individual’s consent must be voluntary and based upon adequate knowledge of the purpose, risks, and potential benefits of a research study. All potential participants should also be informed of their right to abstain from participation or to withdraw consent to participate at any time without reprisal. After ensuring that a person has understood the information, the researcher should then obtain the person’s consent, preferably in writing. [more details here]

This information should be written in language that a layperson can understand and should be included in the screenshot above. In online surveys, consent is usually gained by asking participants to click on a button that indicates that the individual agrees to participate in the study.

This is all missing from the Coursera survey.

Granted, Coursera is a business entity, and it is not bound by the same requirements imposed upon researchers to conduct a survey. Businesses conduct surveys and market research all the time, and none of this applies. But this isn’t just market research. The email and survey introduction have a clear statement of intent: The data will be used for research. Even if Coursera is unaware of the existence of ethical guidelines, Facebook’s emotion contagion study and other news stories on ethics (e.g., Harvard’s hidden cameras efforts) should have provided a moment to pause and ask: Are we doing all we can to ensure that we are treating each other, and our research participants especially, in an ethical and caring way?