Category: learner experience
One of the main arguments that we made in our recent paper on MOOCs, which is also the argument that I continue in this op ed piece published in Inside Higher Ed, is that the field needs to embrace diverse research methods to understand and improve digital learning. The following passage is from our paper, and given that the paper is quite long, I thought that posting it here might be helpful:
By capturing and analyzing digital data, the field of learning analytics promises great value and potential in understanding and improving learning and teaching. The focus on big data, log file analyses, and clickstream analytics in MOOCs is reflective of a broader societal trend towards big data analytics (Eynon, 2013; Selwyn, 2014) and toward greater accountability and measurement of student learning in higher education (Leahy, 2013; Moe, 2014). As technology becomes integrated in all aspects of education, the use of digital data and computational analysis techniques in education research will increase. However, an over-reliance on log file analyses and clickstream data to understand learning leaves many learner activities and experiences invisible to researchers.
While computational analyses are a powerful strategy for making a complex phenomenon tractable to human observation and interpretation, an overwhelming focus on any one methodology will fail to generate a complete understanding of individuals’ experiences, practices, and learning. The apparent over-reliance on MOOC platform clickstream data in the current literature poses a significant problem for understanding learning in and with MOOCs. Critics of big data in particular question what is missing from large data sets and what is privileged in the analyses of big data (e.g., boyd & Crawford, 2012). For instance, contextual factors such as economic forces, historical events, and politics are often excluded from clickstream data and analyses (Carr, 2014; Selwyn 2014). As a result, MOOC research frequently examines learning as an episodic and temporary event that is divorced from the context which surrounds it. While the observation of actions on digital learning environments allows researchers to report activities and behaviors, such reporting also needs an explanation as to why learners participate in MOOCs in the ways that they do. For example, in this research, participants reported that their participation in MOOCs varies according to the daily realities of their life and the context of the course. Learners’ descriptions of how these courses fit into their lives are a powerful reminder of the agency of each individual.
To gain a deeper and more diverse understanding of the MOOC phenomenon, researchers need to use multiple research methods. While clickstream data generates insights on observable behaviors, interpretive research approaches (e.g., ethnography, phenomenology, discourse analysis) add context to them. For example, Guo, Kim, and Rubin (2014), analyzed a large data set of MOOC video-watching behaviors, found that the median length of time spent watching a video is six minutes, and recommended that “instructors should segment videos into short chunks, ideally less than 6 minutes.” While dividing content into chunks aligns with psychological theories of learning (Miller, 1956), this finding does not explain why the median length of time learners spent watching videos is six minutes. Qualitative data and approaches can equip researchers to investigate the reasons why learners engage in video-watching behaviors in the ways that they do. For example, the median watching length of time might be associated with learner attention spans. On the other hand, multiple participants in this study noted that they were fitting the videos in-between other activities in their lives – thus shorter videos might be desirable for practical reasons: because they fit in individuals’ busy lives. Different reasons might be uncovered that explain why learners seem to engage with videos for six minutes, leading to different design inspirations and directions. Because the MOOC phenomenon, and its associated practices, are still at a nascent stage, interpretive approaches are valuable as they allow researchers to generate a refined understanding of meaning and scope of MOOCs. At the same time, it is significant to remember that a wholly interpretive approach to understanding learning in MOOCs will be equally deficient. Combining methods and pursuing an understanding of the MOOC phenomenon from multiple angles, while keeping in mind the strengths and weaknesses of each method, is the most productive avenue for future research.
A computational analysis and data science discourse is increasingly evident in educational technology research. This discourse posits that it is possible to tell a detailed and robust story about learning and teaching by relying on the depth and breadth of clickstream data. However, the findings in our research reveal meaningful learner activities and practices that evade data-capturing platforms and clickstream-based research. Off-platform experiences as described above (e.g., notetaking) call into question claims that can be made about learning that are limited to the activities that are observable on the MOOC platform. Further, the reasons that course content is consumed in the ways that it is exemplifies the opportunity to bring together multiple methodological approaches to researching online learning and participation.
What do learning experiences in MOOCs look like? Amy Collier, Emily Schneider and I have just published a paper that provides some in-depth answers to this question. Here is a copy of the paper in pdf. The paper is part of a special issue published by the British Journal of Educational Technology which can be found here (there are many excellent pieces in that issue, so be sure to read them).
In addition to trying to understand learner experiences, in the paper we describe that we did this study because ”ease of access to large data sets from xMOOCs offered through an increasing number of centralized platforms has shifted the focus of MOOC research primarily to data science and computational methodologies, giving rise to a discourse suggesting that teaching and learning can be fully analyzed, understood and designed for by examining clickstream data”
Our abstract reads:
Researchers describe with increasing confidence what they observe participants doing in massive open online courses (MOOCs). However, our understanding of learner activities in open courses is limited by researchers’ extensive dependence on log file analyses and clickstream data to make inferences about learner behaviors. Further, the field lacks an empirical understanding of how people experience MOOCs andwhy they engage in particular activities in the ways that they do. In this paper, we report three findings derived by interviewing 13 individuals about their experiences in MOOCs. We report on learner interactions in social networks outside of MOOC platforms, notetaking, and the contexts that surround content consumption. The examination and analysis of these practices contribute to a greater understanding of the MOOC phenomenon and to the limitations of clickstream-based research methods. Based on these findings, we conclude by making pragmatic suggestions for pedagogical and technological refinements to enhance open teaching and learning.
We reported 3 main findings:
1. Interactions in social networks outside of the MOOC platform
A number of learners alluded to interactions they have had with individuals who are part of their social networks. These include digital connections with other participants in a MOOC, face-toface interactions with friends and family, and face-to-face interactions with new connections in a MOOC.
Despite the fact that none of the popular MOOC platforms support integrated notetaking at the time of writing this paper, nearly all interviewees reported taking notes while watching lecture videos. Only one interviewee never took notes. However, the tools used to take notes and the subsequent use of notes varied substantially by learner.
3. Consuming content
All individuals participating in this study discussed factors that shaped the ways they consumed MOOC content, shedding light on the context surrounding their participation. Scholars in the learning sciences have long highlighted the critical role of the environment, arguing that learning must be understood as a sociocultural phenomenon situated in context and culture (Brown, Collins & Duguid, 1989). Patterns of MOOC content consumption can be examined by clickstream data, but these contextual factors help explain why learners exhibit particular patterns of participation.
Veletsianos, G., Collier, A., & Schneider, E. (2015). Digging Deeper into Learners’ Experiences in MOOCs: Participation in social networks outside of MOOCs, Notetaking, and contexts surrounding content consumption. British Journal of Educational Technology 46(3), 570-587.
— Bonnie Stewart (@bonstewart) September 22, 2014
A few weeks ago, I notified individuals who filled out my Networked Scholars open online course survey, indicating that my open course was open for registration. I’m excited to see that some colleagues have discovered the course, but it’s time to post the news here too, even though some . I’ve really appreciated the feedback from people regarding the design and content of the course, so if you any thoughts about this, please don’t hesitate to let me know! I am trying to create a memorable and worthwhile learning experience and hearing from you is a significant way to go about doing that. If you have any other thoughts about the course or about what you think makes open online courses engaging, effective, and memorable, please don’t hesitate to send them my way.
2. Through your blog and twitter accounts. The readings and activities will be publicly-available, and you could use your blog/twitter as a discussion/reflection space, so you don’t necessarily need to sign up to Canvas to access this course if you don’t want to.
— Alan Levine (@cogdog) September 22, 2014
I hereby confirm the rumour. We will be using an approach similar to Connected Courses and the distributed syndication model. The official Twitter hashtag for the course is #scholar14. If you choose this route, you can indicate your desire to participate through Canvas (see #1 above) or you can just wait and start participating via your social media accounts when the syndication platform is ready (I’ll write another blog post when we are ready to launch).
First, I want this course to be about learners and their needs, and not just what I think are significant areas to understand. Therefore, I will be asking you to articulate participants to articulate their needs and evaluate their own progress towards their accomplishments. For example, you might already know some of the challenges that academics face when they participate on social media (e.g., see Kansas Board of Regents policy regarding social media use) so you might want to spend more time investigating the relationship between academic freedom and social media. That’s absolutely fine! Or, you might be interested in investigating how you can be more effective in using social media to engage with practitioners. That’s great too! I wrote a little bit about this here.
Second, even though I have experience with and do research on networked scholarship, there are a number of other people who have experience with these topics. Diversity is important, and for this reason, each week I will be hosting a live Q&A panel on Google Hangouts on Air with other individuals discussing the topic of the week. Even though this panel will be live and you will be able to view it in real-time and ask questions, it will also be recorded for those of you who can’t make it.
That’s all for now. I am looking forward to the course.
While designing my open course focusing on networked scholars, I’ll be posting updates here pertaining to pedagogical and design decisions that I’m making. [Aug 20, 2014 update: Course registration is open]
The course is intended to help doctoral students, academics, and other knowledge workers on how social media and networked technologies may support/extend/question their scholarship. The course will also be “wrapped” by a colleague in real-time and colleagues who teach research methods courses will be sharing it with their students. In short, the audience is diverse, their background knowledge varies, and their needs/desires will vary. So, the question becomes, how do you support all learners to achieve what they aspire to achieve?
I’ve been thinking a lot recently about success in open courses. I’m intrigued by discussions of multiple pathways (or dual layer) through open courses and I’ve been reflecting on how to support the different groups of people that might visit (and use) my course. In the GoNorth projects, we had thousands of teachers annually use our digital learning environment and curriculum. To accommodate their needs the curriculum consisted of 3 levels: (experience, explore, expand). This design encompassed varying levels of difficulty and involvement and allowed teachers to adjust the curriculum to local needs. In the edX course Data, Analytics, and Learning that George, Carolyn, Dragan, and Ryan are teaching in the Fall, the learner is given more of that control. The instructors write: ”This course will experiment with multiple learning pathways. It has been structured to allow learners to take various pathways through learning content – either in the existing edX format or in a social competency-based and self-directed format. Learners will have access to pathways that support both beginners, and more advanced students, with pointers to additional advanced resources. In addition to interactions within the edX platform, learners will be encouraged to engage in distributed conversations on social media such as blogs and Twitter.” I like this because of the recognition that learners come to courses with varying needs/wants and that recognition influenced the design of the course.
In thinking about the different needs that students in my course will have, a group of instructional designers and I at Royal Roads have created a scaffold to help individuals define what they want to achieve in the course. This tool will be helpful for self-directed learners and those with enough background knowledge on the topic, but, depending on how it is implemented, it can help novices as well. The scaffold is a Personal Learning Plan (.rtf). I think this might be helpful to others, so I’m tagging it with an open license so that others can use it as they see fit in their own courses. Here’s how it works:
I assume that individuals will enrol in this course to pursue a personal need/ambition (e.g., “I want to learn how education researchers use social media for research and I am at a loss as to where to start”). To support learners in this, I will be asking them to develop a personal learning plan (PLP) as a way to define, verbalize, and be mindful about their goals. A PLP will allow learners to define what they want to achieve by enrolling in the course and reflect on their successes and accomplishments.
Once participants create a PLP they can either keep it private, share it with the instructor, or share it on a discussion board. Sharing it on a discussion board might allow them to be more accountable to the goals they have set and to connect with colleagues that have similar goals. There is one problem here: Let’s assume that the course will be of interest to a couple of hundred people and a hundred of them post their PLPs on a discussion board. That will quickly become overwhelming for everyone. How do we reduce the information available to help learners find each other based on common interests? If learners could tag their post, and the tags became available at the top of the discussion thread, that could help, but alas, that’s not an option available on the platform that I am using. If any of you have any ideas, I’d love to hear them!
Below are two fictitious learning plans as examples. These only have 1 row each, but learners could include as many rows as they need.
The first one is relevant to PhD students
|Goal||Action(s) to achieve goal||Measure of success (i.e. How will I know that I was successful?)||How much time do I anticipate spending to achieve this goal?|
|Decide whether of not to start blogging about my dissertation||- Read assigned material- Participate in discussions||- Make a decision by the end of the course||2 hours per week for the next 4 weeks|
The second one applies to an early-career academic (e.g., a lecturer, a professor, a researcher, etc).
|Goal||Action(s) to achieve goal||Measure of success (i.e. How will I know that I was successful?)||How much time do I anticipate spending to achieve this goal?|
|My social media activity is gaining global following. I want to understand the tensions that I might face.||- Read everything associated with week 2.- Participate in as many relevant discussions as possible in week 2.- Join the live panel discussion during week 2.||- I will write a 200-word journal entry describing potential tensions and challenges that I might face.||7 hours during week 2|
Of course, it is entirely possible, and research has shown, that learners don’t know what they don’t know. A personal learning plan isn’t a panacea, which is why every course needs to include a diverse range of scaffolds and supports. But this is turning out to be a long post, so I’ll save those thoughts for a future update.
As always, I’d love to hear your thoughts. How does this sound? What might be some problems with it? How could it be improved?
I was at the Educause Learning Initiative conference last week (#ELI2014), where I had some interesting conversations and discussions around online learning, MOOCs, research methods, and the future of higher education.
Amy Collier and I presented early results from our qualitative studies looking at learners’ MOOC experiences (if you have not yet responded to our call to share your lived experiences with us, please consider this invitation). Our talk was entitled “Messy Realities: Investigating Learners’ Experiences in MOOCs.” Our thinking is guided by the notion that even though surveys and big data yield insights into general behavioral patterns, these methods are detached and can distance us rather than help us understand the human condition. As a result, the phenomenon of “learning in a MOOC” is understudied and undiscovered. During the session, we shared what we have been finding in our studies, highlighting the messiness of learning and teaching in the open.
Karen Vignare and Amy Collier were also very kind to extend an invitation to a number of us to share our work with individuals participating in the leadership seminar they organized. It was fantastic to hear Katie Vale (Harvard), Matt Meyer (The Pennsylvania State University), Rebecca Petersen (edX, MIT), and D. Christopher Brooks (EDUCAUSE) discuss their work, and once again, I felt grateful that we are having these conversations more openly, more frequently, and with greater intent.
Below are my rough notes from my 5-7 minute presentation. I appreciate parsimony (who doesn’t?), and in the words of D. Christopher Brooks, this is the litany of things I think:
I am a designer and researcher of education and learning. I study emerging technologies and emerging learning environments. I’m also a faculty member , and I have been teaching in higher education settings both face-to-face and online since 2005.
To contextualize my comments on MOOCs, first I want to describe my experiences with them:
- I have facilitated one week of the #change11 MOOC was organized by George Siemens and Stephen Downes in 2011. This MOOC had a distinctively connectivist flavor with each week being facilitated by 1 person.
- I have enrolled in a number of MOOCs, and have even completed a small number of them.
- I have repurposed MOOCs in my own courses. For example, I have asked students to enroll in MOOCs and write about them.
- I have published an e-book with my students, sharing stories of student experiences with MOOCs.
- Finally, I am actively involved in studying learners’ experiences in MOOCs in order to understand the human element in these emerging learning environments.
I have recently come to the realization that I have an ambivalent relationship with MOOCs. My relationship with MOOCs is one of the most ambivalent relationships I have had with anyone or anything. This relationship is more ambivalent than the love-ignore-hate relationship that my cat has with me!
On the one hand, I appreciate the opportunities for open learning that MOOCs provide. I also appreciate how MOOCs have brought us together to discuss issues around technology, teaching, and learning. At the same time, I cringe at the narratives around big data, I cringe at the hype, at the ignorance around what education is and should be about.
I want to talk about two topics today: MOOC research and the MOOC phenomenon.
On MOOC Research
- We don’t know much about MOOCs
- The things that we know about MOOCs are mostly the result of surveys, learning analytics, and big data research
- The existing research and the existing methods that we use are informative, BUT they simply paint an incomplete picture of MOOCs. We should be asking more in-depth questions about learner and instructor experiences in MOOCs
- Qualitative and interpretive research methods can and will help us better understand MOOCs, open learning, and open scholarship
- Descriptions of learner behaviors are helpful, but these descriptions only provide a glimpse and superficial summary of what students experience and what they do in digital learning environments. To give you an example, emerging research suggests that students may be “sampling” courses; a behavior that we don’t frequently see in traditional online courses or traditional face-to-face courses. Nonetheless, “sampling” is not how participants would describe their experiences or the ways they participate MOOCs. To illustrate, consider family-style Mediterranean meals that consist of numerous dishes, where participants sample a wide array of food. If you ask a person to describe this meal, to explain it to someone else, or to simply tell you about the meal, they will likely describe the meal as a feast, they might describe the tahini as lemony, the variety of flavors as intriguing, the whole meal as satisfying. Different people will also describe the meal differently: Tourists might describe the meal as fulfilling, heavy, or even extravagant; locals might describe the same meal as appropriate, or better than or worst than meals that they have had at other restaurants. “Sampling” may be an appropriate descriptor of the act of eating a family-style meal, or exploring a MOOC, but the descriptor does not fully capture the experience of sampling.
On the MOOC as a Phenomenon
MOOCs. The acronym stands for massive, open, online courses. That is not what MOOCs are though. MOOCs are a phenomenon. They represent something larger than a course and should be seen in conjunction to the rebirth and revival of educational technology. They represent symptoms, responses, and failures facing Higher Education. For instance, MOOCs are a response to the increasing costs of Higher Education; represent the belief that the purpose of education is to prepare students for the workforce; represent the belief that technology is the solution to the problems that education is facing; are indicative of scholarly failures; seem to represent the belief that education is a product that can be packaged, automated, and delivered; and, are a response to failures by researchers, designers, administrators, and institutions to develop effective and inspiring solutions to the problems of education (alternatively, they might also represent the failure of existing systems to support creative individuals in enacting change)*.
The MOOC is an acronym that elicits strong feelings: excitement, fear, defiance, uncertainty, hope, contempt…. To address these feelings we have to address the failures of higher education and the underlying causes that have given rise to MOOCs. For this reason, instead of talking about MOOCs at my own institution, I discuss innovations and approaches that I value, including networked scholarship, openness, flexibility, social learning, and the design and development of new technologies.
* NOTE: Rolin Moe and I are working on a paper refining and delineating these. If you have thoughts, concerns, or input on any of these issues, we’d love to hear form you!
I am in Cyprus to meet with a number of colleagues and give an invited talk at ICEM 2012.
Talk title: What does the future of design for online learning look like? Emerging technologies, Openness, MOOCs, and Digital Scholarship
Abstract: What will we observe if we take a long pause and examine the practice of online education today? What do emerging technologies, openness, Massive Open Online Courses, and digital scholarship tell us about the future that we are creating for learners, faculty members, and learning institutions? And what does entrepreneurial activity worldwide surrounding online education mean for the future of education and design? In this talk, I will discuss a number of emerging practices relating to online learning and online participation in a rapidly changing world and explain their implications for design practice. Emerging practices (e.g., open courses, researchers who blog, students who use social media to self-organize) can shape our teaching/learning practice and teaching/learning practice can shape these innovations. By examining, critiquing, and understanding these practices we will be able to understand potential futures for online learning and be better informed on how we can design effective and engaging online learning experiences. This talk will draw from my experiences and research on online learning, openness, and digital scholarship, and will present recent evidence detailing how researchers, learners, educators are creating, sharing, and negotiating knowledge and education online.
As part of my research on digital scholarship and the experiences/practices of scholars in online networks, I am working with the Texas Advanced Computing Center and the newly-established Visualization Lab at the College of Education to understand learner and scholar participation patterns on the social web. Below is our first visualization, which shows interactions between three types of users who are contributing to a hashtag (red, blue, green). It’s a directed graph, with nodes representing users, and edges representing interactions between users. The thickness of the edge represents # of interactions (thick = more interactions). When nodes of a different color interact with each other, the edges take the color of the two node (e.g., when a blue node interacts with a red node, the edge is purple). What does this visualization tell us?
We are still trying to make sense of this, and we are slowly learning from the tutorials that Tony Hirst has created. This is what (i think) this says: First of all, we know that the majority of the people contributing to this hashtag are not having a conversation with each other (#nodes making up the dataset are 3 times the group shown above – this is not shown on the graph). Second, it looks likes there’s a few “central” folk through which conversations occur. Finally, even though interactions happen between red and blue nodes, it looks like the majority of the interaction is happening within those two groups. And that’s important in this situation because one of our hypothesis was that the red group was joining this community to interact with the blue group (if that was the case, we would be seeing more purple in the image above). We definitely need additional ways to evaluate some of these statements, but that’s what it “looks like” from the image above. And here’s where I think data visualizations start becoming really valuable: You can quickly see patterns and ask questions, and continue from there. We have some ideas and hypotheses, but we also want to let the data bring up phenomena that we haven’t thought about. I don’t yet feel confident that I fully understand what I am seeing here, but I am quickly learning a lot! So my question to you is: how would you interpret this? What questions do you have of what you are seeing here?
This blog entry was supposed to go out next week, but I am sharing it today because it is relevant to the entry that George Siemens wrote today.
I gave a talk to Curt Bonk’s class a couple of weeks ago and the central premise of that talk was that we should be designing experiences, not products. This is not a new idea. It goes back to the beginning of my career and it’s a passion that I share with a lot of folks, most notably Aaron Doering and Charles Miller at the University of Minnesota (who incidentally just landed in Sydney for their most recent Adventure Learning project). For example, see Raising the bar for instructional outcomes: Towards transformative learning experiences (2008) and Designing Opportunities for Transformation with Emerging Technologies (2011). A central tenet of the 2008 paper is the following:
There exist “strong pressures to produce mediocre instructional products based on templates and preexisting content.”
That was in 2008. Now consider 2011/2012: Interest in open courses and in large online classes has exploded. The edtech entrepreneur is eager to leverage online education and capitalize on efficiency, by focusing on the delivery of pre-packaged content. Scale and efficiency are key in that if one is able to efficiently deliver content (read: low cost) to large numbers of people, s/he can charge a small fee that will yield high profit. This isn’t a new idea either. David Noble talks about the commodification of education, the attempt to market and sell education as a commodity.
Sebastian Thrun, who was one of the faculty members teaching the Stanford AI class last Fall recently “showed emails from a student who took the AI class, when he could get Internet access, amidst mortar and rocket attacks in Afganistan; and another, a single working mother, who refused to quit the class because it gave her a sense of accomplishment.” Are these statements describing exceptional courses? Are they describing experiences that pull students and engage them to their core? Or are they describing exceptional people? When you provide access to exceptional people (like the two individuals above), they will amaze you, because, well, they are exceptional! How do you design courses that are exceptional, that adapt to all learners, and provide support structures for individuals who are not exceptional? You provide opportunities for personally relevant and meaningful transformation. How do you do that, you ask? Here’s my (free) advice to any hopeful edtech startup: Designing Opportunities for Transformation with Emerging Technologies (pdf).