Professor & Canada Research Chair in Innovative Learning and Technology at Royal Roads University

Ethics and transparency in the Coursera Learner Outcomes Survey


Posted on December 6th, by George Veletsianos in moocs, online learning, open, scholarship. 24 comments

On December 5th, Coursera sent an email inviting individuals to participate in a survey intended to investigate whether participation on Coursera “has had any career, educational, or social impact in [their] life.” The email also stated: “Your survey response will be used as part of a research study conducted by the University of Pennsylvania, the University of Washington, and Coursera, examining the impact of MOOCs.”

Research studies examining the impact of MOOCs outside of individual courses and studies that use methods other than clickstream data, are worthwhile and needed. I applaud Coursera and its partners for the effort to address this research gap.

However, the lack of information pertaining to the research is concerning.

By clicking on the email invitation, potential participants land on a page that describes the research study as follows:

Screenshot 2014-12-06 07.40.12

Both the University of Pennsylvania and the University of Washington have offices in place to support researchers in conducting research in  ways that protect human participants (the UPenn IRB is here and the U of Washington site here). Importantly, these offices are not just regulatory: they provide help and support. The UPenn site for example states that the mission of the IRB includes providing “professional guidance and support to the research community.”

At the very minimum, potential participants should be informed about the study and should provide their agreement to participate in the study. This process is called informed consent. The University of Washington IRB website describes it as follows:

Researchers are required to obtain the informed consent of all participants in human subjects research prior to enrolling those individuals in a study. The individual’s consent must be voluntary and based upon adequate knowledge of the purpose, risks, and potential benefits of a research study. All potential participants should also be informed of their right to abstain from participation or to withdraw consent to participate at any time without reprisal. After ensuring that a person has understood the information, the researcher should then obtain the person’s consent, preferably in writing. [more details here]

This information should be written in language that a layperson can understand and should be included in the screenshot above. In online surveys, consent is usually gained by asking participants to click on a button that indicates that the individual agrees to participate in the study.

This is all missing from the Coursera survey.

Granted, Coursera is a business entity, and it is not bound by the same requirements imposed upon researchers to conduct a survey. Businesses conduct surveys and market research all the time, and none of this applies. But this isn’t just market research. The email and survey introduction have a clear statement of intent: The data will be used for research. Even if Coursera is unaware of the existence of ethical guidelines, Facebook’s emotion contagion study and other news stories on ethics (e.g., Harvard’s hidden cameras efforts) should have provided a moment to pause and ask: Are we doing all we can to ensure that we are treating each other, and our research participants especially, in an ethical and caring way?





24 thoughts on “Ethics and transparency in the Coursera Learner Outcomes Survey

  1. Hi George,
    I’ve posted this as a discussion question to the #rhizo14 Facebook group, in case you are interested. I asked:

    Saw this and wonder what you think? Is it fair to compare an outcomes survey (unclear whether or not it is anonymous) to the clearly ethically questionable activities at Facebook and Harvard? Does a MOOC outcomes survey need university ethics clearance? What would that add to the process? Would it do more harm than good? http://www.veletsianos.com/…/ethics-transparency-coursera-…/

    cheers,
    Rebecca

  2. Rebecca asks a good question about anonymity. If user is logged in the survey data could be connected to activity achievement and demographic data already stored for that user. Do you know if that is the case George?
    Your point about business and research ethics is interesting. In UK I think that Businesses are bound by Data Protection Act on use and sharing of data which should comply with unis DP register entry

    • Hi Frances,

      You asked:
      If user is logged in the survey data could be connected to activity achievement and demographic data already stored for that user. Do you know if that is the case George?

      I can’t answer your question, Frances, but what i can tell you is this:
      – The survey is on surveymonkey and has a unique identifier. I imagine that if Coursera wanted to merge data from different sources, it could.
      – The survey has the following question: Would you be open to talking to a coursera employee about your experience? The survey does not ask for contact information, so I am assuming that either (a) the user’s email is connected to their answers so that s/he can be contacted, or (b) the survey is missing the “provide your information” question.

      You also mentioned that “Your point about business and research ethics is interesting. In UK I think that Businesses are bound by Data Protection Act on use and sharing of data which should comply with unis DP register entry.”

      What’s fascinating to me is the following: Researchers frequently analyze data collected by others (e.g., Netflix has datasets available that individuals can examine). In such cases, researchers are frequently exempt from going back to users to ask if they can use their data. It’s frequently quite difficult to do and the data is often de-identified. In a lot of cases, the data is not *originally* collected for research purposes However, in this case, there is a stated intention that the data will be used for research, which, I think, necessitates a clearer request for consent, because the data *will* be used for research.

      Here is an example of an ethical informed consent statement for MOOCs.

  3. There are several things to say about this:
    – Coursera is very well aware of ethical restrictions. For proof, this contribution by Vivek Goel prior to a conference on IRB, big data and ethics:
    http://cehd.gmu.edu/irb-big-data/participants#vgoel
    – Coursera’s ToS announce that research will be conducted, just as Facebook does. It is not clarified to students or professors that some of it will be unilaterally decided by Coursera, without ethics approval, and could be added to existing courses in later runs.
    – Coursera conducts many surveys, including some during courses (I think the name is “Quick Questions”), with the intent of comparing courses.

    MOOC platforms’ selling points is in great part the possibility of doing research (unlike Facebook or whatnot). Yet they thrive by blurring the line between ethically-conscious research and business-motivated market research. Lots of institutions are unfortunately eager to jump on this bandwagon, in a mutually advantageous arrangement (IRB laundering). Unfortunately they are not questioning the impact that this might have on the profession in the long run.

  4. Hey George, I got here through Rebecca’s link on Facebook and saw your response there. I agree with you that an IRb board would probably exempt this if it contains all relevant risk/withdrawal info etc. I don’t like the idea of universities using a corporate entity to collect data for them without getting IRB approval (and have personally gotten IRb approvals multiple times for MOOC research)
    BUT I agree w Rebecca that this is not as unethical as the facebook issue. Even without explicit consent it is implicit when someone answers a survey that they consent.
    In facebook u mentioned program eval vs research – my uni requires IRB only if u plan to publish results. I think that is flawed coz my program eval can be harmful to participants even if not published
    Yeah IRB processes are flawed and designed for health care not social sciences

Leave a Reply

Your email address will not be published. Required fields are marked *