Assessing digital scholarship (#change11)
It’s digital scholarship week at #change11 and I am in the midst of following the activity on Twitter and a number of blogs. At the same time I am preparing my annual activity report that details the work that I’ve been doing over the past year. In this report, one will find evidence on research, teaching, and service. Evaluation norms for these include: citations, average teaching scores compared to departmental/university averages, journal impact factors, evidence of impact on education and the profession, etc.
One of the issues that often comes up, and one that Martin raises in chapter 11 of his book as well, is the lack of established frameworks to evaluate digital scholarship, digital artifacts, and academics’ digital participation. That’s not to say that all digital participation should be evaluated or that all digital participation is even worthy of evaluation. The lack of frameworks, in addition to indicating that academia may not value digital artifacts, also signifies the difficulty of assessing the impact of scholarship. This is not a new problem, as academia has struggled with figuring out methods with which to evaluate innovations. A similar issue is facing Design-Based Researchers. Collins, Joseph, and Bielaczyc (2004, pp. 40) for instance, note that Design-Based Research generates mountains of data that independent researchers can use in answering their own questions about teaching and learning. But “this would require the community to honor such reanalysis of data with the same status as original research and it would require research journals and tenure committees to take such work seriously.” While other fields have found value in the analysis of secondary data (e.g., bioinformatics), education has yet to make advances in this domain.
Food for thought: Institutions are complex entities that serve numerous stakeholder. How do we create frameworks that value the important work that is being done on digital spaces, while also valuing the cultural norms and values of the institution?
Collins, A., Joseph, D., & Bielaczyc, K. (2004). Design Research: Theoretical and Methodological Issues. Journal of the Learning Sciences, 13(1), 15-42. doi:10.1207/s15327809jls1301_2