Technology and reviewing literature: Researching during a crisis

Dr Jim Turner, Chair of ELESIG

As part of your job, I am sure you have been asked to make suggestions or recommendations of ways of using learning technology. Published learning technology research is sometimes difficult to engage with and can be patchy in certain areas. Finding a recent literature review can help identify effective practice, but have you ever wondered how these reviews are created and more specifically how you can create one? In this blog, we will review a recent webinar on systematic and rapid literature reviews that will help you find your way. There is a side subject to this as well, how developments in technology are supporting the process of reviewing literature, and how this might re-shape our notions of technologically enhanced learning.  

Dr Melissa Bond from the University of South Australia kindly gave up some of her time to talk about her work creating literature reviews. You could say she wrote the book on systematic literature reviews, hold up, she did write the book on this! This emerged from her PhD work and her previous employment supporting staff and students in using technology to conduct systematic reviews. The talk covered her recent work on rapid reviews of teaching and learning literature during the past 2 years. She talked about how she conducted the reviews and the main findings.  

What are systematic and rapid reviews? 

As you can imagine, there is a hierarchy of reviewing processes, starting with the most basic literature review (doing a search on what has been written in this area), right through to a review of reviews (reviewing previous reviews) (fig.1). At each stage, the process is more rigorous and, because of this, more time-consuming.  

Systematic reviews have a set of processes that need to be conducted for it to be truly systematic. These include; establishing and publishing a replicable process, working with others to reduce bias, and having clear inclusion criteria to show why papers were rejected. Rapid reviews are similar, but usually have more constrained inclusion criteria and focus, and may have only one type of research involved.  

Why should I think of doing a more systematic review? Beyond the more obvious benefits, Mel found reading a range of papers helped her understand the assorted styles of writing and research approaches, provided her with a broad understanding of the subject, and identify the research gaps. But there are challenges like; understanding and applying the method, using the technology, finding the time, and defining the criteria and scope.  

The rapid reviews during covid looked at both secondary and Higher Education. Here are the main findings:  

• Not as many studies look at students in primary or secondary years, with a focus on teacher experiences in schools, but student experiences in studies focused on higher education 
• Zoom and other similar tools were the most used types – almost half the studies 
• Not much on students with disabilities – whose voice is being heard in this research? 
• Not much research on assessment tools or social networking tools 

The presentation was very engaging and interactive. Through audience questions, Mel established that the audience were newbies to reviews, that we could see a wide range of opportunities, but that time was the key constraint. The following are a few reflections on the session. 

Reflections 

Using technology to enable the reviewing of literature: Mel showed us how she was using EPPI-Reviewer, a software specially designed for systematic reviews. Once you have established the criteria, key terms, and databases, this tool helps by pulling in the abstracts and presenting them for review and/or rejection (see fig 2)  

EPPI-Reviewer also uses machine learning to help sort and suggest papers. It can also continue this process as new work is published. It is also possible to allow others outside of the research team to view the database and run different queries. I am familiar with using citation and bibliographic analysis tools. However, the growing possibilities to support learning is interesting. Traditionally, this is seen as an area for libraries, but you can see eventual crossovers as AI and machine learning help shift, sort, and share knowledge. Once the rules of the systematic review are created, the technology can create an open resource for other researchers to explore the database and do further analysis. This creates a virtuous cycle around established criteria that can help others to examine the data in diverse ways. The systems also continue to gather the data, meaning the research is always ongoing. 

Reducing the bias: Mel reflected a number of times on the lack of systematic reviews to include a wider range of voices. Reviews often only include papers published in English. Her recent reviews highlight the lack of publishing in Africa, Oceania, the Middle East, and South America. There is also a tendency to not include ‘grey literature’ published outside of academic journals. There is a need to find ways to support and include these voices for the benefit of everyone, as it distorts our understanding of learning across the globe.  

The quality of the literature out there: All the findings of systematic reviews require a good standard of research in the first instance. Mel hinted that there is room for improvement here, and links back to ELESIG’s mission to help support the development of research skills within our community.  

Lack of reviews in education: Systematic reviews are quite common in scientific disciplines, but Mel notes that there are far fewer in education. This is a big opportunity for our area, which brings me to my next point.  

Need to support each other: Very few attending had done a systematic review at the session, which points to a lack of knowledge, skills, and resources in this area. This provides us with an opportunity in the ALT community to work together on identifying and conducting reviews. This could help by sharing the obvious benefits but reducing the challenges such as time and resources. 

Link to video and slides 

Video: ELESIG Webinar: Emergency remote education during the COVID-19 pandemic 

Slides: Student Engagement und digitale Medien: Wohin jetzt? (weebly.com) 

How to join ELESIG:  

Join the conversation and hear more about our work but signing up to our jiscmail subscription or follow @ELESIG 

About Author:  

Dr Jim Turner (Senior Learning technologist, LJMU) is currently Chair of ELESIG.