How is the use of technology in education evaluated? A systematic review
Section snippets
Background and motivation for this study
With the continual influx of new and emerging technologies available for use in education, it is critical to evaluate the degree to which learning technology usage contributes to learning and teaching (Iriti, Bickel, Schunn, & Stein, 2016). There are different motivations for integrating technology into learning, for instance, to improve student learning outcomes, improve access to learning, and enhance learner motivation (Bower, 2017). Many high quality meta-analyses show consistent,
Methodology
The methodological approach used in this paper was a systematic literature review. Systematic literature reviews play a vital role both in supporting further research effort and providing an unbiased synthesis and interpretation of the findings in a balanced manner (Kitchenham et al., 2010). This type of review attempts to collate relevant evidence that fits the pre-specified eligibility criteria to answer specific research questions (Moher, Liberati, Tetzlaff, & Altman, 2009, p. 3). The review
Contexts being evaluated
The sample of 365 studies captured a wide range of learning technology evaluation contexts. Almost half (48.9%) of the studies related to university or college level education, more than a third relating to school education (either secondary or primary), with the remaining studies involving adult education, early childhood education, or studies across levels (see Table 3). These proportions broadly align with the previous studies (e.g., Kirkwood & Price, 2014), with the high proportion of
Implications and critical reflections
The fact that studies across the technology-enhanced learning research papers focused on the evaluating learning, affective elements, behaviors, design, technology, pedagogy, presence and institutional environment, was not in and of itself surprising (ref. Table 6). However, the proportion of studies focusing on each aspect was in some cases unexpected by the research team. For instance, only 1.4% of the papers evaluated ‘institutional environment’. McGill, Klobas, and Renzi (2014) argue that
Conclusion
This paper presented findings from systematic review of the evaluation of technology usage in education based on 365 papers published in Computers & Education between 2015 and 2017. The papers analyzed encompassed a diverse range of education levels, disciplines, technologies (as outlined in Table 3, Table 4, and Table 5, respectively). Through thematic analysis of the literature this study found that empirically, the evaluation of learning technology use tends to focus on learning (78.6%),
Declarations of interest
None.
Acknowledgments
This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.
References (110)
- et al.
Understanding the MOOCs continuance: The role of openness and reputation
Computers & Education
(2015) - et al.
Measuring self-regulation in online and blended learning environments
The Internet and Higher Education
(2009) - et al.
The influence of TPACK contextual factors on early childhood educators' tablet computer use
Computers & Education
(2016) - et al.
An update to the systematic literature review of empirical evidence of the impacts and outcomes of computer games and serious games
Computers & Education
(2016) Place presence, social presence, co-presence, and satisfaction in virtual worlds
Computers & Education
(2012)- et al.
A systematic literature review on serious games evaluation: An application to software project management
Computers & Education
(2015) A meta-analysis of the impact of technology on learning effectiveness of elementary students
Computers & Education
(2017)- et al.
Learning through playing Virtual Age: Exploring the interactions among student concept learning, gaming performance, in-game behaviors, and the use of in-game characters
Computers & Education
(2015) - et al.
How features of educational technology applications affect student reading outcomes: A meta-analysis
Educational Research Review
(2012) - et al.
A systematic literature review of empirical evidence on computer games and serious games
Computers & Education
(2012)
‘Disruptive technologies’, ‘pedagogical innovation’: What's new? Findings from an in-depth study of students' use and perception of technology
Computers & Education
The use of mobile learning in PK-12 education: A systematic review
Computers & Education
Antecedents and consequences of perceived knowledge update in the context of an ERP simulation game: A multi-level perspective
Computers & Education
Dialogic teaching and iPads in the EAP classroom
Computers & Education
A systematic literature review of games-based learning empirical evidence in primary education
Computers & Education
Assessing the effects of gamification in the classroom: A longitudinal study on intrinsic motivation, social comparison, satisfaction, effort, and academic performance
Computers & Education
A contextual game-based learning approach to improving students' inquiry-based learning performance in social studies courses
Computers & Education
Augmented reality and pedestrian navigation through its implementation in m-learning and e-learning: Evaluation of an educational program in Chile
Computers & Education
Examining online teaching, cognitive, and social presence for adult students
Computers & Education
Systematic literature reviews in software engineering – a tertiary study
Information and Software Technology
Psychometric analysis of the performance data of simulation-based assessment: A systematic review and a Bayesian network example
Computers & Education
Digital card games in education: A ten year systematic review
Computers & Education
Effects of different interactions on students' sense of community in e-learning environment
Computers & Education
The effects of computer-based games and collaboration in large groups vs. collaboration in pairs or traditional methods
Computers & Education
Critical success factors for the continuation of e-learning initiatives
The Internet and Higher Education
Effectiveness of virtual reality-based instruction on students' learning outcomes in K-12 and higher education: A meta-analysis
Computers & Education
Educational virtual environments: A ten-year review of empirical research (1999–2009)
Computers & Education
TPACKing: A constructivist framing of TPACK to analyze teachers' construction of knowledge
Computers & Education
The effects of learner-generated videos for YouTube on learning outcomes and satisfaction
Computers & Education
Research on ICT in K-12 schools – a review of experimental and survey-based studies in computers & education 2011 to 2015
Computers & Education
How games for computing education are evaluated? A systematic literature review
Computers & Education
Attitude, digital literacy and self efficacy: Flow-on effects for online learning behavior
The Internet and Higher Education
Social presence in relation to students' satisfaction and learning in the online environment: A meta-analysis
Computers in Human Behavior
The effects of integrating mobile devices with teaching and learning on students' learning performance: A meta-analysis and research synthesis
Computers & Education
The role of social media in higher education classes (real and virtual) – a literature review
Computers in Human Behavior
The role of collective efficacy, cognitive quality, and task cohesion in computer-supported collaborative learning (CSCL)
Computers & Education
Augmented reality trends in education: A systematic review of research and applications
Journal of Educational Technology & Society
Observing interaction: An introduction to sequential analysis
A review of research on mobile learning in teacher education
Journal of Educational Technology & Society
e-Portfolios enhancing students' self-directed learning: A systematic review of influencing factors
Australasian Journal of Educational Technology
A meta-analysis of blended learning and technology use in higher education: From the general to the applied
Journal of Computing in Higher Education
Understanding information systems continuance: An expectation-confirmation model
MIS Quarterly
The revised two‐factor study process questionnaire: R‐SPQ‐2F
British Journal of Educational Psychology
Taxonomy of educational objectives, handbook 1: The cognitive domain
The handbook of blended learning: Global perspectives, local designs
Deriving a typology of Web 2.0 learning technologies
British Journal of Educational Technology
Technology integration as an educational imperative
Transforming qualitative information: Thematic analysis and code development
SUS-A quick and dirty usability scale
The use of serious games in science education: A review of selected empirical research from 2002 to 2013
Journal of Computers in Education
Cited by (118)
Towards teaching-sensitive technology: a hermeneutic analysis of higher education teaching
2024, International Journal of Educational Technology in Higher EducationA meta systematic review of artificial intelligence in higher education: a call for increased ethics, collaboration, and rigour
2024, International Journal of Educational Technology in Higher EducationOnline vs in-person learning in higher education: effects on student achievement and recommendations for leadership
2024, Humanities and Social Sciences CommunicationsThe Realities of Evaluating Educational Technology in School Settings
2024, ACM Transactions on Computer-Human Interaction