Elsevier

Computers & Education

Volume 133, May 2019, Pages 27-42
Computers & Education

How is the use of technology in education evaluated? A systematic review

https://doi.org/10.1016/j.compedu.2019.01.010Get rights and content

Highlights

  • Eight foci of learning technology evaluations emerged from a systematic review.

  • The main foci were learning outcomes, affective elements, behaviors and design.

  • Of the 365 papers analyzed, 66.6% used at least one instrument from a prior study.

  • The 22 instruments used in multiple studies to evaluate sub-categories are reported.

  • The results assist learning technology evaluation and characterize the field.

Abstract

There are a large variety of methodologies, contexts and perspectives that have been used to evaluate the use of technology in education. The vast array of literature involving learning technology evaluation makes it challenging to acquire an accurate sense of the different aspects of learning that are evaluated, and the possible approaches that can be used to evaluate them. This study conducted a systematic review of how technology use in education has been evaluated, based on 365 papers published in Computers and Education between 2015 and 2017. The papers analyzed encompassed a diverse range of education levels, disciplines and technologies, that were broadly commensurate with characterizations of the learning technology field from other studies. The analysis found that the evaluation of learning technology use tended to focus on eight themes: learning outcomes, affective elements, behaviors, design, technology elements, pedagogy, presence, and institutional environment. In the majority of studies (66.6%) at least one established instrument was used to evaluate the effectiveness of learning technology usage, however, a wide variety of instruments were observed within sub-themes of evaluation. The 22 instruments used to evaluate an aspect of learning technology usage in more than one study are identified. Evaluation trends for different disciplines, technologies and educational levels are also established. The findings provide an analytical framework that educators and researchers can utilize when evaluating the use of technology in education, and could potentially inform the development of new, more holistic and more robust evaluation methods.

Section snippets

Background and motivation for this study

With the continual influx of new and emerging technologies available for use in education, it is critical to evaluate the degree to which learning technology usage contributes to learning and teaching (Iriti, Bickel, Schunn, & Stein, 2016). There are different motivations for integrating technology into learning, for instance, to improve student learning outcomes, improve access to learning, and enhance learner motivation (Bower, 2017). Many high quality meta-analyses show consistent,

Methodology

The methodological approach used in this paper was a systematic literature review. Systematic literature reviews play a vital role both in supporting further research effort and providing an unbiased synthesis and interpretation of the findings in a balanced manner (Kitchenham et al., 2010). This type of review attempts to collate relevant evidence that fits the pre-specified eligibility criteria to answer specific research questions (Moher, Liberati, Tetzlaff, & Altman, 2009, p. 3). The review

Contexts being evaluated

The sample of 365 studies captured a wide range of learning technology evaluation contexts. Almost half (48.9%) of the studies related to university or college level education, more than a third relating to school education (either secondary or primary), with the remaining studies involving adult education, early childhood education, or studies across levels (see Table 3). These proportions broadly align with the previous studies (e.g., Kirkwood & Price, 2014), with the high proportion of

Implications and critical reflections

The fact that studies across the technology-enhanced learning research papers focused on the evaluating learning, affective elements, behaviors, design, technology, pedagogy, presence and institutional environment, was not in and of itself surprising (ref. Table 6). However, the proportion of studies focusing on each aspect was in some cases unexpected by the research team. For instance, only 1.4% of the papers evaluated ‘institutional environment’. McGill, Klobas, and Renzi (2014) argue that

Conclusion

This paper presented findings from systematic review of the evaluation of technology usage in education based on 365 papers published in Computers & Education between 2015 and 2017. The papers analyzed encompassed a diverse range of education levels, disciplines, technologies (as outlined in Table 3, Table 4, and Table 5, respectively). Through thematic analysis of the literature this study found that empirically, the evaluation of learning technology use tends to focus on learning (78.6%),

Declarations of interest

None.

Acknowledgments

This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

References (110)

  • G. Conole et al.

    ‘Disruptive technologies’, ‘pedagogical innovation’: What's new? Findings from an in-depth study of students' use and perception of technology

    Computers & Education

    (2008)
  • H. Crompton et al.

    The use of mobile learning in PK-12 education: A systematic review

    Computers & Education

    (2017)
  • M. Darban et al.

    Antecedents and consequences of perceived knowledge update in the context of an ERP simulation game: A multi-level perspective

    Computers & Education

    (2016)
  • M. Engin et al.

    Dialogic teaching and iPads in the EAP classroom

    Computers & Education

    (2015)
  • T. Hainey et al.

    A systematic literature review of games-based learning empirical evidence in primary education

    Computers & Education

    (2016)
  • M.D. Hanus et al.

    Assessing the effects of gamification in the classroom: A longitudinal study on intrinsic motivation, social comparison, satisfaction, effort, and academic performance

    Computers & Education

    (2015)
  • G.-J. Hwang et al.

    A contextual game-based learning approach to improving students' inquiry-based learning performance in social studies courses

    Computers & Education

    (2015)
  • J. Joo-Nagata et al.

    Augmented reality and pedestrian navigation through its implementation in m-learning and e-learning: Evaluation of an educational program in Chile

    Computers & Education

    (2017)
  • F. Ke

    Examining online teaching, cognitive, and social presence for adult students

    Computers & Education

    (2010)
  • B. Kitchenham et al.

    Systematic literature reviews in software engineering – a tertiary study

    Information and Software Technology

    (2010)
  • S. de Klerk et al.

    Psychometric analysis of the performance data of simulation-based assessment: A systematic review and a Bayesian network example

    Computers & Education

    (2015)
  • M. Kordaki et al.

    Digital card games in education: A ten year systematic review

    Computers & Education

    (2017)
  • N. Luo et al.

    Effects of different interactions on students' sense of community in e-learning environment

    Computers & Education

    (2017)
  • J.-F. Martín-SanJosé et al.

    The effects of computer-based games and collaboration in large groups vs. collaboration in pairs or traditional methods

    Computers & Education

    (2015)
  • T.J. McGill et al.

    Critical success factors for the continuation of e-learning initiatives

    The Internet and Higher Education

    (2014)
  • Z. Merchant et al.

    Effectiveness of virtual reality-based instruction on students' learning outcomes in K-12 and higher education: A meta-analysis

    Computers & Education

    (2014)
  • T.A. Mikropoulos et al.

    Educational virtual environments: A ten-year review of empirical research (1999–2009)

    Computers & Education

    (2011)
  • M.W. Olofson et al.

    TPACKing: A constructivist framing of TPACK to analyze teachers' construction of knowledge

    Computers & Education

    (2016)
  • C. Orús et al.

    The effects of learner-generated videos for YouTube on learning outcomes and satisfaction

    Computers & Education

    (2016)
  • M. Pérez-Sanagustín et al.

    Research on ICT in K-12 schools – a review of experimental and survey-based studies in computers & education 2011 to 2015

    Computers & Education

    (2017)
  • G. Petri et al.

    How games for computing education are evaluated? A systematic literature review

    Computers & Education

    (2017)
  • D.D. Prior et al.

    Attitude, digital literacy and self efficacy: Flow-on effects for online learning behavior

    The Internet and Higher Education

    (2016)
  • J.C. Richardson et al.

    Social presence in relation to students' satisfaction and learning in the online environment: A meta-analysis

    Computers in Human Behavior

    (2017)
  • Y.-T. Sung et al.

    The effects of integrating mobile devices with teaching and learning on students' learning performance: A meta-analysis and research synthesis

    Computers & Education

    (2016)
  • P.A. Tess

    The role of social media in higher education classes (real and virtual) – a literature review

    Computers in Human Behavior

    (2013)
  • S.L. Wang et al.

    The role of collective efficacy, cognitive quality, and task cohesion in computer-supported collaborative learning (CSCL)

    Computers & Education

    (2012)
  • J. Bacca et al.

    Augmented reality trends in education: A systematic review of research and applications

    Journal of Educational Technology & Society

    (2014)
  • R. Bakeman et al.

    Observing interaction: An introduction to sequential analysis

    (1997)
  • E. Baran

    A review of research on mobile learning in teacher education

    Journal of Educational Technology & Society

    (2014)
  • J. Beckers et al.

    e-Portfolios enhancing students' self-directed learning: A systematic review of influencing factors

    Australasian Journal of Educational Technology

    (2016)
  • R.M. Bernard et al.

    A meta-analysis of blended learning and technology use in higher education: From the general to the applied

    Journal of Computing in Higher Education

    (2014)
  • A. Bhattacherjee

    Understanding information systems continuance: An expectation-confirmation model

    MIS Quarterly

    (2001)
  • J. Biggs et al.

    The revised two‐factor study process questionnaire: R‐SPQ‐2F

    British Journal of Educational Psychology

    (2001)
  • B.S. Bloom

    Taxonomy of educational objectives, handbook 1: The cognitive domain

    (1956)
  • C.J. Bonk et al.

    The handbook of blended learning: Global perspectives, local designs

    (2006)
  • M. Bower

    Deriving a typology of Web 2.0 learning technologies

    British Journal of Educational Technology

    (2016)
  • M. Bower

    Technology integration as an educational imperative

  • R.E. Boyatzis

    Transforming qualitative information: Thematic analysis and code development

    (1998)
  • J. Brooke

    SUS-A quick and dirty usability scale

  • M.-T. Cheng et al.

    The use of serious games in science education: A review of selected empirical research from 2002 to 2013

    Journal of Computers in Education

    (2015)
  • Cited by (118)

    • Towards teaching-sensitive technology: a hermeneutic analysis of higher education teaching

      2024, International Journal of Educational Technology in Higher Education
    • The Realities of Evaluating Educational Technology in School Settings

      2024, ACM Transactions on Computer-Human Interaction
    View all citing articles on Scopus
    View full text