Selecting a Learning Management System: Advice from an Academic Perspective

Clayton R. Wright is an international education consultant; Valerie Lopes is a professor in the Centre for Academic Excellence, Seneca College; T. Craig Montgomerie is professor emeritus in the Department of Educational Psychology, University of Alberta; Sunday A. Reju is a professor, Namibia University of Science and Technology; and Seb Schmoller was chief executive of the UK's Association for Learning Technology until 2012.

Introducing or replacing an institutional learning management system (LMS) should involve all stakeholders. Although information and educational technology staff obviously play a key role in the LMS selection process, the successful addition or change in an LMS requires collaboration among IT personnel and the academic staff, as well as the consideration of students' needs.

"Basic technology resources, such as the institution's website and the CMS [LMS], are the most pervasive and most valued" by students, as these "basic" technologies "have the greatest impact on student success."¹ Educators should heed these statements, as they are based on the responses of 113,035 students from 251 colleges and universities in 13 countries to a survey conducted by the EDUCAUSE Center for Analysis and Research.²

Selecting an LMS that helps an institution achieve its academic plan is exceedingly important for students, faculty, and educational administrators. To achieve this, educational organizations must:

• become familiar with the advantages and disadvantages of available software;
• establish an LMS selection committee, a decision-making process, and selection criteria; and
• apply these criteria to determine the most appropriate LMS, given the characteristics of each institution and its faculty and student body.

In this article, we provide questions to be considered by the faculty, instructors, teachers, and students involved in LMS selection. Some of the questions posed also are relevant to the interests and concerns of other stakeholders, including administrators and technical support personnel. This article does not recommend one LMS over another, but lists a number of considerations that educators should scrutinize during the selection process. Whether selecting the first LMS or switching from one LMS to another, the basic considerations are the same. It is inevitable that institutions will move from one LMS to another at some point, simply because the needs of institutions change and systems evolve at different rates.

Definition of an LMS

LMS systems are known by various names, including course management system (CMS), learning content management system (LCMS), virtual learning environment (VLE), virtual learning system (VLS), learning portal, or e-learning platform. Each term might have a slightly different meaning, depending on your interpretation. Perhaps it should be called an instructional management system, as the system's parameters are usually set by instructors rather than by students. However, throughout this article, we'll use the term LMS. An LMS is comprehensive, integrated software that supports the development, delivery, assessment, and administration of courses in traditional face-to-face, blended, or online learning environments.

Institutions use LMS software to plan, implement, facilitate, assess, and monitor student learning. The software centralizes course preparation; educational content and resources; the delivery and tracking of student activities, such as discussion and collaboration; the administration of assessment activities; and the accumulation and presentation of marks and grades. All of these activities are conducted behind a virtual wall that provides a measure of authentication, security, and privacy. Recent LMS software also provides an array of information about student activities that instructors and administrators can view from different perspectives. This information can be analyzed to detect patterns that might suggest how students can be better supported. The LMS helps institutions maintain the integrity of their educational programs and enables faculty to effectively and efficiently develop courses, deliver instruction, facilitate communication, foster collaboration, and assess students. An LMS can be used to support traditional face-to-face instruction, as well as blended and online educational environments.

Most educators in the English-speaking higher education market are probably aware of LMS software such as Blackboard (which also owns Angel and WebCT), Desire2Learn, Instructure Canvas, Moodle, Pearson LearningStudio (previously known as eCollege), and Sakai.³ However, according to Craig Weiss,⁴ 552 companies/developers currently offer LMS software. (Weiss provides links for each company and a brief description of the software; he also indicates whether the software supports mobile learning and online/offline synchronization.) Although vast, the Weiss list does not include LMS software such as Edmodo, an LMS developed primarily for use in K–12, or international software, such as the Thailand Cyber University (TCU) LMS or the Knowledge Environment for Web Learning (KEWL) [http://eteaching.uwc.ac.za/index.php?module=news&action=viewstory&id=gen14Srv59Nme26_60660_1320139211] software developed at the University of the Western Cape in South Africa.

Each LMS is different. Some give users the opportunity to implement a variety of approaches, such as content-centric, activity-centric, network-centric, linear, and branching. Some systems are more effective for delivering asynchronous instruction, while others are better at providing synchronous instruction. Some LMSs can deliver content and allow students to access their grades via mobile devices, while others cannot. Thus, instructors must identify the approaches used and those that could be used to promote learning at their institutions. Then, they must closely examine the LMS tools and features and how they might be used. This takes time and commitment. In general, selecting the right LMS depends on many factors, including the age of the students and the type of instruction and learning experiences the instructors and institution want to provide.

LMS Selection and Implementation Challenges

As noted above, educational institutions use LMS software for a variety of reasons. However, despite the many benefits that can accrue when an LMS is implemented, the selection and implementation processes sometimes fail when institutions lack:

• Leadership, not only by management and academic leaders, but also by those who have political influence within the institution.
• Commitment to the process, as it takes time and resources before evidence of success emerges.
• Organization-wide buy-in and appreciation for what an LMS can and cannot do.
• Stakeholder involvement in the selection process.
• Alignment with the education plan or direction of the institution.
• Congruency with how instructors teach. Implementing an LMS can itself lead instructors to reconsider their teaching methods.
• Recognition of the cultural changes required to achieve success. Resistance to change can arise, especially among those familiar with an existing LMS, as they know its shortcomings and have developed workarounds.
• Organizational preparedness during implementation — that is, knowing who will be responsible for what.
• Training for instructional designers, instructors, students, and information technology specialists.
• Quality support, including pedagogical and technical support.
• Focus on designing quality courses.
• Student and instructor computer literacy skills.
• Student access to computers, the Web, and/or the LMS.
• Adequate access to the Internet for students and staff.
• User-friendliness of the software. The software must be easy to use, even for novices.
• Funds required for hardware, including servers, network infrastructure, backup storage, backup power supply, air conditioning for the hardware, and computers/digital terminals.
• Due diligence by the LMS selection committee to adequately address the needs and concerns of the potential users, verify the information provided by vendors and external experts, and ensure that the selected LMS can actually perform the tasks requested by users.
• Effective project management to ensure that the selection process and implementation succeed.

Further discussion about why LMS implementations fail is offered in the article "Tips to Help Avoid eLearning and LMS Project Failure."⁵

Implementing or replacing an LMS can lead to significant upheavals within the organization and substantial expenditures of time and money. Although determining the return on investment (ROI) within the teaching environment can be difficult, you must consider the costs of investing in an LMS. These costs could be compared to those of administrative systems used by your finance, human resources, payroll, and purchasing departments. Following such a comparison, the cost to obtain, lease, and/or provide technical support for the LMS might not seem so outrageous. Once an LMS is installed, it becomes a mission-critical system that instructors and students will depend on 24/7. That is, it will become the most valued software (or possibly the most disliked software) at the institution. Also, you must ask, "What is the cost to the organization if this investment is not made?" For example, fewer students might enroll, or alumni might reduce their donations if they do not view the institution as being innovative and competitive. When selecting an LMS, you must carefully examine both the short- and long-term effects of its implementation.

Proprietary, Open-Source, and Cloud-Based Alternatives

Deciding whether to obtain a proprietary, open-source, or cloud-based LMS is like choosing a religion — it depends on what you believe in. Depending on your educational priorities and how the software is configured, any of the three forms might meet your needs. In any case, use care in making general statements about proprietary versus open-source LMS software, as they share many advantages and disadvantages. Thus, you must examine each LMS using criteria set by potential users. Regardless of your choice — and prior to selecting a suitable LMS or LMS alternative — you must list the features that are important to your institution. And you must separate needs from wants and wishes. In addition, you need to determine what impact the selection may have on your current operation. For example, will the LMS solution:

• Provide the reliability/stability, flexibility, scalability, and security the institution needs?
• Easily integrate with existing systems — both software and hardware?
• Require IT staff to receive additional training? If training is required, can it be provided online and how much will it cost?
• Necessitate the hiring of additional staff with skill sets that differ from those possessed by existing staff? How many staff will be needed to support the LMS solution?
• Require extensive maintenance and support over time? Do vendors offer tiered support plans that can be covered by existing budgets?

Many additional questions must be considered by IT personnel. A number are noted in the appendix, "Selecting an LMS: Questions to Consider."

Proprietary LMSs

Proprietary LMS software is developed and owned by a profit-generating entity that does not let users access and make adjustments to the computer code that determines the structure of the software and the activities it can execute. It is a closed system from the perspective of the organization that deploys it. Examples of these types of LMS include Blackboard and Desire2Learn. Depending on your perspective, there can be several (debatable) advantages to implementing a proprietary LMS:

• It is reliable, because it is built by professionals who are paid to build an effective and efficient product.
• It is backed by a company with a record of successes that you can research.
• It is current, because the product must be competitive, though there might be some lag time between when a new idea is generated and when it is implemented.
• It links with various enterprise software systems, such as those that process financial, human resources, and administrative data. This is particularly true if the LMS is produced by a large multinational company such as SAP, which makes enterprise software focusing on business operations.
• It is supported by a company that provides training, technical support, and warranty service.
• It can be hosted by the company that makes it or leased to the user.

Not everyone is a fan of proprietary LMS software. Instructors and institutions might select an alternative for many reasons, including the following:

• Proprietary software can be expensive, especially for institutions with small enrollments. Also, it entails an annual license fee, which seems to keep rising, while the cost of obtaining many open-source LMSs is negligible. However, proprietary licensing fees can represent only 20–25 percent of the total cost of software ownership.⁶ Once a system has been purchased or leased, the costs of maintaining proprietary and open-source software could be about the same.
• Users and their organizations cannot access the underlying source code and thus cannot adjust the software, add features, or correct bugs immediately. Users instead must make a request to the proprietary software company and hope that the company will respond in a timely manner. The company might respond quickly, but it might not make the change immediately, as it must ensure that the suggested change does not adversely affect other users of the software.
• Proprietary software might not be kept current. It might be designed to address the basic needs of some institutions, but other institutions that are exploring various ways to engage students might have more advanced needs. Note, however, that open-source software might not be kept current, either, as it depends upon an active community of software users, some of whom write code only when they have the time. Whether proprietary or open source, the software must evolve by taking into account new instructional methods, enhanced security features, hardware, and computing architectures.
• Proprietary software companies might discontinue an LMS. If a company develops new LMS software, will it allocate sufficient funds to keep previous software relevant and current? If one company's LMS is acquired by a different company through purchase or takeover, observations of the current LMS market indicate that the newly acquired software is likely to gradually disappear.
• Some users consider proprietary software cumbersome and restrictive; in search of simpler software, they often migrate towards open-source products. The challenge for software companies is to produce software that meets a wide variety of needs. Eliminating features to simplify software requires hard choices. Some instructors might require social media to be integrated in the LMS, others might not. Some might want a spell checker, others might use it infrequently. To serve diverse needs, the LMS must be flexible and host a variety of features that some users might not use. Also, institutions might require software that ensures the privacy and security of personal data and lets them track the educational activities of students. These features can be difficult to build into "simple" software.
• Proprietary software might be designed to be installed on a server, not on personal computers or laptops. However, if the proprietary software can be installed on personal devices, courses can be developed offline and later transferred to the institution's server.
• The license agreement that accompanies proprietary software might constrain how the software is implemented, distributed, and administered. Proprietary license agreements might prohibit institutions from distributing the software to students with limited financial resources unless an additional fee is paid. Thus, students of lower socio-economic status might not have the ability to obtain proprietary software. The institutional purchase of this type of software may increase the digital divide — that is, the gap between those who have access to information technology and those who do not. This digital divide is a significant issue for educators in developing countries.

Open-Source LMSs

Open-source LMSs are developed by individuals or consortia for many reasons, but usually because they believe they have a "better idea" about how an LMS should function. Moodle, the highest-ranked LMS software,⁷ was developed by Martin Dougiamas, a programmer who became frustrated with the LMS used at Curtin University and designed his own system. "Martin's background in education led him to adopt social constructionism as a core theory behind Moodle. This is revolutionary, as most CMS systems have been built around tool sets, not pedagogy. Most commercial CMS systems are tool-centered, whereas Moodle is learning-centered."⁸

The development of Sakai, another well-known open-source LMS, was begun in late 2003 by a consortium of four traditional universities — Massachusetts Institute of Technology (MIT), Michigan, Stanford, and Indiana. The institutions "saw a common cause to jointly develop an Open Source Virtual Learning Environment."⁹ This initial small group has expanded to over 350 educational organizations that use Sakai. The Sakai project [https://sakaiproject.org/about-sakai] is described as a "vibrant community creating technology that enhances teaching, learning and research" through the spirit of collaboration and community.

Some of the (debatable) advantages of employing open-source LMS software for instruction and learning follow. An open-source LMS:

• Can be easy to obtain, as many are free, especially those that provide a basic level of service.
• Lets users examine the source code and make changes and enhancements.
• Lets users obtain any available updates, which are accessible to everyone.
• Is built by a collaborative community open to new ideas.
• Is often the result of the efforts of people who are passionate about open-source software.
• Can often be set up by an instructor without involving technical support from the institution.
• Can be simple to use and consequently is frequently selected by faculty when the proprietary system fails to meet their needs.
• Avoids restrictions that institutional administration might impose.
• Enables collaboration with others, as the software is accessible and can be used by anyone.

An open-source LMS can also entail some (debatable) disadvantages:

• Although open-source software might appear to be free, funds are required to host, maintain, and upgrade the software as well as cover additional storage and database support. Personnel must be hired to authorize access to the software, create course files, monitor the software's efficiency, manage the databases, perform backups, and help people use the software.
• Often, resources are needed to integrate open-source LMS software with existing administrative systems. Unless this is done, it might be difficult for the institution to track all student activities and marks, transfer the marks to existing accreditation systems, and handle the people required to support course delivery. Also, IT staff may need to dedicate time to implement and establish administrative systems, which are already incorporated into many proprietary LMSs.
• Technical support can be a significant issue, as institutional technical staff might be unfamiliar with the programming language on which the open-source LMS is based. In that case, technical staff will need extensive training on the programming language or external professional services must be hired to provide support. Either way, there is a cost.
• The quality of the code used in open-source LMS software might be suspect. However, according to the annual Coverity Scan Open Source Report, the difference between code quality in proprietary and open-source software is minimal; open-source software can be just as reliable as proprietary software.
• The robustness of open-source software has been questioned. A software is robust if it can handle numerous and varied transactions at the same time. Although this can be a concern, open-source software such as Moodle is being used by large educational institutions such as the Open University of the United Kingdom, which has an enrollment of more than 240,000 students.
• Institutions might have problems deciding which "competing" routine to implement and support. Developers of the same open source software LMS sometimes write alternative routines that do essentially the same thing; thus, institutions must decide which routine to implement. Producers of proprietary systems provide a single approved method for handling a particular situation.
• Open-source LMS software might not provide the security and privacy settings required by educational institutions.
• When something goes wrong with an LMS, who is ultimately accountable? Because open-source software is developed by a community, no one member of the community can take responsibility for any mishaps that might occur.
• When some of the "passionate" people who built the software for free move on to other projects, the open-source LMS might become stagnant. However, in the rapidly changing world of technology, who can predict whether an "established" organization will still be here in its current form? Consider Kodak or Netscape. The advantage of software developed by a community is that when some members leave, others can fill the gap.
• The institution will likely be concerned about the lack of institutional control over "rogue" LMSs. Because some open-source LMS software can be installed on a desktop computer, faculty members might express their academic freedom by implementing an open-source LMS on their own that is easy to use and meets their needs, but may not be under direct control of the institution's technical department. This puts students in the situation of having to learn to use a variety of systems at the same institution. How would the institution ensure privacy and security for the open-source LMS installed on a professor's computer? Would the institution be liable if the instructor's system is hacked and data adjusted or released to others?

As stated earlier, the arguments for and against proprietary and open-source LMSs depend on your point of view. Several reputable educational institutions, including Athabasca University, the African Virtual University, and the Open University of the United Kingdom, use open-source LMS software such as Moodle and Sakai. In the United States, "Moodle edges out competitors such as Blackboard among smaller colleges"¹⁰ with an enrollment of 2,500 or fewer full-time equivalent (FTE) students; according to George Kroner, a former Blackboard engineer, "about 76 percent of institutions recognized by the U.S. Department of Education fit into that category."¹¹ Currently, Blackboard is the preferred choice of American educational institutions with an enrollment greater than 2,500 FTE, and Moodle is the second choice.

Cloud-Based Alternatives

In recent years, educators have become aware of a third LMS option: an array or mash-up of cloud-based tools and services that are accessible to everyone. Some of the tools that could be used effectively in the educational environment are listed in Jane Hart's Top 100 Tools for Learning 2013. Institutions can use "a toolbox of web resources that might include social bookmarking tools, document sharing applications, social networking sites, timeline tools, and media options available in the cloud."¹²

For example, a blog page, Facebook page, or wiki page could be used as a base or hub where the course topics and activities are outlined. Then, students could be directed to Google Drive to share documents, Dropbox to store and synchronize files, Skype to communicate and hold meetings, Flickr to share photographs, YouTube to share videos, and Camtasia to capture screen shots and record synchronized audio. Thus, students could use software in an educational setting that might be familiar to them from their personal lives or workplace. However, the use of web-based tools in an educational setting is frequently based on the assumption that all students can access and use these tools. This assumption might not be valid, especially for students who have had limited exposure to the web due to socio-economic or cultural reasons or who live in rural areas where bandwidth might be limited. For example, foreign students might not have had the opportunity to use the web extensively or might come from or live in countries where access to many web-based systems is forbidden.

Using an array of cloud-based tools instead of an LMS offers several advantages:

• Cost. The cost to use many cloud-based tools can be negligible, although there might be an additional cost for advanced features, such as faster download speeds and increased storage capacity.
• Greater selection of tools. Instructors and students can choose the specific tools they need for a range of assignments.
• Familiarity. Many students are already familiar with tools such as Facebook, Flickr, and Twitter. However, as noted, some students might not have access to or familiarity with these tools.
• Shift from content to activity focus. The use of online tools might shift the teaching and learning process from a content-centered to an activity-centered learning environment as students use the various online tools to create artifacts and to communicate, network, and collaborate.
• Increased access to artifacts. When students use software in the cloud, they can continue to access it when they are no longer completing a course or enrolled in an institution. Thus, the artifacts that they have produced during a course are portable — they are available to students anywhere and at any time, including after they graduate.

However, if you choose an array of cloud-based tools instead of a traditional LMS, several concerns arise, including the following:

• Authentication. Proprietary and open-source LMSs can provide authentication, gradebooks, and assessments within a closed secure environment. How will these be handled when web-based tools are used? Institutions typically must ensure that enrollment, assessment, and grade information is hosted on a server that they control.
• Security and privacy. Procedures must exist to ensure that students are aware of the risks involved in using cloud-based tools, and they must be aware of the different settings to safeguard their security and privacy. Online tools, such as a blog, often have a default setting that opens the tool to the public; thus, people who are not registered for a course can view the material and add their comments. For some, this is one of the benefits of using freely available online tools — students are exposed to opinions from a wide variety of people and might find their work "peer-reviewed" by the general public. For others, this is a major concern, as outside opinions might take the course off-track or make it more challenging to manage if numerous people comment. It could also expose the instructor — who is providing online feedback to students — to criticism by the general public.
• One tool vs. many tools. Students might be frustrated if they must learn how to use a variety of tools instead of one LMS. Further, not everything is in one place. Students might have to go to one website to complete one educational task, then to another website to complete a different task, and so on, and thus might perceive their learning environment as fragmented. Unless a course's subject matter has something to do with technology or communications, they might feel that they are wasting time learning to use different tools rather than learning the course's subject matter.
• Advertising. The use of "free" software might mean that students are bombarded with advertising. Software developers may expose users to commercial advertising in order to generate funds needed to create the free software. For some instructors and students, exposure to advertising is an inappropriate distraction in an educational environment.

Still, an increasing number of students might want to use various cloud-based software applications that they find online, and some instructors will want to give students greater flexibility in how they complete their educational activities. This gives rise to four key questions:

• How will the institution ensure that enrollment, assessments, grades, and personal information are under strict control, yet still allow instructors and students more flexibility in selecting software for educational purposes?
• How will the institution assemble a variety of software under one umbrella or dashboard so that students can go to one online portal to access it?
• How will the institution provide effective instructor and student support for the third-party software? The greater the number of tools students and instructors use, the greater the range of technical support that the institution must provide.
• What procedures should be in place in case web-based software that the institution does not control is poorly maintained, shut down for repairs, or ceases to operate? If, for example, web-based software is shut down for repairs or upgrades while a student is working on a project, the student might not be able to submit work on time, or some or all of the work might be lost. If an LMS hosted by an institution is shut down for repairs, the institution can easily extend deadlines, as it has knowledge and control of the situation.

Some of these challenges are being addressed by LMS software developers, who are linking LMSs to various social media such that cloud-based tools appear in the LMS dashboard and the software's output can be integrated with various LMS features. Of course, there are cloud-based social media developers like Edmodo, which claims that "First and foremost Edmodo is not an LMS. Edmodo is a free, teacher-centered social learning platform… Edmodo's grassroots usage can thrive alongside official LMS deployments, and in many school districts Edmodo is used as a no-cost LMS alternative."¹³ So the line is blurred between LMS software residing on a single server and social media platforms that reside in the cloud.

Achieving a Balance

Institutions might need to maintain a careful balance between providing security, privacy, technical support, and a consistent institutional image and providing instructors and students with flexible tools that meet their instructional and learning needs now and for the next three to five years. The debate about whether to obtain a proprietary, open-source, or cloud-based LMS alternative is discussed in more depth elsewhere.¹⁴

Some institutions, such as Lynn University in Florida, have decided to use cloud-based iTunes U instead of a standard LMS to provide course content in several programs.¹⁵ Because institutions might need to develop or acquire software that provides attendance tracking, gradebooks, user analytics, and other administrative tasks not currently supported by iTunes U, this cloud-based solution is not for everyone.

Clearly, deciding whether to obtain a proprietary, open-source, or cloud-based LMS alternative can be difficult. However, it might be best to first decide what LMS features and tasks are required to meet the instructional and learning needs of faculty and students. The selection committee can then choose the LMS that best meets these needs, while also recognizing that needs and LMS software will evolve over time.

The LMS Selection Process

The process of selecting an LMS will be shaped by administrative structures and the institution's culture; however, three crucial steps should be taken to initiate the LMS selection process: choosing the right people to sit on the selection committee; establishing an effective, efficient and transparent selection process; and setting criteria for the LMS that are congruent with the institution's strategic directions and the teaching and learning processes it wants to support.

Establish the Selection Committee

Establishing a selection committee for an LMS can be challenging, as it must comprise key stakeholders and representatives from across the institution. Often, educational LMS selection committees are composed of instructional technology specialists, academic champions of educational technology, and administrators. If an educational institution's primary function is to teach and to facilitate learning, then faculty and students — the primary LMS users — should make up the bulk of the committee. If novice LMS users are not members of the committee, how will their perspectives be obtained? How will the difficulties novices might encounter be identified? If students are not members of the selection committee, how can the committee effectively identify the LMS features that will help students complete their assignments? Some of this information can be gathered by surveys, but someone on the selection committee must advocate for students, as they are the ones most affected by an LMS implementation.

Because selecting an LMS affects all instructors and students and is critical to the functioning of twenty-first century institutions, it will take time to select an LMS that meets the educational community's diverse needs. Although the selection committee should have cross-institutional representation, individuals sometimes serve several functions at the institution. For example, an instructional designer may also teach in an academic department and thus be able to represent both the professional development group and a specific faculty. IT personnel must ensure that the LMS selection committee examines privacy and security issues as well as address concerns about integration with existing software and hardware, technical support, maintenance, staffing, and the total cost of ownership.

The following are a few specific suggestions regarding the composition of an LMS selection committee:

• Small Committee. When the number of committee members is small, the selection process can be executed quickly, as fewer people must be contacted and it is easier to mesh appointment schedules. When the committee meets, each member will have a satisfactory amount of "air time" to present a point of view. However, when a committee has few members, it is more difficult to fend off complaints about lack of representation from various faculties, departments, and support groups.
• Large Committee. When selection committees are large, assembling members for face-to-face and web-based meetings can be difficult, and reaching consensus can take time. However, these committees can also represent a broader spectrum of the educational community. If, for example, faculty might resist any changes to the current LMS or acquisition of an LMS, having a large selection committee of 14–20 members is probably advisable. Bowling Green State University had a selection committee of 65 (including faculty, staff, administrators, and students) when it sought to move from one LMS to another.¹⁶ A large committee can consider diverse viewpoints expressed by faculty members, and when the decision has been made, the faculty representatives can explain the process and the rationale for the decision to their constituents. This latter function of committee members should not be overlooked; faculty can identify and address resistance quickly and indicate how specific LMS features will address their colleagues' concerns.
• Primary Committee Members. The selection committee should comprise a cross-section of the primary users — faculty and students. Preferably, at least one faculty member representing each major academic department at the institution should sit on the committee. Also consider including individuals who serve as instructional designers and/or professional development facilitators, not only because they are likely to help faculty develop LMS courses, but also because they often work closely with several faculties and thus are aware of their teaching styles and instructional challenges. Institutions rarely involve students in the LMS selection process, yet they are the ones who will be required to use the LMS, and they will see it from a different perspective. You should also have representatives from the technical or technology department, library/learning resources, and different campuses on the committee.

In addition to these more obvious choices, some institutions should consider including other members as well. People regarded as "movers and shakers" or who have political influence should be included on the committee. Once the LMS selection has been made, they will influence how others perceive the selection process and the final outcome; they might also influence the success of the LMS implementation process. If the institution plans to shift from one LMS to another, those who like the existing software as well as those who do not should be on the committee. Finally, people who have never used an LMS also should be involved in the selection process to help ascertain whether a system is "user friendly."

• Chairperson. The chair of an LMS selection committee should be someone who is respected and trusted by members of the educational institution. It would also help if the chair had experience in "herding cats" (that is, highly independent people who tend to proceed in different directions). An academic or academic administrator is probably best for this position to signal the importance of teaching and learning to the institution. If you choose the head of the technology department or a non-academic as chair, it might signal to all that the decision will be driven by technical or administrative requirements rather than academic needs.

Organize the Committee

Once committee members have been selected, they must be informed of their terms of reference or help prepare a brief document outlining these terms. The terms can be refined once the selection process starts, but committee members must have a sense of the commitment required (in time and resources) and the outcomes expected.

Along with determining how decisions will be made — majority rule or by consensus — the terms of service must address many issues.

• Decide who has the right to vote on decisions. In some cases, the chair of the committee does not vote if he or she is an administrator or technology expert. Similarly, in selection committees, only instructors and students can vote, but all members of the committee can participate in discussions. This latter voting arrangement usually occurs in institutions that place a high value on the teaching and learning process — the institution believes that everyone else is there to support instructors and students.
• Determine the role of committee alternates. Can they attend and take part in discussions? Can alternates vote? If a sitting committee member is unable to attend, can she or he send a substitute or does the chair choose the substitute? This is an important discussion item, as substitutes can change the committee's composition and cause it to review decisions that have already been made. In theory, this is not a serious matter unless substitutions happen frequently. In practice, it depends on who the substitutes are, whether the constituents they represent respect them, and whether they are willing to accept previous committee decisions. To prevent awkward situations that might arise regarding the selection and voting rights of substitutes, you should establish guidelines for substitution at the beginning of the LMS selection process.
• Consider how to define a conflict of interest. Under what conditions must members withdraw from voting or resign from the committee? It is impossible to outline all the potential conflict-of-interest situations that could arise; thus, committee members should be informed that they are expected to act in an honorable and ethical manner. However, they should inform the chair if they feel they might be placed in a conflict-of-interest situation (for example, if they have previously worked for a vendor or sat on a vendor's advisory committee).

Other issues relate to logistics:

• How often will meetings be held — once a week, once a month, once per term, or at specific milestones during the evaluation process?
• How will information from the wider community be solicited to help the committee make its decisions? How will you weigh community input in the final deliberations?
• How much time should be set aside to work on LMS selection? When will the evaluation be completed?

Potential committee members must know how much time is required so that they can make adjustments to their schedules. Some might find the commitment too onerous; if anyone declines the invitation to join the selection committee, the chair must find someone else who can represent the same constituents. If a committee is serious about the work to be done, the LMS evaluation should take six to nine months, but it might take longer than 12 months, especially when multiple campuses or institutions are involved in the selection process and extensive software trials are conducted.

The LMS selection committee's composition and operation send an important message to members of the educational community regarding how the institution makes decisions; it also affects the credibility of the committee's final recommendations.

Obtain Agreement on Purpose and Guiding Principles

Individuals invited or nominated to an LMS selection committee typically have a general idea of what they are supposed to do. Once the committee is formed, the group must finalize its purpose and establish principles that will guide it through the selection process. Thus, members must take into account the institution's philosophy and strategic plans. The resulting statement of purpose might read as follows: "Our goal is to select an LMS, within six months, which will provide flexible instruction and learning options for 30,000 students who may receive their education on or off campus." The institution's administration might provide the initial wording for the goal, but the selection committee must take ownership of it by making it clear, specific, and measurable. All committee members should support the wording of the goal and the guiding principles.

Guiding principles can be established to set parameters, provide additional detail, and specify the manner in which the goal should be achieved. For example, one of the guiding principles could be, "The process will be open and transparent." The guiding principles below were established by the University System of Georgia Learning Management System Transition Task Force.¹⁷

• Recommend a product that meets twenty-first century needs of students and faculty supporting the improvement of retention and graduation rates.
• Recommend a product that will be used for multiple purposes (e.g., academic instruction/research/training/continuing education/economic development).
• Recommend a student-focused minimum LMS suite to maintain affordability and increase efficiency.
• The task force will partner with IT to recommend an enterprise solution with an architecture that provides optimal performance/stability and supports increased enrollments of 100,000 additional students by 2020.
• The work of the task force will be an open and transparent process to include all stakeholders.

Because the University System of Georgia has several campuses, the task force recognized that one software package might not be ideal for each campus, but that faculty and students on any given campus should all use the same LMS.

Outline the Selection Process

To achieve its goal, the selection committee must outline a process that is effective, efficient, reliable, and transparent to everyone. A process is considered reliable if a different but similar cross-section of people yielded the same results. You can achieve transparency by establishing a website on the institutional intranet that lists the committee members, the goal, guiding principles, activities/events, timelines, and meeting records, as well as links to software demonstrations and recordings of committee events. The website should also let educational community members offer feedback.

Members of the educational community should be made aware of which activities are to be completed and when, and who will be responsible for them. A website and/or newsletter could include an overview of each activity's outcomes. Obviously, activities and timelines will change as the evaluation progresses; these changes can be noted on the committee's website.

To ensure that educational community members are involved in the selection process, the committee could organize a variety of events or activities.

• Focus groups. These small meetings can be held for each educational division or department to ascertain their concerns with the selection process. You could also ask participants to list the criteria they would use during the selection process.
• Town hall meetings.  These larger meetings can help determine what potential users want and offer a forum for discussing the benefits and disadvantages of the current LMS. Heated discussions might ensue, but at least the educational community members will have a chance to express their diverse opinions. Also, offering this forum makes it possible to address concerns early rather than later, when strong points of view can have a negative impact on the implementation process.
• Sandboxes. These exploratory environments can give educational community members an opportunity to try the software and complete a survey or evaluation rubric  for each LMS prior to the final system selection.
• Face-to-face meetings or webinars. These team meetings can be held to discuss the features and benefits of each of the finalists.

Specify the Target Features and Functions

The selection committee must determine how an LMS will facilitate teaching and learning within its educational community's context. Once this is articulated, the committee can specify the required LMS features, the functions or tasks it must execute, and the standard it must achieve. (The document "Selecting an LMS: Questions to Consider" offers an extensive list of possible features and functions.) Initially, the description of a particular task might be quite general, but as each LMS is examined and the committee gains more knowledge about how a task is executed, additional details can be added. As a first step, it might be helpful to list all the criteria and desired features, and tasks in a spreadsheet that can be easily expanded and revised. (You could list the features and tasks in rows, for example, and the vendors in columns.) Eventually, these items must be prioritized; you might, for example, group them into categories such as:

• Needs: mandatory, essential, non-negotiable (high priority)
• Wants: important to have (medium priority)
• Wishes: extras, nice-to-haves (low priority)

Although academics are focused on the teaching and learning aspects of an LMS, they should also have an overview of the software's basic technical aspects and total cost of ownership. Technical aspects might include the programming language used to code the software, server and network infrastructure requirements, hosting options, back-up procedures, and availability and quality of support.

For academics to have an overview of the total cost of ownership, they must have a clear idea of the features they want, the number of potential system users, the number of languages the system must support, set-up costs, technical upgrades needed to support the LMS, maintenance costs over the life of the contract, and potential training costs. They need to know the total cost to acquire or lease and maintain the system for X students during each year of a three- to five-year contract.

Obviously, administrators and personnel responsible for large institutional purchases will eventually require specific prices for each item to negotiate the final price, but LMS selection committee members also need a good idea of what each LMS costs prior to making the final selection. If LMS vendors had transparent or readily accessible pricing, it would make it considerably easier to determine the cost to obtain and maintain an LMS. Then, educators would be able to compare pricing in a transparent manner.

Examine the Field

As noted earlier, hundreds of LMSs are on the market; however, it is likely that only a few will meet the selection committee's feature and function specifications.

To obtain a rough idea of what LMS software might be worth investigating, committee members can:

• Search the web for possible LMS candidates.
• Examine articles in educational journals or visit the websites of professional organizations such as the Association for Educational Communications and Technology (AECT) and EDUCAUSE.
• Attend exhibits at conferences such as EdMedia, Online Educa Berlin, and similar events listed on The Chronicle of Higher Education website or Stephen Downes' Blog.
• Monitor online forums that discuss educational technology in general and/or specific LMS software.
• Contact other educational professionals via professional networks such as LinkedIn or Google+.
• Contact colleagues who teach or work at different institutions and ask them about the pros and cons of the LMS they are using.

You can also ask colleagues at other institutions about how quickly their LMS providers respond to requests, and whether those providers offer clear guidance, deliver on their promises, and seem to predict LMS development's future direction with any accuracy. Colleagues may have experienced the migration of courses from one vendor's LMS to another, and it's quite likely that they would be willing to describe any problems that emerged during the conversion. It is also important to ascertain whether an LMS crashes, how frequently those crashes occur, and how long it takes before the system is back in operation. Theoretically, if a selection committee has 15 members and each person contacts two external colleagues, anecdotal information could be obtained for about 30 systems. Practically, however, much fewer than 30 different LMSs will be mentioned.

Once the environmental scan is completed, the committee can develop a short list of the possible vendors. To speed up the selection process, some institutions hire a consultant to identify a shortlist of LMSs for the selection committee's consideration. For this process to succeed, however, the consultant must have an in-depth understanding both of how the institution operates and how it approaches teaching and learning. The consultant can also assist in preparing the request for proposal (RFP).

Vet and Short-List Vendors/Champions

Based on its field examination of possible LMS solutions, the committee could identify three to five vendors that might offer a suitable LMS. If an institution is considering operating its own LMS using, for example, an open-source alternative such as Moodle or Sakai, someone from the organization should be designated to "champion" that solution and prepare information about it comparable to what commercial vendors provide. By employing a champion for open-source software during its LMS selection process, the University of Alberta was able to ensure that different LMS solutions received the same degree of investigation and attention. Vendors and champions should be invited to present their solutions to the selection committee and other educational community members.

Some institutions might follow a more highly structured procedure; for example, they might be legally required to send out an RFP to specific vendors or the committee might have to make the RFP criteria and timelines public to ensure a transparent and impartial process. Vendors could be asked to provide evidence that their software is accessible, flexible, scalable, reliable and stable, robust, efficient, secure, and cost-effective for classroom, online, and blended learning environments. They could also be asked to describe how their LMS supports adaptive learning, badges, competency-based learning, personal learning environments, and prior learning assessment. The designated internal champion for a particular LMS such as Moodle or Sakai should respond to the RFP exactly as if they were a commercial vendor, thus giving the committee a fair way to judge competing solutions. Even if organizations are not required to distribute an RFP, it might be advantageous to conduct an RFP process, as it will force them to define more clearly what they want and provide a structure against which to judge the alternatives.

Vendors and champions who receive the RFP must complete their bid documents by a specified date. Then, the selection committee uses the selection criteria to narrow the number of vendors it would like to see.

Pilot the Software

Vendors who are short-listed should be required to provide references. Preferably, the references should be sought from organizations that implemented this particular LMS at least two years prior to the current date. Champions should be required to make the case that they can effectively implement and support the proposed solution. In addition to presenting demonstrations to the selection committee, vendors and champions should be invited to provide public demonstrations in a face-to-face or online setting.

During this stage, it is important to remember that the purpose of salespeople or vendor representatives is to present their products in the best light and convince the committee that their software can meet every criterion the committee established. The committee's role is to verify the accuracy of all information, whether stated by a vendor representative or presented in the RFP or vendor literature. Each task that the committee specifies must be performed at a level acceptable to the committee. For example, the vendor might state that the software has its own built-in spell checker. However, the committee might find that the feature is basic, but does not include all the words in the Oxford or Merriam-Webster's dictionary, and cannot be customized by adding additional words or changing the spelling of some words. Also, the spell checker might apply to learning resources that are placed in the system, but might not cover words typed in forums or chats. It takes time to try out each feature to ensure that it works as expected and can be adapted or customized to meet the needs of faculty and students. The committee must conduct such evaluations with due diligence.

During the pilot, the selection committee must establish that the software actually has the features it wants and can execute the tasks listed in the selection criteria. To verify that the software actually meets the criteria, the software must be accessible not only to committee members, but also to educational community members. One recommended verification method is for faculty to design and develop three to five courses from scratch using the new LMS and to load several existing LMS-based courses on the new system. The LMS vendor might have a conversion tool that could assist with the conversion process. This activity will demonstrate the new system's capabilities and indicate how easy it might be for instructors and support personnel, such as instructional designers, to use it. Institutions could also ask a few instructors to teach with the LMS during the term. Although this might extend the LMS evaluation process, it can yield pertinent feedback regarding the effectiveness, efficiency, and ease of use from both an instructor and student perspective.

Toward the end of the pilot phase, committee members will have a better grasp of the pros and cons of each LMS, as well as issues that might need clarification. References provided by each vendor or software supplier should be checked and colleagues at other institutions who were contacted early in the process could be contacted again, as committee members can now ask questions based on personal experience with the LMS. Two of the most important questions to ask are: Does the software supplier listen to concerns and respond in a timely manner? Does the software supplier keep promises made?

Select a Solution

Despite the selection process outlined above, a clear choice might not emerge until the selection committee discusses and rates the characteristics of each short-listed LMS relative to the selection criterion. At a minimum, the committee must ensure that the LMS can deliver the non-negotiable features and tasks specified at the beginning of the process.

Ideally, a final decision will be reached by consensus rather than majority rule, as the latter can lead to negativity that will affect the implementation process. Also, the ratings of each vendor's LMS should be made public. Several numerical procedures have been developed to facilitate comparison of different LMS. Sagitec Solutions, for example, has developed the freely available Excel-based Learning Management Systems (LMS) Evaluation Tool [http://info.sagitec.com/LMS-eval-tool/].¹⁸

Questions You Should Consider

The document "Selecting an LMS: Questions to Consider" contains a list of questions that can help committee members select an online course development and delivery platform that is congruent with the needs of various institutional stakeholders, regardless of whether the preferred system is proprietary, open source, or cloud-based. The appendix questions fall into several categories:

1. Background: Frame the search by asking about the institution's strategic direction and philosophy, as well as the needs of the various stakeholders.
2. General LMS Solution: Cover the software's history, reliability, and reputation, as well as the availability of independent reviews and user community activity.
3. Course Design Features: Help instructional designers and faculty adapt existing courses to the new LMS and develop courses from scratch.
4. Teaching and Learning Tools: Facilitate instruction and learning activities, particularly communication and collaborative activities within synchronous and asynchronous environments. The tools affect what students and instructors can view and control.
5. Assessment Features: Assist with the design and administration of assignments and tests, and the assessment and recording of student activities and performances.
6. Accessibility Features: Address the principle of reasonable access and compliance with current accessibility laws.
7. Administrative Features: Affect security, data management, and reporting, and help users administer their courses.
8. Technical Aspects: Cover the basic software, hardware, and network requirements. Each institution's technology specialists will need to add additional, context-specific questions.
9. Cost of Ownership: Help determine the total cost to acquire or lease and maintain the system for X students during each year of a three- to five-year contract.

Several lists of LMS features are available online. For example, the Academic and Collaborative Technology Initiatives unit at the University of Toronto combined several LMS evaluation tools and checklists from the Internet into a single checklist [http://testsoft.ati.utoronto.ca/wordpress/avi/files/2013/03/LMSEvalChecklist.pdf] that is freely available. The Advanced Distributed Learning (ADL) Co-Laboratories contracted the development of Choosing a Learning Management System, which includes "criteria for assessing quality and suitability of LMSs."¹⁹ That section contains "a list of characteristics, features, and functions that a robust LMS should include." As the authors note,

The applicability of items in this list to your situation will probably vary widely; some items may be mission-critical for your organization and some may not be pertinent at all. You need to carefully weigh the importance of each in evaluating LMSs. If you rate your list of LMS candidates simply by all items in the list without weighting each item for its importance to you, it could skew the results, which could lead to a poor final choice for your system.²⁰

Conclusion

As college and university students highly value LMS use,²¹ it is paramount that an institution takes the time to select an LMS that is aligned with its educational plan and meets the needs of various stakeholders. It does not seem to matter whether a proprietary or open-source LMS is selected; each has its debatable advantages and disadvantages. However, it is critical that a diverse group of people representing different academic and non-academic departments, as well as students, be involved in the decision-making process to ensure buy-in and minimum resistance during the implementation process. Proper project management is also required to ensure a successful implementation. If an institution is seeking to make widespread improvements to an existing LMS, it must also ensure widespread stakeholder involvement and effective project management during this process; such re-alignment is likely to succeed only if the existing LMS has the necessary capabilities.

Today's LMSs come with many features. Which features are important to your instructors, tutors, students, and administrators? The answer to this question is unique to each institution. Consider engaging novice and experienced instructors and students to use the software to determine how useful, flexible, efficient, and user-friendly it is. Computer and technical service personnel can assess the reliability/stability, scalability, and security of the LMS. The software may have all the features requested, but may be difficult to use. Thus, the only real test is to use it in an instructional/learning environment and ensure that it meets the functionality needs and technical requirements of the organization. If the institution has an existing LMS, it must ensure that existing course materials can be transferred to the new software. The actual users of the software, the academic staff and learners, should have significant input into the selection of any LMS.

Students want instructors to use LMS features more effectively. According to the ECAR Study of Undergraduate Students and Information Technology, instructors regularly use only 50 percent of their LMS features.²² Instructors and those supporting them must not only understand the benefits of LMS features, but also be trained to use them effectively. Institutions can accomplish this by, for example, inviting faculty members from other institutions to demonstrate and discuss how they use these LMS features when teaching specific subjects. Professional development facilitators could also provide guidelines for instructors to follow in developing effective courses.²³ Such guidelines might require instructors to use otherwise rarely employed LMS features.

A recent study found that "students hold high expectations for anytime, anywhere access to course materials"²⁴ and "prefer face-to-face interactions, e-mail, and the CMS (LMS) as ways to communicate more with their instructors."²⁵ The selection of an accessible, flexible, scalable, reliable/stable, robust, efficient, secure, and cost-effective LMS for classroom, online, and blended learning environments is vital for institutions. However, purchasing or leasing an LMS will yield good ROI only if instructors use it fully to engage and communicate effectively with students and receive support in developing and delivering courses offered though the LMS. The LMS solution must address the needs of all stakeholders and be congruent with the teaching and learning priorities of the institution as outlined by the institution's academic strategic plan.

Acknowledgments

This article is an update of "Selecting an Open-Source Online Course Development and Delivery Platform: An Academic Perspective," Clayton R. Wright's presentation to the Fourth Pan-Commonwealth Forum. The extensive revisions are based on the experiences of educators who have helped others select LMS software in North America, Southeast Asia, and sub-Saharan Africa. The co-authors greatly appreciate the advice and suggestions provided by Judith T. Johnson, who reviewed the manuscript, and the contributions of the following individuals, who reviewed the presentation made at the Fourth Pan-Commonwealth Forum: Mohamed Ally, Athabasca University, Canada; Ng Wai Kong, Wawasan Open University, Malaysia; Lou Siragusa, Curtin University of Technology, Australia; and Ingrid Stammer, Canadian Pacific, Canada.

1. Eden Dahlstrom, J.D. Walker, and Charles Dziuban, ECAR Study of Undergraduate Students and Information Technology, EDUCAUSE Center for Analysis and Research, September 2013, p. 11.
2. Ibid., p. 4.
3. Phil Hill, "State of the Anglosphere's Higher Education LMS Market: 2013 Edition," blog, e-Literate, 9 November 2013.
4. Craig Weiss, "Top Ten Learning Management Systems—July Rankings," E-Learning 24/7 Blog, 16 July 2013.
5. Alison Bickford, "Tips to Help Avoid eLearning and LMS Project Failure," Connect Thinking blog, 13 August 2013.
6. Bradley Wheeler, "Open Source 2007: How Did This Happen?" EDUCAUSE Review, July/August 2004, pp. 12–27.
7. Jordan Barrish, "Top LMS Software," infographic, Capterra, 13 September 2013.
8. Jason Cole and Helen Foster, Using Moodle, 2nd ed., O'Reilly Community Press, 2008, p. 4.
9. Alan Mark Berg and Michael Korcuska, Sakai Courseware Management: The Official Guide, Packt Publishing, 2009, p. 13.
10. Carl Straumsheim, "Moodle for the Masses," Inside Higher Ed, 13 February 2014.
11. Ibid.
12. Seven Things You Should Know About LMS Alternatives, EDUCAUSE, 2010, p. 1.
13. Crystal Hutter "An Open Letter to Superintendents From Edmodo CEO" [https://blog.edmodo.com/2013/06/07/an-open-letter-to-superintendents-from-edmodo-ceo-crystal-hutter/], blog, Edmodo CEO, 7 June 2013.
14. Virginia Backaitis, "Open Source vs. Proprietary Software: There Is No Clear Winner," CMS Wire, 17 July 2013; Jason Cole and Helen Foster, Using Moodle, 2nd ed., O'Reilly Community Press, 2008; Learning to Teach Online: Learning Management System or the Open Web? College of Fine Arts, The University of New South Wales, 2013; and Seven Things You Should Know About LMS Alternatives, EDUCAUSE, 2010.
15. Carl Straumsheim, "iTunes U," Inside Higher Ed, 8 January 2014.
16. Rodney Rogers, "Selection of Canvas," blog, Senior Vice President for Academic Affairs and Provost, Fall 2013.
17. Donna C. Llewellyn, University System of Georgia Learning Management System Transition Task Force Final Report, University System of Georgia, 30 April 2011.
18. Learning Management Systems (LMS) Evaluation Tool [http://info.sagitec.com/LMS-eval-tool/], Sagitec Solutions, 2011.
19. Peter Berking and Shane Gallagher, (2013, May 14). Choosing a Learning Management System, Advanced Distributed Learning (ADL) Co-Laboratories, 14 May 2013, pp. 40–62.
20. Ibid., p. 40.
21. Dahlstrom, ECAR Study, 2013, p. 11.
22. Ibid., p. 23.
23. Clayton R. Wright, "Developing and Reviewing Online Courses: Items for Consideration," The Association of Learning Technology Online Newsletter, November 2011.
24. Dahlstrom, ECAR Study, 2013, p. 22.
25. Ibid., p. 6.