By Olivia Castello, Social Sciences Librarian
Alex Pfundt, Research & Instruction Librarian, & Coordinator of Information Literacy Library & Information Technology Services (LITS)
Bryn Mawr College
With the help of a Curricular Development Seed Grant, funded by the Mellon Foundation, librarians from Bryn Mawr College’s Library & Information Technology Services redesigned our model for one-shot information literacy instruction. We created self-guided, interactive online tutorials that allowed us to flip traditional demonstrations of skills, such as searching the library catalog, requesting books and articles, and finding empirical research. As a result, we were able to revise our in-class lesson plans to focus on active learning activities. We also conducted a research study in three academic courses to assess the efficacy of our flipped classroom model. This paper highlights the development of the online tutorials and instructional model, the assessment study, and ideas for future directions.
During the 2014-2015 and 2015-2016 academic years, Bryn Mawr College Library & Information Technology Services (LITS) staff members received a Curricular Development Seed Grant, funded by the Mellon Foundation, for a project entitled, “Flipping the Library Classroom at Bryn Mawr College.” The goal of the project was to improve information literacy (IL) instruction for students at Bryn Mawr College by exploring new and engaging pedagogical techniques for in-class librarian instruction sessions.
To this end, we created interactive, self-guided online tutorials, using Articulate Storyline e-Learning Software. The tutorials teach basic IL skills, tailored to Bryn Mawr undergraduates and first-year graduate students, such as: searching Tripod (the Tri-College Libraries’catalog), finding empirical research articles in subscription databases, and requesting items through interlibrary loan. We made the tutorials publicly available online and available for more formal course integration in Moodle.
We also collaborated with several faculty members to conduct an assessment study of a flipped model for IL instruction that used the online tutorials to allow more time for interactivity in class. Our study has shown the model to be effective at promoting student learning. As a result, the flipped IL instruction format, which began as a pilot project, is becoming regular practice for the Research & Instructional Services (R&IS) librarians at Bryn Mawr. In this paper we describe the pilot project, our assessment study, the current state of our tutorials and in-class instruction offerings, and opportunities for future development.
Many have noted the benefits of active learning in higher education classrooms, the potential of self-guided online tutorials to save class time and enhance learning, and the benefits of these specifically for librarian-led IL instruction. Our anecdotal experience teaching in-class IL sessions at Bryn Mawr College has also led us to believe that active learning formats lead to more engagement in class and better learning outcomes.
“Bibliographic instruction” has long been a service offered by the R&IS librarians at Bryn Mawr. Traditionally, this has taken the form of 30-90 minute one-shot sessions, where a librarian leads a lesson on research skills, and library tools and services. This method of instruction has had many benefits, particularly when embedded in the context of particular courses and tailored to relevant assignments on which the students are actively working. It also gives students a chance to get to know the librarian with the subject expertise most relevant to the course, who they may contact for further assistance. However, it can be a challenge that students in any one class often have a wide range of pre-existing familiarity with our library, libraries in general, and the research process, and have a wide range of learning styles and disciplinary interests.
In conceiving the flipped library classroom project, we hoped to create a way for students to arrive at our instruction sessions with a common IL level, so that we could spend class time diving deeper into advanced topics and incorporating more active learning. We wanted to find ways to avoid having the instruction session be too slow for some students and too fast for others because of differences in students’ IL backgrounds and skill levels. We also wanted to explore the potential for increased interactivity in our lessons to improve learning and retention of what we teach.
Recent research has shown a notable increase, relative to previous years, in U.S. faculty who believe that undergraduates are arriving at college without adequate research skills for college-level work. Ithaka S+R’s 2015 faculty survey found that 54% of respondents felt their undergraduates had “poor skills related to locating and evaluating scholarly information” (up from 47% in 2012). They also found that there has been an increase in faculty members’ perceived importance of the library’s role in developing student information literacy, with about half of the 2015 respondents agreeing that librarians contribute to developing students’ research skills. These response patterns were especially strong for faculty respondents from the Humanities and Social Sciences, and for faculty working with Baccalaureate- and Master’s-level students.
At Bryn Mawr, we also know from our biennial patron survey that both faculty and students consider librarian-led IL instruction to be important. Over half the faculty respondents in our 2014 survey said they considered “library research instruction in academic courses” to be important or very important. Only 11% felt it was not important. For undergraduates, 74% of international students, and 65% of undergraduates in general, considered this type of instruction important or very important. Meanwhile, answers to our survey question about satisfaction with library research instruction showed that although 80% of faculty were satisfied, the satisfaction rate among sampled undergraduates was lower at 62%.
For these reasons, we designed a pilot project to explore possible improvements to our IL instruction program. Our project had three main goals: 1) to gain experience with developing self-guided online tutorials; 2) to use these tutorials to flip library instruction and have more time for advanced topics and active learning activities; and 3) to assess the efficacy of our flipped classroom model on students’ information literacy.
Developing the Tutorials
As research and instruction librarians whose course-embedded interactions with students typically take the form of brief, one-shot instruction sessions, we are always on the lookout for ways to make better use of our limited face-to-face instruction time. We have been intrigued by other flipped library instruction projects, and interested, specifically, in ways that interactivity and simulation may improve student learning. In Summer 2014, we applied for and received a Mellon Digital Curriculum seed grant to develop and deploy brief, interactive tutorials using Articulate Storyline e-learning software. Our four tutorials are: “Introduction to Tripod,” “Requesting Books,” “Finding Articles,” and “Finding Empirical Research.” We chose to focus on these topics based on our experience of the most frequently asked questions we encountered with students during in-class instruction, research consultations, and reference transactions. Some of the material covered in the tutorials is specific to our library systems, such as how to search the Tri-College Libraries’ catalog, and how to obtain materials from other Tri-College Libraries, E-ZBorrow and interlibrary loan. Other topics are relevant beyond our campus context, such as how to identify empirical research, and how to develop effective searches in free search engines and subscription databases.
Each tutorial takes between 5-10 minutes to complete and is driven by user interactions to navigate through the tutorial screens. Interactive techniques that we employ include a variety of clickable hotspot screens (Figure 1), “checkpoint” question screens with answer feedback (Figure 2), and simulated online searching and navigation (Figure 3).
Figure 1. Examples of clickable hotspot screens where student interaction drives tutorial navigation
Figure 2. Examples of “checkpoint” screens that test material covered earlier in the tutorial and provide answer feedback
Figure 3. Examples of simulation screens that mimic web navigation and searching
After the initial tutorial development, we conducted two rounds of User Experience (UX) testing to identify potential improvements. First, we asked two Library & Information Technology Services (LITS) student employees to attempt the tutorials in different web browser environments (Firefox, Chrome, and Internet Explorer; with or without Flash enabled). After making changes based on their feedback, we asked four more LITS student employees to attempt the tutorials, resulting in further modifications. We chose to test iteratively with a small number of users based on advice that the best results come from multiple small-scale user tests, rather than elaborate one-time testing.
The UX testing resulted in numerous changes that improved tutorial accessibility and usability:
- Moving from a red color scheme to a higher contrast blue scheme
- Creating HTML5 versions as alternatives to Articulate Storyline’s default Flash-based tutorials in order to accommodate browsers and devices without Flash support
- Re-wording some checkpoint question screens and feedback text for clarity
- Splitting one tutorial that was considered too long into two shorter tutorials
- Stylistic changes to promote consistency with other Tri-College Libraries and Bryn Mawr College systems
We have made our tutorials available on two separate platforms. Any tutorial can be embedded as a Sharable Content Object Reference Model (SCORM) package in the learning management system (LMS), Bryn Mawr’s learning management system. This is a format that we have promoted to Bryn Mawr faculty because it allows tracking of measures of student engagement (e.g. tutorial start time, last access time, completion), and also makes it possible to link tutorials to the LMS gradebook for course credit.
We have also made the tutorials publicly available on the Tri-College Libraries’ Research Guides system at http://libguides.brynmawr.edu/bmclibrary-tutorials in both Flash and HTML5 formats. We have promoted these to both faculty and students without any tie-in to particular classes.
Redesigning Instruction Sessions
The flipped classroom model of asking students to complete the tutorials in advance of our in-class sessions has allowed us to maximize the in-person class time for short, one-shot instruction. Based in the principals of Universal Design for Learning (UDL), students have full autonomy to complete the tutorials at their own pace, on their own schedule, and as many times as they need to feel confident about the material. The self-guided format gives students more options to review and re-learn the material, and the interaction driven nature of the tutorials requires active engagement to move through the screens. We see these as major advantages over our traditional bibliographic instruction where students may have difficulty paying attention, find the pace of a demonstration too fast, have language comprehension difficulties, or miss class.
The R&IS librarians use different active learning techniques depending on what is appropriate for particular classes. In humanities and social science classes, we have had success with scavenger hunts, think-pair-share discussions, peer-reviews and other small group activities. Since launching the tutorials, we have been able to take more time to experiment with game-based, peer-learning, group discussion, and other interactive pedagogical techniques knowing that we no longer need to cover the basics.
Assessing the Flipped Classroom
During the 2014-2015 and 2015-2016 academic years we conducted an assessment study of our flipped instruction model. We partnered with three classes in Bryn Mawr’s Department of Psychology and Graduate School of Social Work and Social Research:
- PSYC 203: Educational Psychology –a single section undergraduate course that enrolls both psychology majors and non-majors
- PSYC 231: Health Psychology –a single section undergraduate course that enrolls both psychology majors and non-majors
- SOWK 503: Research-Informed Practice I –a multi-section graduate level course, required for 1st year social work students
The assessment study had three components:
- Tutorials: We asked students in these courses to complete self-guided tutorials in advance of an interactive, in-person instruction session led by a librarian. Embedding the tutorials in the LMS allowed us to gather data on each student’s activity with the tutorials, including number of attempts (avg = 1.08 attempts), time spent (avg = 8 minutes per tutorial), and tutorial completion status.
- In-class session: During the in-class session, the students experienced a hands-on, interactive IL lesson. Instead of demonstrating basic library catalog searching, interlibrary loan procedures, and other topics covered in the self-guided tutorials, the librarian built on, and went beyond, these basic skills. Sessions included brief demonstrations of more advanced topics (such as Boolean search techniques and iterative searching). Students then worked in small groups to brainstorm search terms, construct and evaluate a practice search. There was time for discussion of the search process and for questions and answer feedback from the librarian.
- IL quizzes: To gauge their information literacy at various points in the study, participants also completed short IL quizzes composed of questions based on the then extant Association of College & Research Libraries’ (ACRL) Information Literacy Competency Standards for Higher Education (Figure 4).
Figure 4. Assessment study components included tutorials and information literacy quizzes embedded in Moodle
The data we collected revealed statistically significant patterns that indicate a positive relationship between our flipped classroom model and student information literacy skills. In the 2014-2015 study year, all students were asked to complete the IL quizzes and experienced the interactive in-class instruction session, but only some students were offered the self-guided tutorials. We then compared the quiz scores of students who engaged with the self-guided tutorials and students who did not. Participants who completed all the tutorials had a higher average IL quiz score than participants who did not complete any of the tutorials (two sample T-test: t(27)=2.72, p=0.011). Minutes spent on tutorials did not seem to have any relationship to students’ IL quiz scores. However, an analysis of the variation between mean IL quiz scores of students who completed different numbers of tutorials fell just short of conventional levels of significance (one-way between subject ANOVA: F(2,31)=3.19, p=0.055). Finally, we compared quiz scores from immediately after the in-person session with those taken later in the semester, but did not find any significant differences.
In the 2015-2016 study year, we offered the self-guided tutorials to all participants, but controlled for baseline differences in information literacy. Students took IL quizzes before and after they completed the tutorials and in-class instruction session, and we found that quiz scores improved (Table 1; paired sample T-test: t(41)=-7.15, p<0.0001). The data showed a moderate positive relationship between students’ minutes spent on tutorials and improvement in IL quiz score, however our analysis failed to reach conventional levels of statistical significance (Pearson correlation: r=.4, n=20, p=0.08).
|Quiz||N||Average quiz score|
Table 1. Pre- and Post-Tutorial information literacy quiz scores from the 2015-2016 study year
In both study years, students completed the tutorials, in-class instruction, and information literacy quizzes within an approximately two-week period. Our analyses showed positive relationships, including two statistically significant ones, between students’ tutorial engagement, in-class instruction experience and IL quiz scores. This has led us to conclude that our flipped instruction model had a learning benefit for students in our study, and may have the potential to benefit the broader population of students at the College.
Adoption and Publicity
Since launching the pilot project, we have publicized the tutorials to faculty members via e-mail and have advertised them to students as part of our preparation for upcoming in-class instruction sessions. Several faculty members at Bryn Mawr, beyond our assessment study partners, have incorporated our tutorials into their classes in a variety of ways.
- ENGL 126: Workshop for Multilingual Writers — required LMS course component
- ENGL 127: Advanced Workshop for Multilingual Writers — required LMS course component
- PSYC 231: Health Psychology –required LMS course component
- SOWK 503: Research-Informed Practice I –required LMS course component
- PSYC 284 Health Psychology Lab — optional LMS course component
- PSYC 289 Clinical Psychology Lab — optional referral to the publicly available tutorials
- PSYC 354: Asian American Psychology — required LMS course component
For the tutorials that are freely available online, we have anonymous data on their use and know that there has been some adoption of the public versions of the tutorials. In the 2015-2016 academic year the tutorials were launched in 116 unique user sessions, and the average time spent on a tutorial was approximately 6 minutes. “Tutorials” was also the one of the top search terms (#13) in our Research Guides system for 2015-2016 academic year. Our non-Flash tutorial versions have been used (6% of views). 70% of users were directly referred to the tutorial, which may indicate that students are clicking on the direct links that we have sent them through email.
We have come a long way towards our original goal of partnering with faculty to improve IL instruction at Bryn Mawr. However, our progress so far also points to new directions for our outreach and teaching.
We hope to continue to explore ways of making the tutorials fully accessible to students with visual, auditory, learning, and/or other disabilities. To this end we have eschewed certain Articulate Storyline navigation options, such as drag-and-drop quizzes. We also upgraded to version 2 of Articulate Storyline software, which promises improved accessibility features, such as better support for screen readers. We are also working on creating accessible PDF transcripts that will be available alongside the tutorials.
Self-guided online tutorials may not be appropriate for every learning situation. We would like to add other formats to our menu of IL resources, such as short screencast how-to videos for simple frequently asked questions. We also continue to consider third-party IL modules and how-to documentation, where possible.
The R&IS librarians have had productive experiences conducting UX testing with undergraduates. Our testing with the self-guided tutorials helped us identify concrete accessibility and usability improvements. As we maintain our four Articulate Storyline tutorials, we hope to continue to conduct brief, periodic UX testing in order to identify further areas for improvement.
Since the professors who have used these tutorials in their classes have liked them, we believe it would promote increased adoption to publicize these successful partnerships more. Increasing publicity about the tutorials generally, and the various options for referring students to the publicly available versions versus embedding them in a course, may also give faculty members who do not typically invite librarians into their classes a new way to include IL instruction into their curriculum
Finally, we conceived of and developed the tutorials and assessment tools based on ACRL’s Information Literacy Competency Standards for Higher Education. Revisiting the tutorials since the official adoption of the Framework for Information Literacy in Higher Education has revealed to us that many of the fundamental learning goals map to the Framework threshold concepts Research as Inquiry and Searching as Strategic Exploration. Because of this, we intend to leave the tutorials largely unchanged despite the rescission of the Standards, and will continue to use them in conjunction with new classroom activities we are designing based on the Framework.
Our self-guided tutorials allow students to review general information literacy skills at their own pace and as many times as they need. Our hope is that this will improve students’ depth of understanding of the tutorial topics. Though some of the material presented in the tutorials is specific to Bryn Mawr, or to the Tri-College Libraries consortium, many concepts are relevant to the evolving information environment beyond our campus. Usage data from both the Moodle-embedded and publicly available versions has shown that there has been some faculty adoption of and student engagement with the tutorials.
The tutorials have increased R&IS librarians’ efficiency in using in-class instruction time. They have allowed us to move away from lengthy demonstrations and devote more time to participatory classroom activities that reinforce and go beyond the basic tutorial skills. We have experimented with active learning techniques like think-pair-share, peer-reviewing, scavenger hunts, and others activities where we tie IL skills to course-embedded contexts and topics of student research. Our goal has been to promote better student engagement and enhanced learning and our assessment study indicates that our flipped classroom model for IL instruction has had a learning benefit for our students.
 Bryn Mawr College, “Developing a Liberal Arts Curriculum for the Digital Age,” Blended Learning in the Liberal Arts, accessed June 13, 2016, http://blendedlearning.blogs.brynmawr.edu/seed-grants-2/.
 Adam P. Fagen, Catherine H. Crouch, and Eric Mazur, “Peer Instruction: Results from a Range of Classrooms,” The Physics Teacher 40, no. 4 (April 1, 2002): 206–9, doi:10.1119/1.1474140; Richard R. Hake, “Interactive-Engagement versus Traditional Methods: A Six-Thousand-Student Survey of Mechanics Test Data for Introductory Physics Courses,” American Journal of Physics 66, no. 1 (January 1, 1998): 64–74, doi:10.1119/1.18809; Eric Mazur, “Farewell, Lecture?,” Science 323, no. 5910 (January 2, 2009): 50–51, doi:10.1126/science.1168927; David R. Sokoloff and Ronald K. Thornton, “Using Interactive Lecture Demonstrations to Create an Active Learning Environment,” The Physics Teacher 35, no. 6 (September 1, 1997): 340–47, doi:10.1119/1.2344715.
 Nicole Johnston, “Is an Online Learning Module an Effective Way to Develop Information Literacy Skills?,” Australian Academic & Research Libraries 41, no. 3 (September 1, 2010): 207–18, doi:10.1080/00048623.2010.10721464.
 Sara Arnold-Garza, “The Flipped Classroom Assessing an Innovative Teaching Model for Effective and Engaging Library Instruction,” College & Research Libraries News 75, no. 1 (January 1, 2014): 10–13; Anthony C. Holderied, “Instructional Design for the Active: Employing Interactive Technologies and Active Learning Exercises to Enhance Library Instruction,” Journal of Information Literacy 5, no. 1 (May 11, 2011): 23–32, doi:10.11645/5.1.1519; Julia E. Rodriguez, “A Massively Flipped Class: Designing and Implementing Active Learning Information Literacy Instruction for a Large Enrollment Course,” Reference Services Review 44, no. 1 (February 1, 2016): 4–20, doi:10.1108/RSR-07-2015-0033; Alanna Ross and Christine Furno, “Active Learning in the Library Instruction Environment: An Exploratory Study,” Portal: Libraries and the Academy 11, no. 4 (October 2011): 953–70, doi:10.1353/pla.2011.0039.
 Ilka Datig and Claire Ruswick, “Four Quick Flips Activities for the Information Literacy Classroom,” College & Research Libraries News 74, no. 5 (May 1, 2013): 249–57; Holderied, “Instructional Design for the Active”; Rebecca Oling and Marie Sciangula, “Learning to Leverage: Using Moodle to Enhance F2F Interaction in One-Shots,” LOEX Quarterly 38, no. 3 (November 1, 2011), http://commons.emich.edu/loexquarterly/vol38/iss3/5.
 Lily M. Belfi et al., “‘Flipping’ the Introductory Clerkship in Radiology: Impact on Medical Student Performance and Perceptions,” Academic Radiology 22, no. 6 (June 2015): 794–801, doi:10.1016/j.acra.2014.11.003; Shawn M. Keough, “Clickers in the Classroom A Review and a Replication,” Journal of Management Education 36, no. 6 (December 1, 2012): 822–47, doi:10.1177/1052562912454808.
 Jakob Nielsen, “Why You Only Need to Test with 5 Users,” NN/G Nielsen Norman Group, March 19, 2000, https://www.nngroup.com/articles/why-you-only-need-to-test-with-5-users/.
 National Center of Universal Design for Learning, “UDL Guidelines: Theory & Practice,” last modified November 12, 2014, http://www.udlcenter.org/aboutudl/udlguidelines_theorypractice.
 Association of College and Research Libraries, “Information Literacy Competency Standards for Higher Education.” (Association of College and Research Libraries, January 18 2000), accessed April 12, 2017, http://www.ala.org/acrl/standards/informationliteracycompetency.
 Articulate Global, Inc., “Articulate Storyline 2 Web Content Accessibility Guidelines (WCAG) Compliance,” accessed May 26, 2016, https://www.articulate.com/products/storyline-wcag.php.
 Association of College and Research Libraries, “Information Literacy Competency Standards for Higher Education.” (Association of College and Research Libraries, January 18, 2000), accessed April 12, 2017, http://www.ala.org/acrl/standards/informationliteracycompetency.
 Association of College and Research Libraries, “Framework for Information Literacy for Higher Education,” (Association of College and Research Libraries, February 2, 2015), http://www.ala.org/acrl/standards/ilframework.
About the Authors
Olivia Castello, Social Sciences Librarian, and Alex Pfundt, Research & Instruction Librarian & Coordinator of Information Literacy, are part of the Research & Instructional Services group in Bryn Mawr College’s Library & Information Technology Services (LITS). Bryn Mawr is a residential, liberal arts college for undergraduate women, with co-educational graduate programs, near Philadelphia, PA. Bryn Mawr College Library is part of the Tri-College Libraries consortium, with Haverford and Swarthmore Colleges.
This project relied on the valuable contributions of numerous people. In addition to the original project team members (Christine Boyland, Olivia Castello, Alex Pfundt, and Arleen Zimmerle) thanks to Helen Chang for help with Moodle tutorial deployment and to David Consiglio for statistical analysis suggestions. Thanks to the student participants and faculty partners in our assessment study, President Kim Cassidy and Prof. Laurel Peterson of the Department of Psychology, Prof. Leslie Alexander, Prof. Margo Campbell, and Prof. Becky Laster of the Graduate School of Social Work and Social Research. This project was made possible by Bryn Mawr’s “Developing a Liberal Arts Curriculum for a Digital Age” grant, funded by the Andrew W. Mellon Foundation.