A Tale of Two Sections: An Experiment to Compare the Effectiveness of a Hybrid versus a Traditional Lecture Format in Introductory Microbiology

INTRODUCTION

Online education is becoming increasingly common and offers many benefits, including increased flexibility and compatibility with work schedules, the ability of students to access educational materials from a distance, the ability to self-pace learning, and savings in classroom space and faculty staffing (Garrison and Vaughan, 2008; Means et al., 2010). However, students in online classes may feel isolated or disconnected from both the instructor and other students, often experience a steep learning curve for the online tools, may require high levels of motivation and organization, and are less likely to complete their online courses (see Lim and Morris, 2009; Xu and Jaggars, 2013). In a meta-analysis conducted by the U.S. Department of Education, there was little or no evidence of an overall benefit of online courses (Jaggars and Bailey, 2010; Means et al., 2010). Hybrid classes, which combine the benefits of online learning and face-to-face instruction, are becoming increasingly popular (Garrison and Vaughan, 2008; Means et al., 2010), and a number of studies suggest student outcomes can be equal to or greater than those from traditional face-to-face courses (Amaral and Shank, 2010; Means et al., 2010; Bowen et al., 2014). However, it is often difficult to determine which factors of redesigned courses lead to increased success, as many variables (including the all-important amount of time students spend on the course) usually change simultaneously when courses are redesigned from a traditional to a hybrid format.

Recently, some studies have examined the effect of hybrid courses compared with face-to-face instruction through randomized experimental design. For example, Joyce et al. (2014) randomized 725 students to a hybrid or traditional section of a course on principles of microeconomics. In this study, these authors found that the students in the traditional section scored 2.5% better (p < 0.05) on test scores and grades. In another study, Figlio et al. (2013), randomized 327 students to either a hybrid or traditional section of an introductory microeconomics course and found there was no significant difference between face-to-face and hybrid instruction, with some evidence that face-to-face instruction was better, especially for Hispanic, male, and lower-achieving students (Figlio et al., 2013). Conversely, Bowen et al. (2014), in a study of an introductory statistics course, randomized 605 students to either hybrid or face-to-face instruction across six campuses and found no difference in objective measures (final exam scores, standardized test of statistic literacy, and pass rates) between the two approaches. All three studies sought to determine the effect of a single variable—the time spent in the classroom—and although the overall conclusions are mixed, all three studies showed that any differences seen between hybrid and face-to-face classes are fairly small. The present study was designed to examine the learning outcomes and experiences of students in a hybrid section compared with those in a traditional section of an introductory microbiology course; to our knowledge, this is the first time this has been done for a class in the life sciences at a large public university. Like the studies described earlier, the main difference was in the way in which the lecture material was delivered (online vs. face-to-face), with an attempt to keep other variables constant between the two sections.

At Northern Arizona University (NAU) and many other institutions, faculty members are encouraged to develop hybrid courses both to increase efficiency of classroom space and faculty resources, while maintaining or increasing student learning outcomes. Introductory Microbiology (BIO205) is a large, multisection class that has typically been taught through a traditional lecture format, but enrollment in this class has increased enormously in the past few years (from one to four sections per year). To determine whether use of a hybrid format would be an effective way of maintaining or enhancing student learning, a study was conducted in Spring 2014, wherein two sections were taught in a 16-wk-long semester course by a single instructor (A.E.M.A.). The aim of the study was to determine the effect of a single variable—the effect of replacing an in-person lecture with an online lecture to be watched before an in-person class period, with most other instructor-controlled components of the course kept constant between the two sections.

The specific research goals were: 1) to compare performance of the students in the two sections through analysis of exam and final scores; 2) to determine whether any differences could be explained by demographic differences (major, student standing, prior grade point average [GPA], gender, and ethnicity); and 3) to determine whether any differences in performance between the two sections could be explained by different levels of student engagement as measured by in-class attendance, whether students took advantage of extra credit opportunities, and how students in the hybrid section used the online lecture (i.e., whether they used audio as well as visual components of the lecture and whether/how they took lecture notes).

METHODS

Institutional Review Board Approval

The study protocol was approved by the Northern Arizona University Institutional Review Board (IRB project 572201-1), and students were asked to sign IRB-approved consent forms if they wished to participate in the study.

Design of the Hybrid and Traditional Sections

The two sections were designed to be taught identically, except in the way the material was delivered—online versus face-to-face. For the traditional section, during the first class session of the week, students had a face-to-face lecture. For the hybrid section, instead of the in-class lecture, an online lecture was posted. The two lectures were identical in content and were delivered in the same style—orally with whiteboard-style notes and diagrams. The hybrid section was given a preclass online quiz on the lecture material to ensure it was watched before class (the traditional section did not have a preclass online quiz). For both sections, during the “second” class period of the week, students were provided with a worksheet based on, and extending, the material from the lecture. They were given ∼30 min to complete the worksheet, in groups of three to five. During this time, they received help as needed from both the instructor and peer teaching assistants (TAs). After 30 min, students handed in a group answer sheet, with the names of those who contributed; they kept individual answer sheets for their own records. The instructor then went through the worksheet with the class as a whole, to ensure that students had a thorough understanding of the material before leaving the classroom. After the in-class worksheet session, an online postclass quiz was posted for both sections, based on the lecture material as well as material from the in-class worksheets. This postclass quiz closed before the next class. As the students in the hybrid section had already had a preclass quiz on the lecture material, the postclass quiz for the hybrid section contained questions based primarily on the worksheets. However, as the students in the traditional section had not had a preclass quiz, their postclass quiz also contained questions given to the hybrid section during the preclass quiz (thus, both sections were given exactly the same quiz questions, but the timing of the preclass quiz questions was different).

In addition to the same lecture material and weekly in-class worksheets, both sections had the same midterm and final exams and the same opportunities for extra credit points. Student success in the two sections over the course of the semester was compared using exam and total semester scores (excluding the quiz scores).

Thus the only differences in the way in which the two sections were taught were: 1) in how the lecture material was delivered (online vs. face-to-face); 2) the hybrid section came to class just once a week (as they had the lecture online), whereas the traditional section came to class twice a week (as the lecture was delivered in person); and 3) the online postclass quizzes for the traditional section contained the questions from both the preclass plus postclass quizzes that were assigned to the hybrid section. Additional differences between the two sections, which could not be controlled, were: 1) time of day at which the classes met (12:45 pm for the traditional class and 4:00 pm for the hybrid class) and 2) student demographics (see Participants; Table 1).

The Online and Traditional Lectures

LIVESCRIBE (www.livescribe.com) was used to generate online lectures that contained both audio and visual components. This technology involves use of an electronic pen and paper that records the voice as well as notes, diagrams, and so on as they are written; therefore, it closely resembles what would be seen and heard by students watching a lecture delivered using a whiteboard. The online lecture is then saved as an audio/visual PDF file that can be downloaded like any other PDF. Students can download the lecture notes when they have Internet access and can watch and listen to the lecture at their convenience, with the opportunity to pause the lecture as needed. Online lectures varied from 28 to 57 min, with a mean of 44 min.

Traditional lectures covered the same material (each online lecture corresponding to one in-class lecture). In the traditional section, class time was 75 min. The difference in time taken for the online versus in-class lecture was due to additional time needed in the traditional lecture format to let students finish drawing diagrams, write notes, and ask questions (as noted earlier, the online lecture students could pause the lecture as needed). Students in both sections were welcome to ask questions (face-to-face for the traditional section and online for the hybrid section), although the vast majority of students in both sections did not ask questions during the lecture period.

Assessment of Student Learning

Student learning in both sections was assessed out of a maximum of 450 points, with 1) best 10 of 12 weekly online quizzes (in the hybrid section, there were 12 pre- and postclass quizzes, each worth 5 points; in the traditional section, there were just 12 postclass quizzes, each worth 10 points, as described earlier; in both sections, the online quizzes were therefore worth 100 points total); 2) best two of three midterm exams (worth 200 points total) and a final exam (worth 100 points); 3) participation points, based on group worksheets (worth 25 points, with participation points closely reflecting attendance); and 4) a readiness test, which students were required to pass (unlimited number of attempts) by the end of the second week in order to stay in the class (worth 25 points). Sixteen additional extra credit points were made available to both sections during the semester for taking practice exams and a practice quiz. For comparisons between the two sections described in this paper, the data were analyzed without the online quizzes, because the timing of these was slightly different between the sections.

Exams and quizzes were all developed by the instructor over the course of several years and were designed to test course objectives. More important topics had more test questions, and confusing questions were clarified or eliminated over the years of teaching this class.

Analysis of the Way in Which Students Used the Online Lecture

At midsemester, immediately after the midterm grades were posted and when students were likely to be in a routine, students in the hybrid section were asked anonymously what their midterm grade was and whether they used audio and/or visual components of the online lecture. Use of the audio was then compared with the student's self-reported midterm grade. The response rate was 65%.

In addition, again around midsemester, the instructor asked to see the lecture notes the students had brought to class that day. The students were told that the instructor was evaluating whether there was a relationship between student success and the type of notes taken. The students had not been warned that this would happen, and the 1-d assessment therefore provided a snapshot of what students were typically using for notes during class time. While students were working on the worksheets in class time, the instructor walked around categorizing students’ notes into those that were 1) handwritten, similar to those that would be taken during a traditional lecture; 2) printed-out copies of the online lecture, with or without annotation; 3) brought up on the computer or cell phone; and 4) not brought to class that day. Notes taken by students during class time in the traditional section were not examined.

The Peer TAs

Because of the size of both sections (122 students in the single traditional section and 202 in the hybrid section—split equally into two hybrid subgroups), it was necessary to have help with answering students’ questions as they worked on the in-class worksheets. Three peer TAs were recruited from students who had earned an “A” in BIO205 in the previous semester. Two of these peer TAs assisted with both sections, and the third assisted with just the hybrid section. Each week before the first class of the week, all three peer TAs met with the instructor for a TA meeting, to review the worksheet. In addition, each class had a paid supplemental instructor (SI) provided by the NAU SI program. The peer TAs received independent study credit in pedagogy and the SI received a stipend.

Participants

Student demographics (major, student standing, prior GPA, gender, and ethnicity) for the two sections are shown in Table 1. Class sizes for the two sections were similar—in the traditional section, there were 122 students (who met as a single group twice a week), and in the hybrid section, there were 202 students divided equally into two subgroups (each of which met once a week). Nursing, biomedical sciences, and biology accounted for 71–76% of all students in both sections (Table 1 and Figure 1), but the nursing students accounted for 45% of the students in the hybrid section and 16% in the traditional section (Table 1 and Figure 1). Numbers of students of different standing in the two sections were also skewed, especially for freshmen and juniors: freshmen accounted for 48% of the students in the hybrid section and just 11% in the traditional section, and juniors accounted for 36% of the students in the traditional section and just 12% in the hybrid section (Table 1 and Figure 2). There was a trend for the average prior GPA of students in the two sections to be different (3.15 for the hybrid section vs. 3.27 for the traditional section; p = 0.076), but gender and ethnicity makeup was similar between the two sections (Table 1).

Data Analysis

Total points for the semester, as used in the analysis, were made up of the best two of three midterm exams (each worth 100 points), the final exam (100 points), class participation points (25 points), points for passing the readiness test (25 points), and extra credit points (16 points). The data were analyzed without the online quiz scores (100 points), as the makeup and timing of the quizzes were different between the two sections (see Assessment of Student Learning). Differences between the two sections were analyzed with independent t test or univariate general linear model when additional factors, such as class year or major, were analyzed. Post hoc testing was performed when appropriate, using least significant differences. Pearson correlation analysis was used to examine the relationship between various outcome variables. All analyses were conducted using IBM SPSS statistical software version 22.0 (SPSS, Chicago, IL). All comparisons were considered significant at p < 0.05. Data are presented as means ± SD.

RESULTS

Exam Scores in the Two Sections

Analysis of exam scores from the three midterm exams and the final exam indicated that students in the hybrid section achieved significantly lower scores than those in the traditional section for all exams (p < 0.05; Table 2 and Figure 3). Table 3 shows the effect of gender, major, ethnicity, and class standing on differences in final score (excluding quizzes) between the two sections. As shown in Table 3, there were no significant differences in class performance within each section between males and females or between majors (comparing nursing, biomedical sciences, biology, and all other majors grouped together). There was, however, a significant difference in class performance within each section when white students were compared with all minorities combined: the former had significantly higher total points as compared with the latter within each section (p < 0.001). Similarly, there was a significant effect of class standing within each section (p < 0.01). There was, however, no significant interaction between the type of class (hybrid vs. traditional) and ethnicity or class standing, meaning that the performance differences between the two sections were the same whether in the hybrid or the traditional section.

Post hoc testing revealed that freshmen had lower scores than sophomores, juniors, and seniors (all: p < 0.001), but there were no significant differences between sophomores, juniors, and seniors. When students of the same standing were compared separately across sections, sophomores in the traditional section did significantly better than those in the hybrid section (p < 0.001), but the scores of freshmen, juniors, and seniors were not significantly different between sections (Table 3).

As described in the Methods section, there was also a marginal difference in average prior GPA between the two sections (3.15 for the hybrid vs. 3.27 for the traditional section, p = 0.076). Although this difference between the two sections was just a trend and not significant, prior GPA was associated with overall success in the class: the overall correlation for both sections combined was r = 0.63 (p < 0.001); for the hybrid section, r = 0.62 (p < 0.001); and for the traditional section, r = 0.68 (p < 0.001).

Withdrawal Rates of Students Who Withdrew from the Two Sections

Comparison of withdrawal rates from the two sections indicated slightly higher attrition from the hybrid than from the traditional section (7.8 vs. 3.9%, respectively), although this was not statistically significant. Class standing, major, gender, average prior GPA, and ethnicity of students who withdrew from the two sections is shown in Table 4. Ethnicity, major, prior GPA, and gender appeared to have no significant effect on the different withdrawal rates from the two sections. The only factor that appeared to be associated with the different withdrawal rates from the two sections was student standing: 11 of the 17 students who withdrew from the hybrid section and one of the five students who withdrew from the traditional section were freshmen (recall that 48% of students in the hybrid section were freshmen and just 10% of students in the traditional section were freshmen). Thus, the higher withdrawal rate from the hybrid section is most likely explained by the higher proportion of freshmen in this section, rather than the online nature of the lecture.

In-Class Participation (Attendance) Points in the Two Sections

In-class participation was monitored through the use of group worksheets in both sections (see Methods). As the worksheets were graded liberally, the participation points closely reflected attendance. Students were allowed to miss one class session without penalty. Therefore, their total participation grade was calculated from their best 10 of 11 scores (each worth 2.5 points). Thus, the maximum number of participation points that could be earned was 25 points. The average participation points were not significantly different between the hybrid and traditional sections (23.7 ± 3.0 points [or 95%] vs. 23.7 ±4.0 points [or 95%], respectively) and did not differ significantly between class standings. There was a significant correlation between attendance and total points in the whole cohort (r = 0.36, p < 0.001) and within each section (hybrid: r = 0.37, p < 0.001; traditional: r = 0.41, p < 0.001). This finding is in agreement with a number of studies that show the importance of attendance on student success (e.g., Means et al., 2010).

Extra Credit Points in the Two Sections

Students in both sections were given the opportunity to earn extra credit points for taking and handing in practice exams before an in-class review session for each exam. The maximum number of points that could be earned in this category was 16 (∼3% of the overall grade). The mean for extra credit points was significantly different between sections; students in the hybrid section earned on average 11.7 ± 4.6 extra credit points (or 73%), while students in the traditional section earned on average 13.8 ±3.3 points (or 86%; p < 0.001).

Type of Lecture Notes Taken by Students in the Hybrid Section

Somewhat surprisingly, most of the students in the hybrid section were found to be taking extensive handwritten notes (Figure 4). When final scores were compared with type of lecture notes (assessed for 154 students in the hybrid section), there was a significant effect of note taking when quiz scores were included (p < 0.001) and a trend when the final scores were analyzed without quiz scores (p = 0.076). Post hoc testing showed that students who took handwritten or typed notes had significantly higher final grades than those who did not bring notes to class (either because they forgot to bring them or they had not taken any), whether quiz scores were included or not (p < 0.05; Figure 4). Note taking by students in the traditional section was not examined.

Use of the Audio and/or Visual Components of the Online Lecture

Another possible contributing factor to the difference between the two sections was that not all students in the hybrid section used the audio component of the online lecture, despite efforts to encourage them to do so. Students in the hybrid section were asked in an anonymous midterm survey what their midterm grade was and whether they were using audio, visual, or both components of the online lecture. Of those who responded to this question (61%), 15% of the students reported using just visual, 0% reported using just audio, and 85% reported using both. Self-reported midterm grades and use of audio were analyzed for individual students, and, not surprisingly, those students who used both audio and visual had higher midterm grades than those who used just the visual component of the online lecture (p < 0.01; Figure 5).

Predictors of Final Scores within Each Section

When final scores (without quiz scores) were compared for students within each section, significant predictors of final score in the hybrid section were class standing, prior GPA, ethnicity, and note type; major was not significant. For the traditional section, however, the only significant predictors were prior GPA and ethnicity (note type was not analyzed for this section), although class standing showed a trend (p = 0.073), and major had borderline significance (p = 0.051).

DISCUSSION

In this study, we compared the effectiveness of teaching an introductory microbiology course (BIO205, a sophomore-level 16-wk course at NAU) through a hybrid approach with the effectiveness of a traditional lecture format. In both sections, which were taught concurrently by the same instructor, identical lecture content and in-class assessments were used. The main difference between the two sections was in the way the lecture material was presented: online for the hybrid section (replacing one of the two class periods per week) and in person for the traditional section. Several key findings were made. First, students in the hybrid section did significantly less well overall than those in the traditional section (Table 2). Analysis of this difference identified a significant main effect of class standing, with freshmen performing less well in both sections (Table 3). As there was a much higher percentage of freshmen in the hybrid section than in the traditional section (48 vs. 11%; Table 1), some of the difference in success of the two sections is attributable to the larger proportion of freshmen in the hybrid section. (None of the students who signed up for the class knew that there was a difference between the two sections, so the different proportions of students of different standing in the two sections was most likely due to the fact that, at NAU, freshmen are the last ones to register for classes; the class that was earlier in the day [12:45 pm] filled with students of higher standing first, so the freshmen ended up in the later section [4:00 pm], which turned out to be the hybrid section.)

Second, comparison of students of the same standing between sections (e.g., freshmen vs. freshmen) indicated a trend in which students of all standings did less well in the hybrid section than did their counterparts in the traditional section (e.g., for overall scores, excluding the quizzes, the differences were 3.4% for freshmen, 7.5% for sophomores, 2.9% for juniors, and 4% for seniors; Table 3). The only significant difference, however, was for sophomores. It is possible that because BIO205 was designed to be a sophomore-level course, freshmen generally found the material difficult and juniors and seniors generally found the material easy, regardless of the way in which the course was taught.

The lower success of students in the hybrid course compared with their counterparts in the traditional course could be due to less interaction with the material or a sense of isolation arising from required class attendance being limited to just once a week. We did not examine the latter in this study, but several observations suggest that, on average, students in the hybrid course were less engaged with the material than those in the traditional section. For example, when students were asked what type of notes they were taking from the online lecture, 26% had either simply downloaded the lecture onto their cell phone or laptop or had no notes in class that day (because they forgot to bring them or because they had not taken any notes). Students who brought handwritten or typed notes to class had significantly higher final semester grades than those who did not bring notes to class (Figure 4); this result supports the findings of Mueller and Oppenheimer (2014), who documented the success of students who take handwritten notes. Similarly, failure of 15% of students in the hybrid section to use the audio component of the online lecture (i.e., of those who responded to the anonymous midterm survey) also provided an indication of lack of interaction with the material; not surprisingly, those students in the hybrid section who reported not using the audio component of the online lecture did significantly less well (as judged by their self-reported midterm grade) than those who did (Figure 5). An additional measure of lower engagement of students in the hybrid section comes from the significantly lower number of extra credit points that students in this section earned, compared with students in the traditional section. All these factors likely reflect reduced time spent on task—one of the major predictors of students’ success (Admiraal et al., 1999; Lim and Morris, 2009; Means et al., 2010).

The overall finding in this study—students in the hybrid section were less successful than those in the traditional section—is similar to findings in two other randomized experimental studies that showed a small negative effect of using a hybrid approach compared with a traditional lecture format for teaching microeconomics (Figlio et al., 2013; Joyce et al., 2014). The findings here, however, differ from those in another randomized experimental study on a large undergraduate class (in statistics) that showed no significant difference between the hybrid and traditional approach (Bowen et al., 2014). It is likely that there is no single answer to the question of which is better. The present study, however, adds to the literature on experimental studies comparing hybrid and traditional classes and, for the first time, provides data on an introductory microbiology class at a large public university.

To maximize student success in future versions of the hybrid class, it will be important to 1) ensure that all students listen to the audio component of the online lecture and take handwritten or typed lecture notes and 2) identify more ways (either online or otherwise) in which to engage students in the material. Lack (2013) and Bowen et al. (2014) suggest use of additional sophisticated interactive online materials may be beneficial, but the Means et al. (2010) report indicates that inclusion of more online media is not necessarily more effective in increasing student learning than other approaches (Means et al., 2010). As prior GPA and class standing are two of the predictors of final score in the hybrid section, it may also be worth limiting admission of freshmen to the hybrid version of this sophomore-level class to those who have more than a threshold GPA. The potential benefit of teaching the class through a hybrid approach is large (particularly in increased efficiency of classroom space and use of faculty resources)—especially given that 82% of the students in the hybrid section who responded to the midterm survey said that they preferred this style of teaching (data not shown)—and makes it worth trying to identify factors that will increase student learning outcomes by this approach.

ACKNOWLEDGMENTS

REFERENCES