Saturday, May 4, 2013

Flipping with a MOOC-- A very new approach to teaching for me

This semester (spring, 2013), I integrated my on-campus Duke University class (which I've taught twice before using a "traditional" lecture format) with my online class (which I'd taught once before via Coursera MOOC), both bearing the title "Introduction to Genetics and Evolution."  My on-campus class had 453 students, while the online one peaked at 27,000 enrolled (though MOOC enrollment figures are misleading).  Needless to say, I was more than slightly nervous about this experiment messing up, given the number of students who would be affected!  My initial reaction is that the integration (via "flipped classroom") was a success and thoroughly enjoyable by me (I'll have to wait to see the formal course evaluations before I know how much most of the students liked it), but I learned some lessons for future iterations.


Duke students had more expected of them outside of class than in my prior course iterations.  Prior to every class period, Duke students were expected to watch 3 videos (average ~15 minutes each) in Coursera bearing 80-90% of the primary "raw information content" that I expected them to learn from the course.  This content is what I delivered in years past via traditional lecture format in class, but now students had to set aside ~45 minutes before each class period to get it, perhaps longer if they needed to replay segments or stop to think. 

Following the videos, the students had to take a graded preclass quiz, formatted similarly to their in-class tests and of comparable difficulty.  The purpose was two-fold: for them to identify areas of misunderstanding or gaps in their knowledge, and for me to see what students struggled with from my lectures.  The last question of the quiz was a free-response, "What did you find interesting or confusing?"  I reviewed their progress the night before class, and their answers guided the slides I'd make for the class period.

I set aside the first few minutes of the class period for students to collaborate in completing the preclass quiz.  After that, I spent ~10 minutes going over areas of misunderstanding based on the preclass quiz performance and free responses.  The bulk of the class period was devoted to students working in self-selected small groups on ~7 ungraded practice problems designed to help them integrate and apply the concepts from the lectures.  The teaching assistants (TAs) and I would walk around the room and engage the students to see what problems they were stuck on.  At the end of the class period, I'd go over the answers, and then spend ~5-10 minutes presenting a practical or medical application of the concepts related to the lecture, or recent primary research building from it (the remaining 10-20% of "information content"), often building on what students said they found interesting.

Besides that, the rest of the class format was as before-- the students had a weekly problem set to complete, weekly laboratory sections, a few other online practice problems, and three open-book, in-class tests.  The class period activities were also recorded and made available to any students who couldn't make it due to illness, travel, etc. (again, just like before).


Most of the class's students who answered the informal mid-semester and end-semester course surveys indicated they were pleased (and often pleasantly surprised) with how well the flipped class format worked.  Students did not have to initially assimilate the primary class material at 8:30am as previously, but could work at whatever time of day was optimal for them.  They could also easily pause at any time, or immediately replay anything they missed.  Most reported that the in-class reinforcements were helpful for them to really understand the material and to perform better on assessments (see figures below).  Most appreciated the "reactive" and "interactive" nature of the class, where the class period content responded directly to their areas of interest or confusion.  Some students told me that the format forced them to study as they went along, such that they noticed they barely had to study right before the tests since they'd already done so much along the way.  A handful said they liked the online discussion forums, and found the elaborations by either Duke or online students on some of the material helpful.

Important note: these informal evaluations were done online, so they were not biased in being completed by just the subset of students who regularly came to class. 

Fig 1: Response to question: Did you find that the in-class presentations and exercises were useful and reinforced concepts and approaches from the recorded lectures?  (For scale, 1 was never chosen, and 2 was chosen twice)

Fig 2: Response to question: Did you find that the in-class activities provided you with additional practice that improved your performance in assessments (e.g., problem sets and tests)? (For scale, 1 was chosen twice, by the same people who put "2" in previous question.)
Even some of the more critical students appreciated the outcome of the class format-- one person said snappishly, "Of course we did better on the tests-- we had to work harder."  I'll just say "true."

In terms of my feelings, I perceived that the students really understood and interpreted the material better than in either previous course iteration.  I added more challenging concepts than I had covered in previous years, and I was impressed that the students seemed to really understand them.  I was also impressed that some areas of common misunderstanding from previous years just didn't seem to come up.  I should emphasize-- these are all my perceptions, not quantitatively confirmed conclusions with statistical significance.  But, the class did better on the first test than I've ever seen in my career teaching.

Finally, it was just a lot more fun for me.  A traditional class is like a performance-- you get up on stage and do it, with the only feedback being quizzical looks, laughs, or the very rare hand up for a question.  This was truly interactive-- I started the class period by giving students what they asked for, and then spent much of the rest of the period in one-on-one or small group discussions about the material.  I like teaching for the interactive nature of meeting people and watching them get excited, interested, or at the very least "more knowledgeable" about a subject, and this format really allowed me to experience those pieces.  It also really, really helped me learn what concepts students understood and what concepts need new forms of presentation for future class iterations.  As my colleague Dan Gauthier in Physics said about his flipped class, "I've never felt more attuned to my students' understanding."


Some of the students were unhappy with the added workload of having to watch ~1.5 hours of video each week.  I personally don't find that unreasonable given the small amount of other outside-class work that was expected of them-- I am confident many humanities courses assign more than that in readings each week.  Here was perhaps the most critical response I received on the format: "While the flipped class format benefits many of my fellow students, this format hinders my learning. In a normal class, I would go to lecture and take notes and such, and then do whatever work was required of me, but no more; I normally take a minimalistic learning approach, does as little as possible to succeed. As such, I do not normally watch the online lectures because I find that doing so is more time consuming than just glancing over the lecture slides to find the information needed".  Fortunately, this was a minority opinion (and, in my opinion, one unbefitting of a student at Duke University or any college/ university).

One negative I did anticipate was that a fraction of the students fell far behind (especially between test 1 and test 2) and did not really catch up in a timely manner to do well on the later tests.  If one fails to watch the lecture videos before class, there really is no point to coming to class since the student would be just lost on the material.  At that point, it can become self-reinforcing-- a student falls behind, and the work needed to "catch up" is greater than in a traditional class, possibly making them more likely to further procrastinate and fall further behind.

A possibly related negative I did NOT anticipate was a drop in class attendance... attendance probably was consistently at or below 50% after the third week.  My previous traditional classes were all recorded, so it was not uncommon for people to skip and presumably just watch the recording of my video.  We often had 50-70% attendance in the traditional classes, unlike the <50% with the flipped class.  I was surprised that the fraction of the class that now didn't come to class (one day after a late basketball game dipping to ~20%), even though it seemed like there was "more" reason to come than before since the in-class piece was not basic content just-as-easily assimilated by video.  A few students I talked with cited two reasons 1) the increase in overall amount of outside-class work (the fact they already had to spend 90+ minutes a week on the material made some students less likely to spend even more by coming to class), and relatedly, 2) some simply felt confident with the material from the video, and would thus skip the reinforcement and application in-class.  The negative here is that, for a fraction of these skippers, this confidence was unfounded-- I warned students that the test 2 material was more challenging than test 1, and despite my repeated warnings, there was a noticeable drop in attendance after the high performance on test 1 (and proving my point, a 10-point drop in average grade).

The one class format element that I think "flopped" was the few minutes at the beginning of class for completing the preclass quiz in the classroom-- almost no students ever did this, and most people just arrived 5 minutes late for class to skip that piece.


The biggest surprise to me was the absence of correlation between class attendance and grade.  The three test grades were very highly correlated with each other despite the differences in mean, but there was no clear association between which students came to class and student test performance.  However, upon further consideration, this result ties in with what I mentioned above.  Some of the students who felt they had a good handle on the material just from the primary content delivery in the video recordings didn't come to class (or didn't come regularly).  I think the absence of correlation suggests that many of them were right-- they did understand the material well enough from the video lecture that in-class reinforcement may not have had a noticeable effect on their assessment performance.  Similarly, some of the people who came to class were those who struggled with the material the most-- their coming was a good thing, since they had more practice and better performance as a result.  Some of the students with whom I interacted in the class had clearly watched the video lectures, and were still struggling to assimilate the content.  As such, my hypothesis is that the format, even with its sparse attendance, served to homogenize student performance-- basically "almost everyone" in the class did well in assessments, either due to intrinsic high ability (or prior background) but perhaps little or no extra practice in class, or potentially lower ability (or less background) but with a lot of extra practice in class.

(Obviously, many students who came to class also had high ability and good prior background, too-- I certainly don't mean to imply that only struggling students attended class.  Some came regularly just because they were very interested in the material, too.  Similarly, I know many students who performed poorly yet rarely or never came to class.)

I recall our forward-thinking university provost chatting about this a year ago at an informal dinner-- that with flipped classes, "everyone might do well" in the class.  Some of the literature also says this format "levels the playing field" of classes.  The grade improvement opens the much stickier question of the purpose of grades (which I shall pose but not pursue)-- are we truly grading to assess how well students can understand and apply the material (wherein it's fine for the whole class to have A's), or are we also trying to distinguish students with higher intrinsic ability from others?  Personally, I have no intention of "toughening" my class solely to "counter" high student performance on what I think are perfectly fair and reasonably challenging tests-- doing so directly violates my philosophy of the purpose of teaching and needlessly alienates students.

(Quick note: my question about "all A's" was rhetorical-- not all my class's students received A's... there were many who did not achieve high grades.)

My other surprise was the general absence of engagement of the on-campus students in the online Discussion Forums associated with the online class.  Many of my Duke students said they looked in the forums, but even though I offered the Duke students extra credit for posting in the forums, 80% of the Duke students never posted in the forums, and most of those who did post made 5 or fewer short comments or questions.  A handful really engaged with others in the forums, but <10 students total.


Despite the absence of a significant association of test grade with attendance, I know at least some students skipped class to the detriment of their performance.  This was partially my fault-- in the canonical flipped class format, there are in-class assessments with stable student groups that are assigned by the instructor, thus forcing students to attend and be accountable to their classmates and themselves.  I deviated from this format because I anticipated (correctly, I think) that a lot of students would miss classes each day due to illness, family travel, student activities, and sports.  I did not want to be in the position of having to judge daily whether particular absences were "excused" or not for tens of students.

What I'm considering next year is to have one group-activity question graded each class period (or once per week).  If there are 14 such activities, perhaps I'd allow them to keep their highest 6 grades of the 14 only.  Hence, one has to show up to the class at least that small fraction of the time to get the grade.  However, I would then not accept any excuses-- any sports- or activity-travel, or any illnesses, have to come out of the "8 dropped grades."  This may seem like an exceptionally low bar, but my goal is to get students to come to some classes such that they have the opportunity to assess for themselves whether coming to class is helping their understanding and test performance.  If the answer is no, then so be it.  I just don't want them to "assume" that coming to class is unnecessary without having experience with trying it.  I'm still deciding if this is a change I'll employ or not...


Although requiring a TON of upfront work, this experience has definitely changed my view of teaching.  As I commented in a prior blog post, I feel now that, as faculty become less "lecturers" and more "facilitators" in the classroom, they work with the humanity of students rather than treating students as consumers of prepackaged products.  Teaching this class was very, very enjoyable.  Further, these students are interesting and talented people, many of whom are spending tens of thousands of dollars to be at an elite university, and are receiving some classes in a format of not much greater value than what could be found online.  My statement is not to suggest that the total college experience is what happens in the classroom (clearly untrue), nor that the students themselves are incapable of obtaining more from traditional lecture classes by taking the initiative to engage professors directly.  However, my goal this semester was to add as much value as possible to the in-class experience by making it visibly dramatically more than what is available freely online-- far more opportunities for interaction, extensive reinforcement, "going further" with the material, etc. (in addition to the laboratory sections, of course).  I hope that the vast majority of the students viewed it as successful and beneficial... I guess I'll find out, to some extent, when I get the formal evaluations back!

Thoughts and feedback welcome!

Article in Duke Today, February 11, 2013
Editorial in Duke Chronicle, February 25, 2013
Article in Duke Chronicle, March 5, 2013
Article in Duke Towerview magazine, March 27, 2013
Student project on flipped class discussing this one in particular, April, 2013

Thursday, May 2, 2013

Teaching, and the educational role of the university in the 21st century

When I went to college in 1988-1992, my experience was similar to how many science-oriented students experience college today.  I went to classes, wherein a professor stood in front of the room and told us facts and perspectives about their subject.  In most biology courses, my role was clear-- listen, take notes, and occasionally (genetics, especially), learn approaches to solving particular kinds of problems.  Students asked questions occasionally, but probably over half the class never asked the professor a single question all semester, and no one had the audacity to ask the professor questions after every class.  They may have asked the teaching assistant a few questions, but these TAs were often only marginally more familiar with the material than the students they oversaw.  Honestly, much (though not all) of this so-called learning involved rote, short-term memorization of facts.  If we students got something wrong on an assessment, it was our fault for not having understood the material from its single presentation (irrespective of how well it was presented or not reinforced).

The world is different today.  There are very few facts in the world that I cannot find in mere seconds using my computer or smartphone.  Virtually everything known is accessible to the world on the internet, though there are also a lot of misunderstandings on the internet portrayed as facts.

As I see it, universities have two potential educational roles in this new era (I'm not addressing research roles here).  The first is a service role to the community.  Universities have always been the storehouses of knowledge and understanding, but it's both arrogant and short-sighted for universities to perceive this role as exclusive to their students or in-field colleagues, particularly given the amount of public funding that they receive.  University faculty can serve their communities and the world by providing or "authenticating" facts, evidence, and diverse perspectives in their study areas through the internet and other media.  A university's role in dissemination is not symbolic of arrogance-- university faculty are regularly consulted by the media to interpret new findings or perspectives in their areas given their expertise and training, and this role is merely to be more pro-active.  The public's interest for such reliable sources is there-- if one is diagnosed with cancer, would one prefer to just Google "cancer" and look up information on whatever site comes up (perhaps "")?  Or would one prefer to get information from the National Cancer Institute of the NIH?  Presumably the latter, and this example illustrates the public's perceived value of "authenticated" information.  Similarly, if I wanted to learn about genetics or psychology or economics or art history more broadly, I'd love to take a free online course (or "MOOC") from a practitioner who has an advanced degree in the area and was hired by a university as an expert in that area.  Neither of these features guarantee that the information will be presented coherently or that the presenter won't be wrong or that there aren't better resources, but it's a safer way to start the road to learning than a random internet search.  MOOCs are not the only means of public dissemination, but they are a good one that is both effective and engaging.  Freely providing knowledge is not only an important gesture by universities to their communities, but arguably an obligation, and it can also facilitate learning of their on-campus students (see below).

The second role I discuss is the university's primary one-- to help their enrolled students learn.  This learning can no longer be rote, short-term memorization of facts-- such "learning" trivializes the role of the university relative to the internet.  Instead, we need to engage with students directly, and in a manner that far exceeds what is possible through the internet or free online MOOCs.  Our courses need to go beyond fact dissemination-- we need to engage students both individually and in groups to assess how well they are interpreting and applying the concepts we're presenting them.  The flipped class is one means of achieving this goal-- students get the primary content in some way outside the class period, and their understanding is assessed.  This assessment step is critical-- students learn what elements of the material they didn't correctly interpret or apply the first time, and faculty receive feedback to correct frequent student misinterpretations and misapplications in their presentations.  The faculty then spend the class period clarifying areas of confusion directly in response to the student feedback, and then reinforcing true understanding of the material with new problems, applications, and engaging discussions.  The format forces faculty and students to interact bidirectionally in the learning process, and this bidirectionality has obvious benefits both to student understanding and faculty teaching strategies.  It's also personally satisfying for both parties, as faculty become less "lecturers" and more "facilitators" in the classroom, they work with the humanity of students rather than treating students as consumers of prepackaged products.  Relatedly, I've become a firm believer in "open-book" assessments, too, for two reasons-- 1) the world is essentially "open-book" so assessing in a situation where simple facts cannot be quickly checked is (usually) unrealistic, and 2) it forces the faculty member to produce questions that are not merely regurgitation of facts presented in the course, and thus better assess student "understanding" on a higher level.

None of what I've said above is novel or revolutionary.  However, many faculty and students are too comfortable with standard lecture formats for our classes (especially in the sciences and social sciences-- less-so in the humanities and interpretive social sciences) and are resistant to changing the roles, particularly given the upfront work involved.  While our time is limited, the goal of all universities (both students and faculty) should be to promote the best learning possible, so isn't this worth the investment?  Similarly, a lot of people view MOOCs as a threat to our universities-- we're giving away for free what students paid thousands of dollars to receive.  Some have said that MOOCs are a means for the "elite" universities to secure their position and displace others by disseminating content possibly (and often incorrectly) perceived  to the broader public as "better" than what a good state or liberal arts school may provide.  I argue that, if colleges or universities fail to provide opportunities for MUCH more mentoring and learning in their on-campus classes than what happens in topically equivalent MOOCs, they're wasting their students' time and money.  MOOCs can educate the public and can be a tool to enhance or supplement available on-campus classes, but they are no replacement for an on-campus undergraduate education should be.

Times have changed, and forward-thinking universities are beginning to change accordingly.  It's up to universities and their faculty to keep up with these changing times.  If universities don't change quickly, prospective students will soon figure out which schools are least likely to provide a return on their investments...