Visual Abstract
Abstract
Background In graduate medical education (GME), there are many barriers to achieving a personalized learning process with standardized learning outcomes. One way to support this is through mobile-friendly digital blackboard videos. We sought to measure the effect of a mobile-friendly video curriculum on resident satisfaction, knowledge, and clinical skills during a nephrology rotation.
Methods This was a prospective, controlled, nonrandomized trial. The control group consisted of internal medicine residents who completed our inpatient nephrology consult rotation as usual. The classroom group had the same clinical experience, but also had access to a library (Nephrology Immersion Classroom) of mobile-friendly, nephrology-themed, digital blackboard videos. In a postrotation assessment, we measured resident satisfaction, clinical knowledge using 15 multiple-choice questions, and nephrology-specific clinical skills.
Results Of the residents in the classroom group, 77% enrolled in the online classroom, and the majority reported using the classroom occasionally or frequently. A majority found it very easy to use (86%) and strongly recommended having similar videos for other rotations (77%). We observed improved report of rotation-specific clinical skills, but no difference in short-term knowledge between the two study groups.
Conclusions A mobile-friendly, digital video curriculum for internal medicine residents on an inpatient consult rotation was well utilized, highly rated, and associated with improved nephrology-specific clinical skills. Continued evaluation and incremental improvement of such resources could enhance implementation of GME core curricula.
Introduction
“It’s time to change the way we educate doctors” (1). Historically, scheduled learning activities for internal medicine (IM) residents have consisted of large group lectures held during a noon conference or academic half day. The reliance on lectures has persisted due to the ease of scheduling, programmatic inertia, and faculty unease with implementing new teaching activities. However, varied schedules, competing clinical demands, and postcall duty-hour limits frequently interfere with conference attendance. Even with the provision of food in one example, resident noon conferences were attended, on average, by only 35% of the residents over 2 years (2). Lecture attendees frequently cycle between episodes of attention and inattention (3), disagree with the faculty on the ideal content (4,5), and their learning outcomes may not improve, especially when compared with active learning strategies (6,7). Taken together, resident learning activities should be reformed to better meet trainees’ needs, accommodate their fluid schedules, maintain their attention, and promote knowledge retention.
As medical education reformers have called for standardized learning outcomes along with an individualized learning process (8), digital videos have emerged as a promising and popular learning platform with the current generation of learners (9⇓–11). One style of digital videos is called the digital blackboard video, or pencast. These short, visually stimulating, and dynamic tutorials can help explain difficult concepts, organize existing knowledge, and teach clinical reasoning (9,10). At Duke University School of Medicine, we successfully implemented a digital blackboard curriculum for medical students, for whom the addition of a library of pencast videos to usual course materials was associated with a significant improvement in performance on the physiology exam (12). Therefore, in light of questionable efficacy, inattention, and poor attendance at large group lectures, we sought to create and study the implementation of a mobile-friendly pencast video curriculum for IM residents.
Within IM training programs, residents perceive the care of patients with kidney disease as being too complicated or difficult, and a quarter of those surveyed would have considered nephrology if it had been “taught well” (13). A recent empirical study of patient complexity found that nephrologists do indeed care for the most complex patients, according to a variety of complexity markers (14). Because of the recent and substantial decline in IM residents applying to nephrology fellowships (15), improving the quality of our educational program for IM residents is paramount. Therefore, we sought to understand the feasibility and effect of the Nephrology Immersion Classroom, a mobile-friendly library of digital pencast videos, for IM residents on a nephrology consult rotation.
Materials and Methods
This was a prospective, nonrandomized study that included second- and third-year IM residents completing the nephrology consult rotation at either Duke University Hospital or the Durham Veterans Affairs Medical Center. Residents were paired with a consult team comprised of an attending and fellow. Rotation length varied from 1 to 4 weeks, depending on residency program scheduling needs. During each week on the consult rotation, residents attend weekly educational conferences offered by the nephrology division, including journal club and renal grand rounds.
The control group comprised all IM residents who completed the consult rotation between July 1, 2017 and June 30, 2018. No changes were made to the structure, organization, or composition of the nephrology consult rotation during this time. The classroom group consisted of all IM residents who completed the consult rotation between July 1, 2018 and June 30, 2019. The classroom group differed only in having private access to the Nephrology Immersion Classroom: the mobile-friendly library of 32 blackboard-style, narrated videos and associated review articles. At our institution, the nephrology rotation is not required for all residents, and a majority of residents do not select it as a top preference.
The videos were authored by a single faculty member (J.K.R.) and constructed using digital-art software (Autodesk Sketchbook), screen-capture software (Screencast-o-Matic), USB tablet and pen (Wacom Bamboo Tablet), and a USB microphone (Blue Snowball iCE Condenser Microphone). The content for the entire curriculum was chosen after surveying IM residents (N=5) and nephrology faculty and fellows (N=21), who were asked to prioritize clinical nephrology topics taken from the American Board of Internal Medicine Certification Blueprint (16). This process resulted in a list of core topics chosen for the video curriculum (Table 1). We reorganized the broad core topics into subtopics for a series of 6- to 15-minute videos. We restricted the videos to this length based on the available empirical evidence for maximizing student engagement with online videos (17,18). Each major topic was covered by one to six videos, resulting in a total of 32 videos. For the classroom cohort, all residents were given invitations to join the Nephrology Immersion Classroom on the first day of the rotation. The classroom was hosted using Google Classroom, a learning-management system that hosted private links to all of the videos in a mobile-friendly format with search capabilities. The classroom also included access to high-yield review articles related to some of the core topics. All videos were hosted on YouTube using unlisted links, so the videos were only accessible through the classroom. Table 1 includes links to each video by clicking the video title.
Clinical nephrology topics and corresponding videos in the Nephrology Immersion Classroom
Figure 1 shows a representative screenshot of the Nephrology Immersion Classroom. To access Google Classroom, the participants needed a Google account, and if they did not have one, we asked them to create one for the purposes of the study. Classroom participants were also encouraged to download the free Google Classroom smartphone application. Classroom cohort residents were allowed continued access to Google Classroom for the duration of their residency training.
Nephrology immersion classroom visualization. Links to the mobile-friendly pencast videos and high-yield review articles are bundled together, arranged by topic, and searchable.
To assess perceptions of the Nephrology Immersion Classroom, residents in the classroom cohort were given a postrotation survey with questions related to the use and quality of the online classroom, including frequency of use, ease of use, when and where it was used, and recommendations for other rotations. We also included a section for comments in response to the following questions “what went well with the online nephrology classroom?” and “what are some things that would make the classroom better meet your needs?”
To assess self-reported skills, we collected responses from the rotation evaluation form administered by the IM residency training program. This evaluation form asks residents to rate their ability (on a five-point Likert scale) to perform certain rotation-specific skills. All residents completing the rotation in both cohorts were invited to complete the rotation evaluation at the direction of the residency training program.
To assess nephrology knowledge, residents in both cohorts were invited to participate in the knowledge pre- and post-test. Residents were invited to take the pretest on the first day of the nephrology rotation and complete the post-test within 1–4 weeks after completing the rotation. Nephrology medical knowledge was assessed using a set of 15 multiple-choice questions (MCQs) in the pretest, and a separate set of 15 MCQs in the post-test. These case-based questions were modified from validated examination questions used in the Kidney Self-Assessment Program. MCQs were chosen and directly linked to core content, as reflected in the video curriculum and the current practice of nephrology. A $5 Starbucks gift card was given to the participant as compensation for each completed survey. The gift card incentive was implemented after the first academic quarter of the control year, due to initially low response rates. Assuming an α of 0.05 and SD of 20, to detect a mean difference of 15% on the knowledge assessment with 80% power, we anticipated a sample size of 28 residents would be needed in each group. In addition, we collected and examined performance on nephrology questions from the IM In-Training Examination, a formative knowledge assessment completed by IM residents. For categorical residents, the exam was taken in the second year of training. For combined program residents (medicine-pediatrics and medicine-psychiatry residents), we used scores taken from the third year of training. We also examined responses related to acquisition of rotation-specific skills based on the anonymous rotation evaluation administered and collected by the IM training program. This study was reviewed and approved by the Duke University Institutional Review Board (project number 00087609).
We conducted all quantitative analyses in STATA version 14.0 (Statacorp, College Station, TX). Normally distributed data are reported as mean±SD; non-normally distributed data are reported as median and interquartile range. The analysis of knowledge assessment was restricted to residents who completed both pre- and post-tests. For these analyses, we used paired t tests to compare the means within groups, and unpaired t tests to compare means between the groups. Responses on the rotation evaluation regarding self-reported skills were anonymous, and we used unpaired t tests to compare the mean score between the groups for each item. Two-tailed P values <5% were considered statistically significant.
Results
During the control year, 39 of 80 eligible IM residents completed the nephrology consult rotation. All received invitations to take pre- and post-tests and 13 completed both (33% response rate). The majority of pretests were completed on the first day of the rotation, and post-tests were completed a median of 26 days (range, 1–117 days) after the end of the rotation. During the classroom year, 41 of 86 eligible IM residents were assigned to the nephrology consult service. Two residents were excluded from the study because they also participated in the control group. Out of the final cohort of 39 classroom residents, 24 (61% response rate) completed both pre- and post-tests. The majority of pretests were completed on the first day of the rotation, and post-tests were completed a median of 5 days (range, 1–84 days) after the end of rotation. A greater proportion of respondents in the classroom cohort were postgraduate year 2 compared with the control year (64% versus 54%, P=0.42). In terms of when participating residents completed the nephrology rotation, survey respondents were equally distributed throughout the academic calendar, with the exception of the first quarter of the control year, which had fewer respondents compared with the other academic quarters.
Knowledge Assessment
Table 2 shows performance on the pre- and postrotation knowledge assessments for residents who successfully completed both tests (33% response rate in the control cohort; 61% response rate in the classroom cohort). In both cohorts, we observed significantly higher exam scores on post-tests compared with pretests, but no significant differences in the mean post-test exam scores or change in scores between the control and classroom groups. The results were no different when restricting the analysis to residents in the classroom cohort who enrolled in the Google Classroom, or to those who reported more than occasional use of the classroom (data not shown).
Performance on pre and postrotation knowledge assessments
Table 3 shows performance for both groups on nephrology questions from the IM In-Training Examination. Mean scores (percent correct) were similar between the control group and the classroom group. We also calculated mean scores among classroom group residents who reported using the digital video curriculum (any resident who reported using it occasionally, frequently, or very frequently, including one resident who did not complete a post-test survey, but notified the authors that they used the curriculum very frequently). In this restricted analysis, classroom users had higher mean scores on the In-Training Examination nephrology questions compared with the control group. This difference, however, was not statistically significant (P<0.26).
Resident performance for nephrology questions on the IM In-Training Examination
Nephrology-Specific Clinical Skills
Using the end-of-rotation evaluation administered by the IM training program, we compared resident self-assessment of their ability to perform particular skills on a five-point scale (1, never; 2, infrequently; 3, sometimes; 4, most of the time; 5, always). Table 4 shows each rotation-specific skill and mean (SD) response for each cohort of residents. On average, residents in the classroom group (97% response rate) reported higher levels of readiness with all of the skills compared with residents from the control group (74% response rate). The difference in the following two skills were statistically greater in the classroom group: prioritizing a workup for acute kidney failure and recognizing the indications for a kidney biopsy.
Resident self-assessment as determined by ability to perform rotation-specific skill
Feasibility of the Nephrology Immersion Classroom
A total of 30 residents (77%) electively joined the online classroom, and 66% completed the postrotation survey. Table 5 shows usability metrics related to the web-based Nephrology Immersion Classroom. The majority of respondents reported using the classroom occasionally or frequently. Classroom users watched videos mostly at home or at work during down time. A majority of users (86%) found the classroom very easy to use, and most users (77%) recommended having video tutorials available for other clinical rotations. Table 6 lists representative comments found on the postrotation survey written in response to prompts about what went well and could have been improved regarding the nephrology classroom and rotation.
Usability and perceptions of the Nephrology Immersion Classroom
Narrative comments about the Nephrology Immersion Classroom
Discussion
In this study, we examined the usability and effects of an online video curriculum on IM residents’ experience during an inpatient nephrology consult rotation. Overall, enthusiasm and interest in the Nephrology Immersion Classroom was high. Despite the short rotation length and elective nature of our curriculum, the majority of IM residents enrolled in the online classroom. These residents reported modest to heavy use of the video curriculum, typically at home or at work during down time. Although we observed no significant improvement on MCQs from pre- to postrotation, the video curriculum was rated as very easy to use, and the majority of users strongly recommended having similar resources for other clinical rotations. Despite having a greater proportion of junior (postgraduate year 2) residents in the classroom cohort, we observed higher self-reported readiness to perform nephrology-specific skills, namely prioritizing a workup for ARF and recognizing the indications for a kidney biopsy. Residents in the classroom cohort reported high levels of satisfaction with the video curriculum and it appeared to have improved self-reported clinical readiness.
In contrast, reported use of the classroom was variable and our residents identified some opportunities for improvement. In the narrative comments, residents made statements suggesting there was not enough time to watch some of the longer videos (two were >15 minutes). Also, residents commented on the need for multiple modalities. For example, one resident recommended a transcript of the video, whereas others recommended a single summary sheet with high-yield algorithms or flowcharts. These comments suggest that additions like these may contribute to greater learning at the point of patient care. Another proposed opportunity for improving the classroom is the addition of knowledge self-assessments, which could help the learner check knowledge retention after watching some of the videos.
Historically, scheduled learning activities in graduate medical education (GME) have relied heavily on the delivery of large-group, didactic lectures. With evolving medical education research and concurrent advances in technology, GME programs are now embracing learning activities that better personalize the learning process, foster self-directed learning behaviors, and reserve the classroom for active learning. In this paradigm, the Nephrology Immersion Classroom represents successful implementation of a mobile-friendly video platform for self-directed learning while on a clinical rotation. The advantage of this is multifold. First, the current generation of learners are comfortable with using digital learning tools in place of traditional lectures (9⇓–11). Second, short concept videos have been empirically associated with high levels of engagement, and mobile-friendly videos allow high level of user control (i.e., location, time, and viewing speed) (17,18). Third, access to a library of short concept videos allows customization of the learning process: users can self-select topics based on perceived need, interest, or relevance to their patients’ problems. Large-group, lecture-based curricula suffer from a “one-size-fits-all” approach and “just-in-case” learning, whereas a pencast video library facilitates a personalized approach with “just-in-time” learning.
Although the residents in our study reported high levels of satisfaction and modest use of the pencast curriculum, questions remain about how such a curriculum should best be implemented. The purpose of this study was to measure satisfaction and short-term knowledge from simply having elective access to a library of clinically focused pencast videos after starting an inpatient consult rotation. We observed no difference in short-term clinical nephrology knowledge in our classroom cohort compared with a control cohort of residents. Given the breadth and depth of the curriculum (along with competing clinical demands), GME trainees likely need more time to fully engage with such a curriculum. If implemented longitudinally, however, it is unknown whether a pencast curriculum would enhance long-term knowledge outcomes over the duration of GME training. Also, it matters how such a curriculum is structured within global GME learning activities. Some GME programs have reported success using short videos to help achieve procedural readiness (19,20), and others have successfully implemented flipped classroom activities using online videos (21,22). In the context of flipped classroom learning activities, a video pencast curriculum could easily serve as the preparatory material to be viewed before case-based active learning sessions. The effects of flipped classroom teaching on learning outcomes in GME has been mixed, despite high levels of resident satisfaction (21⇓⇓–24). Recent work by Graham et al. (25) describes successful implementation and improved long-term knowledge outcomes with a flipped classroom for IM residents on an ambulatory rotation. In an accompanying letter, Wortzman and Saddawi-Konefka (26) suggest that the success of this flipped classroom was due to IM residents having dedicated time in their weekly schedule to do the preparatory work, because prior studies in GME showed poor compliance with the prework (27,28). When surveyed, 66% of IM residents thought that the flipped classroom is not feasible on inpatient rotations, mainly due to the absence of protected time (25). Therefore, assigning pencast videos as preparatory work for flipped classroom activities probably works best during ambulatory rotations, where provision of study time is feasible. In summary, the Nephrology Immersion Classroom serves as a potential prototype for GME learning in the 21st century: it could serve the preparatory modules for flipped classroom experiences or as a structured repository of short, mobile-friendly videos for self-directed, practice-based learning and improvement.
Our study has a number of limitations. First, self-reporting in the postrotation survey is subject to recall bias. Second, we observed incomplete response rates for both pre- and post-test assessments in both cohorts, and the knowledge outcome analysis was restricted to paired samples. Despite an incentive to boost survey completion, these low response rates underpowered the analysis and could have also introduced a nonresponse bias to the other survey items. It is possible that digital videos improved resident knowledge, but the 15-item examination was unable to detect a difference between the two groups. Although major elements of the nephrology rotation were unchanged between the two rotation years (other than addition of the classroom), we were unable to control for potential residual confounding given the nonrandomized nature of the study. There is also at least one resident who reportedly watched all pencast videos, but never submitted the post-test knowledge assessment and survey, further contributing to nonresponse bias.
Hosting the Nephrology Immersion Classroom, a mobile-friendly pencast video curriculum, for IM residents on an inpatient rotation was associated with high levels of resident satisfaction and self-report clinical skill development. Adding this video curriculum did not improve short-term medical knowledge, as measured by case-based MCQs, when compared with the usual nephrology consult rotation. The classroom was rated as very easy to use and the majority of residents strongly recommend having similar video tutorials for other rotations. To improve the classroom, we will add images of diagnostic algorithms, flowcharts, and embedded quizzes for self-assessment. We found that a mobile-friendly video curriculum is well utilized, highly rated, and open to broad application elsewhere in GME curriculum.
Disclosures
All authors have nothing to disclose.
Funding
This study was supported by the American Society of Nephrology William and Sandra Bennett Clinical Scholar Award.
Acknowledgments
This research was previously presented in poster form at the American Society of Nephrology Kidney Week 2019 Scientific Session, Washington, DC.
Author Contributions
J. Roberts was responsible for resources, software, validation, and visualization; J. Roberts and M. Wolf wrote the original draft, provided supervision, and were responsible for data curation, formal analysis, funding acquisition, investigation, and methodology; and all authors conceptualized the study, were responsible for project administration, and reviewed and edited the manuscript.
- Received April 8, 2020.
- Accepted August 19, 2020.
- Copyright © 2020 by the American Society of Nephrology
References
Jump to section
More in this TOC Section
Original Investigations
Cited By...
- No citing articles found.