Over the past several years Wentworth has invested in access to the data collected by our LMS, Blackboard. In addition, we have participated in a Colleges of the Fenway joint evaluation of the impact of LMS use on our campuses (B.L.E.S.). Based on some of the results of the B.L.E.S. project and other internal projects, we are now beginning to mine the LMS data to examine how the tools are being implemented and to see what impact they may have on student success. We are NOT trying to determine if individual instructors are using Blackboard, trying to judge the value of their use of Blackboard or create a mandate to use Blackboard. Instead, through this analysis, we are initially identifying patterns of LMS tool use that are tied to high impact practices that Learning Innovation and Technology wants to propagate. Additionally, as instructors engage with LIT’s instructional designers in developing or revising courses, they can discuss specific tools and pedagogical practices that B.L.E.S or other research show can positively impact student learning here at Wentworth.
Keep in mind the that the patterns identified in the B.L.E.S. are associations and can’t directly be attributed to specific LMS tool use. The tool use that is associated with higher student achievement may instead be associated with classroom or pedagogical practice. So, while we have been looking at LMS tool usage, we are not strictly looking to change/increase LMS usage but we are looking to improve overall teaching and learning. The data we have comes from the LMS, so the analysis will be LMS centric. B.L.E.S. is our first pass at analyzing teaching patterns at Wentworth, it is not the endpoint.
As an example, B.L.E.S. found a positive association between using the Blackboard groups and assignments tools and student grades in the analysis of Mechanical Engineering courses. The data suggest that the use of these tools increases student grades, but is it the use of the tools or something else? We know many courses in Mechanical Engineering (and other disciplines) have students engage in group projects. It’s likely in this case that the associations we are seeing are not the tools per se but the group work. Because Mechanical Engineering faculty have students work in groups and many choose to use Blackboard, they activate the groups and assignments tools in their courses.
In other cases, the associations found suggest ways to improve instruction for all students. We see that the number of files posted in a course on Blackboard has a negative impact on our poorer performing students. Who are these students? This group includes, among others, students for whom English is a second language, students with learning disabilities and students who lack academic skills because they are first-generation college students. Even if an instructor explains in class what file is needed for an assignment, or which files are optional readings, this group of students might have problems later if the information is not repeated in context with the files. So for this group of students repeating the information with the file in Blackboard can decrease anxiety around identifying the correct file due to language challenges. Not only does this benefit students with challenges but it benefits all.
In addition to looking at the association of Blackboard tool usage and student performance, we have also been exploring overall patterns of LMS use. Are there combinations of tools that are common or unique to Wentworth? To explore this question, we’ve looked at all course sites created in Blackboard from 2014 – 2018 and the use of approximately 20 Blackboard tools. Using Latent Class Analysis to identify naturally occurring groupings within the data, we found 6 different archetypes in use at Wentworth during this time, Non-use, Content Repository, Communication, Evaluative, Groups, and Holistic. Each Archetype is characterized by a specific group of tools being used. When compared to other analyses of Blackboard use, we find that the Groups archetype is unique to Wentworth. This is not surprising given the frequent group work that occurs in our courses. (Click on the image below to see full-size version).
Looking at the archetypes analysis it appears that we have a lot of courses using Blackboard to store files. Combining this with the B.L.E.S. results suggest that we may want to provide training to ensure that students are provided with helpful scaffolding.
It’s important to keep in mind, the analytics projects we are embarking on are not intended to identify individual instructor practices but to look for patterns in the data that can enlighten our teaching practices at Wentworth. We are not looking to prescribe the use of specific LMS tools but to encourage practices, that may include supportive LMS usage, that help our students reach the outcomes we have set for them in our courses. And when we suggest a tool, we want to provide evidence that supports its best use.