Designing for Learning Analytics

Introduction

Learning analytics is the practice of using data about learners and their context for the purpose of understanding and optimising learning, and the environment in which it occurs (Siemens et al., 2011). The primary aim of learning analytics is to improve student learning outcomes through the learning process and the specific environment that the learning is taking place in.  

The data we analyse is often captured through the interactions of the student within the learning design. The value of this approach is that the student data is captured in an authentic learning setting as the learning happens (Siemens, 2013). In a broad sense, this provides educators with a new form of feedback. The feedback it provides is vast and varied, some examples include student engagement, progression or performance. Furthermore, this data can also be used to evaluate the effectiveness of our learning designs. As students interact with our learning designs the data captured can provide pedagogical recommendations where learning behaviour differs from, or aligns with, our pedagogical intention (Lockyer & Dawson, 2011).  

However, we may find that the data captured by the learning management system, or through the online activities that students engage in, do not provide the insights we desire. Therefore, we must consider the design of learning analytics within our learning designs in order to capture the most relevant and useful data.  

A perspective of learning analytics in learning designs

One approach to design for learning analytics is to adopt an evidence-based framework. The Analytics Layers for Learning Design (AL4LD) framework developed by Hernández-Leo et al. (2018) provides a structured and organised perspective for embedding learning analytics into a learning design. It enables educators to use common language and outlines a set of tools and examples to illustrate each layer, and provides clear implications for practice.  

The AL4LD framework has three layers:  

1. Community Analytics: The metrics and patterns of design activity within a community of teachers and related stakeholders.  
2. Design Analytics: The metrics of design decisions and aspects that characterise learning designs prior to their delivery.  
3. Learning Analytics: The data collected while the learner experiences a learning design. This includes engagement, progression and achievement aligned with design intent.  

This article will focus on the Learning Analytics layer as it is most relevant for an instructor or teacher who is exploring how to embed learning analytics into a learning design.  

The Learning Analytics Layer

The Learning Analytics layer involves the data collected while the learner experiences a learning design. The data is generated by the learner’s behaviour from within the design or from the actions of other involved participants, such as other students, tutors or the instructors. The data collected here can provide evidence of design impact when it is as a result of learning activities and interactions.  

The framework categorises data into 5 distinct types: Profiles, checkpoints, process, performance and satisfaction . These will be outlined below:  

Profiles  

This data revolves around the profile of the students who participate in the learning design. This includes demographic data and academic data.  

An example could be to look at student demographic data and ask, how many students speak English as a second language?  

Checkpoints  

Checkpoints indicate that a student has met a requirement for learning by accessing the relevant resource of the learning design. It is worth considering the limitations of this data as checkpoint data in and of itself does not provide an insight into the learning process, understanding of how students are learning or their quality of learning.  

An example might be to explore vUWS site login data and ask, have students logged into my vUWS site this week?  

Process  

Process data provides an insight into learning information processing and knowledge application within tasks that students complete as part of a learning design. It uses data in relation to the student presence and usage behaviour within activities that learners engage with. Data can be used to inform teaching during the semester or used to inform reflection after the end of semester. The data varies between activities due to the inherent nature of what the activity is.  

An example of a common activity to explore might be to look at a Panopto video and ask, how much of my Lecture 1 video did students watch? When did they “drop off”?  

Performance

Performance includes student responses and assessment related data. It is important to be mindful of non-assessable performance data and how students may or may not apply themselves to it.  

An example could be the use of a weekly quiz and asking the question, do students understand this week’s content?  

Satisfaction

This data type refers to data relating to student satisfaction and preferences. Be mindful of over surveying students and keeping the surveys concise and focused. Consider the questions that are asked and how the data will be analysed.  

An example is the use of self-reporting activities and ask the question, do students feel confident in their own understanding of the concepts?  

Using the framework in practice  

We now understand what the Analytics Layers for Learning Design (AL4LD) framework is and have considered some examples of each data type, we now want to explore how we might apply a strategy to use these ideas.  

The follow steps provide a method for reflecting on the current learning design and considering how we might embed learning analytics into the design with the intention on generating the desired insights.

1. What do I want to measure?

When we are considering the AL4LD framework and thinking about implementing learning analytics in our own learning designs, the first thing we should be thinking about is “ what do I actually want to measure?” followed closely by “why do I want to measure this?” .  

Perhaps there are specific activities that students engage in which we want an insight into, new learning materials we want to evaluate or a threshold concept we want to ensure students understand. In thinking about what you want to measure, it is useful to consider and develop specific questions.  

2. What data is available to you?

Now that we have some specific questions to ask, we need to consider what data is available to answer the questions. The first step is to review the data that is currently available to us. For example, explore the vUWS data or the tool specific data that the activity or learning material is contained in.  

Perhaps data is not available to answer the specific question and we need to use a proxy measure. A proxy measure measures something that is similar or related to what we would like to measure, and we make an inference about it – i.e. it is an indirect measure.  

On many occasions, a single measure or proxy measure will not be sufficient to answer our question(s). We may need to draw on multiple measures or data sources and triangulate the data to provide the insight we are looking for.

3. Modifying the learning design

To collect the data we are after, we may need to make changes to our learning design. This might be small adjustments, or more significant modifications. It is often convenient to consider the least significant modifications before moving into more significant changes.  

Reconsider the tool being used  

There may be multiple tools available that perform the same function but offer up different insights into the student learning process through data. Or, perhaps the tool we are using has multiple methods of implementing it so we may need to consider how we are using a specific tool.  

Modify the activity  

It may be that the learning material or activity does not capture the data you are looking for in order to answer your question(s). Therefore, you may need to modify the activity to provide the insight you want.  

Designing a secondary activity  

In some instances, we may want to design a specific activity to collect data. When designing these activities, it is important to ensure that the activity is meaningful for the student, has educational value for the student and is not just a data collection tool. We should be strategic with these types of activities and must not to overwhelm the student with too many activities.  

Conclusion  

Learning analytics can provide educators with feedback on student engagement, progression, performance, and their learning process. Furthermore, it can also provide insights on the effectiveness of the learning design.  

The Analytics Layers for Learning Design (AL4LD) framework provides a structured and organised perspective for embedding learning analytics into a learning design. It enables educators to use common language and outlines a set of tools and examples to illustrate each layer, and provides clear implications for practice.  

With clear goals, specific questions and careful planning, the AL4LD framework can be leveraged to generate useful and meaningful insights for the instructor.  

References  

Hernández-Leo, D., Martinez-Maldonado, R., Pardo, A., Muñoz-Cristóbal, J. A., & Rodríguez-Triana, M. J. (2018). Analytics for learning design: a layered framework and tools. British Journal of Educational Technology, 50 (1), 139-152. https://doi.org/10.1111/bjet.12645  

Lockyer, L., & Dawson, S. (2011). Learning designs and learning analytics. Proceedings of the 1st International Conference on Learning Analytics and Knowledge , 153–156. https://doi.org/10.1145/2090116.2090140  

Siemens, G., Long, P., Gašević, D., & Conole, G. (2011). Call for Papers, 1st International Conference Learning Analytics & Knowledge (LAK 2011). Retrieved from https://tekri.athabascau.ca/analytics/call-papers  

Siemens, G. (2013). Learning Analytics: The Emergence of a Discipline. American Behavioral Scientist , 57 (10), 1380–1400. https://doi.org/10.1177/0002764213498851