By Stephen Downes
Nov 07, 2025
Transcript of Inpost Training and Educational Data Analytics: An Overview

Unedited Google Recorder Transcript

 

Okay. Hi, everyone. As you know, I'm Stephen Downes. Presentations, titled training and educational data analytics and overview, and for the sake of the audio recording which I am doing. But it's November 6th, 2025. Um.

 

I'm just going to share my slides. Because what would a presentation be without slides really? Okay, Window Nine possible Windows. Let's pick. Oh, right, I need it to actually be running okay.

 

10 possible Windows. Sorry about this, folks. I'm just slow. There we go.

 

Yeah. Okay. So, you should be seeing the. Nice mountain and the total slide. If not, please let me know. Um, I'm just gonna work my way through these slides. That said, I do invite any of you to ask any questions or make any comments as we go along. Uh, I can handle the interruptions that's perfectly fine. Um, I'm planning for basically a one hour session. I may well and sooner than that. It's always a bit hard to gauge exactly when I'll finish, um, either way. I'm here for as long as you need me, um? For any comments, questions, Etc? Um, I don't see the chat. So, if anybody? Types anything into chat. I would need somebody to actually interrupt and tell me that something has been typed into the chat. All good. Okay, thank you. So, there's three basic sections to this. Uh, first of all, I'll do an overview of Education and Training data standards. Uh, second, I'll look into what we're actually up to when we're doing analytics for decision making. Uh, training and education. And then, finally, I'll raise an incomplete set of issues and considerations. And, of course, that's something that could go on forever. So, I'll put a limit on that. So, we begin with education and Training data standards. So, what we have here are some fairly typical analytics architectures. Um, and basically, you can see that they involve some sort of admin system. Some sort of learning materials area and then the actual analytics, including data storage and preparation, um, and then data visualization and then feedback, which feeds back into the learning system. This is a pattern that's repeated throughout learning analytics. We can see here. On the right, another? Uh, representation of the same concept. We have an analytics engine, an intervention engine, which is a recommender or something that prompts human in in intervention. Um. And a learning, adaptation, and personalization engine, which again creates semantic content based on an analysis of the learning material and the individual's learning needs. Uh, we see this same sort of pattern come out as well. In analytics architectures for Canadian Forces systems on the left, we have, uh, one for the line vehicle crew training system on the right. An early version of analytics for weapons effects simulation. On the left, we we look at the student training systems very similar to. Uh, what we see outside? Uh, the forces contacts. There's a learning management system. Um. That ties into the defense learning Network. And then there are various capabilities created that build and manage the synthetic environment. Uh, on the right, we have similarly a data communication Network, the various systems that are used, and again, an x-con capability. So? Analytics and and training and? Education analytics would be based on this sort of system, a learning management system, a weapons effect simulation system, a vehicle crew training system, and we see. So these sorts of systems in use well across education and Training. There's there's a lot of variability in them, but at heart, they're all pretty much the same. Um. This is a base architecture that we can use in order to talk about learning analytics generically. It's a it's called the apperian diamond. Aperion is. A learning and Development. Software Consortium. Um, that supports open source learning management Systems. And this captures all the different aspects of alerting analytic system and learning analytics architecture. So, we have the learning activities themselves. We have a record of the learning that has taken place. We have a learning data analysis. Into, which is fed information about the student. This analysis results in some sort of action. For example, a Content recommendation. There's also a communication of the analysis, typically in the form of a dashboard. The remainder of this section will be to go through each of these six sections in more detail. So, before we do that, though, I want to do a high level look at some of the network communication standards. When we're doing, you know, when we're talking about any training system of this type and in particular, a distributed training system, which something like, um? Effect simulation would be because it's not all in one place. It's in a whole bunch of different places. It's of particular importance to keep in mind the different types of communication standards. We begin at the signals level noticing on reflection. As I look at this that? You know, the signals were mostly familiar with, like Wi-Fi and 4G or LTE, you know, cell phone signals, but we could even include that with that, things like a laser signal emitted by a weapon simulation that's a signal as well. At the signals level, it has no actual content, but it is a physical Sim signal, and that's the sort of thing that we're referring to in signals level.

 

For an actual communication to take place from one system to another system. Protocols you're familiar with already probably include http for the web, FTP, or file transfer protocol SMTP or simple mail transport protocol. And there's a bunch of others. Some of them are. Asynchronous like HTTP? Others are synchronous and actually create a persistent socket connecting two separate systems. After the protocol has been established, the next level. Of communication standards are at the data level. This describes how the data itself is encoded and and how it is accessed by one system from another system. The most common type of data level standard on the web right now is rest or representational State transfer. There used to be a thing called web services. And then wsdl web services. Description language. This still exists. It's still around. It's still used to support a lot of systems, but I'd say it's. It's past its peak Json, LD, JavaScript object notation, uh, linked data. Is another common. Standard used to represent data that's sent from one system to another. It's not really a level per se, but authentication is an important set of communication standards saml. Forget what the simple access markup language. No, that can't be right. Or open ID.

 

Collections, uh, I should have, should have reviewed those. Um. These are are all standards for establishing. That the person who was accessing the data has the authentication to access the data. Um, and then at the actual application level, um, you can describe things like web services at the application level, and that's why we have web services description language here, but also. Application specific data encoding like HTML for a web document docx for? A Microsoft Word document. A lot of. Definition languages that we see in. And military applications are actually at this level, so we have military scenario, definition, language and Coalition battle management language. These are at the application level. And, and so a lot. That's where a lot of the discussion is focused, but the point here is that all of these need to be kept in consideration when talking about a system, such as weapon effect simulation. Another aspect. Okay. Now, we're moving into the the six areas of the Imperial Diamond. So, the first is learning activities and by learning activities. What I mean is any content or resources or software or programs or anything like that. That actually creates learning, so it might be a textbook. It might be an exercise. It might be a simulation. It might be a template of some sort. Gold standard for this is IEEEE 1484.12.1 2020. Which is the standard for learning object metadata often referred to Simply as learning object metadata or Lom? Began as a standard created by the IMS Consortium instructional management Systems, and it has various. Versions metadata for learning resources. A remlr is another example. In the world of simulations, we have. Distributed simulation, engineering, and execution process or DC, again, a standardized process for building federations of computer simulations so that one simulation can interact with another simulation. For example, one simulated tank can interact with another simulated tank. Um, and that was also published as an IEEE standard. These are just some examples of Standards. There are actually dozens and dozens of them. The input for data encoded in any of these standards comes from. Well, whatever it was that was used to create the learning activity or learning resource, typically for documents and courses. They would come from content authoring systems, and they might be encoded in something like scorm, shareable courseware, object reference model. Simulation systems templates might be used and in weapons effects. Simulation. We have battle plan templates that can be. Developed and used in order to create a weapons effect simulation. Related standards to learning activities include competency definitions. These are used to establish learning objectives and assessment standards in learning materials. The IEEEE data model for reusable competency is an example of this as well. The American Advanced distributed learning competencies and skill system? Is used for the same purpose, and there are more standards that may be used as input for learning activities. This would include task standards, qualification levels, learning design, Etc. Here's the set of standards around the creation of learning activities. Learning records stores, the second. What a learning record store does is. Contains records of the learning activities that took place. Uh, we see as an example of some learning record store data in the diagram on the right, so? A learning event might consist of the session. The session would have an ID. The learner would have an ID. There might be a number of attempts. There might be extended data there might be a sequence of events that took place in the learning event. Have a name, start time, end time, Etc, and then there may be extended data as well there. What might this extended data be that depends completely on the event, and different events can produce different kinds of data. For example, if a person is attempting to take a test, then the the event data might be the answers and the test score that resulted. If a person is attempting to do something in a simulation, then the event data might record the steps that they took and the result of those steps. Learning record store can contain data in a variety of formats and what the learning community has done has been to develop a standard for transfer of information from the learning activity to the learning records store. This standard is called the experience API or X API that's formerly known as Tin Can. But then, somebody? Well, some company trademarked it, so they moved away from that name and went to X API. It's also formalized as an IEEE standard. So, what experience API does is take all of these elements that you see in the diagram and? Create a way to represent them for each type of learning activity. There's an experience API. Application profile, and so. Different kinds of data can be represented and therefore transferred using X API, so it's not a single way of representing learning activities you're not locked into a single way of doing it. You use xapi almost like your content envelope, and then you design your data accordingly. Just like we have a specific transmission standards.

 

And then we have application. A specific protocols. There are other types of Records that can be transferred in this way. Ims instructional management system. You might recall them from a few a few moments ago. IMS caliper is another mechanism and the. Tripoli 2997 Enterprise learning record is another type of data that might be encoded inside a learning record store. There's a whole bunch we could talk about when we're talking about learning record stores. This is a very simple representation. But learning activities data can become voluminous. Um, and it can come from multiple simultaneous sources, and in fact, that's why we have a learning record store. We don't just store all the data inside a single learning management system. We have a separate and specific learning record store because learning. Activity data may come from anywhere. It may come from a person working inside a learning management system to do, uh, you know, to take a test or to learn some Theory, or it may come from a simulation system where they're actually trying to put something into practice. Uh, it may come from different places, you know, perhaps, at, uh, Gagetown, perhaps at Wainwright, uh, it may be from different systems from different manufacturers. The X API makes sure that they all send this data into this learning record store in a standardized way, and then the learning record store serves as a central repository for an individual's learning, no matter where it took place.

 

Another set of data. That's, uh, it's kind of outside the diamond formation, but it's certainly relevant to any sort of data analysis that takes place. Is the student information system. Now, we call it the student information system here, because that's the name that is used in most, uh, educational institutions, although. Mean could refer to any Personnel, information system or staff information system that you can think of. Basically. What we're doing in a student information system is we're creating data specific to each individual. In the training environment, and I might add, and this will come up again in the future. Additional parts of this presentation.

 

The students information system. We may also want information about groups of individuals or cohorts of individuals so that we're not simply recording the the performance of one single person. Each individual has an individual, but how a company or a platoon might perform. As a collective entity, we'll come back to that. There are some standards again, not surprisingly. IEEE p2997 Enterprise Learner record, which we just referenced in the previous slide, also applies here. Advanced distributed learning, which, uh? For the U.S military, they have their own Enterprise learner record very similar to IEEE. Fact, the standard IEEE is based off the ADL Enterprise learning record. That's a very common pattern that we see in the world of Standards. The input for a student information comes from a wide variety of places, not surprisingly. Would include basic biographical information, such as the name, address, Etc, employment in a military context would be their unit. Uh, their platoon of their rank, their specialization. Uh, any other organizational information. It may include their educational background, a history of the courses that they've taken the, the results of the courses, any training that they've taken. Uh, whether it was required or optional, any credentials that they may have earned, uh, any badges if the organization is using digital badges? And it may record the individual's competencies and the evidence used to support that competency. All of this feeds into the system as well, so that when we have. A learning event that was recorded in a learning record store combined with the student information system information about that person. It gives us a fairly complete picture of who was doing the training and what happened during the training. Why do we need that? Well, the I don't want to call it the core that is kind of the core. Of the the whole system is, uh, the learning analytics part. This is basically.

 

One or a number of applications. Of a variety of types that takes in this kind of information and is used to analyze the data and produce useful results about that data, and I'm being very vague here, because we're going to talk about this quite a bit more in the latest parts of this presentation. We're just worried about the standards. There are various standards coming that we're talking about when we're talking about learning analytics. We're talking a lot about data management. And, and, and, uh. Data. Data pools in general, so. Were impacted here by data management standards, the DND CAF data strategy, for example. There's even there are larger. Data governance Frameworks, not just for the forces, but for the government as a whole. Um, data interoperability is. Is an ISO standard, and then there are data standards themselves, which describe the way the data is organized, stored, or represented inside a learning analytics system, and there's I've got a link there to a position paper that describes a number of those. And then there are the various tools that are used for learning. Analytics on this data are P.

 

Large system called cognos, and there are more. So, what comes into the learning analytics part? Well, it's the learning data itself, which might be in the form of X API or caliper. There are models. The models might come from machine learning. Uh, the models might come from neuro as a set of neural network weights. For example, chat GPT or Claude, Etc. There's a wide range of models. Or algorithms that can be used to analyze the data. As well. Typically, there will be contact contextual information, uh, applied as well, in order to generate the the analytics. One simple example of a context or a contextual application would be the question you were trying to answer by doing the analytics. So in in the area of learning analytics, we get the whole idea of prompt engineering. We, we can also set the analytical process against a background of Base documents. For example, battle test standards or something like that, um? The placement of this documentation is accomplished through. Something called retrieval, augmented generation or rag, and then the analytics might actually access external tools. This is relatively new in the world of analytics. Something called the model context protocol. Not a standard, yet a long way from being a standard, but that's being used by a variety of AI systems in order to connect the analytics that they're doing with tools that can help them do that. Um. So, after the analytics, we want to present or communicate the result of that analytics in some way. This is an area where there's not really a whole lot of standards. There are numerous tools. Um, you know, like, uh, for example, uh, the social network analysis and pedagogical practices or snap, uh, diagnostic instrument, uh, there are learning object context ontologies. There are very visualization types such as bar, tart bar charts, or bubble charts, Etc. I've linked to a list of a bunch of them.

 

There may be standards applicable, and there would be another domains for sure. Just around presentation of data in a dashboard generally. Um, for example, accessibility principles. Hdi principles for best presenting the information. Some of this is formalized. The accessibility principles are others less so, and it's an open question. Which, if any of these would apply to dashboards that are being employed in our specific context, there's a whole guidebook on learning analytics dashboards that I found useful, and I've put a link to it there. Um, the input for the dashboard. Uh, is often in a form of queries. Uh, that would be created in the design of the dashboard and then applied to the analytics database and so typical query. Uh, query languages would be used like, um, SQL Sparkle or graph query language, and these vary depending on how the data has been stored in the data analytics process.

 

Um. And the other part of the diamond that comes after analytics is action. Um, what do you expect your system to do after? The analytics itself. Um. This can be done either automated. It can happen automatically, or as is much more typical in our environment. It would be a users instructor selected activation, so we'd be presented with information on the dashboard. We'd look at it and then decide okay. Based on that information, then I'll activate this training module or that simulation or whatever. Um. But in? Uh, learning management Systems. This happens automatically. And so. Uh, there's a term for that that I'm trying to think of, which Escapes me at the moment. But. You know, basically.

 

You know these. These systems are quote unquote, types of personalized learning, and basically they they take the results of the previous learning event, plus all the other information that's relevant. And then they automatically trigger the next thing in a sequence or in a process of learning. Um. I meant to say this in the. When I was actually talking about the analytics themselves. Uh, in our context and weapons affects. Simulation contacts. There's actually very little in the way of data analytics that happens. Um, and the way to think that doesn't mean that our whole Diamond structure is irrelevant, but Um, what it means is the the analysis is still happening. Somebody is still analyzing the data, but instead of using machine learning or neural networks, uh, to do the analysis, we actually have. Octs who are doing the analysis? They take in the data. They look at all the different things, and then they they create the after action report. And so we might think of that as a human based data analytics. I know, that sounds kind of weird, but in this framework, that's the task. And the outcome that's being performed here.

 

So, that's the analytics part of it. Let me pause just for a second, uh, because I've been talking for a bit, see if there are any comments or questions.

 

Okay.

 

All right. Let me move on, um? That was the longest section, so though it seems like we're falling behind in time or not, really? Um, so?

 

There's a theory of analytics called single Loop and double Loop learning. Um, it's credited here to our aggress, uh, I think it's been around forever personally and. This is really characteristic of the sorts of things that we're doing with analytics. So?

 

Single Loop, basically, is the application of some kind of training methodology. To an individual or a group. I know, that sounds very clinical. It's more personal than that. Well, don't use that language anyways. And then the result of that application. So we run some people through simulation. They do the simulation. Then you look at the result and see how well they did. That's the first Loop, but the second L. Actually evaluates the training itself, so we run the people through the simulation. We look at what resulted, and then we go back, and we ask ourselves, is the simulation doing the kind of job that we wanted the simulation to do? Um, and that's the. That's the double Loop right. And you know, it's represented here as assumptions, you know why we do what we do, but basically it comes down to. We're evaluating our methodology, our training methodology. So we, we can look at that from the perspective of learning analytics. The results are always going to be about the same right student performance and structure performance program performance. I would include, you know, platoon performance? Or you know, any other? Type of, uh, training entity that we're considering. The the sorts of techniques that we're looking at adaptive learning. That's the, uh, the phrase I was searching for a couple of seconds ago, where the training activity is prompted automatically by the system, so adaptive learning. But there are other elements that go into our learning activities and learning resources, for example, learning objectives, learning plans, self-regulated learning if the people are learning on their own. There's actually a whole set of standards for learning sequence and and. Learning design and then the dashboards that we use. So that those two things? Representing all the. The learning intervention, as it's often called an literature and then the result of that learning. But then we go back to the assumptions, and we're doing a different sort of thing, uh. For example, we might predict. Based on previous work that if we run our platoon through such and such a simulation, they will acquire such and such a skill. So that's our prediction. Then, we look at how they did when we did that. If the prediction is not right? We don't just blame the platoon, we we go back, and we question our assumptions that led to that prediction in the first place. So that we're considering the totality of the learning event. Various other. Assumptions and mechanisms for testing assumptions, modeling metrics, model selection, cross validation, Etc. Are are used to do this. Now, these single loops and double Loop. Don't occur just in the learning activity learning assessment context. They're actually applied across the board through the entire learning system. And you know, we looked at a number of these different Learning Systems. The different elements of the diagram of the of the diamond. And so we, we see that represented in the diagram on the right. Um, we can see, for example, one side of Loops involving instructors and iOS and x-con another set of. Uh. Uh, impacting learning, designing, and learning analytics. Another, the feedback and after action review with the students. That would be the actual assessment of learning itself, and then even, uh, a loop involving the whole concept of learning analytics as it relates to learning science. So, we're always doing this, right, we, we're applying a thing, then we're questioning.

 

Thing that we should be applying. Are we getting the results we would want to get? So? You know, it's far more than just, you know, analytics that recommends the next piece of content to a person that's a very small part of it. Uh, analytics really speaks to the overall learning strategy and learning design, not just in the particular case, but you know, the entire concept of learning, as it is understood by the institutional organization.

 

Over time. Um, and as organizations develop the learning analytics. Develop as well. Um. Generally, you know, I'm back in the pre-history of of educational technology. Um. The first step was just to become aware of what was happening, keeping records. Which seems like it wouldn't be a major Innovation, but at one point it was keeping basic reports logging events that actually happened. Now, I I reflect on my own. Employment history and my own training through my career and. Being aware is actually a challenge. There isn't a single complete record of all the learning that I've done. Over my 25 30 years of professional experience, which is kind of funny, I mean, I don't even have that. Um, then the next step is yes, go ahead.

 

Oh, I'm sorry, Sean Francois, please go ahead.

 

Sure. Yeah, training validation is a very good way of putting it. Yeah.

 

Sure. Please, go ahead. Yeah.

 

Yeah. Yeah, it's obviously much more, uh, stringent, than our my own organization. Um. But hey, that happens. So anyhow? Um, moving up through the evolution of learning analytics. Now, we're, you know? We see this more. The looking at? The organization, the students, the faculty through various dashboards. Reporting tools, data integration, Etc. That leads on to what is presented here in this chart as organizational transformation. Uh. The you know the this? This report comes from my friend George Siemens and and the people that he's been working with, and they're talking about predictive models, personalized learning, and measuring by impact. You know, there are lots of ways you could represent those, and I would consider those to be a subset of organizational transformation. Um, similarly, I would consider the definition of. Learning validation that was just described here, not to be a complete description of what could be done from the perspective of assessment and validation of the the training methods and methodologies, but that's part of what this project is about. And then, finally, you even look at sector transformation um, which in our context might be of forces-wide data sharing Innovation.

 

Um. So, what are we doing this, um? Or actually, I'll say, perhaps a bit differently. What are we doing when we're doing this? This is the chart. Here's just a snapshot from a number of years ago, so don't don't worry about the precisement placement of the lines. But what I want to indicate here is that our Focus changes over time and continues to change over time, but the sorts of things continue. Uh, the sorts of things that might be done in learning analytics, Behavior modeling, performance modeling, assessment obviously, modeling of students. So, what is your stereotypical student? Uh, you know, what is the average Soldier? You know? What types of students do you have or soldiers? Do you have? Things like student support and feedback. Uh, curriculum and curriculum design, assessing domain knowledge. What is the domain knowledge? How much of it did somebody retain, uh, sequencing? How best to sequence. In what order should we present stuff and then instructor support the sorts two ways of looking at that right? The sorts of things that we can do to support instructors or in our case octs. And secondly. Of support, our instructors or octs. Best position to provide in the learning event itself. Uh. Applications of analytics. There's, there's a bunch of different kinds of analytics that can be applied to the data that's collected. This is not a complete list, but it's, but it's a typical list I act. It's, there's even this list is long enough. I presented it in two parts causal mining to find causal relationships clustering to find similarities. Discovery with models to? Uh, you know? Use a model in a new context distillation of data in order to represent it in intelligible ways. Because a lot of patterns and sequences, and that you just look at the raw data, you wouldn't spot it. But if you distill the data, then it comes out knowledge tracing to evaluate the Mastery of skills outlier detection, all kinds of reasons to do outlier detection. I just put in there identify different individuals. I think you get the the value of outlier detection prediction, uh, here, especially to validate models, uh, process mining, to analyze event logs. That's really good for spotting sequences or typical behaviors. Recommendation in order to? Suggest the next activity or next resource relationship mining. This is really useful, especially when we're looking at cohorts or Collective learning. I haven't really talked about that at all, but we, we could. Statistical analyzes to find relationships among variables, including causal relationships, among others, social network analysis. Again, really useful in Collective training. Text mining. Take raw data in text. Like somebody's, uh, you know, hadn't written report or typed out report, and they extract say a sequence of events or whatever from that visualization. For dashboards, and uh, non-negative Matrix factorization in other words. Saying, previously completed. Test results in order to analyze student test data. A longer term application of this could be automated assessment. Not sure that would be relevant here, but it certainly is in The Wider educational environment. I've done my own breakdown of AI applications based on the sort of outcome that we would like to see from data analytics generally. Uh, the the first four of these descriptive diagnostic predictive prescriptive? Throughout learning analytics literature. Um, I've added two generative and deontic, and we characterize each of these by the sword of question that the analytics is trying to answer. So descriptive obviously answers the question what happened? Diagnostic what kind of thing happened? You see the distinction there, right? Predictive what will happen? Descriptive, and this is the recommendations part, right? How can we make it happen? A generative which? Didn't doesn't even exist in learning analytics literature to the last few years, but now we've seen all over with generative Ai and large language models make something new, right? Take all the stuff that you've seen in the past. Make something new. And then deantic nobody's talking about this yet, but it's trust me. It's coming, uh, answering the question what should happen based on all of the data? What are the the normative principles that apply in normative principles might include, for example, Community standards definition of what is bad and good definition of what is fair recommendations for changing laws, regulations, Etc and even things like content moderation or. Distress or other psychological applications.

 

Okay, that's a lot. Uh, I'll pause here again and see if there were any comments or questions.

 

Okay. So a few issues. I'm not gonna do all of the issues. But a few that are, you know, worth highlighting and in particular in the context that we're working in? So, right off the top privacy and security? A lot of people in the world of data analytics in the world of AI generally. Talk about! Uh, individual privacy. Uh, and that's kind of what this chart represents, but there's also Collective privacy. There's also proxy privacy where you're managing the privacy of other people. So, there's, you know, a whole stewardship aspect. And then there's the actual privacy plus perceived privacy, and then what might be called privacy calculus where you're weighing? Uh, you know? Privacy measures versus the risks. Of disclosure. Now, everything I've said about privacy here also applies to security, right? We have Collective security, individual security, proxy security, how security is perceived, and then the security disclosure calculus. Tons of stuff that could be said about this. Certainly, an issue that anybody working in learning analytics needs to be aware of? Second thing is data sources these. Pie charts, or, you know, they're just samples? Of a study. Looking at different kinds of data sources. They don't stand for anything in particular, except that. Um. I want to draw attention to the fact that data does come from different sources. And some sources are better than others. Some sources are more appropriate than others. In the sense that some of them are relevant to the learning tasks that we're trying to complete, and others are not. And which is which, at the beginning of any analytical process, is an open question, for example.

 

Suppose we're in the weapon effect simulation environment, and the suggestion is made that we collect biometric data from the individuals who are taking part in the simulation, so their heart rate, for example, their skin temperature. Uh, how hard they're breathing, stuff like that? Could do it. It is a data source could be fed into the, uh, into the database, and then have learning analytics applied to it. The question is? Is that data useful and relevant? And that's the kind of question that needs to be asked for any sort of data. Um. Another example, and this is often used as an example in. University education circles student evaluations after the fact? Um, our student evaluations, a good source of data about, say, the quality of the instruction that they received. Well, there's the strong correlation between student evaluation results and the grade received, but there might be a much weaker correlation between student evaluation and the quality of instruction. So, the question has to be asked is student evaluation data? It and useful in this particular data analytics scenario. And again, these are questions that can't be answered generally, but need to be answered for the specific context in the specific event.

 

Any sort of analytics. Um. Is liable to issues of bias and misrepresentation. And here. I'm not just talking about bias in the sense of discrimination against minority group. Any source of systemic misrepresentation of what is actually happening in a learning environment? Is a bias. Uh, sometimes we might want to buy us our data collection in a certain way. In neural networks, bias is is a parameter that's actually set that determines the sensitivity of particular neurons to data. So, if you turn up the bias. Your neurons become really sensitive to certain type of information, and you do that because you want your analysis to focus on that type of information. So sometimes bias might be relevant. The other thing I want to point out here, we've got. Different types of bias for different aspects of the analytics workflow. So, I got an analytics workflow picture here, right? Data collection structure and cleaning model training. HP tuning model picking feature selection xai analyzing whatever that is and causal analysis. So, all of those are steps. In the data line data analytics part. Well, all of those are steps in one way of doing data analytics, right? I mean, any of those steps? We can introduce bias. So, in data collection, obviously, picking this data, not that data data structuring, organizing it one way. And not another way, I mean, any set of labels that you apply to data implies a pre-conceived ontology, a preconceived way of looking at the world. So, I'm not saying any of these are wrong. They're all unavoidable. The main thing to keep in mind here at each step of the workflow there is bias involved. We need to ask, what is that bias? And is that bias appropriate for the type of analysis that we wish to undertake? Finally, data literacy. Huge broad area which I won't get into in any sort of detail. But data literacy is an essential aspect of any kind of data analytics and. Uh, you know, even if we're talking about officer controller trainers doing the after action review based on their own Reflections based on their own experience, uh, and what they saw in the actual simulation exercise? Data literacy even applies in that sort of circumstance. Because the OCT is sitting there considering a wide range of data, assessing its relevance, trying to put it together. And that involves data practices, because basically they're working with data. Um, data literacy comes in a variety of shapes and forms. I did an analysis a number of years ago for ADM. And described data literacy in terms of or as characterized by different roles. So, what counts as data literacy for one type of person is going to be distinct from what counts as data literacy for another, and you can describe. That in terms of one of these charts, like I put here, these lines don't actually represent any actual measurements. But each of these boxes would have a value that is different depending on the different role that you undertake. And I see a question.

 

Okay.

 

This is the second last slide. Which also should be helpful.

 

And yeah, that's stable. Letter receipt, the last issue. And I've I've touched on this in a number of occasions. So far, we've had lots of discussions about it. Among ourselves is the, uh, question of individual versus collective analytics. Individual attributes are different from Team attributes. Individual outcomes are different from Team outcomes. Um, a process that we can describe for a team is different from a process that we can describe for an individual, not in all cases, but in many cases and the description of the task environment and the task itself is going to be different. So there were going to be aspects of collective analytics that don't apply. In the case of individual analytics, and I've listed a few here. Uh, communication coordination, Mutual performance monitoring, uh, you know, watching out for each other backup behavior? Um, collaboration, that is working together toward a single known objective and conflict management. These are just examples. It's not a full set of examples, but it's the sort of thing that analytics needs to be. Needs to take into account when being performed on a cohort or a collective as opposed to an individual. That's everything. Um. So, like I said, it's about an hour. Um, and uh, I'll turn it back over to the room to see if you have any questions or comments.