R U Asking the Right Questions to your content provider?

I’m the worst when it comes to questions. I often will ask way too many questions to a vendor, regardless of if they are a learning system, e-learning solution, other providers or a 3rd party publisher. I will usually say, okay this is the last question, then follow-up with another question.

Perhaps it is a question that just popped into my mind, or it could be based on what I heard with the previous inquiry.

Sometimes in my inquiries, I will ask a question and then say “Correct”? And the vendor will either say yes or no.

I find that vendors often are used to the same questions over and over again, thus their response is somewhat robotic, I get that. I will dive deep (usually far beyond) what their usual response is, to truly get at the root if you will. I want to know from all different perspectives, from my role as an analyst, to the role of those in L&D and Training, and even to the role of folks who are not in either field but are the ones making the decisions and overseeing the whole thing.

The goal here is to know, simply that. I never allow a vendor to drive the route. They do that anyway, especially at trade shows, when folks ask them questions. They can steer this way or that way, but with me, I am the speed bump on the road.

When it comes to data, I often talk about it as the story for your learning and training. In past posts, I’ve covered it from the learning system standpoint.

This post though is different.

Rarely do folks who purchase off-the-shelf courses/content from a 3rd party publisher, ask that publisher questions, let alone ones tied around data they are capturing. If you are using an aggregator, such as GO1 or OpenSesame, then those questions go directly to them, as they are the head, per se, of the content/courses that sit on their platform, which you have purchased access to – with all those publishers. Later this month, a showdown between GO1 and OpenSesame and a new 3rd party aggregator (global brand name) – but not yet launched, with take place. Each of these vendors were asked questions, questions of relevance, which for key ones will be posted as part of the analysis.

This is mentioned here, because a lot of the questions were around data that they capture, how it is used to analyze engagement of the courses and content, usage and so forth.

A key question that I ask any 3rd party publisher, is whether or not, the client (regardless of if they purchased the content/courses direct and then added it to their learning system, or they sit on the platform, sort of a B2B, or they selected the courses/content from an e-learning marketplace that is available from the learning system vendor).

The Initial Questions

Remember these are questions, that you will ask your 3rd party publishers – questions that will help you with the story for your learning and training. I hear way too often from folks (buyers) who see usage drops, or say employees dislike the system. Then I dive into it, and courses/content plays a key role.

Systems whose data doesn’t really help your learning or training, although the perception might be there, is just one part of the entire story. The other, the key to the kingdom if you will, comes from your 3rd party content publisher.

For those of you, who have no background in L&D or Training, don’t fret, these are questions you should ask. For those of you in L&D and Training, what I have found is that some ask questions, others do not, and the questions themselves may not be the right ones.

There will be questions that are tailored to one publisher (i.e. you purchased courses/content from a specific publisher – thus if you purchased from three publishers, you ask the same questions) OR to an aggregator (GO1, OpenSesame, Udemy, Coursera), which have multiple publishers on their platform.

Let’s resolve that.

Data Capture (Regardless if it’s one publisher or an aggregator)

• What data are you capturing, or can you capture with the courses/content we have purchased?
• What types of data do you capture with your off-the shelf courses and content? (This is general and not specific to what you may or may not purchase) This is usually my first question when it comes to data.
• What data can you share with us or willing to provide to us? (Sometimes the retort is “what data to you want?”, which since you may not know, is obtuse)
• How do you track the data and analyze it? Do you use trend lines? Track and compare or correlate – based on say usage and engagement?
• Do you have data scientists in-house? (If they are capturing data, someone built those data sources, so the answer you are seeking is Yes. Especially if the vendor has algorithms in place – aka machine learning, which many refer to as A.I.)

Usage and Engagement

In an ideal world, a publisher would be able to drill down to a course that has a TOC and tell you how many times X learner went into a certain chapter, how often, time per visit, and then a total amount of time. Too many publishers, even systems for that matter, go cumulative, which tells me, what exactly? If you have a course and your 30% of your learners are going only into chapter three, and none of the other four chapters wouldn’t that be of more use to you, then say, the publisher or system telling you that 30% of your learners went into the course, without getting more specific?

• If the publisher can drill into specific chapters of a course (assuming it has a TOC), where did the learner go? How often? Is there a trend with other courses that are similar to the subject chapter, and if yes, what is that learner doing?

Example: The learner goes into a Microsoft Excel course and looks only at pivot tables. They, go in there several times, maybe for a minute or more (after all, they are learning how to do pivot tables, and likely have Excel actually open, and thus doing what they are learning). There are other courses in your system, from the same publisher, that are Excel related. One is a short video around macros and pivot tables. Is the learner going into that video too? If yes how long? Let’s say the video has a TOC (you can do it with some). Where are they going? Are they going to say only a scenario or a specific section? Again, the more data you have, around usage and engagement, the better it will be to select other courses similar, to find other publishers that align to the same area, or remove those from the learner’s catalog options, that are not relevant to that learner.

• What courses/content is the learner selecting? Which ones are being accessed the most? Do you track trends on a specific learner’s usage and engagement or only total (if they do)?
• In the catalog or courses we purchased (if single publisher), what courses are the most popular by our learners in terms of usage? Which ones are the most popular by the amount of time? Which are the least popular by usage? (This data will help you weed out unpopular ones – it is due to subject? Course design? or what?)
• If you are with an aggregator, such as GO1 or Open Sesame for example, course usage by learner and information as the previous bullet point is the same, however there is a slight twist. Aggregators find that people tend to purchase the per seat, you get access to all the publishers – courses and content. Thus, finding out usage and engagement goes a further step – what publishers are the most popular for our learners, (ideally individual too), and of those publishers, what subject – topic- category? Which ones are the least popular? Are there any, that are not being used?
• Define popular? This is key, because what you think “popular” is, isn’t necessarily how the vendor identifies popular. To me, popular is based on usage, however, I have found publishers who use a formula that includes “ratings” of the course/content, number of views, completions (yuck – worst factor to include) and other data.

LinkedIn Learning’s “popular” honestly makes no sense. I’ve written about in the past, but in doing a random sampling (to see if there is a trend), LL will show popular based on the number of learners, but have courses, that are not in popular, even though they have more learners than those in popular.

Take a look (there is no audio)

If you are providing LL to your end users – they are seeing a screen like this – and thus popular, I suspect to them, will be based on learners – i.e. usage with high access being the key indicator, but it clearly is not.

On your system, with the publisher or aggregator you go with (LL requires you to go to their site directly), ask them around popular.

Ratings, can be a factor for some publishers, but even then, specifically, is it an aggregate? Do the learners know what one star means compared to five star? On your back-end, i.e. the publisher, what do they define as a one star or two stars or three and so on? Can they provide data as ratings tied to the course(s) to the subject areas and level of engagement? If we go back to the Excel Pivot Table example, are the ratings based on the entire course – say Excel 201, or is it based specifically or can it be, based on a specific chapter? Can a learner change the rating? Does the publisher identify that, i.e. what it once was, to what it is now? If yes, do they time stamp? OpenSesame for example, does exactly that with their courses/content. They can show what rating that learner gave for the course, and if they changed it, what is it now, and they time stamp.

This is at the course level though.

Segmentation, drill down with publishers is a definite challenge. And I hope that publishers start to take notice, that by chapter if available, is more of relevance, than the total course, especially if the learner isn’t going thru the entire course – which honestly, they shouldn’t – if it does not require to be completed. Usage and Synthesis is tied directly with interest, and as a result, a trend will appear that benefits the learner and you.

If the publisher offers scenarios – i.e. real world sims, rather than just an assessment, engagement data is very important. What is that learner doing – in the sim? What data can the publisher provide in scenarios or sims?

If the course has additional materials that someone can view or download, can the publisher provide any data around that? Let’s say the Excel 201 course has five additional resources a learner can download, Carolyn, the learner, downloads only one of the five, even though she looked at all of them. By examining this data, you can ask – why? Is it directly tied around what she is only looking at? Does it indicate something or some other area of interest to her? And if yes, are there courses or content that can be offered to her along that subject?

If other learners for that course, are selecting only the same resource, a green light bulb should be flashing in your brain – this is a trend, and important indicator. Why, should be your first thought.

Other data

• Is there or are there any other data that the publisher or aggregator provide to you?
• What do they recommend for you to know?
• How can they provide the data to you? .CSV only? Can they provide it as a visual dashboard or with graphs and other visualization?
• Can you receive the data at X time or Y time – i.e. some type of schedule where they send it to you?
• If the data shows poor usage, ratings, etc – can the publisher remove the course or content from your system or hide it so your learners can’t access? (This is a big one if you use an aggregator). Will they notify (again, if you are using an aggregator) the publisher? Can they – the aggregator provide any data from other buyers – Not their names – but tied around that specific publisher? In other words, is the aggregator aware or tracking poor usage, unpopular of each publishers they have in their system? What data – i.e. the aggregator – as it relates to all their publishers – do they track? Can they provide you with any of that data?

Bottom Line

I left out, the curation angle – i.e., how often does the publisher update their content? How do they decide or what factors are part of their decision-making process to remove or add new course(s)/content? How many people are on the team (i.e. is it more than one person), who makes that decision? What factors do they consider when adding a new course/content – is it based on trends, or client feedback or some combo or what?

For 2022, the latest trend is mental well-being. Does the publisher have any of the courses/content? If yes, how many, and what are the topics?

I always ask publishers whether it is their responsibility or that of their client, to provide information and assistance on what data and how the client can tap into it to improve the learning/training of their end-users.

Some say the client.

Others, say themselves – i.e., the publisher.

The latter is what you want.

Because

If you don’t ask,

Trust me,

They won’t tell.

E-Learning 24/7