MOOC Design Challenges

Yishay Mor has been promoting a survey from the MOOC Design patterns project asking for MOOC design challenges ( It was a simple bit of work to reverse engineer the recommendations Allison and I had developed last year for our PL-MOOC study.

Here are the design challenges i came up with.

1. Personalisation: how can we ensure learners get what they want from a MOOC

The rigid syllabus and structure of many MOOCs provides a somewhat take it or leave it option for MOOC content. But adult learners are different to learners in formal education: they come with different levels of starting knowledge (e.g. compared to students doing a second level biology class, all of whom will have done a first level science class) and different expectations and goals. Imposing goals on them (‘read or view this content’) can be demotivating. It would be better for MOOCs to scaffold self-directed and individual and collaborative learning within a loose framework. in this way, learners can personalise their learning, making it serve their specific aspirations. This is especially true in professional contexts, where MOOCs provide opportunities to bring learners with complementary expertise together, who may learn more from their peers than the course content. Even in non-professional domains, learners could be encouraged to personalise their experience by bringing their own content – choose a book or poem that means something to them, as part of an English course, explore their locale for courses about geography and the natural world.

2. Lasting : how can we ensure that learners effort in learning on a MOOC has some ongoing benefit for them.

I suppose the answer is we can’t, in that learners ultimately have to take responsibility for this themselves. But course designs should encourage learners to articulate and share action plans for using their new knowledge as a way of helping learners think about the value of what they are learning and to get them to engage deeply with the course content that is being delivered. MOOCs should also be designed so that on completion a learner has evidence that they have learned – something they have created, not just a certificate as evidence that they have participated.

3. Capitalise on the diversity of motivation, expectation, prior knowledge and experience that is inherent within all MOOC cohorts.

By their very nature, MOOCs attract a broad range of learners. These learners may differ in their motivation (e.g. learning for fun, or to address a specific learning need, or to gain accreditation), expectation (e.g. that the learner will gain access to high quality learning materials, or broaden their network) and prior knowledge and experience (ranging from those who have strong theoretical knowledge but no practical experience to those who have no formal knowledge but a wealth of experience). All these different types of learners cannot remain motivated and engaged within a rigid curriculum with fixed content. Instead, course designs should encourage learners to determine how they interact with others, supporting learners who need support while affording self‐regulated learners the freedom and flexibility to interact as they wish. From a course design perspective, some learner interactions could be scaffolded, for example by matching learners with similar intentions (Milligan et al, 2015). Flexible design could extend to certification, with achievement being linked to personal goals and progress where possible.

Adult learners bring a wealth of experience to their learning. Designing tasks which capitalise on this by encouraging the learners to build on existing knowledge (where possible and appropriate) and share their experience can enrich the learning experience for all by exposing learners to real world experience and new practices. Engaging with real world examples can be motivating and provides learners with evidence that they can use for their own personal development.

4. Integration: MOOCs don’t exist in isolation from a learner’s other activity. Help learners to break the boundaries between the course and the rest of their life by encouraging them to discuss ideas from the course with their external network.

Discussion with peers about learning is high value as it is localised and contextualised, and there is a pre-existing trust relationship. Interaction with an external network encourages breakdown of barriers between learning and other areas of life where that learning may be applied. To complement this (and to support learners who do not have external networks or who seek specifically to broaden their network), course designers should encourage focused communities to develop. Communities could emerge around language (mooc participants can lack confidence to engage when they are not confident about expressing themselves in a second language), or role (e.g. school teachers and university lecturers in parallel communities) or motivation.

These challenges are adapted from 6 recommendations that Allison Littlejohn and I drew up at the conclusion of our Professional Learning in MOOCs project )( The recommendations can be found on figshare at:

PL-MOOC study reflection



We’ve now reached the end of the funding period for PL-MOOC and, while there is still some analysis to conduct and some papers and outputs to write, the final report is written and the main findings synthesised. Allison and I have tweeted about the recommendations and the main findings but I thought i would take an opportunity to reflect more broadly on the study over here.

Professional Learning
Our research interest at the Caledonian Academy is in professional learning – the learning for work that an individual undertakes to support their practice. We’re interested in the increasing responsibility that professional learners must take for their learning, monitoring their learning needs continually to maintain his or her expertise. Our focus is primarily on informal learning, and the learning networks that professionals maintain. We tend to ignore more formal learning activities but we understand that formal learning has a role to play and indeed how formal learning fits with organisational systems within which professional learners are operating. In her writing on ‘Integrative Pedagogy’, Tynjälä has described four different types of knowledge that make up professional practice – the formal theoretical knowledge you can get from books; the practical knowledge you get from doing the job, the socio-cultural knowledge you get from being part of a community of professionals, and finally the self-regulative knowledge you use to keep on top of the continual learning you have to do. A critical element of each professional’s self-regulated learning is to assimilate learning of all four types of knowledge through an ‘integrative pedagogy’

We’ve seen the rise in popularity of MOOCs over the past few years and thought a lot about how MOOCs might serve professional learners: they provide an excellent mechanism to bring learners with common interests together, they are flexible and open, providing opportunities for close integration with practice. But can they support professional learning. While our previous study ‘SRL-MOOC‘ looked at learning in a cMOOC (Change 11), for this study we focused on an xMOOC, the format favoured by the major mooc providers (EdX, Coursera, Futurelearn). We explored the learning behaviours of health professionals studying the ‘Fundamentals of Clinical Trials’ MOOC. The MOOC was designed by Harvard Medical School, Harvard School of Public Health, and Harvard Catalyst, The Harvard Clinical and Translational Science Center and offered through the edX initiative founded jointly between Harvard University and MIT. Our reason for selecting this MOOC was because a) the course was likely to attract a high number of participants working in the health domain, allowing us to examine how professionals learn and b) the course design was typical of the so-called ‘xMOOCs’.

Its probably best to read the report and formal summaries to get a full picture of the findings, but in a nutshell, we didn’t see much evidence of professional learning occurring in the Fundamentals of Clinical Trials MOOC. There was little in the way of integration of theory and practice that is essential for effective professional learning to occur. Activities not central to the course (discussing in the forum,or elsewhere) were given less priority than viewing videos and completing assessments. Evidence of impact on practice was limited. Now we should point out that the Fundamentals of clinical trials course doesn’t claim to offer professional learning, and the course participants overwhelmingly like the course. But somethings going on here. Health professionals signed up for the course … so there seems to be a demand for learning from the community. We asked professionals about their expectations/motivations and can demonstrate that these were articulated in terms of practice – meeting professional learning needs. Yet when we asked about goals (4 weeks into the course), these were articulated not in terms of practice, or even in terms of the learning content, but instead were articulated in terms of consuming content, passing assessments and completing the course. I think the design of the course shaped their expectations.

But if there is a demand for professional learning, can xMOOCs (Downes is right, we’re not talking about cMOOCs, which  would more naturally support it – though our study of change11 suggests that it is not so simple) be designed to support it. Building on the recommendations provides some pointers:

  1. MOOCs bring together learners with a great diversity of motivation, expectation, and prior knowledge and experience. Don’t expect everyone to start from the same point – nor indeed finish at the same point. Even if the course is content focused, you should allow your learners to personalise their learning to ensure that they are focusing their attention on what they need and want to learn. Personalising learning (and goals) also enables the learners to clearly link their learning to practice.
  2. Help your learners to find others to learn alongside – they might want to find people who speak the same language, or who work in similar contexts, or who have a similar level of experience. In our studies we have  learners expressing frustration at not finding peers (of each of these types and others) to learn alongside, often because they don’t have the confidence to look for them themselves – in a MOOC they are probably there.
  3. Capitalise on the real world experience that professionals bring to their learning by getting them to specify problems and provide solutions. Get learners to create content and contribute it back to the course as resources for other learners (now and in the future).
  4. As well as fostering networks within the course, remember the pre-existing networks that the learners already have outside the course. In our study of Change 11 we saw that some learners kept their internal and external networks separate – seeing Change 11 as a course, while others linked their networks, recognising that the course cohort was merely an extension of their existing network.

We’ve put our findings and ideas out there, but of course the study was by its nature limited and more work needs to be done, not least trying to put them in practice. We welcome responses and reactions to the recommendations we formulated, and hope others can take the instruments we developed and use and refine them.

I really enjoyed working on this study, the compressed timescale meant that we had to make decisions quickly, and work through challenges. Thanks to Allison for the many discussions we had throughout the last 9 months, to Obiagele Ukadike from Harvard for being our contact within the course team, and to Lou McGill for conducting the interviews.

This study was funded by the Bill and Melinda Gates Foundation as part of the MOOC Research initiative led by George Siemens. Thanks.

And of course a special thanks to the participants of the course for answering our survey, and agreeing to be interviewed.

For a readable summary of Integrative Pedagogy, try Tynjälä, P. & Gijbels, D. (2012) Changing World: Changing Pedagogy. In P. Tynjälä , Stenström, M-L and Saarnivaara, M. (eds.), Transitions and Transformations in Learning and Education. pp 205-222. Dordrecht, Springer.


MOOC Research Conference

2013-08-28 17.39.35

Back a few days from mri13 and now that the experience has settled in somewhat I thought I would try and reflect on the conference and what I learned about the current mooc landscape and its impact on our research here at the Caledonian Academy.

First off, thanks to George Siemens and Co. for bringing together a diverse group of people with key knowledge and perspectives and no shortage of ideas and opinion. I thought the conference was about perfect size, though the attendees were sometimes a little thinly spread due to the many parallel sessions. This also meant I only saw a minority of presentations from other MRI grantees (even though I prioritised these over presentations which might have had more general appeal to me), so I need to go back to the abstracts and hope I can find some online presentations for the ones I missed (ours [Professional Learning through Massive Open Online Coursesis here). 

The conference started off on a high with a charismatic performance from Jim Groom describing his work supporting learners as creators at UMW.
In the CETIS PLE project (Milligan et al, 2006, Wilson et al, 2007) we described a utopia of learner-centred learning in a mature online landscape but my own experience at institutional level has taught me how difficult this is to achieve at scale (is it a coincidence that my work has moved into more informal contexts – professional and workplace learning). Jim’s work has shown that learner-centred learning can be supported within formal settings. However, opening the conference with a presentation showing what learning could be like backfired a little for me because over the next few days I saw too much evidence of learning taking second place to other considerations – content, delivery, administration, data. At points I was left wondering whether the legacy of the MOOC bubble of the last few years will be a million hours of talking heads video lectures.

In some presentations I was left wondering whether the whole canon of literature on online and distance learning has been cast aside. A key element of our project has been to try and understand the MOOC we are studying as an online learning event (before then going on to explore the learning occurring within it). Having a robust theoretical base is essential for us as we go forward to design tools and recommendations intended to inform the design of future MOOCs. Another disappointment was that some of the presentations seemed a little dumbed-down. In one presentation, we got the ‘MOOC: every letter is negotiable‘ line, while one presenter asked us to think when a cMOOC would be best and when an xMOOC was appropriate (like I’m not allowed to consider a million other ways of delivering a course online). Surely a MOOC research conference can support  a base level of dialogue beyond this.

Having complained (I did enjoy the conference), I should say that I saw some really thought-provoking presentations that I will follow up on (see abstracts for the MRI grantees here – sorry i couldn’t find presentation links).

  • Bruno Poellhuber described an MRI study based across a number of Canadian Universities using SRL to explore motivation and engagement in MOOCs. Over the years, our work has been focused increasingly on motivation and I think it is a key issue in MOOC research, and one which has to be addressed by any MOOC provider or designer.
  • Rebecca Eynon and Nabeel Gillani from the University of Oxford presented the most interesting SNA focused talk I attended. Part of their study focused on the vulnerability of networks, removing nodes to show how those networks dissolved or degraded. This is a mixed methods study, combining big data analysis with qualitative approaches (interviews) to really understand what is going on in the course they are studying.
  • Martin Weller presented data from two complementary studies. In the first, his colleague Katy Jordan had collected and analysed a large amount of data on MOOC participation. The emerging visualisations (some with large error bars) begin to highlight some key patterns. For example: it looks like you can now start to predict participation rates in MOOCs. [update: see Martin’s blog post on Completion Data ] In the second, Martin showed how he had adapted course analysis approaches used at the OU in an attempt to describe the form of a range of MOOCs. [update: see Martin’s blog post on Learning Design of MOOCs] Although this work is still in its early stages (I liked the fact that the findings presented by all the MRI grantees at the conference were ‘interim’) I think it shows great promise.
  • Although not a MRI grantee, I liked Shirley Alexander’s presentation describing the signature pedagogy (Learning 2014) being implemented at UTS in Australia. I suppose that brings us back to Jim Groom and learner centred learning. And away from MOOCs.

I’m looking forward to the MRI projects coming to fruition. I hope they make their research results easy to find online.

Personally, it was great to think and talk about research for an extended period of time (both at the conference and at the airport/on the flights at either end). It was also great to meet other researchers in the community and makes some personal connections. The main lesson I learned: I realised that while we don’t do ‘big data’ analysis at the Caledonian Academy, it shouldn’t stop us thinking of how we can try to complement our studies with data-driven studies: I think that in the long run, combined approaches (cf the Oxford study) are going to be the most informative.


Productive MOOCs


One of my frustrations with MOOCs is that while they provide value for the individual, their (lack of) collective value is a missed opportunity. Although I’ve had a blog post in the works for almost a year, I could never work out how to write it concisely. A recent post by Rachel Harris of Inspire Research  provided a slightly different angle to address the issue.

Rachel is evaluating the ALT ocTEL MOOC and one of her evaluation questions is ‘Can new knowledge be generated via critical discourse in a MOOC?’. With our other research interests here at the Caledonian Academy we are really interested in knowledge creation and collective knowledge. So,  in the comments I said:

I’m firmly of the belief that knowledge creation is a key component of learning, and successful MOOC learners will (through writing and responding to blog posts) create new knowledge and advance their thinking.

Of course I am talking about cMOOCs here (xMOOCs will have to wait for another day, but Dave Cormier has a related post from last weekend which explores similar issues). My comment continued with this observation (from participation in cMOOcs, and from researching Change11)

But one of my concerns in observing other MOOCs is that while each individual might advance their knowledge, there is little in the way of collective knowledge advancement. What this means in practice is that there is lot of duplicated knowledge produced during the MOOC.

Wouldn’t it be great if cMOOcs could be made more ‘productive’ – instead of advancing many people’s knowledge a little by re-creating the same (or similar) new knowledge again and again, can MOOCs be structured to stimulate the creation of new knowledge in a more coordinated way. Can you bring the learners together to produce something entirely novel as they learn? This is in the true spirit of connectivism.

Aside from my desire to see new knowledge created, there is actually an important learning observation here. As I went on to say in the comment on Rachel’s post:

 To the active participants, this can be frustrating as the consequence is that there is too much ‘noise’ in the system.

You might read a number of blog posts by your fellow participants, each making the same point. In the Change11 MOOC, some participants found this boring, others (particularly the less confident) decided not to articulate their own view because they thought it had already been presented and didn’t want to ‘just repeat’. But articulating what you have learned is important so this is a missed learning opportunity.

On a more positive note, the emergence of self-organised spaces in the Chage 11 MOOC (the facebook group, the One Change a Day group) showed that co-creation could succeed, but it was only a minority of participants who managed to engage in this coordinated way.

Solutions. I’m not sure I have any,  but here is one idea**: Can MOOCs be used to encourage interdisciplinarity? An interdisciplinary*MOOC could bring together experts from different fields to explore a topic from different perspectives and adopting each other’s methods etc. The ultimate product could be the new knowledge created as a result, perhaps a series of working papers. The immediate flaw in this is that it doesn’t seem very massive (or at least the productive element might be the result just of the collaboration of the instructors), but if you can structure the MOOC participants as disciplinary ‘tribes’ then you can focus groups on creation of specific knowledge – e.g. initially discipline sub-groups could cooperate to co-create a disciplinary view, then be re-assorted into inter-disciplinary groups to explore the overlaps and tensions. oriented towards a more specific goal that matches their existing knowledge and motivation,  they can work together to co-create something that can be contributed back to the group. A structure such as this might also help counter the tension/imbalance between homophily and heterophily that I think is inherent in MOOCs.

* by interdisciplinarity, I mean ‘integrating knowledge and methods from different disciplines, using a real synthesis of approaches’ – taken from which in turn is from a paper by Stember (1990)

**I’ll dig out the notes for the original blog post and write more at a later date.

Lurkers, lurking and labels

Conic Hill from Drymen primary school

Our SRL-MOOC study has been getting some attention thanks to a post by Phil Hill which built on the categories I had assigned to learners in Change11 that were emerging from our data analysis. There’s been some discussion going back and forth in the comments (where I think it ought to be), but a comment from Debbie Morrison last night on my use of the term ‘lurkers’ might benefit from a more extensive discussion, so I have written this post.

Debbie wrote:

I realized from reading your comment your choice of the term of lurker was not in a negative context – however the term does have a negative connotation associated with it – to ‘lurk’ according to the Webster’s dictionary, ” to lie in wait in a place of concealment especially for an evil purpose”, though I realize a modern definition has been added recently, “to read messages on an Internet discussion forum (as a newsgroup or chat room) without contributing”.

And it’s this second definition here that is critical. The term ‘lurker’ is already used in the academic literature. Discussing lurking in traditional online courses, Rovai (2000) describes lurkers as “… learners who are bystanders to course discussions, lack commitment to the community, and receive benefits without giving anything back”. This is exactly what we observed, so while I thought about whether to adopt a different term (I played around with lurking, but found it unworkable).  I eventually concluded that Rovai’s definition described what we were observing.

Connectivist MOOCs (such as Change11) must accommodate learners of all types to satisfy Downes’ (2009) ‘Diversity’ and ‘Openness’ criteria. In practice, as long as there is a balance of these different types of learner, then lurkers can be accommodated, and the evidence from this study is that lurkers can learn effectively in connectivist environments: taking the knowledge they acquire to their external networks.

In contrast, in xMOOCs, where structures of peer-grading are put in place, or people are placed in small groups, then lurkers can be disruptive if they don’t participate in the designed learning environment.

Downes, S (2009) Connectivist dynamics in communities.  Available online February 26 2013. [changed – had previously posted the wrong Downes ref]

Rovai, A. P. (2000). Building and sustaining community in asynchronous learning networks. The Internet and Higher Education, 3(4), 285-297.