The first edition of IRRODL for 2012 is now out, and one of the 15 articles is the one I spent a lot of last year working on with Carmen Tschofen
Tschofen, C. & Mackness, J. (2011) Connectivism and Dimensions of Individual Experience. International Review of Research in Open and Distance Learning. http://www.irrodl.org/index.php/irrodl
Having both been participants of the first MOOC (CCK08), we have observed and reflected on developments over the past four years. This paper is a result of those observations, discussions, reflections – namely around our perceptions of
a growing tension between the elements of connectivity believed to be necessary for effective learning and the variety of individual perspectives both revealed and concealed during interactions with these elements.
The Research Process
We didn’t start with the intention of writing a paper and our original discussions, which also included Matthias Melcher, were all related to learner autonomy. In fact I signed up for CCK11 with the specific intention of focusing on gaining a better understanding of the meaning of learner autonomy and set up a wiki to gather my thoughts and then invited Matthias and Carmen to join me. Discussions were deep and intense, often highly convoluted and we produced pages and pages of wiki script, so much so that ultimately when we did decide to write a paper, it was difficult to see the wood for the trees.
Our first submission to IRRODL was not accepted. It wasn’t rejected as such – we were told that it wouldn’t fit with the special issue we submitted it for (and to be honest we had tried to ‘make it fit’) and were invited to rewrite it and resubmit. But by this time the conversations in the blogosphere had moved on and so had our thinking, so we ended up submitting a completely new paper, although it does still include reference to learner autonomy.
The Peer Review Process
Our paper was reviewed by three people – I assume blind reviewed. The outcome was a bit bizarre. Reviewer A’s feedback included many ‘Excellents’ and ‘Satisfactorys’, Reviewer B thought the paper ‘Unsatisfactory’ and needing major rewriting and gave us very comprehensive feedback as to why. Reviewer C’s feedback included many ‘Satisfactorys’ . We didn’t receive any guidance from the Editor as to which Reviewer to believe. So we didn’t do a major rewrite :-), but we did make quite a number of more manageable changes.
Reviewer B strongly objected to our use of blog posts as sources of information, and I have to say that we rather strongly objected to his/her objection. There were at least two reasons why we thought reference to blog posts was legitimate for this paper.
First, most of the conversations about connectivism and MOOCs happen in blogs – published research is as yet quite limited. We do not believe, as Reviewer B appears to, that all blog posts are inferior to published journal articles, nor that a writer can only be quoted from a peer-reviewed publication.
Second, we were quite often writing our paper in response to the changing conversations that were happening in the blogosphere at the time. In fact we were worried that our paper was going to be out of date before it was even published, and even on the very final submission added a last reference, which we had just come across.
Finally (and very pertinent to us) is the fact that neither of us works for an academic institution, nor do we live within easy access of a university library. By force of circumstance, the Web is our main source of information, papers, articles etc. and there is no convenient catalogue to tell you where to look. I think, in the years to come, there will be more and more people like me who start doing research as self-employed people, or post retirement and who will be looking to the Web for their sources of information.
Frustrations
- We wanted to publish the paper in its submission form (i.e. before peer review) on our blogs. We could see that some MOOC participants were discussing the very issues we were writing about and it would have been great to discuss them with a wider group – but despite three requests to IRRODL for permission to do this, we didn’t receive a response to our request. Even a ‘No’ would have been better than nothing. As it was, we didn’t know whether posting to our blogs would count as pre-publication and therefore jeopardize our chances of having the paper accepted – so we didn’t. Perhaps we should have been braver!
- Despite the fact that IRRODL’s turn around time is fast compared to other journals, it still feels slow in terms of the potential for discussion and how fast everything moves on the Web. We submitted the paper in October, which is not that long ago in terms of actual days, but it is in terms of my thinking. I doubt that IRRODL could have published any quicker, so I’m not sure how this mismatch between author and publisher could be resolved. One of our purposes in writing the paper was to generate discussion, but things have moved on.
The pleasures
A while back I wrote a post with the title Pas de Deux Online Partnerships ; I feel this is exactly what happens for me when working on these papers. I get to work with people who can raise my thinking and understanding to levels that I would not be able to reach alone. The discussions we have on the wiki are highly stimulating and ‘out of the glare’ of public blogs, we can really dig deep, reflect, get to know each other and challenge each other.
We have discussed whether we need to publish. Could we just publish on our blogs and leave it at that? Matthias and I tried this (See https://jennymackness.wordpress.com/2010/09/10/the-riddle-of-online-resonance/), and I now feel that our work is more likely to be read if it is published in a journal, especially an open journal. I also feel that after spending almost a year intensely working, reading, thinking, discussing, disagreeing, challenging, compromising and finally agreeing, it feels right to publish and celebrate ‘the fruits of our labour.
A big thank you Carmen – and also thanks to Matthias for your valuable contributions in the early stages.
Jenny Connected 
First – let me say. I read and loved your paper and would encourage others to read.
Second, thanks for your comments on review/editorial process – these are valuable for authors/ reviewers/ editors. I am curious to know the value of Reviewer b’s comments – did they make a difference?
You are opening up the writing/review editorial process and this is useful – it’s a process that fascinates me http://francesbell.wordpress.com/2012/01/29/comparing-two-publication-channels-academic-journals-and-blogs/
Hi Jenny and Frances,
Thanks to you, Jenny, for laying out the ups and downs of the process so well. And thank you, Frances, for the positive response. It was quiet a tangled thicket for a while, to be sure!
To follow up a bit on Jenny’s process discussion, I should clarify that I do live within close proximity to any number of universities and colleges. As neither a student nor a faculty member, however, the degree to which I can use their materials is very limited. Interlibrary loan requests for university materials through the public libraries to which I do have access are sometimes honored (usually slowly). Sometimes I can enter academic libraries, and if I look like I know where I’m going, I can usually muck about in the open stacks:-) But I can’t access some electronic catalogues without an institutional account, and I can’t, of course, check anything out. (Can’t use the copier in some locations without an ID-linked copy card, either!) Sometimes I can make a desperate, specific request, and friends with access will “smuggle” things out to me using their privileges, but I hesitate to over-use this option. It all gives new meaning to the idea of “hacking” learning, in any case.
I personally found the review process unsurprising. The comments of Reviewer B meant that we more sharply defined the parameters in which we accepted and thus discussed connectivism, and they meant we provided a brief defense of open scholarship methods in our responses. If these responses or defense had been insufficient and the paper not published, I would have been equally as satisfied to publish our results, as Jenny mentions, in a blog. This actually may have been one of the most significant differences in terms of our goals for the paper.(Jenny- your take?)
I appreciate Jenny’s arguments for formal publication, and am perfectly happy to have taken this route for this foray. But the discrepancies among the reviewer perspectives confirmed my previous suspicion that the process is really hit-or-miss, and that validity is in the eye of the beholder and determined by the beholder’s context. That said, consensus among reviewers, good or bad, probably also would not have changed the fact that we felt that we had done what we could at this stage, that the ideas we were exploring would be sufficiently introduced into the broader conversation (and that was really the primary intent), and that it was time to offer these thoughts in whatever format was available. This was particularly true because, as Jenny notes, the conversations kept moving along, and by trying to adhere to publication and review processes, some windows for connection actually seemed to close along the way, while other topics (such as introversion) were/are rapidly becoming a focus in popular culture in ways that are perhaps less nuanced than would be useful in our discussions.
Ultimately, in that were were not trying to produce something definitive (and I suspect this expectation was at the core of Reviewer B’s objections), the review and publication process seemed cumbersome for a discussion that was based in and developed at least in part by an ongoing “feed” rather than deep-diving “search.”
Hi Frances – thanks for your comment – and to Carmen also for expanding on the process we went through and the things we thought about so clearly.
I agree with Carmen that Reviewer B’s feedback was helpful and as I mentioned very comprehensive. Not only did we receive full comments on IRRODL’s reviewer feedback sheet – but also comments directly onto our paper. As Carmen says, the comments from Reviewer B, in particular, but from all the Reviewers helped to focus our minds on what we considered important and what counted as justifiable criticism. There were instances, for example, where I felt that the Reviewer had not understood what we were trying to say. Of course, that could be because we did not express ourselves clearly enough, but, for example, we did feel in responding to the Reviewer that we needed to explain that our paper focussed on self-determination and not self-direction. However, the Reviewer’s comment meant that we tried to clarify this in the paper, which was a good outcome.
I agree with everything Carmen has said about access to libraries – and also that the formal publication route was more important to me than it was to her, so I’m grateful to her for going along with me. Would you have read the paper Frances (and thank you for your positive response) if it had just been posted here on this blog?
I also think Carmen’s last point is an important one. We suspected that our paper might not quite ‘fit’ and more than once wondered whether IRRODL would know what to do with it. So we are grateful that they didn’t reject it out of hand.
Thank you for sharing your thoughts and experiences, Jenny! I have a paper in the same IRRODL issue, and followed up with some of my thoughts here: http://www.veletsianos.com/2012/02/02/author-experiences-with-journals-and-publishing/
George – thanks for you comment. I have replied on your blog.
Jenny
This is a really thoughtful post. Indeed, the frequent lack of agreement in peer review analysis is indicative of poor reliability, that is, peer review is not a good measure of quality of any type. That leaves us with nothing more than a subjective process of gate keeping. But, you are also right that it really doesn’t matter if we have access to interactions that, “can raise my thinking and understanding to levels that I would not be able to reach alone”. There is a need for a social media of research between blogs and journals. MOOCs may be an important development as they grow and mature in process and capabilities.
PS I do think your paper’s final premise is correct. Anther measurement example is the neuropsychological assessment of cognitive differences leading to learning disability identification. The disability is not in the cognition, but in the connection; or maybe the restricted nature of some connections. (Note – Tests measuring the Big 5 or Myers Brigs could be questioned based on validity, but that is not the case with the high quality of current neuro-psych instruments)
Hi Howard,
Thanks for your response about the paper. The rapid developments in neuropsychology contributed significantly to our sense that the conversations on the topics we were discussing were “moving on,” and for this reason, I would agree that the Big 5 and Myers Briggs, while widely accepted and of interest to us in this context, may soon be regarded as transitional tools of understanding. In terms of understanding cognitive differences and new learning contexts, I think the exploration of connection is interesting both situations of potential restriction as well as in (using a term relatively new to me) neural propagation depth (Heylighen).
The implications of neuro-anything are going to take us into uncharted waters, whether in psychology or the (potentially related) investigation of “learning analytics,” which I could see being based not just on on external activity measurements but also possible measurements in cognitive activity (spooky, but possible). Ran across this the other day: “Neuroscience’s explanations and the traditional ones compete; they cannot both be right. Eventually we will have to choose between human narrative self-understanding and science’s explanations of human affairs. Neuroeconomics, neuroethics, neuro-art history and neuro lit crit are just tips of an iceberg on a collision course with the ocean liner of human self-knowledge.” http://onthehuman.org/2012/01/final-thoughts-of-a-disenchanted-naturalist/
It will indeed be interesting to see if a new “middle ground” for research may emerge from MOOC developments.
Thanks Howard for your comment. I have been thinking about your comment that peer review is no more than a ‘subjective process of gatekeeping’. George Siemens first wrote a lengthy post expressing his dissatisfaction with peer review in 2007 – http://www.connectivism.ca/?p=91 and then again in 2009 – http://www.connectivism.ca/?p=160 – so it seems that this dissatisfaction has been ongoing for years.
How could the process be improved? When I worked in Higher Ed, we were always aware of the difficulties associated with ensuring that students’ work was assessed accurately and fairly. I think there were 3 things that helped with this:
1. Knowledge of the subject and experience of marking (so a level of expertise)
2. Very clear marking criteria which had been discussed and refined many times for a given assignment
3. And perhaps most importantly – moderation. So from time to time we had to meet, justify and agree our marking with other tutors. A student would never have received 3 different marks on an assignment.
Of course peer review is not marking. The authors don’t have to change their paper in response to reviewers’ comments, if they can justify this response. But I am aware that there is a power relationship between authors, reviewers and publishers, which cannot be ignored.
It’s a pity there isn’t more dialogue between reviewers and authors. Have you any thoughts about how the process could be improved? How would you see ‘social media of research between blogs and journals’ developing?
Jenny, it is a great idea that you draw the connection between marking and peer review. Just like the honorable intention of “fair” marking and assessments tends to destroy what they want to measure, petrified rituals of intended quality assurance with journal reviews may stifle timely, lively and innovative exchanges, if they try to suppress blog publications.
Hi Matthias – thanks for your comment….
>petrified rituals of intended quality assurance with journal reviews may stifle timely, lively and innovative exchanges, if they try to suppress blog publications.
In my teaching career I often felt, particularly in Higher Education, that if the students failed, then the failure was more mine than theirs, or at least as much mine as theirs – and I have never understood how a PhD student can be failed. It really should never get to this point. If there has been enough on-going dialogue and peer-review, which is where the learning happens, a failing student can withdraw with grace and dignity long before it comes to the assessment.
Assessment/publication is so ‘final’ and failing in these circumstances is destructive. It is not the same as making mistakes as you go along, learning from them and adjusting your approach and thinking accordingly. In an ideal world by the time work is submitted for assessment/publication it should be a foregone conclusion that it will pass. The problem with open discussion of writing for journal articles is all those issues of plagiarism and copyright!
Hi,
Quite an extensive discussion of open access publishing, peer review, the issues of time delays, and other related concerns can be found in Hacking the Academy: Hacking Scholarship. http://www.digitalculture.org/hacking-the-academy/hacking-scholarship/#scholarship-jackson
Hello,
I think that in our field (I tend to call it learning technologies, but others call it educational technology) we tend to make generalizations that don’t serve us well. Peer review is an example of this. All of the papers that I have published have been improved as a result of peer review. Even in the cases where minor revisions were required. The same with others. Have I disagreed with some of the reviewers? Of course! Have I thought that some reviewers didn’t understand my paper or were trying to push their own agenda! Of course. The point that we have to realize is that alternative systems will neither be “fair,” nor necessarily without perils. This is not to say that peer review couldn’t be improved, but it’s to say that we need to think critically about these issues. Here’s what we had to say about peer review in a recent paper: “Peer review is the first example of how seemingly non-negotiable scholarly artifacts are currently being questioned: while peer review is an indispensable tool intended to evaluate scholarly contributions, empirical evidence questions the value and contributions of peer review (Cole, Cole, & Simon,1981; Rothwell & Martyn, 2000), while its historical roots suggest that it has served functions other than quality control (Fitzpatrick, 2011). On the one hand, Neylon and Wu (2009, p. 1) eloquently point out that “the intentions of traditional peer review are certainly noble: to ensure methodological integrity and to comment on potential significance of experimental studies through examination by a panel of objective, expert colleagues”, while Scardamalia and Bereiter (2008, p. 9) recognize that “like democracy, it [peer-review] is recognized to have many faults but is judged to be better than the alternatives”. Yet, peer review’s harshest critics consider it an anathema. Casadevall and Fang (2009) for instance, question whether peer review is in fact a subtle cousin of censorship that relies heavily upon linguistic negotiation or grammatical “courtship rituals” to determine value, instead of scientific validity or value to the field, while Boshier (2009) argues that the current, widespread acceptance of peer review as a valid litmus test for scholarly value is a “faith-” rather than “science-based” approach to scholarship, citing studies in which peer review was found to fail in identifying shoddy work and to succeed in censoring originality. The challenge for scholarly practice is to devise review frameworks that are not just better than the status quo, but systems that take into consideration the cultural norms of scholarly activity….”
From: Veletsianos, G. & Kimmons, R. (2012). Networked Participatory Scholarship: Emergent Techno-Cultural Pressures Toward Open and Digital Scholarship in Online Networks. Computers & Education, 58(2), 766-774.
Hi George – thanks for sharing the excerpt from your paper and the reference. It seems that peer review is of ongoing concern! Jenny
Hi Jenny,
I’ve been reading your article because I’m very interested in the theme, but it takes time to comment, I am so slow.
About different reviewers I have a simple comment: it is always good and useful to receive serious assessment or critics. Without feedback loops we could think that we are good at whatever we say. Sometimes it takes time to accept or understand the feedback ..
I will comment your article in my blog in near future I promise
Hi Heli – I agree that it’s alway good to receive critical feedback, so I’m looking forward to receiving yours, when you have the time. Ideally authors would receive a lot more feedback than they do. Jenny
Hi Jenny,
Here is my first response to your fantastic paper. Thanks to Carmen and you for the precious sharing.
John
Hi Jenny, the paper looks great, looking forward to reading it. I’ve also gotten widely divergent reviews – the worst was for AERA this year, where we suggested a symposium on MOOCs and P2PU – two reviewers both gave us very high grades and said highly relevant, looking forward to participating in your symposium etc… And yet we didn’t make it…
You mention all the ideas you discussed on the wiki – is this something that’s publicly available? It would be fun to see what you were talking about that didn’t make it into the paper.
Hi Stian,
With Jenny’s and Matthias’ agreement, I’m responding to your question about our work in the wiki, although I don’t speak for them in detail. As you’ll find cited in our paper, the wiki is probably best described as a more private or quiet space that allowed a “deeper, more chosen openness.”
The ambiguity and paradox involved in ideas of trust, identity, privacy and openness in connective and networked environments are certainly topics which meld with those discussed in our paper. While making the wiki publicly available is something we thought about, we have been left with a sense that, when considering the spectrum of “open” and “walled garden” networked and connective learning options, a small-group wiki is another essential space, and that shifting the parameters of this space after the fact might hinder trust and the productivity of “more chosen openness” in future interactions.
Will look forward to any thoughts you have on the paper!
Carmen
Hi Carmen, Jenny and all,
As I have also worked with Jenny and Roy in the past, using a number of wikis, in our CCK08 research, I shared some of the views of Carmen. There are however, tacit knowledge that one may say never easily shared while in open and public space, but could be possible when one is learning in a chosen open-but-closed space. This involves trust, privacy and a sense of openness, where not everybody is comfortable with, when sharing with weak ties.
John