All A Twitter in the Annals of EM

The Annals of Emergency Medicine has published a few feature articles on Web 2.0, FOAM and social media over the past few years (such as this). The latest is titled ‘Social Media and Physician Learning: Is it All Twitter?’ (free PDF). Michelle Lin has summarised the key points from this article over at ALIEM, and asked me for my comments. So here is my perspective.

The article itself features nothing that hasn’t already been discussed over and over on countless FOAM blogs and Twitter conversations, but it is always nice to see these issues being packaged for mainstream dissemination.

Firstly, yes, we all know that FOAM and social media lacks an ‘evidence-base’ — just like much of medicine and almost all of education, there are no RCTs or similarly high level studies demonstrating benefit. This is partly because the use of social media in medical education is relatively new, partly because it is difficult to assess, and partly because those who find it useful just want to get on with it. Attempts to define the utility of social media and FOAM in medical education are to be welcomed.

There is a discussion of peer review, and whether it should be pre- or post-publication. I believe the traditional peer review process used for medical scientific publications is flawed (see  Time to Publish Then Filter?The Wisdom of Crowd Review and Peer review: a flawed process at the heart of science and journals). Yet, until now, in the context of scientific publication, peer review has reminded me of Churchill’s assessment of democracy: “… the worst form of government except all the others”. In my mind, post-publication peer review must be explored — and Michael Callaham’s commentary at the end of the ALIEM post makes a lot of sense on the whole. In some ways, post-publication peer review is already happening in the FOAM world using social media. Blogposts, such as this from the Intensive Care Network, have led to corrections in high profile journals such as the New England Journal of Medicine. Online journal clubs abound and there are entire blogs dedicated to critical appraisal (such as Emergency Medicine Literature of Note and EM Nerd). However, post-publication peer review needs to be formally integrated into the medical scientific publishing model. Indeed, Pubmed Commons is now being trialled.

I will pull up Michael Callaham on one point though — I think he underestimates the potential impact of social media when he states:

“The more I think about it, I don’t see social media as providing anything truly new in terms of approach or content. It is simply a much improved technological mechanism for providing a prompt public forum for the thoughts of many who did not get to participate in the process before, and who previously did not overcome the barriers of slow correspondence and publication. Increasing their participation would be a good thing.”

While I agree that free and open debate — the anvil on which ‘Truth’ is forged — existed before social media, my experience tells me that social media transforms this utterly. We must not underestimate the impact of technology (after all weren’t the discovery of fire, the invention of the wheel and the Gutenberg press simply technological advances?) Social media has revolutionised my capacity to track and participate in cutting edge controversies and discussions of the merit of breaking research. It enables me to exchange ideas with some of the brightest minds in my field anytime and anywhere. This is amplified even more so for those in remote locations and for those with limited resources.  Importantly, whether you like social media or not, it is here and it is here in a big way. Increasingly social media is integrating into day-to-day living and anyone interested in the exchange of information ignores it at their peril. The game has changed.

Michelle also hits the nail on the head with her comment that scientific publication and knowledge translation are different things (together with my colleagues Paul Young and Dashiell Gantner, I have an editorial for Critical Care and Resuscitation on the role of social media in knowledge translation currently in press). FOAM is not scientific research — it is a way of disseminating, discussing, dissecting and deliberating over the products of that research — as well as issues where research findings do not apply, or do not exist. FOAM is more akin to editorials and commentary articles in journals  (which are usually not peer reviewed but written by invitation) than to the research articles themselves. Pre-publication peer review can be of use in FOAM for fact checking and so forth, but it also runs the risk of diluting arguments and opinions before they’ve had a chance to live or die in the melee of truly open discussion. What we should strive for in FOAM, however, is scholarship — appropriate referencing to journal articles and the FOAM that came before.

Over to you.

PS. Impact factors suck!

 

Print Friendly

Comments

  1. says

    A really good post touching on a point I have thought about a lot lately, due to my doctorate and my work. It is good to try and rationalise as much as possible the decision we take in education, especially when social media (and FOAM) are used. However, when I hear “lack of evidence” I feel we are trying to enclose something fluid, dynamic, uneven, hybrid -- such as FOAMed within the walls of limiting epistemological frameworks. What is evidence? Why do we need to evaluate “technology based educational interventions” by using numbers? It appears that the only acceptable way to do MedEd research or report on educational experiences is to ground them with an evidence based medicine hierarchy.
    We do need to know what works and what doesn’t, but this can be done without trying to transform behaviours into numbers, through the support of pedagogical theory, through sharing, discussion and confrontation.

    • says

      I echo your thoughts about making FOAM fit into previously created principles and frameworks. That being said, I think hypothesis-driven assessments of what work and doesn’t is reasonable. Perhaps a more qualitative, ethnographic methodology approach can address these questions of “why” and “how”? Now THAT’d be cool, wouldn’t it?

  2. says

    Thanks for this eloquent response to my and Mike’s comments. It’s a conversation that we’ve had before and will continue to have until we reach a critical mass and tipping point for culture change. I think we’ve cross at least one small hurdle by getting Mike to engage with the FOAM community. I’ve been secretly hoping to get him on the blog for the past 4 years. No luck until now, despite us working in the same department! Hopefully it won’t be another 4 years before we get past another hurdle…

    P.S. Love the new LITFL redesign. Awesome!

  3. Michael Callaham says

    I stand corrected about fire and the wheel! You are right. It’s not a small thing.

    Re impact factors, I certainly do agree. They have so many flaws. First of all it’s only based on citations of a paper by other authors, a very narrow measure of impact. Page views, downloads, etc. are much more of an indicator of the broader audience. But the impact factor particularly makes no sense in today’s world -- that is, the journal impact factor, which just means the average number of citations received by all the core articles in the journal. Even the most elite journals have lots of articles that are never cited by anyone; the less elite journals typical have half or more of their articles never cited by anyone. Ever. So what we really should be seeing is article-specific “impact factors”, which would also be much more meaningful to the author (and promotion committees). They’re pretty easy to calculate too if you have access to a medical library and the ISI Web of Science. I’ve found it pretty interesting to calculate my own, and been surprised by which articles have turned out to accumulate large numbers of cites, and which do not.

    • says

      Thanks Michael.

      Clearly the comparison I made was a gross over-exaggeration designed to elicit a response… Yet the ‘quantitative’ impact of social media on the availability, speed of access, and dissemination of ideas, together with the interaction it facilitates, seems to create a ‘qualitative’ shift -- although from a reductionist viewpoint it is still just sharing and discussion as we have always done, it now feels like something ‘different’ going on.

      Impact factors for individual papers and authors would clearly be an advance. A shift away from the journal impact factor could greatly change the landscape for medical publishing, especially for revered journals like NEJM. Also the true measure of any scientific endeavour in medicine is effect on clinical practice… There remains a sizeable research-practice gap and translation probably has little to do with the quality of scientific research. Furthermore, such a focus neglects quality research that confirms that what we are already doing is right.

      Somethings are hard to measure!

      Chris

  4. says

    A really nice post leading to constructive and insightful comments (I note that these discussions are regularly reaching ‘mainstream’ journals now and hopefully are here to stay….)

    Annalisa’s comments are really important. Epistemology essentially looks at what knowledge is and how it is encapsulated for a particular subject. Annalisa points out we are not sure what all the facets are in the process of the evaluation of Social media/FOAM. We are stuck using one epistemological approach but there maybe many others. I’d take that further and say we are sometimes confused about whether we are measuring FOAM, Social media or both. Jan Green’s article discusses ‘online collaborative learning’ versus traditional mediums. I’ll stoke up a bit of a debate by saying that social media actually wasn’t defined in the article. From my point of view social media allows an accessibility, that FOAM defines the scope of, but being deeply provocative here, you are still learning through text and images on a screen. Just what is it that we are saying is better than what….?

    This is a distinct process from peer review which again is a topic that just won’t go away! I think the FOAM community should be aware that however passionately we feel about the problems with pre-publication review very few of us have taken a plunge to just publish traditional academic literature entirely in a post-publication context {would be grateful to for examples other than Simon Carley’s http://stemlynsblog.org/apls-estimation-formulas-do-not-safely-predict-weight-in-uk-children-st-emlyns/}

    I do feel very aggrieved however with Dr. Mesko’s comments in the Jan Green article:

    “These processes should be totally separated to make sure content that is academic must keep its academic nature”

    Hope I am not taking this out of context but why is my academic opinion any less outside of a journal rather than within it? i.e I am taking this to read those who use FOAM are less academic…..

    But as I said in a previous blog on the subject http://rolobotrambles.wordpress.com/2013/01/19/peer-review-pointless-perfunctionary-or-practical/

    “Academics will continue to discuss peer-review into the next decade

    IF #FOAMed is good enough it simply won’t matter”

    Damian :)

    • says

      Interesting comments. I still feel like we may be lumping too many things together when discussing quality assessment and peer review. In my mind, there’s original research, and then there’s medical education (also known as knowledge translation). In an oversimplification, original research aims to discover “truths” in medicine, and medical education aims to effectively teaching these “truths” to frontline providers to improve patient care. To me, the FOAM movement primarily targets at accelerating the latter. So why are we trying to apply a one-size-fits-all solution (i.e. peer review) to both. Perhaps we look at comparing FOAM content to print textbooks instead? They undergo a less rigorous peer review and yet are often accepted as lore. Why is that? Plus open-access blogs and podcasts do a much better job at accelerating the dissemination of medical knowledge. Let’s keep research and education separate.

      As for Damian’s point about defining and differentiating social media and FOAM, I struggle with that sometimes. For me, I see SOCIAL MEDIA as digital tools that facilitate more real-time communication and collaboration. I see FOAM as a movement which is a subset of the broader networked idea of OPEN EDUCATIONAL RESOURCES (OER) which encompasses the broader (not just medical) education community. It’s been around for a long time, has lots of amazing resources, and has historical lessons that we can learn from. I’m just scratching the surface about OER and finding many parallels. I hate reinventing wheels and so I’m starting to read more about it.

      As for traditional peer review, I actually do still think there is a role for this in the scientific community for original research despite its flaws. I think I’m in the minority within the FOAM community, but that’s just my opinion until a better, feasible, valid solution is made available. It’s certainly better than nothing.

      My 2 cents.

      • says

        Thanks Michelle -- entirely agree with concern over amalgamation of assessment and peer review (I need to find the right way to draw the Venn Diagram though!).

        The difficulty is that some FOAMed initiatives are not a translation of evidence base research into practice they are a real-time discussion and breakdown of ongoing clinical practice. This is challenging area in which definitions of context and content have yet to be, or are only just being, formed .

        Concede the SoMe/FOAM differentiation is not black and white. I would be interested to know where you think OER sits in an SoMe circle. I agree FOAM is a good contender for sitting within an OER domain. I wonder how OER sits within definitions of SoMe as it it is not beholden to one particular type.

        These discussions are always useful to move the paradigm of FOAM forward!

      • says

        Hi Michelle, Damian and all the other discussants

        The continued rattling on about peer review is not driven by FOAM users as such -- apart from ALIEM and BoringEM, few FOAM creators have experimented with it. It is simply a reaction to our critics, and I agree I don’t think it needs to apply to FOAM and I don’t think the criticism is valid. Incidentally, the peer review we’ve seen on ALIEM and BoringEM has not really been traditional peer review -- more like expert commentary on an article and editorial input.

        As for bagging peer review -- there is still a role for it due to it being the Churchillian default. But most people involved in scientific research and publishing agree that it IS flawed and would like something better.

        Clearly FOAM overlaps with OER. But it does have distinctive features: it has developed into a community in it’s own right, many involved are not ‘officially’ educators and it is thoroughly integrated (but distinct from) social media. Clearly history tends to repeat, and we are ignorant if we do not seek to learn from what came before.

        Knowledge translation is an important part of education and part of FOAM, but only part. As I wrote in the post, FOAM and education also deals with issues for which there is no research or which research does not apply. Tacit knowledge sharing is a critically important part of FOAM, for instance.

        Great discussion everyone

        Chris

        • says

          Oh…
          ..and I actually believe FOAM is more akin to editorials, commentaries and perspective pieces in medical journals than textbooks -- though there is definitely overlap and exceptions. Most FOAM creators are not seeking to define a curriculum or a body of knowledge, most of the topics are controversial and deal with recent research or other issues. Also, with respect to knowledge translation, textbooks are woeful and terribly out of date. FOAM is essentially editorials and commentaries with even less lag time, and no need for an Editor-in-Chief’s invitation!

          Chris

Trackbacks

Comments