More on commenting and the publishing reform

Ingrid Daubechies asks on Math 2.0:


Suppose most mathematical research papers were freely accessible online.

Suppose a well-organized platform existed where responsible users could write comments on any paper […]

Would this be, or evolve into, a useful tool for mathematical research? What features would be necessary, useful, or to-be-avoided-at-all-costs?

This is not a rhetorical question: a committee of the National Research Council is looking into what could be built on top of a World Digital Math Library, to make it even more useful to the mathematical community than having all the materials available. This study is being funded by the Sloan Foundation.

There’s good stuff in the comments, especially here and here. I’ve said before that having comments on papers is not my highest priority, and I can think of other improvements on a comparable scale (significant, but without overhauling the whole system) that would add more value. So, in case anyone is interested and for future reference, here’s my take on a few specific issues that seem to come up again and again. In this post, I’ll stick to relatively small stuff, generally of the kind that could be set up initially by, say, NRC without much help from the community, as per the question I started with. There are of course bigger fish to fry, from the creation of new journals to rethinking funding mechanisms for science. But that’s for another time.

Errata and post-publication review. This comes up in every discussion of comments on papers, and for good reasons. If I could only improve one thing about the publishing system, this would be it. It’s a fact of life that published papers aren’t perfect, and it would be best for the community to have a system for fixing errors or omissions in the literature, including those found after publication. That could mean posting a corrected version of the paper, or else a separate errata pointing out which parts of the paper are incorrect (and how to fix them, if this is possible). I don’t even think that the basic principle here is controversial; it’s the implementation of it where opinions differ.

In traditional publishing, this is a complicated process involving essentially a new publishing cycle and a separate publication, which moreover appears years later and is not attached in any way to the original paper. Many authors (and editors?) avoid errata unless absolutely necessary, for example in cases where the mistakes are fundamental and unfixable. Others prefer to take it into their own hands and point out the corrections on the arXiv and/or their own webpages, on the perfectly reasonable assumption that the interested reader is more likely to find them there, and that it’s better to post them right away than wait for the journal.

With electronic journals, it should be easy and immediate to incorporate either errata or a corrected “post-publication” (and clearly marked as such) version of the paper, and archive it together with the published article. I don’t know how many journals do this already, but it certainly happens on the arXiv, where authors have been known to post corrected versions of papers after journal publication. I’m hoping for a similar shift in formal publishing. Acknowledging and fixing errors in published papers should not be a rare and momentous event, something only done in the most grievous of circumstances, but rather the normal thing that people do when someone points out that one of their theorems is missing an assumption.

I’m not in favour of errata via unmoderated comments, and that’s just from the point of view of the end user (ignoring all issues of civility and commenting etiquette). As a reader, I’m interested in having either a correct version of the article, or else a short document explaining clearly and concisely what corrections should be made. I’d like to be able to retrieve that information quickly without having to fish it out from long and argumentative comment threads that might or might not lead to it. I don’t need clarifications of misunderstandings that I never would have thought of in the first place, or long lists of minor typos listed one per comment, or quarrels about American vs. Canadian spelling and punctuation. (It would happen, too. I’ve had referee reports that took the time to point out for example that a full stop should be inside quotation marks. I didn’t care much, so I moved that full stop, then had it restored back to its original place by the journal copyeditor. – Aren’t you happy that I’m telling you all about it instead of staying on topic?) I’m sure that such comment threads could be fun to read, if one has time for it. But more often than not, I don’t.

I don’t know if the World Digital Math Library could do much about any of this. Errata should probably be the responsibility of the journals and the primary archivers, but there might be some sort of “best practices” guidelines on this.

The Evil Authors. At this point, someone always brings up the case of the Evil Authors who refuse to acknowledge and correct serious mistakes in their work. The only solution, if you believe them, is to point out those mistakes in public comments on the journal web page.

I would argue that this is a very imperfect solution, leading to Wikipedia-style edit wars if both sides dig in their heels, and not necessarily leaving the reader any wiser. It’s easy to forget that “authors” and “commenters” are actually the same people in different roles. The Evil Author will have commenting privileges, too. He could maintain his version in the comments. He could claim an error in someone else’s paper where there is none, then refuse to back down and retract the comment. Actually, it doesn’t even take the Evil Author for this to happen. There are enough discussions on the internet that get out of hand, even where everybody has the best intentions.

Editorial intervention is one way to resolve this. If there’s a dispute, or if the author is unable to respond (deceased, retired from mathematics), an editor gets notified, seeks additional opinions as appropriate, and publishes an errata (possibly overruling the author’s objections) or not, depending on the actual status of the paper. This is additional work for the editor, but I don’t expect that it would be anywhere as bad as moderating comment threads.

(I suppose that you will not find this satisfying if you don’t trust the editors, either, or if you’re thinking of a different system where the entire peer review, not just the errata, takes place in public comments. I’m all for creation of such journals, preferably in many different formats and flavours. It’s not clear to me that they will not have the same problems as traditional journals, but in any event, we need more choices, not fewer.)

In my experience, it’s rare for authors to actually defend incorrect mathematics. It’s far more common to admit the errors but argue that they are unimportant, or to claim that the missing argument is “routine” when it’s not. Even that does not usually get as contentious as matters of priority and attribution, and yes, if the omissions or misstatements are serious, I’d treat it on par with incorrect math.

The question is where to draw the line. The pro-comment argument starts with the Evil Author pushing clearly incorrect papers, then makes one huge leap to claiming that the only way to stop the Evil Author is to have free-for-all comments, then another to concluding that everyone should be able to post whatever they want on the official home page of the paper, for the sake of peer review and correcting mistakes. I disagree. Errata is one thing, but if you think that some argument should have been written up differently, or if you have an alternative one that you like better, that’s your preference, not a mistake in the paper. I’m not arguing that such contributions are not valuable: they can be, and there should be more and better ways to archive them, see below. That’s not at all the same.

Short contributions. One thing that would be immensely useful, and that the NRC (for instance) could actually implement, is an arXiv-style repository for short notes and contributions. That could include expository notes, discussion of results from other (published or not) papers, minor extensions, alternative proofs, examples, heuristic arguments, and so on. There is already a wealth of such material posted on individual webpages, and I think many of us would welcome a centralized, stable repository for it. Instead of having such notes posted as “comments” on some published paper (many notes might not have a natural “parent” paper, or might have more than one), I’d rather have an independent archive with a well designed labelling system and search function, so that I could find for instance all notes on a given topic, or citing a specific article. (Comments are fine, as long as the author has the option of turning them off. See below.)

Comments on papers. The model being proposed is, apparently, a huge database of papers with comment functionality. Let’s call it IMathDb, the Internet Math Database.

Right off, it strikes me as very different from pretty much any well functioning discussion forum that I know of, mathematical or otherwise. Those are organized vertically and chronologically rather than horizontally. When someone makes a post or starts a thread, the activity is immediately visible to all users, so they can flock there and engage in conversation. The site encourages discussion and has a critical mass of users who are active posters rather than just readers. Many of the posters are “regulars” who know the lay of the land, hang out on the site just to talk with each other, and sometimes intervene to moderate the discussion. The site is also active enough so that most regulars will check it at least daily, possibly more often when a subject of interest is talked about. This is important, as it keeps the conversation alive and assures that there’s a range of opinions represented, rather than just one or two individuals talking to each other.

I’m mentioning this because I’d like for there to be more discussion boards for mathematics, with different scopes, audiences, cultures. I’d be happy to see internet-based math hangouts without Math Overflow’s reputation scores, with original posts not restricted to questions, with less restrictive policies of adhering to the topic (what’s off topic to you might be a very interesting line of inquiry to me), with different moderation systems.

The IMathDb, with its horizontal structure and comments attached to papers, sounds instead more like IMDb, or Yelp, or RateMyProfessors for that matter. The structure of the site would encourage posting “reviews” and “complaints” rather than having actual conversations about mathematics. Customer ratings on a scale from 0 to 5 stars would not look out of place, although I would hope that this is not being proposed.

The first problem with this, at least for me, is that I’d rather have conversations about mathematics then read other people’s reviews of papers. Moreover, when I talk about mathematics, I’d rather not limit myself to talking about one paper at a time, but discuss a topic instead, or connections between several different papers.

The second problem is that such a database could be easily misused. The reviews could be taken at face value by deans, provosts, or hiring and promotion committees, regardless of their reliability. The comments could be counted and tabulated. There are too many precedents for this. I’ve been on a prize selection committee where the evaluation process was based in part on MathSciNet reviews. I pointed out that they had never been intended for that purpose, but was overruled. There have been similar problems with the Featured Reviews. In 2008, the International Mathematical Union produced a document on using citation statistics, with emphasis on (and many examples of) how not to use them. I expect that a similar (but probably longer) document will soon be needed to explain internet comments to scientists, university administrators and program directors at funding agencies.

Finally, and briefly, because this is getting too long: the matter of author’s participation. I’m happy when people talk about my papers, online or otherwise. I really am. But I don’t especially want to see all those discussions, let alone respond or moderate, and conversely, I’m guessing that people might want to be able to have such conversations without me watching over their shoulder. Decentralization rules.

The problem with centralized comment pages is that they present additional obligations for the authors. Most proposals for journal comment pages (that I’ve seen, anyway) assume that authors would participate in the discussions and would help moderate the comments. I’m guessing that IMathDb would be seen similarly. I’ve said already that other ways of talking about mathematics work better for me. Additionally, I have a full time job already. I’m not able to offer individual tutoring to everyone who might want to read my papers, or to moderate and participate in internet discussions on a website where I never volunteered for it, possibly on a schedule that conflicts with mine. We can’t keep accepting more commitments just because something or other might be, in theory, a good thing to do. Because in practice, it won’t get done. I’ve seen it many times.

Hope that all this helps, somehow.

Endnote: There is much else that doesn’t fall under the question at hand, but warrants attention anyway. In particular, I have not mentioned gender issues at all in this post. Don’t worry, more on that is forthcoming.

Advertisements

Comments Off on More on commenting and the publishing reform

Filed under academia, journals, mathematics: general, publishing

Comments are closed.