Peer review is obsolete

In a provocative mood of evening musings:

I don't think peer review is working particularly well for showcasing the best work in higher-ranking journals. I suspect there are common flaws in duplications of work, overstatements, etc. A small number of overworked reviewers with little immediate feedback will inevitably let things slip through - especially for more interdisciplinary papers, which have a broader range of things that might go wrong outside the reviewers' specialties.

Having a reviewer check work is terrible efficiency, now that typesetting programs allow for more intermediate steps to be easily incorporated, print page limitations are no longer an issue, and the background code can be bundled along with things. And papers should be "living" - corrected and updated continuously, rather than just in one early cut.

What do we need? Central storage repositories for projects that can be trusted to be long term resources as academics migrate from post to post or leave for different fields entirely. Filtering based on citations and references - what is actually used? The ability for social network style "liking" of papers if they explain things cleanly.

Go deeper - can we have people sign their names to checking a particular derivation, reproducing a result? Then there is public attribution of those who have checked things carefully and staked their names on it. "Liked" "Used" and "Reproduced" can all be done based on interest. Good work should rise by all three attributes.

Still useful are editors (professional as well as impromptu) to collect sets of "most interesting" papers in particular subjects and curate histories of topics. Most interesting are not necessarily the most recently created - new shifts in subject matter can bring renewed interest to decades-old work.

Papercraft spacetime: An equatorial slice through Schwarzschild

Take a sheet of paper and ignore it's thickness. Welcome to a slice of flat spacetime! It's even persisting through time, so we have three of the dimensions in the paper. We've only ignored one.

Now think about a equatorial slice through Schwarzschild. There's curvature: the usual coordinates use a circumferential radius coordinate, so r is the distance around a circle at that point divided by 2π. On the flat paper, if we draw a bunch of concentric circles around a point, each one has a distance r to the centre and a circumference 2πr. And then the distance between a circle at r₁ and a circle at r₂ is r₁-r₂. But in Schwarzschild it's different: the distance between two circles is

which is longer than r₁-r₂, by a little bit far away from the center, increasing toward the horizon.

So let's make a slice of schwarzschild out of paper!

Version 1: This is not perfect, but I thought that the programming environment Processing would be good for this and some other little ideas I have.

I figured that I would divide the spacetime slice around the centre like a pie and figure out the shape of each chunk, then tape them together. So the first step was to figure out how I might draw a slice in flat spacetime:

void setup()
{
  size(400,400);
  background(255);
  smooth();

  int segments = 16;
  int radii = 10;
  int centrex = 200;
  int centrey = 350;
  float spacing =30;
  stroke(128);
  for(int  i= radii; i>0; i--){
    arc(centrex,centrey,2 * spacing *i,2* spacing*i,-PI/2 - PI/segments,-PI/2+ PI/segments);
    line(centrex-spacing*(i-1)*sin(PI/segments),
    centrey-spacing*(i-1)*cos(PI/segments),
    centrex-spacing*i*sin(PI/segments),
    centrey-spacing*i*cos(PI/segments));
    line(centrex+spacing*(i-1)*sin(PI/segments),
    centrey-spacing*(i-1)*cos(PI/segments),
    centrex+spacing*i*sin(PI/segments),
    centrey-spacing*i*cos(PI/segments));
  }
}

And then, keep the arcs to make the same circumference of circle, but add extra linear spacing so that they follow the Schwarzschild relationship (note: math needs to be double checked)

void setup()
{
  size(400,800);
  background(255);
  smooth();

  int segments = 8;
  int radii = 20;
  int centrex = 200;
  int centrey = 700;
  float basespacing =20;
  float horizon = 80;
  stroke(128);
  float oldx = centrex-(horizon+ radii * basespacing) * sin(PI/segments);
  float oldy = centrey-(horizon+ radii * basespacing)*(cos(PI/segments));
  for(float  radius = horizon + radii*basespacing; radius >= horizon; radius = radius - basespacing){
    // The spacing is modified to shift circumference 
    // circles up so they are the appropriate distance from the horizon
    float horizondist = sqrt( radius * (radius - horizon) )- 0.5*horizon*log(horizon/radius) 
        + horizon * log(1 + sqrt(1-horizon/radius));
    println(radius + " "+(radius-horizon) + " "+(horizondist));
    arc(centrex,centrey-horizondist+radius,2 *radius,2*radius,-PI/2 - PI/segments,-PI/2+ PI/segments);
    float newx = centrex - radius * sin(PI/segments);
    float newy = centrey-horizondist+radius - radius * cos(PI/segments);
    line(oldx,oldy,newx,newy);
    line(-oldx+2 *centrex,oldy,-newx+2*centrex,newy);
    oldx = newx;
    oldy = newy;
  }
}

But this isn't quite right, because the lowest arc, which is the horizon, at which point the proper distance between two radial coordinates blows up - doesn't need to be an arc. The paper will be curved down into a funnel shape here, and a flat bottom of the paper will curve around with all the pieces stuck together. So some of the arc-ness is coming from the change in angle of the sides in the curved space time, and I need to take that into account.

Curation vs. Review

Back to the theme of curation, a nice post from Science in the Open titled It’s not information overload, nor is it filter failure: It’s a discovery deficit.

The most relevant paragraph for me:

Filtering before publication worked and was probably the most efficient place to apply the curation effort when the major bottleneck was publication. Value was extracted from the curation process of peer review by using it reduce the costs of layout, editing, and printing through simple printing less. But it created new costs, and invisible opportunity costs where a key piece of information was not made available. Today the major bottleneck is discovery. Of the 500 papers a week I could read, which ones should I read, and which ones just contain a single nugget of information which is all I need? In the Research Information Network study of costs of scholarly communication the largest component of publication creation and use cycle was peer review, followed by the cost of finding the articles to read which represented some 30% of total costs. On the web, the place to put in the curation effort is in enhancing discoverability, in providing me the tools that will identify what I need to read in detail, what I just need to scrape for data, and what I need to bookmark for my methods folder.

And now, to go reread that article I was supposed to review this week...

John Wilbanks talks about opening science

Two links liveblogging a talk about open science:

Ethan Zuckerman and David Weinberger at Joho the Blog

Math and physics are, as mentioned, probably more open as far as sciences go. But it is remarkably hard, and I am a long way from, living up to my own ideals of open research.

Education in an online world

A post from Daniel Lemire's blog plays into the ideas I have about the future of education:

Getting serious about online teaching: "In this new online world, professors are not content providers. They provide structure and motivation. They are role models. And most importantly, by their reputation, professors can provide certification."

See also the Edge World Question Center response by Martin Rees, which Daniel Lemire also mentioned earlier (in a post with lots of interesting comments):

A level playing field: "Traditional universities will survive insofar as they offer mentoring and personal contact to their students. But it’s less clear that there will be a future for the ‘mass university’ where the students are offered little more than a passive role in lectures (generally of mediocre quality) with minimal feedback."

Some thoughts about this vision of the future:

An intensive one-on-one experience doesn't scale to hundred-student undergraduate lectures. How does this sort of "weed-out course" translate into this future university? The content of such courses can perhaps mapped to some required courseware, but will students want to risk much time spent on the idiosyncratic requirements of a particular teacher who may not accept them? Institutionally agreed upon curriculums could still be important.

Project portfolios would probably be important credentials. How will students get feedback from projects? What other metrics will be useful? I suddenly picture the university providing an online gaming-style system of students charting their progress and achievement points, hanging out in chatrooms looking for teammates on the next project quest. World of Warcraft as university model. You don't get a degree, but you can link to your profiles on different university-servers.

Stuck on this carousel my little eye can catch one-million-year-old light.

A quote I highlighted many years ago, a footnote which resonated particularly after a high-school sweetheart complained that I over-analysed a sunset:

Poets say science takes away from the beauty of the stars — mere globs of gas atoms. Nothing is "mere". I too can see the stars on a desert night, and feel them. But do I see less or more? The vastness of the heavens stretches my imagination—stuck on this carousel my little eye can catch one-million-year-old light. A vast pattern—of which I am a part… What is the pattern or the meaning or the why? It does not do harm to the mystery to know a little more about it. For far more marvelous is the truth than any artists of the past imagined it.

–Richard Feynman, The Feynman Lectures on Physics

Making science

Penelope Trunk says the first thing to do when starting a blog is to pick a topic. I have been thinking about blogging for a while, to balance some of my prodigious appetite for information accumulation with synthesis and output, but choosing a sustainable focus is difficult. Here, though, is an attempt, in three scenes:

Scene 1

Very shortly after I arrived in Germany for my postdoc, armed with my contract letter from the institute, I went to set up a bank account. Despite the banker's tentativeness about speaking English, things went smoothly. After everything was set up, though, the banker looked curiously at my contract, tilted her head to look at me, and asked “But, what do you make?”

I like this aspect of German, using the verb machen to talk about what one “makes” rather than what one “does” for a living (though of course the verbs aren't exactly one-to-one). I didn't have a quick response at the time—stammer, point at the contract again, grin at her suggestion that I make neutron stars—but the question stuck with me in a more philosophical sense: what is the result of my work? What do I create?

Scene 2

I'm at a workshop dinner for Peer 2 Peer University, which I was invited to via the amazing Pippa. People are talking a lot about the future of education, expanding access, going beyond existing institutions. When they ask me what I do I say that I work for The Man. As someone embarking on a so-far fairly traditional academic career it's a good prompt to think about what is valuable and useful in existing institutions, and what form those things might take ten plus years in the future: I end up talking to people about curation and mentorship, as well as accreditation and infrastructure.

Berlin has a cool community of people interested in making stuff and creative commons. It's good to talk to people doing interesting things outside of academia, and engaging with them complements various thoughts about the future of science that I've been following online. I'm always fascinated by discussions of meta-research, e.g. I started following Michael Nielsen's blog after finding his essay on the principles of effective research as a grad student.

Scene 3

Some LIGO friends of mine have a tradition of posing for pictures at conferences: everyone with a fist in the air, hailing the camera with a hearty “For SCIENCE!” Working in science means being part of an interesting international community, and I feel an allegiance to that community. I really like my work! I will talk about my own research at the drop of a hat—to friends, to strangers at bars, over dinner, at parties… And I am an idealist; trying to guide my own research choices by what leads to the best science.

What is science, then? Here's a working definition I suggested in conversation with another interesting person met in Berlin: the collective communal understanding of reproducible results. Communication is important in my view of science: “It's not really knowledge until you've communicated it to others in your field.” and “the public pays my salary, the public owns my research; my job is to make it well known to the public”. I want to work on doing this.

Denouement

So: science. I want to talk about the science I do, what I make, the particular scientific problems and contexts that interest me, and why. And I also want to talk about meta-science: how we create science, how we learn and communicate and teach, and how these things might change in the future.