Tuesday, August 14, 2012

Neandertal inbreeding or incomplete lineage sorting?

Update:  Good grief, I just realized the title of this post should be "interbreeding" not "inbreeding."  Inbreeding makes no sense.  Sorry about that.

Via Nature's blog, we learn of the pending publication of new papers examining the curious observation of Neandertal alleles in modern Eurasian genomes.  To review, when the Neandertal genome was published, researchers discovered a curious pattern: Neandertal gene sequences matched Eurasian sequences significantly more often than African sequences.  This was interpreted as evidence of interbreeding between the Neandertals and the ancestors of modern Eurasians.  There are other possible explanations, one of which is incomplete lineage sorting.  Here's how that works:

For any population of critters, the gene pool (all the copies of genes in each organism in the population) is a mixture of genes that are very similar and some genes that aren't all that similar.  When the population divides (as it does during speciation), the two new populations still have some genes that are very similar (implying a recent divergence) and some that aren't so similar (implying a more ancient divergence).  If you want to reconstruct speciation events based on gene similarities, you have to be careful that you take into account this pattern of lineage (population) sorting, or you might come up with a really wrong answer.  Some genes will show a close relationship between the populations, and some will show a more distant relationship.

When the original Neandertal genome paper came out, all the attention went to interbreeding to explain the similarity of Neandertal and Eurasian genes, and I think that was well justified.  Since incomplete lineage sorting is a random process, I would not expect to see any modern human population significantly more similar to Neandertals than any other modern human population.  Modern human populations diverged well after the Neandertal divergence, so whatever genes were still very similar to Neandertals should have been evenly divided.  Even more interesting is the geography: the ancestors of the modern humans with Neandertal-like genes come from the same geographic region as the Neandertals.  If incomplete lineage sorting is to blame for Neandertal-like genes in modern humans, that geographic pattern would just be a weird coincidence.  That's not to say that one couldn't make a case for incomplete lineage sorting, but we would need some new evidence to really seal the deal (such as new samples of Eurasians that don't have Neandertal-like genes or Africans that do).

According to Nature, PNAS was supposed to publish a new paper this week arguing for the incomplete lineage sorting model, which they did.  The press have headlined these results as Neanderthal breeding idea doubted, Neanderthal-human similarities not due to mating, and Humans didn't breed with Neanderthals.  Reasons to Believe is going to have a field day.  The new paper by Eriksson and Manica is largely a theoretical study, modelling what might have happened under a particular population divergence.  According to the abstract,
We show that the excess polymorphism shared between Eurasians and Neanderthals is compatible with scenarios in which no hybridization occurred, and is strongly linked to the strength of population structure in ancient populations.
Compatibility does not equate to "Neanderthals and modern human ancestors didn't interbreed," although it certainly raises questions about the possibility of alternative explanations.

Meanwhile, the original authors uploaded a manuscript to the arXiv server that uses linkage disequilibrium to establish a probable timeframe for the interbreeding.  Linkage equilibrium happens when recombination breaks up blocks of genes that are connected on the same chromosome.  Linkage disequilibrium is the condition of genes that are co-inherited more often than we would expect by chance, because recombination hasn't broken them up just yet.  Given what we know about human recombination, Sankararaman et al. estimate Neandertals interbred with modern humans around 47,000 to 65,000 years ago.  That implies that the Neandertal-like genes in modern humans are of recent origin in the modern human genome, which is consistent with interbreeding but not incomplete lineage sorting.

At this point, I'm still pretty strongly in favor of the interbreeding model, especially with the new linkage disequilibrium analysis, despite the theoretical results of Eriksson and Manica.  I'll close with this quote from one of the authors of the linkage disequilibrium analysis:
“It’s been an issue for several years. They were right to work on this,” says Reich. But now “it’s kind of an obsolete paper,” he says.  (from Nature's news blog).
Feedback? Email me at toddcharleswood [at] gmail [dot] com.