This sounds kind of strange to me. I will try to present some of the facts of early medieval Jewish demographics, to the limited extent that I understand them. First of all, Jews continued to work in a range of occupations, including agriculture, into the 11th century. Haym Soloveitchik recently showed how deeply Jews were involved in wine production in Germany, into the 14th century. Though it doesn't seem to be explicit there (his emphasis is on their economic involvement), halachic considerations demand that at least some of the physical work was being done by Jews. [H. Soloveitchik, Principles and Pressures: Jewish Trade in Gentile Wine in the Middle Ages, Tel Aviv 2003, pp. 80-90 and passim]
Update: I need to add this quote from a short piece on the same topic that Soloveitchik published in English.
"And, in an era before Jews are identified with credit, neither did their
Christian neighbours see them as creditors, anymore than customers of today view Bloomingdale's or Marks and Spencer as creditors rather
than as merchants."
Haym Soloveitchik, 'Halakhah, Taboo and Moneylending', The Jews of
Europe in the Middle Ages (Tenth to Fifteenth Centuries): Proceedings of the
International Symposium held at Speyer, 20-25 October 2002, ed. Christoph
Cluse, Turnhout 2004, p. 301
Second, they lived in very small communities. These communities tended to consist of one or two extended families, which gave rise to the problem of convening a court that did not include relatives of either side in a dispute. [H. Soloveitchik, The Use of Responsa as a Historical Source: A Methodological Introduction, Jerusalem 1990, p. 30]
Third, nuclear families tended to be fairly small (Grossman thinks that Stow's conclusion, that the rate of growth was almost negative, but they were still not very large - four children). [A Grossman, The Early Sages of Ashkenaz, 2nd ed., Jerusalem 1988, p. 441]
Doesn't it make more sense to simply assume that all this created a very limited gene pool?