It’s pretty self evident once you think about it. They are merchants who love emeralds (their currency) and they have big noses. Golems are a part of jewish tradition; it is said that the most powerful rabbis were able to summon golems, so the fact that villagers can summon them too also adds to the theory. It is also worth mentioning how the player is incentivised to build camps where villagers are kept in horrible conditions, and those who can’t work are discarded.