Fairgrounds backers turn rhetoric ugly as some begin to wonder if anti-referendum effort looms