Hah. Yes. The one-dimensional up/down mechanism leaves a lot to be desired. Your response could be so wildly engaging that literally everyone on HN has voted on it; so controversial that the masses and the bots are evenly split between loving and hating it; and so immaculately well-written that no one can think of anything to add or to counter it with which wouldn't embarrass themselves. That's usually what I tell myself after I get a single downvote and no reply ;)
I think eventually HN should see something getting a lot of votes and count everything as an upvote. (presumably if it is bad long before then a mod should have noted and removed the bad post)