I noticed that every comment had gray icons, even ones that I hadn't voted on yet. I dug around the javascript file and found that it was checking my IP against entries in the database. When I checked the table in the database, I noticed that my IP had been added to the entry for every comment. I'm not sure why this happened, but I deleted all the rows from the table and it looks to be working now. But I'm a little worried it may happen again.