OK, I'm going attempt to clear this up because the answer seems obvious to me.
Scenario 1
Poster X has a rating of 4.4. You decide that the input that Poster X is contributing to the forum is worth 4 stars. By giving this person 4 stars you are lowering their average rating, thus reducing their points. The more votes a person has, the more votes that will need to be cast to alter that persons rating.
Scenario 2
Poster X has a rating of 4.4. You decide that the input that Poster X is contributing to the forum is worth 5 stars. By giving them 5 stars you're raising their average rating, thus increasing their points. But wait, Rater X has just decided that the input that Poster X is contributing is only worth 2 stars.
Your vote was .6 stars above the average causing the point to increase.
Rater X's vote was 2.4 stars below the average causing the points to decrease 4 times more than your votes increase, causing there to be a net loss of points.
If you and Rater X were reading the post at the same time, but Rater X rated Poster X before you rated Poster X, then the net amount of points would be reflected after your vote was placed because you had not refreshed the page prior to making your vote.
Hope this clears it up,
Brian