Yeah. It is kinda annoying. Like writhing and puking, dizzying sensation.
And I believe, If your eyes are closed and then put your finger in that area, it won't effect you. So I think it is somekind of us thinking "danger" as we can see the finger obstructing our view and nearly poking our eyes.
It happens strongly when you feel very conscious about it (trying to watch the pointed finger directly). I tried a lot to push my glasses back with one finger and I don't feel it.