Penis size issues; 11.5 cm length 5 inch girth?

This is seriously awkward and rather shallow but I feel compelled to seek some advice. Firstly I consider myself to be rather intelligent and considerate. In general I am an easy going person but now I am increasingly feeling low and reclusive due to my penis size. I know this sounds ridiculous but there have been numerous instances where women have shown an interest in me but I build this defensive mechanism and fail to reciprocate. I realise yahoo answers might not be the right forum to discuss such issues but today a girl I would consider to be utterly beautiful asked me out for a coffee but foolishly I kindly declined. I just feel so insecure and fear the time when I will inevitably get really intimate with a woman. Is the size of my penis really a problem or am I overreacting? Thanks in advance.