Why does my left crank arm keep becoming wobbly?

I've replaced my left crank twice now on my GT Aggressor XC3 09. The axel is wearing the bit where it fits onto the axel away very quickly. I know they can become wobbly with over/under tightening the bolt. I did about 20 miles on it and it was loose once again, it's becoming very frustrating. At first the arm seemed solid but it later became loose. Can getting grit and dirt make it come loose? Aghhh please help!