Sand worms.
I was getting severely out fished by a friend at first and took notice. The difference was that he would jig his bait occasionally.
I started to do the same thing and noticed that often when I went to lift (jig) the bait the spot was already on the hook but I never felt the fish take it. I'm not sure if this is a "spot" thing or if the fish were just lethargic.