Apparently this has happened to others!
I have found some reasons this is happening to me. Peggy Osterkamp discusses these reasons in this post: too much draw, not enough slack in the weft, or the weft is pulled too tightly at the selvedge. It could be any of these reasons!
I also found a great discussion on this topic in a Weavolution forum.
Food for thought and I need to sit down and figure it out. I may just get this right by towel number 4!