I was reading this letter on Prabhupada'a categorical take on this issue. In my mind it settles it. Please read below.
You have asked what is meant by illicit sex. Sex should be used only in marriage for begetting nice children to raise in Krsna Consciousness. Krsna says in the Bhagavad-gita that I am sex life performed according to religious principles. Sex life for any other purpose means illicit sex. The use of contraceptive method for sex enjoyment is very sinful. Restraint in the matter of these four sinful activities is achieved by tasting the superior mellows of Krsna Consciousness.
- Srila Prabhupada letter to Mr. Suresh Chandra
18 June 1973