Der Cody:

There are grave/mortal sins, venial sins, and things which may not be sin at all but are not fitting for a Christian. I have no idea what you have said and thus cannot speak to it directly; I don not know what "drug references" means. I would guess this would fall into the category as unfitting.

St. Paul tells us in 1 Corinthians 10:23, "'All things are lawful,' but not all things are helpful. 'All things are lawful,' but not all things build up." And in 1 Corinthians 6:12, "'All things are lawful for me,' but not all things are helpful. 'All things are lawful for me,'^] but I will not be enslaved by anything."

These passages imply that while something might be technically permissible, that "something" is not prudent for a Christian to do because it is not helpful to their spirituality, does not buildup their faith, or causes them to be habituated to something.

This sounds like what is happening to you. You described a habit of improper jokes. St. Paul does say, "Let there be no filthiness, nor silly talk, nor levity, which are not fitting; but instead let there be thanksgiving" (Eph 5:4). 

As far as whether or not what you are doing is venial, why does it matter? We are not to sin at all (venial or grave). We are also not to act in a way unbecoming to our Christianity, even if there is no sin. So it does not matter. It is something you need to work on stopping.

As for Confession, the Church requires us to confess grave sins. But, the Church also says that venial sins are good to confess too. Confession is a healing and strengthening sacrament. Thus, it is good to bring to Confession venial and even unbecoming acts that may not be technically sin.

Force of habit can reduce culpability, but never decide that for yourself. It does not matter where lies your culpability, you need to work to overcome the improper habit. Do not try to discern where the lines are drawn. Only God knows for sure. If it is improper, confess it and work to stop it.

God Bless,
Bro. Ignatius Mary