The term "invincible ignorance" was probably first used by Pope Pius IX in the encyclical Quanto Conficiamur Moerore (1863). The concept, however, can be traced as far back as Origen in the third century.
In the fourth century St. Augustine wrote:
Although there is One present everywhere who in many ways through His creation beckons to hostile servants, instructs believers, comforts those who hope, encourages those who work, aids those who try, and hears those who pray, you are not considered at fault if you, against your will, are ignorant; however, if you are ignorant because you fail to ask, you are at fault. You are not blamed because you do not bind up your wounded limbs. Your sin is that you despise Him who wishes to heal you. No one is denied the knowledge of how to seek advantageously what, to his disadvantage, he does not know, and how he must humbly confess his stupidity, so that He who neither errs or toils when he comes to give aid may help the man who seeks and confesses. What a man through ignorance does not do rightly, and what he cannot do, even though he wills rightly, are called sins because their origin lies in free will. … [W]e call sin not only what is properly called sin because it is committed from free will and in full knowledge, but even that which must follow from the punishment of sin. Thus we speak of nature in one way when we refer to man’s nature as he was first created, faultless in his own class; and we speak of it in another way when we refer to the nature into which, as a result of the penalty of condemnation, we were born mortal, ignorant, and enslaved by the flesh. Of this the Apostle says, “We also were by nature the children of wrath, as were the others.
...
Therefore, if blessedness for us consisted of fine speech and if it were considered a crime to err in speech and grammar in the same way as when we err in the activities of living, no one would denounce an infant because it set out from this point to pursue eloquence. Clearly, however, a man would rightly be condemned if by the perversity of his will he had either returned to babbling like an infant, or had remained at that first stage. So even now, if ignorance of the truth and difficulty in behaving rightly are the natural points from which man begins his ascent toward the blessedness of wisdom and tranquility, no one properly condemns the soul because of its natural origin. But if a man refuses to strive for excellence, or wills to step back from where he set out, he justly and properly suffers punishment.
The Creator of man is in all respects to be praised: whether because from the beginning He instills in man the capacity for the highest good, or because He aids man in attaining this good, or because He completes and perfects man’s progress; and He justly ordains the justest condemnation for sinners—either those who, from the first, refuse to strive for achievement, or those who slip back from a higher state—according to their just deserts. Besides, we cannot say that God created an evil soul on the basis of the argument that it is not so great as it has the power to be if it advances… [On Free Choice of the Will, III.xix, pp. 129-130; 138]