You're absolutely right! "Irregardless" is in the same league as the double-negative constructions "I didn't say nothing" and "You mean you didn't find nothing at all?" These are nonstandard syntactic constructions that contain two negatives but are intended to have a negative meaning. On the contrary, however, the use of the two negatives results in a positive meaning, which makes the construction semantically erroneous.
(The correct equivalents of those two double-negative constructions are, of course, "I didn't say anything" and "You mean you didn't find anything at all?")