The question is how did Hazal equate disregard for Hol HaMo'ed as "Idol Worship"?
My late Rav, R William Cohen of W. Hartford told us a possible reason whilst studying Masechet Makkot. That is during the early Xtian era, disregard for the Jewish Holiday Cycle was one of their first breaks with Jewish Tradition.
Thus by dissing our Calendar, they were Crossing Over [pun intended] to the Greco-Roman Calendar and abandoning the Torah Yearly Cycle. This abandonment was tantamount to abandoning Torah Judaism altogether, because had they kept the Mo'adot, they may have found a path back towards Torah Judaism