CBS' firing of Stephen Colbert has seen the Late Show time slot nosedive in ratings — and it's taking the network with it.Since CBS handed Byron Allen the 11:35 p.m. slot formerly occupied by Colbert's The Late Show — with Allen Media footing all production costs under a "time buy" arrangement — Nielsen data reviewed by Status show nearly two-thirds of the audience Colbert was drawing toward the end of his run has fled.And media experts say that massive damage won't stay contained to the late night time slot."Within the television industry, it was figured out quite quickly that a popular late-night program would provide a lift to a morning show," University of Maine communications and journalism professor Michael Socolow told Status. "Early audience studies revealed that people habitually left their TVs tuned to a channel, and they wouldn't switch channels the next time they turned on the TV unless they did not like what they were watching."Late local news is also exposed. "People who wanted to watch Colbert, a lot of them probably watch late local news," said Bill Carter, the veteran New York Times media reporter and author of multiple books on late-night television. By canceling Colbert, Carter said, CBS "diminished itself."CBS told the Daily Beast, "We're proud to partner with Byron Allen on a new business and programming model for late night that proactively addresses a network daypart that was cost-prohibitive to continue. "With this 'time buy' model, we have shifted an hour that was losing roughly $40 million annually to $15 million in profit — a $55 million swing."Ratings data suggest Jimmy Kimmel's ABC program has absorbed a significant share of Colbert's former viewers.Allen, 65, said from the outset he wasn't chasing Colbert's audience. "At the end of the day, I'm not trying to replace Colbert," he told NPR last month. "I am not trying to hold on to his audience because Comics Unleashed has been around 20 years and has its own audience."