If you tell me this, I will laugh. I'm polite, so I'll try not to laugh in your face, but I will laugh.
Sometimes the further discussion reveals that while they would never watch TV, they do listen to This American Life on NPR. Or the BBC. Or play video games. Or read mysteries, a favorite among the academic crowd. Or they make an exception for PBS, because PBS. Or they go to the most recent depressing and obscure foreign film that they can find and brag about it. (Which can be good, but the bragging? not so much.)
Folks, it's all the same. Here's the big secret: One is not morally superior to the other. All are ways of giving the brain a vacation, of distracting it so that it can stop beating you up for a while about the work you're not doing and give you a breather and fresh ideas so that you can do it.
But seriously--if you're not giving your brain a distraction, you're not giving it a rest. The productivity police may think that rest isn't necessary, but they're writing self-help, not creative work.
One of the best takedowns of this attitude I've ever seen, and the reason I refer to this attitude as "I don't own a television machine," is from an episode of The Dick Van Dyke Show called "I'm No Henry Walden." The premise is that a Robert Frost-type poet named Henry Walden* has had Rob and Laura Petrie (Dick Van Dyke and Mary Tyler Moore) invited to a high-toned literary party where no one knows who they are or understands what they do.
If you want to see the whole thing, including a brilliant double-talk performance by Carl Reiner that I swear I have seen many times given as a paper at MLA, the link is below. If you just want to get a flavor of it, including the immortal "I don't own a television machine," go to 11:33.
*From Henry David Thoreau & Walden Pond. Fun fact for the Orson Welles crowd: Henry Walden is played by Everett Sloane, who was Mr. Bernstein in Citizen Kane.