In the novels, Denethor went mad because, like Saruman, he had been unwisely glancing too much into his palantiri. I thought this made a lot more sense than him just going crazy rather unconvincingly as he did in the movies.