With Madonna already in his diocese preparing for her Saturday night gig at Vector Arena, Bishop Dunn said: "There's always a fine line, isn't there, about artistic expression and how free artists can be."
During an interview with RadioLive on Thursday, host Mark Sainsbury asked if Bishop Dunn was just being "an old fuddy-duddy". The Bishop replied, "I don't really care, quite honestly".
"I find it hugely offensive. I think there are boundaries, especially in our society in New Zealand now, where we have such a mix of cultures and faiths that artists have to be a little bit sensitive to values that are held by people in our society."
"I'm sort of a bit wary of censorship," he said. "But I would bet every penny I've got that Madonna wouldn't dare say or use lyrics that were offensive to Islam, as she feels free to use with regard to Christianity, that's just a fact. People would respond with outrage."
Unlike Bishop Dunn, Minister Glynn Cardy of St Luke's Presbyterian Church in Remuera thought Madonna, like all artists, should be "free to express their ideas".
My personal opinion is that Madonna is an artist and like most artists uses her experience and understanding of her culture in her work," Minster Cardy told Stuff on Thursday afternoon.
"Religious, and particularly Christian, experience is part of her context. She uses Christian symbolism and language in a provocative manner to challenge the institution of Christianity and individual believers to re-examine our thoughts and prejudices," he said.
Minister Cardy said he hoped Madonna will always be free to express her ideas "regardless of whether I agree with those ideas and ways of expressing them or not."
"Institutions like the Christian Church can be, and have been, greatly blessed by the criticism of artistic provocateurs beyond our gates."
But the head of the New Zealand Catholic church chose not to wade into the controversy.
Cardinal John Dew, the Catholic Archbishop of Wellington and the leader of the church in this country, wouldn't comment.
His spokeswoman said: "Cardinals do not comment on other Bishop's or Cardinal's personal views and individual opinions. Bishop Pat was exercising his own personal opinion."
The Like A Prayer singer has never been one to shy away from controversy, regularly incorporating religious motifs and Catholic symbols into her stage show.
Madonna's Rebel Heart Tour, which will see the singer perform for the very first time on the New Zealand stage, is said to include a "holy water" stripping segment, which features scantily-clad dancers and "sexy nuns" gyrating on cross-shaped stripper poles.