Vice President Mike Pence on Tuesday argued against pulling down Confederate monuments, saying that it was an attempt to “erase parts of our history just in the name of some contemporary political cause.”

Pence told Fox & Friends on Tuesday that “I’m someone who believes in more monuments, not less monuments.”

“Rather than tearing down monuments that have graced our cities all across this country for years, we ought to have been building more monuments,” Pence said. “We ought to be celebrating the men and women who’ve helped our nation move toward a more perfect union and tell the whole story of America.”

The debate over Confederate statues has taken on renewed urgency after a white supremacist and neo-Nazi rally this month in Charlottesville, Virginia, left one dead and 19 injured. President Trump has defended the Confederate statues, calling them “beautiful” and saying they reflect the country’s history and culture.

Asked by Fox’s Ainsley Earhardt if the Confederate monuments at the Capitol should be removed, Pence said, “it’s important that we remember our past and build on the progress that we have made.”

Most of the Confederate monuments were not erected right after the Civil War but rather several decades later during Jim Crow and the Civil Rights Movement.