Mr Mandela, 95, led South Africa’s transition from white-minority rule in the 1990s, after 27 years in prison.
He had been receiving intense home-based medical care for a lung infection after three months in hospital.
In a statement on South African national TV, Mr Zuma said Mr Mandela had “departed” and was at peace.
“Our nation has lost its greatest son,” Mr Zuma said.
The Nobel Peace Prize laureate was one of the world’s most revered statesmen after preaching reconciliation despite being imprisoned for 27 years.
He had rarely been seen in public since officially retiring in 2004.
“What made Nelson Mandela great was precisely what made him human. We saw in him what we seek in ourselves,” Mr Zuma said.
“Fellow South Africans, Nelson Mandela brought us together and it is together that we will bid him farewell.”
Earlier, the BBC’s Mike Wooldridge, outside Mr Mandela’s home in the Johannesburg suburb of Houghton, said there appeared to have been an unusually large family gathering.