He denounced the Israeli blockade imposed on Gaza and deplored the devastation caused by the Israeli offensive in December and January against the Palestinian territory.
He visited sites destroyed during Operation Cast Lead and a US school completely destroyed in an Israeli air raid.
"I am very affected. I must hold back my tears at seeing the destruction that was inflicted on your people," he said.
"I came to the American School. It taught your children, it was funded by my country and I see that it was deliberately destroyed by bombs dropped by F-16 manufactured in my country. I feel partly responsible for what happened and all the Americans and the Israelis must have the same feeling," he added.
"It's not good to see such destruction, but either is it to see rockets falling on Sderot. All this violence must stop," he added.
He was expected to pass on a letter from the parents of Gilad Shalit, an Israeli soldier that Gaza militants, including Hamas, seized in a cross-border raid almost three years ago, and who remains in captivity.
Shortly after entering Gaza, Carter's convoy of white UN 4x4 vehicles sped through the streets, stopping briefly in the area of Ezbet Abed Rabbo, one of the most ravaged during Israel's war in the territory.
'We're not animals, we're human beings'
As Carter briefly got out of his vehicle to take a look at the damage, one of the residents who lined the streets ran up, yelling he wanted to talk to the former US leader, and getting into a brief shoving match with bodyguards.
"They all come here and look at us like we're animals and then they go home," said Majid Athamna. "We're not animals, we're human beings."
"If he wants to come and visit us, he has to listen to us."
Carter was also due to visit a hospital before speaking at the headquarters of the UN agency for Palestinian refugees UNRWA later in the day and meet Hamas prime minister Ismail Haniya.
Israel launched a war on the Hamas rulers of Gaza on December 27 in response to rocket fire, an offensive that killed more than 1,400 Palestinians and 13 Israelis before ending in mutual ceasefires on January 18.
Rebuilding has been stalled by the blockade that Israel had slapped on Gaza in June 2007 after Hamas, a group pledged to the destruction of the Jewish state, violently seized power in the territory.
Since then Israel and Egypt, which controls Gaza's only border crossing that bypasses the Jewish state, have kept the impoverished territory of 1.5 million aid-dependent people sealed to all but essential humanitarian supplies.
Israel has insisted that the blockade is necessary to prevent Hamas from arming itself, but human rights groups have slammed it as collective punishment of a territory where the majority of the population depends on foreign aid.