Seventy years on, as Hungarians still grapple with the past and many support the far-right Jobbik party, the government finds itself in conflict with the Jewish community over Hungarians' role in the Holocaust.
The Open Society Archives brought that role to the fore on Saturday, opening up to the public 76 buildings where Jews were gathered during the war. Many were either slaughtered on the spot or transported to death camps.
In the courtyard of one building, Marton, now 84, talked to about 100 people about his personal memories.
He said he watched helplessly as Hungarian fascists deported his mother from their home in an ornate building in central Budapest. She was first taken to rural Hungary. When she escaped and came home, an informer reported her.
"A day after she came back the building manager showed up with a fascist commando and took her away again," Marton said.
"She was deported to the camp in Bergen-Belsen and died there, two days after the camp was liberated. She received food and her eviscerated body could not handle it."
He remembered that on one occasion, at a building across the street, the fascists responded to gunfire from a top-floor window by herding every Jewish resident into the street. They took the women and children to the ghetto, but not before executing the men in front of their families.
Buildings were earmarked for Jews with a large yellow Star of David, many of them housing hundreds of people, several families crowded into each apartment. Time and again fascist commandos raided such houses and killed dozens of Jews at a time.
Of the 2,000 buildings so designated, 1,600 still stand in the Hungarian capital, and on Saturday residents of the 76 participating houses, many of them Holocaust survivors, welcomed visitors and told them about the horrors.
The Open Society Archives called the events the Day of Starred Houses, designed to spread out across Budapest in the hope of making the events as visible as possible.
Although Jewish culture has flourished in Hungary since the fall of Communism and Budapest is again home to one of Europe's largest Jewish communities, anti-Semitism is a growing problem that Jews and others view with alarm.
Budapest now has about 100,000 Jewish residents, compared with more than 200,000 before the Holocaust.
The government of center-right Prime Minister Viktor Orban found itself in confrontation with the Jewish community as it commissioned a World War Two memorial that Jews think whitewashes the responsibility of Hungarians in the Holocaust.
Although Orban has repeatedly pledged zero tolerance for anti-Semitism and said Hungarians were indeed complicit in the genocide, he rejected full blame for it and said without German occupation there would never have been death trains leaving Hungary at all.
Many Jewish organizations have boycotted government efforts to commemorate the Holocaust anniversary in protest.
After the far-right Jobbik party gained an unprecedented 21% in a parliamentary election in April, tens of thousands of Hungarians took to the street to protest the rise of the far right in the country.
To people like Tamas Marton, that is little comfort. To this day he lives in the same house where the atrocities happened to his family during the war. For decades he used to live side by side with the building manager who turned his mother in.
"He lived right in this building," he said. "Of course he tried to avoid me. He never really looked me in the eye after that."