Organizers of the sitdown protest said police physically lifted over 20 people, including a number of Holocaust survivors, from the planned site of the monument in the capital, Budapest.
The government says the structure – which will depict Hungary as an angel being attacked by a German eagle – will commemorate all the victims of the country's occupation by Nazi Germany.
But several Jewish organizations and historians say the memorial absolves Hungarians of their active role in carrying out the deportations of Jews to Nazi death camps.
Some 600,000 Hungarian Jews perished during the Holocaust, almost all after the German occupation began on March 19 1944.
Building work on the monument started on April 8, two days after Prime Minister Viktor Orban won a second consecutive landslide election, and since then protestors have been trying to block construction.
The memorial is due to be completed by May 31.
Hungary's largest Jewish organization, Mazsihisz, has criticized Orban for breaking a pledge to discuss the monument with them.
Protest organizer Gabor Popper told AFP the demonstrators knew they could not stop the monument being built.
"If the government really wants to build it, and it seems they do, then the most we can do is somehow delay its construction," he said.
Earlier Tuesday, former prime minister and opposition politician Gordon Bajnai sent an open letter to Hungarian president Janos Ader asking him to "do everything he can" to stop the monument being built.