Around 100 protesters, some wearing yellow stars pinned to their coats, held candles outside the building in the town of Esztergom, 31 miles north of Budapest.
"It's disgusting that anti-Semites are able to get away with this provocation in modern Hungary," a Holocaust survivor, who didn't wish to give her name, told AFP.
Jobbik became Hungary's third largest party when it won around 17 percent of the vote at the last parliamentary election in 2010, but it has often been slammed as racist towards the ethnic Roma community and for anti-Semitic remarks.
In 2012 a Jobbik deputy called for a list of Jewish nationals to be drawn up for "national security reasons."
The building, formerly a synagogue, has served as the town's cultural centre since 2006. A number of protesters who made their way into the building past a police line were removed by Jobbik stewards.
"Nazis get out of the synagogue," shouted one demonstrator.
Inside, Jobbik party leader Gabor Vona told around 200 supporters that his party was the victim of hysteria by leftist politicians and Jewish leaders. Describing his party's manifesto ahead of the next election parliamentary election on April 6, he said Jobbik didn't want to "disrespect or provoke anyone, including the Jewish community".
"As a legal party which won the votes of around one million people in 2010 however, we have as much right to hold an event here as anyone," Vona said. The building served as the town's synagogue from its construction in 1888 up until the end of World War II.
Most of the Jewish inhabitants of Esztergom, around 500 people, were killed in Nazi death camps in 1944