A group of men threatened Baruch with death and called him a "crazy Jew" as he walked in a mall with his pregnant wife several years ago, prompting him to limit when he wears his kippah—a Jewish skullcap.
"It's never happened again because I chose to hide my identity as a Jew in public," he said.
There is a similar ripple of concern among many of the estimated 200 Jews living in the Southeast Asian country of 260 million people, with most of them centred in a remote corner of the sprawling archipelago.
Manado on Sulawesi island is one of the few places that Indonesia's remaining Jews—mostly descendants of traders from Europe and Iraq who were once thought to number around several thousand before World War II—feel comfortable showing their faith.
A 62-foot-tall menorah, possibly the world's largest, stands near the town of Tondano—around 20 kilometres (13 miles) south of Manado -- where Baruch holds regular services at a modest, red-roofed synagogue.
The Shaar Hasyamayim synagogue is Indonesia's lone house of worship for Jews after the only other one in the city of Surabaya was demolished in 2013.
It had been the site of anti-Israel protests for years, and was sealed off by religious hardliners in 2009 and left to decay.
Indonesia has long been praised for its moderate brand of Islam, but more conservative forms of the religion have taken center stage in recent years, driven by increasingly vocal hardline groups.
Tensions in the Middle East, particularly between Israel and the Palestinians, spill over here and deepen religious divides.
Thousands of hardliners demonstrated in Jakarta when US President Donald Trump announced last year that the American embassy in Israel would be moved to the contested city of Jerusalem.
"There is still a lot of anti-semitic sentiment in Indonesia," Baruch said.
"Generally speaking, Indonesians don't differentiate between being Jewish and Israel. They think Jews and Israel are the enemy of their religion and state," he added.
"There is no denying that tolerance is fading in our country."
The size of the Jewish community makes it almost invisible so Jews have not been the target of Islamist militants like some of Indonesia's larger religious minorities.
A wave of deadly suicide bombings at churches in Surabaya last year highlighted the threat to minority groups, while Shiites and Ahmadis—regarded as heretics by some majority Sunni Muslims—have also been the target of violence.
Still, Indonesia's Jews are on the radar of some groups.
Monique Rijkers' efforts to bridge the divide with a TV program about Judaism drew the ire of the Indonesian Muslim Students Association, which she claims reported her to government and broadcast regulators.
"They demanded that I be fired and that the program be cancelled," said Rijkers, founder of Hadassah of Indonesia, a non-profit organisation that offers cultural education programs Centered on Israel, Jews and the Holocaust.
Indonesia's Jews face some practical challenges, too, such as finding kosher food in a country where it's not widely available.
Another hurdle is that Indonesia has long allowed for only six different religious categories on all-important ID cards—Islam, Protestantism, Catholicism, Buddhism, Hinduism and Confucianism.
The cards are crucial for accessing government services, and for doing things such as registering marriages and births, meaning most Jews lie and put "Christianity" on the documents.
Even some Muslim Indonesians learn that taking an interest in anything Jewish can raise eyebrows.
Sapri Sale, who started teaching a Hebrew class in Jakarta a year ago, has been studying the language since the 1990s and compiled what he says is the world's first Hebrew-Indonesian dictionary.
But his interests got little positive feedback at home.
"I was called Sapri the Jew," he said.