Azoulay, a high-profile businessman and advisor to Morocco's King Mohammed VI, who is a player in the Middle East peace process, is the driving force behind the Andalousies Atlantiques festival of Judeo-Arab music, whose sixth edition ended this weekend.
"Essaouira throughout its entire history and its entire way of living was a synthesis between Muslims and Jews," Azoulay told AFP. "It was not something artificially constructed, it was natural."
"And this festival is a reconstruction of that reality as it was historically. It is not cosmetic, it is real."
Morrocan-born Rabbi Haim Louk performs with the Moroccan orchestra Zyriab (Photo: AFP)
The opening concert at the three-day fest improbably featured an 80-year-old singer-rabbi, Haim Louk, backed by a Moroccan band who drew thunderous applause from the audience -- people of all ages and social class, women wearing headscarves and others in western gear, tourists, foreigners, Jews and Arabs.
Azoulay grew up in the town, which then had a big Jewish community, and returned after a successful banking and communications career in France with the idea of reviving the local economy.
An obvious path was to turn the town into a cultural hub to reflect its past, and a number of festivals including the world's leading festival of pulsating Gnaoua (or Gnawa) music now take place in the town.
"The changes in the town have been tremendous," Azoulay said. "Twenty years ago there was no airport. The hotels here now employ hundreds of people."
Azoulay grew up in a building in the kasbah where a Jewish family lived on one floor and a Muslim family on the next. "It was so normal that it was banal."
"When you see a concert such as Haim Louk, it is very moving," he said. "It is a reflection of what was and what is today in Morocco, and it is a step in the right direction in terms of our values.
"I would challenge anyone to take that social and cultural cohesiveness away from us, because of a political situation in which people are at odds with each other," he added.
Describing himself as spiritually Jewish, but also a Berber who is strongly influenced by Arab-Islamic history and culture, Azoulay said this meant he could enjoy Mahler, Um Kalthoum and Andalusian music.
"When people can sing and play together on stage in Hebrew and in Arabic, it is beyond symbolic, it is real. It is about reconciliation," he said.
"And when you see the standing ovation that a Moroccan Muslim public gave a Moroccan Jewish artist, you see maybe they could pay attention elsewhere."