Israel wants to bring them back. To do so, it plans on recruiting a legion of ultra-Orthodox Jews, who because of their dedication to prayer and study, have been unable or unwilling to join the workforce, putting a heavy weight on the economy.
The job of a diamond polisher, however, is unique, said Bumi Traub, president of the Israel Diamond Manufacturers Association. It need not disrupt their pious lifestyle.
"The profession is fitting. You deal with the rock, and if you need to go pray, no one will bother you," he said.
The door to Traub's office requires a fingerprint scan. Security is tight in the four-building exchange where annual turnover of trade reaches $25 billion.
About a third of rough diamonds produced in the world each year pass through the Jewish state and diamonds account for more than a fifth of the country's industrial exports.
It was a natural sector to develop when Israel was founded 64 years ago, since the small stones have been choice merchandise for generations of Jews who had to quickly flee from riots and persecution.
The plan to revitalize manufacturing will cost millions of dollars and the diamond sector, for the first time, is turning to the government for help. The government, eager to get as many ultra-Orthodox working as possible, is on board.
The global financial crisis has taken a toll on the diamond trade, and Israel was not spared. Turnover was nearly halved at the outset in 2009, though in 2011 it returned to pre-crisis levels. A smaller drop is again expected for 2012.
The damage has been moderate compared to other major hubs such as India, according to Yair Sahar, president of the Israel Diamond Exchange.
"In other centers the leverage was tremendous, as opposed to here where we were much more conservative," he said, referring to the low level of debt among Israeli firms. "We entered the crisis more prepared, so to speak."
There have, however, been other problems.
The price for raw material has risen faster than that of the final product, eating away at profits. And a money laundering and tax evasion scandal at the start of 2012 scared away some customers. The investigations have ended and, so far, no one has been charged.
The diamond trading floor in Ramat Gan, a suburb of Tel Aviv, is the biggest in the world. Armed guards escort non-members and on one wall are mug shots of problematic dealers whom customers are urged to avoid.
Diamonds change hands freely across the rows of long dark tables that line the hall. On one side a seller could be local. A buyer across the way could represent some anonymous client on a different continent.
They scrutinize the stones under a magnifying glass, weigh them on sensitive scales and when a deal is reached they say "mazal ubracha", a Hebrew phrase recognized in centers around the world meaning "luck and blessings."
In 2011, rough diamond imports to Israel topped $4.4 billion and $7.2 billion in polished diamonds were exported. Every second diamond sold in the United States, according to value, came from Israel.
But only $1.5 billion of the stones were cut and polished locally, a much lower percentage than a decade ago. The rest were sent abroad to foreign firms or Israeli-owned factories.
"Once, everyone who sat in this room was a manufacturer," billionaire dealer Lev Leviev said at the opening of a Gemological Institute of America (GIA) laboratory in September. "There was not a diamantaire who was not a manufacturer, and over the years we lost it."
Salaries were just too cheap to compete with, he said, first in India, the world's biggest importer of rough diamonds, and later in China.
Israel has subsisted on larger, high-end stones whose owners pay more to have them manufactured close to home. But industry leaders hope to change that, in part because polishers in developing countries are demanding more money.
"I think we are there, more or less. With rocks of one carat plus, I think we are in a place where the (wage) gap doesn't justify running to manufacture abroad," said Sahar.
The GIA decision to open its lab in Israel was a first step. Manufacturers can now have their diamonds graded and evaluated in Israel rather than sending them to the United States.
"It's critical for the growth, for the international branding of the export business, and we think that we're a good partner to help the manufacturing grow," GIA President and CEO Donna Baker told Reuters when the lab opened.
By cutting costs and allowing increased turnover, it will add between $30 million and $50 million a year to the industry.
At the peak of manufacturing in the 1980s, there were 20,000 people cutting and polishing diamonds in Israel. That has dropped to about 2,000.
"There is no new manpower. Most polishers are 50 years old and up," said Roy Fuchs, who owns a factory a few minutes' walk from the exchange. "If they don't invest and bring in new blood, there simply won't be manufacturing."
To make it happen, the industry realizes it needs help, and for the first time, it is looking for assistance.
"It's not easy. You need cooperation with the government," said Udi Sheintal, the Israel Diamond Institute's managing director. "Here in the middle of Ramat Gan, you don't get incentives. There are only incentives for certain populations, like the haredi."
The term haredi, which in Hebrew means "those who tremble before God", refers to people who strictly observe Jewish law. They dress in traditional black outfits, the men do not shave their beards and they spend their days in study and prayer.
Some 8-10% of Israelis are haredi. For the most part they live in insular communities, are exempt from mandatory military service and, according to the Bank of Israel, less than half of ultra-Orthodox men work.
The issue has created a rift in the mostly secular Israeli society and put a strain on an otherwise robust economy. The government has already earmarked $200 million over the next five years to encourage haredi integration in the work force.
Many in the new generation of ultra-Orthodox are open to the idea of getting jobs. The key is finding one that fits, said Bezalel Cohen, 38, who has worked for years to promote employment among his fellow haredim.
"The diamond industry's initiative (to hire ultra-Orthodox)has potential to really succeed," he said. "As long as the pay and training is proper, it should take off."
Aside from helping to pay the salaries for newly hired haredim, the government will offer grants to small exporters and marketing support.
The Trade Ministry's diamond controller, Shmuel Mordechai, said the government backs the idea and has funded similar programs in other financial sectors. It would have helped even earlier, he said, but the diamond industry was never interested.
"They lived in their bubble, they said, 'Don't bother us, don't help us.' In recent years, because of difficulties in the industry and because we opened up our tools to them, they understand," he said.