The meeting was initiated by the Yesha Council in a bid to resolve what settler leaders characterized as a legal issue, but Olmert told them he will be acting in accordance with a High Court of Justice ruling and will not postpone the demolitions.
However, the two sides agreed to continue their dialogue.
Settler leader Pinchas Wallerstein told Ynet Olmert rejected Yesha's request out of hand.
"Olmert told us 'forget about it, I have no intention to postpone the demolition. I intend to abide by the court ruling.'"
Wallerstein added: "We were willing to do everything in order to prevent a rift within Israeli society…to our great regret, due to various considerations that may partly stem from political considerations, there was someone interested in razing the homes."
Meanwhile, the Yesha Council called on the public to flock to Amona for a "resolute struggle against the intention to raze and destroy."
"In areas blocked by the army, organize in groups and start walking in order to reach Amona at any price," the Council advised its supporters.
Meanwhile, IDF troops at a West Bank roadblock stopped two buses carrying youths heading to Amona.
Earlier, a military order was signed allowing troops at roadblocks to prevent settlers from passing through if they are suspected of planning to reach Amona and disrupt the evacuation.
Troops prepare for evacuation
Meanwhile, IDF Central Command Head Yair Naveh and Police Chief Moshe Karadi visited the base where troops train ahead of the evacuation of the illegal West Bank outpost of Amona.
Naveh, who spoke to the soldiers and police officers at the base, told them the IDF is preparing for extreme scenarios.
"The evacuation of Amona will take place on Wednesday. If there's a need to go back and clear the Hebron marketplace, it will happen on Thursday or Friday," he said. "We're not dealing with disengagement. This is a law enforcement mission we've been tasked with."
Naveh also told security forces: "You'll be equipped with bats and helmets, and (tear) gas will be used with the approval of a brigade commander or a higher ranking officer."
"We must make plenty of arrests and be determined," he said. "A masked person should be detained at once. If we face even the smallest problem in evacuating women – male troops will also take part."
Police Chief Karadi said: "Our mission is to finish the evacuation successfully and without casualties, and to that end we have 1,600 police officers from the best units in the Israel Police."
Efrat Weiss contributed to the story