The move is said to be geared towards improving access for worshippers to the site, which is considered holy by Jews.
The site is part of the same ancient wall that used to surround the Second Temple, and lies several hundred meters north of the Western Wall, one of Judaism's most sacred sites.
The scaffolding was located under an arch in the small courtyard abutting the wall. A number of Palestinian homes are built on top of the arch.
An AFP correspondent confirmed the scaffolding had been removed, saying a small but steady stream of worshippers and tourists were visiting the site.
Sheikh Azzam Khatib, head of the Islamic Waqf in Jerusalem, confirmed the scaffolding had only recently been removed and said officials had put up a new sign identifying the location as "The Little Western Wall."
"We think they want to expand that place to let more worshippers in because only limited numbers can go there at the moment," he told AFP, explaining that the scaffolding was installed to reinforce homes affected by digging in the area.
"When they were digging for the (Western Wall) tunnels, some houses were damaged and they put up this scaffolding to support the houses," he said, referring to work ahead of the controversial tunnels which caused angry rioting and bloodshed when they were opened in 1996.
Israel's biggest tourist attraction
Locals confirmed the scaffolding and a number of metal support girders had been removed in the past few days, expressing fears it could lead to a collapse of the nearby buildings.
"In the 1970s, part of the wall collapsed and it's only a matter of time before it happens again," Raaf Shaadi told AFP, saying his family had been living in the neighborhood - also called Shaadi - for more than 300 years.
Shaadi said police and municipal officials had come to take down the scaffolding several days ago, telling locals the walls were stable and not in any danger of collapsing.
Construction work in and around the Old City has historically been one of the most contentious issues in the Israeli-Palestinian conflict and has a history of triggering unrest.
Each year, more than eight million people visit the Western Wall, Israel's biggest tourist attraction, which is revered by Jews as the last remaining remnant of the Second Temple.
Above the wall is the area known to Muslims as the Haram al-Sharif or Noble Sanctuary, which houses the Al-Aqsa Mosque compound, the third-holiest site in Islam.
Israel considers Jerusalem its eternal and indivisible capital, a claim not recognized by the international community. The Palestinians want east Jerusalem as the capital of their promised state.
- Follow Ynetnews on Facebook