The memorial site was erected in a bloody area, which is known as the "Russian Babi Yar" as more than 27,000 people were murdered there.
Authorities claim that the plaque was placed there illegally and was therefore removed, but the Russian Jewish Congress says it will take legal action in a bid to reverse the decision.
In August 1942, as part of the battles on the southern front of Russia, Nazi soldiers arrived at an area called Zmiyevskaya Balka and massacred 27,000 people, more than half of them Jews. This is the highest number of Jewish people murdered in a single incident during World War II.
According to Yad Vashem figures, from August 1942 to February 1943 Hitler's soldiers murdered 15,000 to 16,000 Jews in the Rostov area, which is considered one of the most violent cities in Russia today.
A memorial site for the victims erected in 2004 mentioned the anti-Semitic background of the murder and the victims' religion. The new plaque includes no mention of Jews, but simply states that it is in memory of "the peaceful citizens of Rostov-on-Don and Soviet prisoners-of-war".
According to the RJC, the decision to replace the former plaque was unauthorized.
According to the BBC, Rostov's Deputy Culture Minister Valery Gelas Admitted that the rules for historical memorials had been broken, but said the new plaque would remain.
He told Moscow Echo radio that the wording was in line with historical research and data presented to the Rostov cultural authorities.
Tamara Pletneva, a Communist member of parliament on the Duma committee for nationalities, said it was time to "forget our bitterness and live in peace".