The rabbi said a few fans had shouted "Jews out!" and other abuse and raised their arms in a Hitler salute - a crime in the Alpine republic - but that police showed no interest and told him to calm down, saying it was "only soccer".
Vienna police said they were investigating a complaint after the incident in the Austrian capital, which took place ahead of a match between Rapid Vienna and the Greek team PAOK last week.
The rabbi, who did not want to attract further personal attention by giving his name, said the fans were German-speaking but had worn T-shirts supporting the Greek team.
Oskar Deutsch, head of Austria's Jewish community – which has shrunk to about 9,000 from about 200,000 before Nazi Germany's 1938 annexation of the country – said it was a further sign that Austrians were becoming more openly anti-Semitic.
Last month, far-right Freedom Party leader Heinz-Christian Strache attracted little high-level censure after posting what critics considered to be an anti-Semitic cartoon on his Facebook page.
Deutsch said, "Parallels between the feeble reactions from politicians and the public to Strache's anti-Semitic caricature and this event ... are clear to me, and lead to a lowering of the inhibition threshold for anti-Semitic aberrations."
'Come on, it's only soccer'
He said the Vienna police chief had told him by telephone of his dismay at the incident and had promised a full explanation.
Strache's Freedom Party is running neck-and-neck with the conservative People's Party in opinion polls with about 20% of the vote ahead of a parliamentary election next year. The Social Democrats are leading with about 28%.
The rabbi said a second policeman he had complained to about the abuse had smirked and said, "Come on, it's only soccer."
The Simon Wiesenthal Center, a US non-profit organization that fights anti-Semitism, wrote a letter to Austria's Interior Ministry expressing its outrage at the incident.
"Law enforcement cannot be neutral in the face of hate crime," wrote Shimon Samuels, the Center's director for international relations.
Austria's Interior Ministry had no immediate comment.
Last month, in the wake of the Strache cartoon affair, Austria's Chancellery repeated its "rigorous" rejection of all forms of anti-Semitism, racism and incitement to hatred.