The Hague Appeals Court ruled that a law which makes it mandatory for all people older than 14 to carry ID cards and show them to police upon request does not have a religious exemption.
The man, whose name was not released due to privacy laws, had argued it was against his religious beliefs to carry anything but his clothing on the Jewish Sabbath.
The ruling didn't say why police approached the man.
The law was introduced in 2005 amid a wave of anti-immigrant and anti-Muslim sentiment. Carrying ID cards hadn't previously been mandatory in the Netherlands since the Nazi occupation in World War II.
Rabbi Pinchas Goldschmidt, president of the Conference of European Rabbis, said in response that "the problem of carrying an ID card on Shabbat is not new, and even in Nazi Germany religious authorities discussed whether and how should one go out to the public domain with certificates.
"I hope the Jewish community in Holland reaches an agreement with the authorities on this issue," he concluded.