Britain's Ministry of Defense issued a statement saying ""HMS Montrose was forced to position herself between the Iranian vessels and British Heritage and issue verbal warnings to the Iranian vessels, which then turned away,"
The incident occurred almost a week after British Royal Marines boarded an Iranian tanker, Grace 1, off Gibraltar and seized it on suspicion that it was breaking sanctions by taking oil to Syria.
Earlier on Wednesday, Iranian President Hassan Rouhani said Britain would face "consequences" over the seizure of the Iranian tanker.
Tensions between Iran and the United States and its allies have risen sharply since Washington stepped up economic sanctions against Iran and moved to bring the country's oil exports to zero as part of a "maximum pressure" policy to make Iran halt actions that it said undermined regional security.
Iran has responded to the sanctions by starting to breach limits put on its nuclear activities under a 2015 deal with world powers.
Several oil tankers were attacked in waters near Iran's southern coast in May and June, for which the United States blamed Iran. Tehran denied any involvement.
Last month, Iran shot down a U.S. drone near the Strait of Hormuz, prompting President Donald Trump to order retaliatory air strikes, only to call them off.
The U.S. officials, speaking on the condition of anonymity, said Wednesday's incident happened as British Heritage was at the northern entrance of the Strait of Hormuz.
"The Royal Navy HMS Montrose, which was also there, pointed it guns at the boats and warned them over radio, at which point they dispersed," one of the officials said.
"It was harassment and an attempt to interfere with the passage," the other official said.
The United States hopes to enlist allies over the next two weeks or so in a military coalition to safeguard strategic waters off Iran and Yemen, Marine General Joseph Dunford, the chairman of the Joint Chiefs of Staff, said on Tuesday.