TEL AVIV—The death of three Israeli hostages in Gaza shot and killed by Israel’s own military Friday in a case of mistaken identity has sparked fresh anger among the families of captives kidnapped by Hamas and renewed questions about the conduct of the war.

That is translating into growing political pressure on Prime Minister Benjamin Netanyahu to strike a deal to get back the hostages, scores of whom were snatched during Hamas’s Oct. 7 deadly assault on Israel and are believed to still be held by the U.S.-designated terrorist group.

The Israeli government says its war aims are to uproot Hamas and release the hostages. But those goals now appear contradictory. The deaths on Friday underscored the difficulties of pursuing an aggressive air-and-ground campaign in a densely populated urban space while also trying to save hostages.

The three hostages killed Friday had stripped off their shirts and carried a white cloth on a stick in an apparent effort to show potential rescuers that they were unarmed and not a threat, but they were shot anyway in what Israeli military officials described as a tragic violation of rules of engagement that prohibit shooting anyone who is surrendering.

On Saturday night, thousands of Israelis came out across the country to call for the release of the hostages, and some have started to call for Netanyahu to resign.

So far, Netanyahu has shown no signs of backing away from the strategy, despite U.S. pressure on Israel to shift to more targeted, less destructive raids.

“We will achieve all of our goals: eliminating Hamas, releasing all of our hostages and ensuring that Gaza doesn’t again become a focus of terrorism, incitement and attacks against the State of Israel,” Netanyahu said Sunday.

He has insisted that military pressure on Hamas is what will push the group to the negotiating table, attributing the war campaign with securing the release of 110 hostages during a temporary cease-fire last month.

Former Israeli hostages are now warning that the military’s campaign is endangering the lives of those who remain inside Gaza.

Raz Ben Ami, who was held captive for 54 days, said at a rally in Tel Aviv that after her release, she told Israel’s cabinet members that hostages could be caught up in the fighting.

“Unfortunately I was right,” Ben Ami, whose husband remains a captive, said on Saturday. “Military action alone won’t save the lives of the hostages.”

On Sunday, Lt. Gen. Herzi Halevi, Israel’s senior military officer, told a group of commanders and soldiers that if civilians appear to be surrendering, soldiers cannot shoot them. “If it’s two Gazans with a white flag coming out to surrender, why would we shoot at them? Absolutely not,” he said. “That’s not the IDF.”

The army also released new details about the three hostages Sunday, sharing photographs of what they believe to have been their hiding place with signs calling for help written in leftover food.

Israelis are broadly supportive of the war and the government’s aim of removing Hamas from the Gaza Strip. But polls show an overwhelming majority of Israelis no longer have faith in the prime minister.

Guy Laron, a senior lecturer at the Hebrew University in Jerusalem, said Netanyahu won’t change his war strategy because his right-wing, religious and ultranationalist coalition partners won’t remain in his government if he does a deal with Hamas that would include the release of large numbers of Palestinian prisoners.

“The entire conduct of the war is dictated by the politics of preserving this specific coalition, this specific composition of parties,” he said.

The ground operation so far has had limited success in freeing hostages. One attempt at liberating hostages ended with two Israeli soldiers being severely wounded. In the beginning of the ground offensive, Israel freed one hostage through military operations.

Friends and family mourn Alon Lulu Shamriz, one of three Israeli hostages who were mistakenly killed by the Israeli military while being held hostage in Gaza by the Palestinian Islamist group Hamas, at his funeral in Shefayim, Israel, December 17, 2023. REUTERS/Violeta Santos Moura TPX IMAGES OF THE DAY

Hamas has claimed that Israel’s intense air campaign killed hostages. Israel has confirmed that at least 20 captives held in Gaza are dead, but hasn’t said how they died.

Defense Minister Yoav Gallant said on Saturday that “Hamas only responds to force” and that without Israel’s aerial bombardment and ground campaign, “not one of the hostages would have returned.”

“The pressure has two purposes, but unfortunately there are also prices,” he added.

Israeli officials also note that more than 100 hostages were returned alive during the weeklong truce last month that coincided with some of the heaviest fighting and bombing of the war.

But Israeli troops have so far not been able to find and free other hostages. As the war has gone on and combat has intensified, commanders say, they increasingly find themselves in combat facing an enemy that dresses in civilian clothes, making the job of identifying potential hostages on the battlefield difficult.

Questions about the feasibility of rescuing hostages come amid mounting international concerns over the high Palestinian death toll in Gaza. Even the U.S., Israel’s staunchest ally, has become increasingly concerned over the growing humanitarian crisis in Gaza and is pressuring Israel to wrap up its ground offensive.

Israel’s military command structure widely acknowledged that the military’s own rules of engagement were violated during Friday’s hostage killings. Such instances “are the singular, not the rule,” said Israel Ziv, a retired general. “The rules of engagement are very clear, and [forces] are acting in accordance with them.”

Speaking to Army Radio on Sunday, Gil Dickmann expressed concern about his cousin Carmel Gat remaining in Hamas captivity, saying that Friday’s mistaken killing of three hostages showed that “pressure on [Hamas leader Yahya] Sinwar’s neck is also pressure on our hostages’ necks.”

—David S. Cloud contributed to this article.

Write to Shayndi Raice at Shayndi.Raice@wsj.com