TEL AVIV—Israel has executed a series of daring military operations, including attacks in enemy capitals and commando raids in the heart of the Gaza Strip, that threaten to set off a new escalatory spiral in the Middle East.

The military is betting the response will be manageable—and that it is more important to win the fear of its adversaries after the security failure on Oct. 7.

“I don’t think there’s any doubt that the need to demonstrate that the [Israeli military] is not only competent but to demonstrate that it is brilliant in terms of its execution has created a much greater willingness to take risks,” said Aaron David Miller, a senior fellow with the Washington think tank Carnegie Endowment for International Peace.

Israel said Tuesday it killed a top Hezbollah military commander , Fuad Shukr, in Beirut, blaming him for a rocket attack in northern Israel that killed 12 young people. Hours later, Hamas political chief Ismail Haniyeh   was killed in Tehran . Israel didn’t take responsibility, but both Iran and Hamas blamed it for the attack.

Those attacks followed other operations that Israel carried out in ways that brought heavy civilian casualties and international concern. In a strike that killed Hamas military leader Mohammed Deif in July, Israel dropped eight 2,000-pound bombs and, according to Gaza health authorities, killed 90 Palestinians in a designated humanitarian zone. Earlier, Israel chose to launch a hostage-rescue operation in the heart of a Gaza marketplace in the middle of the morning, triggering a street fight that left as many as 274 Palestinians dead , according to local health officials, who didn’t say how many were combatants.

It isn’t clear whether the strikes on top leaders will significantly degrade the capabilities of Hezbollah or Hamas, both U.S.-designated terrorist groups. But they have led to fiery rhetoric and warnings of tough reprisals.

Mourners gather during the funeral of Hezbollah senior commander Fuad Shukr, who was killed in an Israeli strike, in Beirut’s southern suburbs, Lebanon August 1, 2024. REUTERS/Alkis Konstantinidis/File Photo

FILE PHOTO: A demonstrator holds up a poster of assassinated Hamas chief Ismail Haniyeh during a protest to show solidarity with Palestinian prisoners, outside the United Nations offices in Sanaa, Yemen, August 3, 2024. REUTERS/Khaled Abdullah/File Photo

Iranian Supreme Leader Ayatollah Ali Khamenei said the attack in his country would engender a harsh response. Hezbollah leader Hassan Nasrallah , speaking Thursday at Shukr’s funeral, said the killing in Beirut had moved the conflict into a different phase of escalation.

“You don’t know what red lines you have crossed,” he said. “We’ve entered a new phase different than the previous phase.”

Concerns are now growing that Iran could initiate a large-scale attack along with aligned militias in Lebanon, Yemen and Iraq, a scenario the U.S. has been trying to avoid for months. Diplomats from the U.S. and around the region are now engaged in a flurry of meetings and calls to persuade Iran to avoid responding in a way that leads to an escalatory spiral, people familiar with the efforts said.

Israel is expecting a response to the killings and is hoping it will remain manageable. In a similar situation in April, Iran launched a heavy barrage in retaliation for a strike attributed to Israel that killed a senior Iranian military officer in a diplomatic building in Damascus, Syria. Israel managed to down virtually all of the more than 300 missiles and drones aimed at its territory, but only with the help of a U.S.-led military coalition .

Israel’s bigger concern has been restoring deterrence. The Oct. 7 attacks led by Hamas overran military bases and kibbutzim, leaving 1,200 people dead and 250 as hostages. The assault shook Israeli society not only due to the brutal loss of life, but also because of the shame of the failure of a military once viewed among the mightiest in the Middle East.

Recapturing that edge from both a military and intelligence perspective has been top of mind for the Israeli security establishment since then.

Some of Israel’s aggressive moves have caught its counterparts off guard. U.S. and Arab mediators had worked feverishly in the days after the rocket attack from Lebanon that killed 12 Druze young people to head off an Israeli retaliation that would spark a regional war. U.S. officials said Wednesday that they expected the Beirut attack but were caught off guard by the killing of Haniyeh in Tehran.

The U.S. now fears that the killings might unleash a cycle of even fiercer reprisals that could include attacks on American forces in the Middle East. They have also severely dimmed the chances of a cease-fire and hostage deal to pause the fighting in Gaza, which has been one of President Biden’s chief foreign-policy goals. On Wednesday, senior administration officials were negotiating with Israeli, Qatari, Iraqi and Saudi officials to see if there was a way to reach a deal.

The Biden administration has a lower tolerance for risk than the Israelis, said Mark Dubowitz, chief executive of the Foundation for Defense of Democracies, a Washington-based think tank. For the Israelis, the threat from Iran and the regional militias it supports is existential, he said.

“I think Khamenei and Nasrallah might misunderstand the resolve to go into full-scale war,” Dubowitz said. “There are people who believe that this is an opportunity, especially against Hezbollah. This is the time.”

Some Israelis believe the constant fire of Hezbollah rockets, which have displaced around 60,000 Israelis from the country’s north, has given the country a legitimate cause to go to war, he said.

Israel has long relied on targeted killings to try to undermine its enemies, including former leaders of Hezbollah and Hamas, and senior Iranian security and intelligence figures. Israeli officials say such killings degrade their enemies by removing key leaders who are hard to replace, while at the same time forcing those who are still alive to act more cautiously, thereby making them less effective.

“We live in a region where no one appreciates anything but military force and the willingness to use it,” said Giora Eiland, a former Israeli national security adviser. Assassinations “help Israel to regain not only self-confidence but also some abilities to deter our enemies.”

A Palestinian child dressed as a Hamas militant stands next to an empty coffin in Beirut before a march honoring Ismail Haniyeh. Photo: Manu Brabo for The Wall Street Journal

War can be an opportune time to eliminate enemies, because daily hostilities are a given, while in peacetime the same killing could risk a new outbreak of violence, an Israeli security official said.

Israel is willing to take greater risks in eliminating its enemies during this war given the severity of the threat, the official said.

“We are fighting for our lives,” the official said.

But there has long been debate in Israel over the strategic value of targeted killings. Israel has killed senior members of Hamas and Hezbollah in the past. Yet both have become stronger over time.

“It was painful, all the assassinations for Hamas, but it seems that all these moves cannot really change the ideology,” said Michael Milshtein, a former head of Palestinian affairs for Israeli military intelligence. “When in Israel, people spoke about strategic impact…it was an illusion more than any kind of serious assessment.”

Write to Shayndi Raice at Shayndi.Raice@wsj.com and Dov Lieber at dov.lieber@wsj.com