Hezbollah is dealing with some disagreement among its ranks over how to respond to a series of devastating attacks on the Lebanese militant group, according to people familiar with the group’s deliberations. With no good options available, the group faces one of the most consequential decisions in its four-decade history.

Hezbollah, the world’s most heavily armed nonstate paramilitary group, fired its first missile at the commercial capital of Tel Aviv on Wednesday, its boldest response yet to a dramatic series of strikes by Israel. It must now choose whether to unleash more of its advanced weapons, striking deeper into Israel and potentially triggering a full-scale war, or hold back and risk diminishing its reputation as one of the fiercest fighting forces in the Middle East.

“This is the single most crucial moment for Hezbollah since it was created,” said Rym Momtaz, a security analyst at the Carnegie Endowment for International Peace with expertise in Hezbollah. “Hezbollah has no good options.”

A full-scale war, such as the one Hezbollah fought with Israel in 2006, would likely be devastating for Lebanon, exacerbating its economic crisis and eroding Hezbollah’s support among the population. Still, if the group doesn’t respond in kind to the recent attacks, it could undermine the deterrence that Hezbollah spent decades building against Israel, largely with military and financial help from Iran.

Israel says the recent attacks were aimed at stopping Hezbollah strikes that have continued for nearly a year and allowing Israelis to return to parts of northern Israel.

A series of attacks by Israel have left Hezbollah in disarray. Last week, thousands of Hezbollah pagers and walkie-talkies exploded , and an airstrike in Beirut killed a group of elite military commanders . On Tuesday, another Israeli airstrike in southern Beirut killed Hezbollah’s top missile commander, Ibrahim Muhammad Qabisi.

There is disagreement among Hezbollah members over how to respond, the people familiar with Hezbollah’s discussions said. Some members say the group has been too cautious about escalating the conflict. They argue that Hezbollah should retaliate now, taking advantage of the anger in its own ranks and the broader Lebanese population.

Hezbollah members also have expressed frustration in conversations with Iranian Islamic Revolutionary Guard Corps officials that Tehran hasn’t stepped in to support its Lebanese ally, the people said.

But Hezbollah’s leadership wants to avoid what it sees as a trap set by Israel: Getting Hezbollah to be seen as starting a regional war that draws in Iran and the U.S., the people said. The group’s leadership is trying to figure out how to re-establish deterrence without going to war with Israel, these people said.

It isn’t clear how much Israel’s attacks have degraded Hezbollah’s weapons. Until recently, the group possessed an estimated 150,000 missiles and rockets, about 10 times the number it had in 2006. Israel’s military says Hezbollah has fired 9,000 projectiles at Israel since last year, and the recent strikes have destroyed tens of thousands of munitions.

“I believe Israel disrupted the group’s military capability, at least temporarily, and made it harder to respond coherently,” said Daniel Byman, senior fellow at the Center for Strategic and International Studies in Washington who co-wrote a recent study of Hezbollah’s arsenal.

Hezbollah is believed to still possess a vast arsenal , including hundreds of guided ballistic missiles. A Hezbollah official said Israel’s recent strikes had destroyed fewer munitions than during the 2006 war.

On Monday, Hezbollah fired tactical surface-to-surface rockets, known as Fadi, with a range of up to 65 miles, into Israeli territory. Aside from the Qadr-1 medium-range ballistic missile fired at Tel Aviv on Wednesday, it has yet to deploy its most-advanced weapons, including an estimated tens of thousands of unguided ballistic missiles with a range of 80 to 180 miles, and hundreds of guided ballistic missiles with a range of 150 to 300 miles.

Some current and former Israeli security officials said they were surprised that Hezbollah didn’t retaliate with more force after Israel’s strikes in recent days. Emergency and municipal authorities, who were briefed by the Israeli military weeks ago in preparation for possible Hezbollah strikes, were told to expect 4,000 missiles and rockets each day and to prepare for possible daily casualties in the thousands.

The strikes targeting Hezbollah launchers likely helped subdue the response, said one official, who added that the military wasn’t counting on its ability to consistently pre-empt those attempts.

One possible response by Hezbollah could be to launch hundreds of nonguided rockets to overwhelm Israel’s air defenses and target nonmilitary sites. When Iran attacked with more than 300 rockets and drones in April, Israel relied on assistance from the U.S., Jordan and other countries to shoot them down, even though Tehran had telegraphed its attack in advance.

One complication is that Hezbollah’s longer-range precision missiles are more visible when being set up, potentially exposing them to Israeli strikes, Byman said. The risk is amplified by the recent attacks, which have shown Hezbollah to be thoroughly infiltrated by Israel.

The pager and walkie-talkie attacks also likely complicated—but didn’t destroy—Hezbollah’s communications. Most of the group’s communications take place through coded language over landlines and human couriers, not pagers and radio.

Hezbollah’s predicament is partly of its own making. The group has fired rockets at Israel since Oct. 8—the day following Hamas’s attack on Israel—and has refused to halt its attacks until there is a cease-fire in Gaza, an ultimatum that has now backed it into a corner.

“It tied its fortunes to the Gaza front, which it cannot control, is constrained by the dire Lebanese situation, was pulled into an attritional war it wasn’t built to fight, and has been outgunned and outfoxed by Israel,” said Momtaz, the security analyst.

Even after Hezbollah’s top military commander, Fuad Shukr, was killed in an airstrike in Beirut in July, the leadership overwhelmingly refused to separate the fighting in southern Lebanon from the conflict in Gaza.

Hezbollah officials have been concerned about penetration by Israeli intelligence since it began shelling northern Israel last October. Officials were concerned about leaks and Israeli infiltration and were trying to plug holes in the organization, they told The Wall Street Journal at the time. By the end of that month, Hezbollah operatives were being killed at a rapid clip, in areas that hadn’t previously been exposed as housing weapons facilities.

Hezbollah officials believe Israel’s recent attacks are aimed at offsetting the tactical win the group scored in northern Israel, where some 60,000 Israeli residents have been evacuated, by exacerbating the mass displacement and humanitarian crisis in southern Lebanon, the people familiar with the matter said.

Hezbollah’s leadership still believes Israel is hesitant to launch a ground invasion, which would be harder to pull off than airstrikes and likely cause significant Israeli casualties, the people said.

There is precedence for diplomacy. The most recent war between Israel and Hezbollah, in 2006, ended when the Lebanese and Israeli cabinets voted for a United Nations Security Council Resolution, which positioned U.N. peacekeepers in south Lebanon. As part of the cease-fire, Hezbollah agreed not to place armed forces south of the Litani River, 18 miles from the Israeli border—a requirement Israel wants the group to adhere to today.

Hezbollah’s decision in the coming days could have momentous consequences for its future. It could cost the group credibility in Lebanon, where it is also a powerful political party, a welfare provider and a stronger military force than the national army—all founded primarily on resistance to Israel.

“Hezbollah are damned if they do and damned if they don’t,” said Aram Nerguizian, a nonresident senior associate at the Center for Strategic and International Studies with expertise in military dynamics in the Middle East.

“The risk for the group is that the longer this goes on—and the longer the Israelis are able to avoid getting into a ground campaign that only benefits Hezbollah—the more rapidly the group’s credibility, monopoly on national security politics, and de facto political hegemony are degraded,” Nerguizian said.

Write to Sune Engel Rasmussen at sune.rasmussen@wsj.com and Summer Said at summer.said@wsj.com