Israel’s military said Thursday it killed five Palestinian gunmen, including a militant leader, who were hiding in a West Bank mosque, as Israeli forces pressed a major operation targeting militants in the Palestinian territory.

The armed forces launched air-and-ground raids overnight Tuesday centering on the cities of Jenin and Tulkarem. The Israeli military said the operation is aimed at preventing terrorist attacks originating from the West Bank. It comes amid heightened fears that the Palestinian territory could become a third front for Israel as it fights Hamas militants in Gaza and exchanges fire with Lebanon’s Hezbollah movement.

The military said that during the fighting in Tulkarem, its forces killed a man named Muhhamad Jabber, known by the nom de guerre “Abu Shujaa,” who was the leader of a militant network in Tulkarem’s Nur Shams camp, where Palestinian armed groups have been battling Israeli forces in recent months.

A member of the Israeli border police was lightly injured in the fighting and taken to a hospital, the military said. Another Palestinian militant was arrested, the military added.

Explosions from the fighting in Tulkarem blew open the side of a building and left rubble including concrete blocks and a door strewn in the street, according to video footage taken by local residents and seen by The Wall Street Journal.

At least 17 Palestinians have been killed since Israel launched the West Bank operation earlier this week, according to WAFA, the official news service of the Palestinian Authority, which has limited powers of self-rule in parts of the Israeli-occupied West Bank. The death toll doesn’t distinguish between civilians and combatants.

The operation poses strategic risks for Israel, analysts say. Its armed forces are stretched thin after almost 11 months of combat in Gaza and an ever-escalating conflict with Iran-backed Hezbollah in Lebanon. The military relies heavily on part-time reservist soldiers who say they are exhausted from the longest war Israel has fought in decades.

The current operation in the West Bank involves hundreds of soldiers, including forces that have been taken into the field from training courses and reserves as reinforcements, said Shlomo Mofaz, a former senior official in Israeli military intelligence. “The army needs to know how to be spread across several arenas,” said Mofaz. “It has little choice.”

Israel hasn’t disclosed troop levels in its West Bank operation but said regional brigades as well as combat engineers were taking part. It didn’t provide an estimated end date for the maneuvers.

The West Bank, which Israel seized from Jordan in 1967, is home to three million Palestinians and along with Gaza is considered a core part of any future Palestinian state that would exist alongside Israel in a possible diplomatic settlement. Some half a million Israelis now live in settlements that have carved up the territory into smaller pieces.

Palestinian militant groups that oppose the Israeli occupation of the area—including the armed wings of Hamas, Palestinian Islamic Jihad and Fatah—have long posed a danger to Israel, with some armed groups gathering strength in recent years.

The Palestinian Authority, which was set up as a part of peace negotiations in the 1990s and envisioned as an interim step to an eventual Palestinian state, has slowly ossified as Israeli settlements grow and militant groups gather strength. Its weakening authority also poses a risk for Israel because its security forces cooperate with Israel in efforts to contain Palestinian armed groups.

—Abeer Ayyoub and Anat Peled contributed to this article.

Write to Jared Malsin at jared.malsin@wsj.com