The International Court of Justice declined to order Israel to cease military operations in Gaza while it weighs South Africa’s claim that the country is committing genocide against Palestinians.

In a series of near-unanimous votes, the world court ordered Israel to ensure that its military not violate the Genocide Convention, punish incitement to genocide and report back to The Hague on its compliance. But it stopped short of ordering the fundamental objective South Africa and its allies sought: an end to Israel’s military response to the Oct. 7 terror attacks Hamas launched from Gaza.

The court also called for the immediate and unconditional release of hostages Hamas took from Israel.

The decision represented a rare win for Israel before a United Nations body and could act as a counterweight to the growing international opposition to its military campaign in Gaza. More than 25,000 people, the majority women and children, have been killed there since the start of hostilities, according to Palestinian authorities, whose figures don’t distinguish between combatants and civilians.

South Africa, backed by the Palestinian Authority, filed the genocide charge earlier this month before the world court, a judicial arm of the U.N. It seeks a preliminary order requiring Israel to cease military operations in Gaza while proceedings over the genocide claim play out.

Israel, in response, said it was acting in self-defense to free hostages and eliminate a terrorist threat following the Oct. 7 massacre of some 1,200 people that Hamas launched from Gaza.

During arguments two weeks ago in The Hague, South Africa portrayed Israel’s military response to the Hamas attack as furtherance of a campaign against Palestinians that started with the establishment of the Jewish state in 1948.

To demonstrate Israel’s intent to wipe out Gaza’s Palestinians, lawyers for South Africa cited furious comments from Israeli leaders following the Hamas attack, among them Defense Minister Yoav Gallant’s remarks to troops on Oct. 9 that Israel was “fighting human animals” and must “eliminate everything.”

In rebuttal, Tal Becker, legal adviser in Israel’s Foreign Ministry, called the genocide charge “libel, designed to deny Israel the right to defend itself according to the law from the unprecedented terrorist onslaught it continues to face.” He said that while Israel obeys the laws of war, those very laws contemplate that war necessarily involves civilian casualties. Hamas, he said, exacerbated such suffering by placing its fighters and supplies in civilian areas and facilities, converting them to legitimate targets.

To bolster its case, Israel declassified war-cabinet decisions as evidence that it was working to ensure humanitarian aid reached Palestinians inside the Gaza Strip and suggested promoting the construction of field hospitals. Israel’s legal team also released daily directives by its military to soldiers that attacks “will be solely directed towards military targets.”

The world court has 17 members: 15 permanent members, including its American president, Joan Donoghue, and two temporary judges, one each appointed by South Africa and Israel. The other judges on the bench are from Australia, Brazil, China, France, Germany, India, Jamaica, Japan, Lebanon, Morocco, Russia, Slovakia, Somalia and Uganda.

Two years ago, in a Genocide Convention case Ukraine brought against Russia, the ICJ was strikingly unified in ordering Moscow to cease military operations in Ukraine; only the Chinese and Russian judges dissented.

The U.S. and other Western nations have dismissed the genocide charge against Israel as unwarranted. Speaking earlier this month in Tel Aviv, Secretary of State Antony Blinken called the accusation “particularly galling, given that those who are attacking Israel—Hamas, Hezbollah, the Houthis, as well as their supporter, Iran—continue to openly call for the annihilation of Israel and the mass murder of Jews.”

But in the developing world, South Africa’s case has gained support from countries ranging from Brazil to Bangladesh and multilateral groups such as the Organization of Islamic Cooperation, a group of Muslim-majority countries.

South Africa’s support for the Palestinians’ quest for statehood has been consistent since the ruling African National Congress party came to power in 1994. The Palestinian cause is widely seen as analogous to the ANC’s own struggle against apartheid.

“South Africa really has a moral responsibility to always stand with the oppressed because we come from a history of struggle, a history of striving for freedom, a history of believing that everybody deserves human dignity, justice and freedom,” Naledi Pandor, South Africa’s foreign minister, said earlier this month. “This is the only reason that we have taken this major step as South Africa.”

—Alexandra Wexler, Fatima AbdulKarim and Shayndi Raice contributed to this article.

Write to Jess Bravin at Jess.Bravin@wsj.com