BRUSSELS—NATO was created to fight a shooting war against Moscow. Now Secretary-General Mark Rutte wants the alliance ready to fight unconventional battles against unseen enemies.

Sabotage, hacking and terrorist-type attacks in the U.S. and Europe are escalating, Western security officials say. Politicians and intelligence agencies finger Russia, China, Iran and the West’s other adversaries for most of the incidents, but attributing blame is often difficult. Responding can be even harder.

Rutte and national leaders in the North Atlantic Treaty Organization say they must boost cooperation and prepare response plans before the string of low-grade attacks reaches a level closer to hostilities that could trigger military action.

Russia presents the most immediate threat , Rutte and other allied officials say, because it is using what are called hybrid attacks in response to the West’s support for Ukraine’s resistance to Moscow’s invasion .

“Russia is really ramping up,” Rutte said in an interview. NATO must “make sure that deterrence is there and that the Russians don’t try anything which is really risky.”

He said NATO and its members are responding by expanding exchanges of intelligence, conducting more exercises, boosting cyber defenses and enhancing protection of critical infrastructure. The alliance is now developing a plan for response to hybrid attacks that it aims to adopt at its annual meeting next summer in The Hague.

NATO members have primary responsibility for response to attacks, handling them at the national level, Rutte said, but the alliance can play a critical supporting role by pooling information and sharing best practices.

“We are already working very hard on making sure that we exchange information, [and] that we have the intelligence-gathering,” he said.

Secrecy and mistrust have long impeded intelligence-sharing among allies. It has improved significantly since Russia’s invasion of Ukraine, Michal Koudelka , director of the Czech Republic’s Security Information Service, said.

NATO in May launched a Maritime Center for Security of Critical Undersea Infrastructure, based near London at its existing Allied Maritime Command. In July, members agreed to create a new Cyber Defense Center to combat increasingly sophisticated threats. The center, to be based at NATO’s military headquarters in Belgium, will bring together military, civilian and industry experts.

Europe, rather than the U.S., has so far borne the brunt of attacks, although Russia and China have also tested U.S. resolve.

Russian hackers in 2021 shut down Colonial Pipeline , the largest fuel-transport system in the U.S., by breaking into its networks. Moscow blamed the hack on rogue elements, but the shutdown prompted President Biden to call Russian President Vladimir Putin and warn him that Russia had crossed a red line.

U.S. officials say China has installed malware in American infrastructure from energy grids to port cranes. Authorities have accused China of hacking U.S. cellphone networks and utility companies.

Rutte, who served as prime minister of the Netherlands for 14 years until July and took over at NATO in October, said Western leaders over the past decade have increased their understanding of adversaries’ untraditional hostilities.

“I think we lost our naiveté, all of us, and clearly came to the conclusion that collective defense” entails combating hybrid attacks, he said.

For more than a decade, Moscow has toyed with the politics of its perceived rivals through hacking and influence operations, but its efforts went into overdrive after Russia’s invasion of Ukraine in 2022, said Eric Green , a former senior adviser to Biden on the National Security Council.

Russian plots to send incendiary devices onto U.S.-bound cargo planes and a plot to kill the chief executive of Germany’s Rheinmetall have served as wake-up calls about just how serious the Russians have become, according to Green.

“Since the war started, they have drastically raised their risk tolerance,” Green said. “Three or four years ago, if someone proposed shooting the head of Rheinmetall, that would have been stopped. But now the attitude seems to be, ‘OK, let’s do it’.”

Russia, which has set up a hybrid-warfare headquarters in Moscow, is playing a game that is kinetic as well as psychological, diplomats and intelligence specialists say. So countering these attacks is a nuanced mix of hardening Western defenses against cyber forays and sabotage, but also playing down the significance of attacks when they are successful.

“What we don’t want is a situation in which everything that happens anywhere is suddenly attributed to the Russians,” a senior NATO official said. “It makes them look bigger than they are and more effective than they are. It instills anxiety.”

Fragile undersea fiber-optic lines and pipelines of NATO countries have emerged as a particular concern. Since Soviet times, Moscow has fostered a special arm of the military to map out pipelines and, more recently, fiber-optic cables for possible sabotage. Its operations have been beefed up as relations between Moscow and the West collapsed over Ukraine, the senior NATO official said.

China, which has been amplifying Russian disinformation campaigns against Ukraine and NATO, appears to be complicit in some undersea sabotage , the NATO official said.

Twice recently Chinese ships damaged undersea cables in Europe, the official said. One of them, the Yi Peng 3 bulk carrier , was surrounded by European warships last month, after it dropped an anchor in the Baltic Sea and dragged it for more than 100 miles, severing two critical data cables.

Finnish Foreign Minister Elina Valtonen said in an interview that NATO’s new center for monitoring undersea infrastructure will be an important asset in preventing attacks and for quickly repairing parts that are sabotaged. Companies running the pipelines and undersea cables are adding sensors to their equipment that are now feeding into the monitoring center, the senior NATO official said.

“In order to build deterrence, we do not just need to have the right response,” Valtonen said. Members of NATO need a “shared picture of what’s happening across the alliance.”

Moscow’s use of hybrid tactics, which it has honed over the decades since the Cold War, is likely to mushroom further in the coming years, in part because its conventional war in Ukraine has lately been so costly, said Seth Jones , a political scientist and defense expert at the Center for Strategic and International Studies.

The U.S. and Europe have so far been timid opponents against Russia in its hybrid tactics, Jones said, in contrast to the days of the Cold War, when the U.S. ran its own robust covert operations inside the Soviet Union.

“I think there’s a very strong rationale to be thinking about the growing threat that the Russians pose,” he said, adding that discussion of a coordinated response from NATO has been building for years.

“We really have to ramp up,” NATO’s Rutte said of allied preparations for unconventional warfare. “We really have to take a big next step to make sure that we protect ourselves.”

Write to Alan Cullison at alan.cullison@wsj.com and Daniel Michaels at Dan.Michaels@wsj.com