ABOARD THE HNLMS LUYMES—Belgian Navy Commander Erik Kockx was patrolling the Baltic Sea recently when he got word that a ship on NATO’s watchlist was acting in a suspicious manner. After leaving a Russian port, it had slowed down while passing near a pipeline on the sea bottom. The Luymes sailed toward the tanker to investigate.

Kockx leads a task force in the North Atlantic Treaty Organization’s new mission to police the inland sea that its members share with Russia. NATO in January launched the operation, dubbed Baltic Sentry, after a string of undersea cables and pipelines were damaged by ships—many with links to Russia—that had dragged their anchors.

“We are functioning as security cameras at sea,” said Kockx, whose usual duty is clearing unexploded mines from the busy waterway.

No proof has been found that Moscow ordered or orchestrated the destruction, according to officials familiar with the investigations, though suspicion of it runs high in NATO countries.

Baltic Sentry taps at least 10 ships under NATO command at any given time and splits them into two task groups. It also uses many more ships from the navies, coast guards and police forces of the eight alliance countries bordering the Baltic. New undersea drones are keeping a watchful eye on pipes and cables. NATO surveillance planes from the U.S., France, Germany and occasionally the U.K. take turns scanning the seaway from high above. NATO has also strengthened its military presence on the Baltic, said U.S. Army Gen. Christopher Cavoli , who serves as supreme allied commander for Europe and launched the mission.

Many of the units involved were already performing similar duties. Now they communicate and cooperate much more, orchestrated by the Centre for Critical Undersea Infrastructure that NATO’s Allied Maritime Command established last year, officials say.

Baltic Sentry has largely relied on European forces, but late last month a U.S. Marine Corps detachment of around 40 Marines arrived in Finland for what NATO calls vigilance exercises, entailing drones and small craft. The unit, which is operating alongside Finnish forces, is practicing small-island skills that could prove useful in other regions, such as the Pacific , officials say.

NATO’s goal is to prevent more damage to subsea infrastructure and respond faster if something occurs. In a twist more familiar to police forces and intelligence agencies than militaries, success translates into a lack of action; cops and spies can be most effective when they prevent bad things from happening.

When Kockx’s ship recently approached the suspect tanker, which was part of Russia’s “shadow fleet” of illicit fuel transporters, it regained speed and sailed on.

“It’s quite difficult to prove that our presence helps. It’s hard to say that if we hadn’t been there, something would have occurred,” said Kockx as cargo ships floated by in the distance. But just as drivers become more attentive when they see a police car, “we’ve seen behavior get better,” he said of seamanship. “Ships know they’re being watched.”

In a sign of Baltic Sentry’s unusual nature, the ship Kockx used to inspect the suspect tanker wasn’t a combat vessel. It was a Royal Dutch Navy hydrographic survey ship normally used to inspect the sea bottom. The Baltic, a central theater in two world wars, is littered with wrecks and explosives that still pose danger. Surrounding NATO members are world leaders in finding and disposing of sea mines, officials say.

Today the Luymes’s advanced sonar, submersible probes and other gear are valuable for checking on cables and pipelines. As a precaution, the ship also carries two .50-caliber machine guns mounted on its deck.

Commercial traffic on the Baltic ranks among the world’s densest, with more than 1,500 ships plying its waves on any day, so policing it all is difficult. Further complicating NATO’s sentry duty initially was a lack of comprehensive information about all the critical infrastructure snaking across the sea’s muddy bottom. Details of pipes and cables have traditionally been kept by national governments or private companies. Nobody had a picture of everything.

“We had a very steep learning curve,” said Royal Danish Navy Capt. Niels Markussen , director of the NATO Shipping Centre at Allied Maritime Command outside London.

NATO’s new undersea infrastructure center last year assembled the first unified map of the Baltic’s floor. It tapped in-house software developers to create an application powered by artificial intelligence to analyze traffic patterns, both current and historic, dubbed Mainsail.

“If you know what the norm is, you can more easily identify what’s not normal,” said Royal Norwegian Navy Lt. Cmdr. Dani Johannessen , commanding officer of the speedy mine hunter HNoMS Hinnoy. NATO’s mix of AI and human vigilance aims to flag unusual behavior such as how ships navigate, the ports at which they call, and their reactions to radio calls from patrolling ships.

Johannessen’s ship, a unique Norwegian catamaran design that can rise up on an air cushion to sail into minefields with reduced risk of detonation, is also proving valuable to Baltic Sentry. It carries a variety of undersea gear including a tethered probe resembling a small orange torpedo called Minesniper and a maritime drone that looks like a tiny submarine. It can also launch divers if a task requires humans.

While patrolling for saboteurs, NATO ships maintain many normal routines. The Hinnoy crew regularly drills defensive measures to fend off assaults from small boats—like the suicide bombers who killed 17 U.S. sailors on the USS Cole in the Persian Gulf in 2000. Two of Johannessen’s sailors boarded a speedboat to pose as attackers zooming toward the ship. Other young men and women in the crew donned combat gear and fired off red plastic blanks from machine guns around the deck.

The Hinnoy, the Luymes and ships from Sweden, France and Belgium in Kockx’s task force also drill for coordination at sea to improve operational readiness. Officers say they are already seeing results, cutting their response time upon learning of suspicious activity to one hour, from 17 hours in October.

Still, daily operations are complicated because so many countries and offices are potentially involved.

“There’s a complicated mix of authorities that belong to ministries of the interior, coast guards, police, international organizations, commercial organizations…And, in each case, we work very carefully with the relevant authority,” said Cavoli, the NATO commander, on launching Baltic Sentry.

Officials hope that ship crews bent on damage who last year might have thought they could get away undetected or unpunished will now fear consequences and, at least, losing money by being detained.

“We can’t put a ship over each mile of infrastructure,” said Markussen. “It is deterrence we are working on.”

Write to Daniel Michaels at Dan.Michaels@wsj.com