An online nude-photo scam is ensnaring thousands of teen boys and causing emotional trauma.

Scammers posing as teen girls befriend boys online, share nude photos of a girl and then ask for nude photos in return. Once the boy reciprocates, the schemer demands money be sent by a peer-to-peer payment app and threatens to share the boy’s photos with his social-media followers if he doesn’t pay.

That is how law-enforcement officials and child-protection experts describe a growing wave of online predators targeting teens. Previously, online sextortion—as they call it—largely involved pedophiles blackmailing kids into sending photos or videos. These new scammers focus on money, law-enforcement officials say.

Three years ago, the National Center for Missing and Exploited Children received fewer than 10reports of this sort of financial extortion. Last year, the congressionally mandated nonprofit received more than 10,000—and has already received 12,500 this year.

In a survey of more than 6,000 teens and young adults in the U.S. and five other countries conducted by Snapchat’s parent company, 65% of respondents said they or their friends have been targeted in schemes where unknown attackers obtained explicit personal imagery or other private information, then threatened to release the material to friends and family. The majority of those approached were boys.

Boys are easily lured because they respond to sexual photos more readily than girls do, says Lauren Coffren, NCMEC’s executive director. As soon as boys send their own photos, she says, “The immediate response back is, ‘I’m going to ruin your life if you don’t pay me.’” More than a dozen teen boys in the U.S. have killed themselves in instances involving these scams, she adds. By law, cases involving child sexual materials need to be reported to NCMEC.

An international crime

I recently spoke to a mom in Hatfield, Mass., who said her 15-year-old son was targeted in a sextortion scheme in July. Someone claiming to be a teenage girl followed him on Instagram, and they chatted via direct message. The two then added each other as friends on Snapchat, where they exchanged nude photos.

As soon as the boy shared a picture of himself, the perpetrator told him to send $200 or the photo would be shared with his Instagram followers, she said.

The boy sent $30 via Apple Cash, then another $40, then $25 from a gift card. The person shared his photo with at least one of his friends, via Instagram. His mom received notifications from Apple about the money transfers and asked him about it. She said he started crying and confessed to her all that had happened.

She called local police and had her son provide statements. She also reported the scam to the Federal Bureau of Investigation, which notified NCMEC, according to documents I reviewed.

Scammers such as these are often based in West Africa, outside of U.S. legal jurisdiction, according to statements from the FBI and Department of Homeland Security. However, there have been some arrests made. In August, two Nigerian men were extradited to the U.S. in an extortion case that involved a Michigan teen’s suicide.

The Massachusetts teen’s mom had him block the offender, which ended the threats. But she says she’s horrified to think what could have happened if she hadn’t discovered the scam. The boy told his mom he is terrified about the impact such a blunder could have on his future.

Scammers typically try to get photos via Snapchat, since its disappearing messages leave less of a trail, law-enforcement officials say.

“There are some bad actors that seek to exploit some of Snapchat’s hallmark features, but we are determined to make sure that Snapchat is a hostile environment for this kind of activity,” says Jacqueline Beauchere, Snap’s head of global platform safety.

An Instagram spokeswoman says the Meta-owned platform removes content and accounts that attempt to extort, harm or solicit inappropriate imagery.

How to protect teens

Teens don’t have to feel helpless if they become targets of sextortion. Here are things parents and teens can do:

Talk to your kids. Make teens aware of these types of scams so they can spot them while online. (One giveaway: Girls generally don’t ask boys to share nudes.) If a boy is solicited by a stranger, it’s probably not a teenage girl.

Advise your kids not to engage with anyone who asks for a nude photo or threatens to create fake nudes. Some bad actors coerce teens into sharing their social-media passwords. Make sure your child knows never to share account passwords. They should block anyone who starts requesting nudes or making threats, and flag any incidents to the social-media network.

Protect their social-media accounts. Snapchat and Instagram have default settings that can make it hard for strangers to find and message accounts whose account holder’s age is listed as under 18. Teens should keep their accounts private and only accept friend requests from people they know. Also, they should set up two-factor authentication on all social-media accounts.

Keep an eye on their online friends. SnapchatInstagram and Discord allow parents to see who their kids are communicating with and following—and who is following them.

Supervise their payment apps. The criminals behind these extortions usually instruct teens to pay them via peer-to-peer payment apps. You can restrict whom your kids can send money to by creating sponsored accounts for teens on Cash App and Venmo. For teens who use Apple Cash, you can do what the Massachusetts mom did: Set up Apple Cash Family for money limits and transaction notifications.

What to do if a child shares photos

Stay calm. Yes, your kid made a mistake. Blaming him might make things worse. Explain that he is a victim of a crime and that you need to work together.

Don’t pay. Paying extortionists or providing them with account passwords doesn’t stop the crime, Coffren says. The scammers might keep asking for money or share the photos anyway.

Report the incident. The tech companies can only stop criminals if they know about it. Snap recently created a new reporting category to expedite these cases: Under Snapchat’s reporting menu for “nudity or sexual content,” the first option is “They leaked/are threatening to leak my nudes.”

Though Snapchat is known for its disappearing messages, if the company identifies content involving illegal behavior, it retains the messages for an extended period to aid law enforcement.

Social-media companies are required to report all cases involving child sexual content to NCMEC, but victims and their families should report the incidents to NCMEC, too. The center can coordinate law enforcement and work with tech companies. People can make a report to NCMEC’s CyberTipline online or by calling 800-843-5678.

Local police departments aren’t usually equipped to handle this kind of crime, but most states have Internet Crimes Against Children task forces.

Get photos taken down. Teens can create a digital fingerprint, known as a hash, of their nude photo, and report it anonymously to NCMEC via the center’s Take It Down program. The photo never leaves the teen’s device, but NCMEC can share the hash with social platforms to search for copies.

—For Family & Tech columns, advice and answers to your most pressing family-related technology questions, sign up for my weekly newsletter.

Write to Julie Jargon at Julie.Jargon@wsj.com