WhenKate MiddletonandPrince Williamstepped out alongsidePrince HarryandMeghan Markleat Windsor Castle followingQueen Elizabeth’s death, many hoped thereunionwas the first step in reconciliation between the brothers and their wives. However, thesurprise outingwas difficult for both couples behind the scenes.

In his new bookOur King: Charles III, author Robert Jobson wrote that Kate later said the outing was “one of the hardest things she’s ever had to do.”

Meghan felt similarly about the walkabout, where the two couples viewed tributes toQueen Elizabethand greeted mourners gathered outside.

“That’s the longest they had ever been away from the kids, and then on top of it, knowing what’s being said out there — it was very difficult,” a source tells PEOPLE exclusively in this week’s issue.

Kate Middleton, Prince William, Prince Harry and Meghan Markle.Chris Jackson - WPA Pool/Getty

WINDSOR, ENGLAND - SEPTEMBER 10: Catherine, Princess of Wales, Prince William, Prince of Wales, Prince Harry, Duke of Sussex, and Meghan, Duchess of Sussex on the long Walk at Windsor Castle on September 10, 2022 in Windsor, England. Crowds have gathered and tributes left at the gates of Windsor Castle to Queen Elizabeth II, who died at Balmoral Castle on 8 September, 2022. (Photo by Chris Jackson - WPA Pool/Getty Images)

Meghan, 41, andPrince Harry, 38, were in Europe for a series of charity appearances while their two children —Prince ArchieandPrincess Lilibet— remained in California whenQueen Elizabethdied in September. The Duke and Duchess of Sussex extended their trip to attend the funeral events.

According to Kensington Palace,Prince Williaminvited his brother and sister-in-law to join them — but it was Kate who came up with the idea.

“She didn’t want her or William to have any regrets,” a friend of Kate’s told PEOPLE.

Prince Harry, Meghan Markle, Prince William and Kate Middleton.Chris Jackson - WPA Pool/Getty

WINDSOR, ENGLAND - SEPTEMBER 10: Prince Harry, Duke of Sussex, Meghan, Duchess of Sussex, Prince William, Prince of Wales and Catherine, Princess of Wales look at floral tributes laid by members of the public on the Long walk at Windsor Castle on September 10, 2022 in Windsor, England. Crowds have gathered and tributes left at the gates of Windsor Castle to Queen Elizabeth II, who died at Balmoral Castle on 8 September, 2022. (Photo by Chris Jackson - WPA Pool/Getty Images)

Can’t get enough ofPEOPLE’s Royals coverage?Sign up for our free Royals newsletterto get the latest updates on Kate Middleton, Meghan Markle and more!

“Things are strained,” a palace insider says.

Prince Harry cover rollout 5/1

WhilePrince Harryand King Charles have had “positive conversations” ahead of the coronation, there has been no communication betweenPrince HarryandPrince William, and an insider says Harry’s appearance at the crowning ceremony will do little to ease tensions.

“I don’t think the coronation and a big conversation can be conflated,” says an insider.

WhilePrince Harryattends the coronation solo, Meghan willremain in Californiawith their two children. King Charles' crowning ceremony falls on Prince Archie’s4th birthday, and PEOPLE understands that the Duchess of Sussex will spend the weekend celebrating with him and his sister, 22-month-old Princess Lilibet.

Meghan Markle and Prince Harry.

Meghan, Duchess of Sussex and Prince Harry, Duke of Sussex clapping during the Opening Ceremony of the One Young World Summit 2022

“It’s an elegant solution,” royal biographer Sally Bedell Smith tells PEOPLE, adding that Archie’s birthday “gave them a reasonable out for Meghan not to come.”

But those closest to the couple know that the media backlash Meghan faced in the U.K. duringQueen Elizabeth’s funeral last year — and in the months since — played a significant role in her decision not to come.

“Meghan wants to be there to support her father-in-law, but at the same time, the scrutiny she receives outweighs the support,” the close friend says. “There’s always going to be that other side challenging their reasoning, and who wants to put themselves in that position?”

source: people.com