Simon Cowell and Lauren Silverman's baby-making affair began four years ago, with the socialite wife more than just a one night stand for the "X Factor" judge, according to US Weekly.

The outspoken reality TV personality, 53, is caught in a twisted love triangle with Silverman, 36, and her husband Andrew, 37, who is a close friend of Cowell. Though the pregnancy came as a shock, both Cowell and Silverman had been seeing each other privately since 2009.

"Lauren has loved Simon forever," said a source close to the New York socialite, wife and mom to 7-year-old Adam told US. "And they started having an affair four years ago."

Though the relationship began as a fling, the two got very close quickly and carry a strong torch for each other. And they will welcome a baby in February, and plan to co-parent together.

"Simon and Lauren are together," said a Silverman family source to the celebrity news magazine. "They talk 100 times a day."

[For our first story on Cowell's baby, click HERE.]

The dad-to-be befriended the Silvermans, who were then-newlyweds, at Barbados' Sandy Lake Resort in 2004. Quickly, the Silvermans and Cowell began fast pals, spending lots of time together aboard the small screen judge's yacht and at his Los Angeles guesthouse.

As the marriage started crumbling, Silverman turned to Cowell for support. The two began "hooking up" and spending more and more time with each other. And by January, the relationship had turned serious and Silverman was consulting a divorce attorney.

Silverman's husband has filed divorce papers citing adultery and naming Cowell as a co-respondent, which means he could be called to the witness stand.

Tell us what you think on the new shocking reveal about Cowell's affair in the comments section below!