I was playing in adventurer's league the other day, and one other player and I were discussing on trading spells. He was a pact of the tome warlock and had the "Book of Ancient Secrets" invocation. Now as we wanted to trade spells, I was just giving him my find familiar, my DM said we both had to pay 50Gp for him to gain the spell, and we both had to make a check without modifiers for it to succeed.

We stopped it for now, but I tried to argue to him that only he needs to pay the price and that he didn't need to make the check because this was a spellbook, or at least only one the one scribing it and as an arcana check. So with him being new, he said we should go and search for the answer, he did this because he heard other people say this was right. Is it?

Only the person copying the spell pays the price. The chance of failure only applies to copying from scrolls.

The Your Spell Book sidebar in the PHB states:

For each level of the spell, the process takes 2 hours and costs 50 gp. The cost represents material components you expend as you experiment with the spell to master it, as well as the fine inks you need to record it. (PHB p.114)

The DMG states (emphasis mine):

When a spell is copied from a spell scroll, the copier must succeed on an Intelligence (Arcana) check with a DC equal to 10+ the spell's level. (DMG p200)

Thus, only one person needs to pay the cost, and the chance of failure only applies when copying from a wizard spell scroll. Since you are trading spells with another, the two of you may agree to split the costs; however, the cost is assessed once per spell.

Note: Both characters have to spend the downtime needed for copying spells

Related Questions:

Is there a difference between “spell scrolls” and “spells on a scroll” for copying purposes?
How do you calculate the chance of failure when copying spells in Adventurers League?