I have a guest player who is secretly an evil "plant" inside the party causing trouble. He's a sorcerer who wants to use Subtle Spell metamagic to stop the party from delving into his master's old lair, and he has a custom ring item that works as a hands-free arcane focus. He and I agreed that he'd give me signals about what to cast and where, and I'd roll the damage and keep track of the lost sorcerer points.

So the party is sneaking past a sleeping giant and the sorcerer signals fireball on the party to wake the giant. After the battle, the cleric says her passive Perception (which is pretty high, admittedly) should have let her see where the fireball originated from, pointing out that fireball says:

A bright streak flashes from your pointing finger to a point you choose within range

Now, I assume the pointing finger is part of the somatic components of fireball and thus not necessary with the Subtle Spell metamagic. And for spells that obviously originate from the caster like cone attacks, it wouldn't be particularly possible to hide the location of the caster. But fireball's streak effect doesn't seem to be a mechanic of the spell, more just flavor text.

I addressed the problem in-game by rolling a Sleight of Hand check against the cleric's passive Perception using the sorcerer's stats and managed to keep it hidden and "it happened so fast you saw it came from behind you but nothing else", but is that the right solution?

But the more I think about it, the more complicated it seems. Does it follow the stealth rules where any attack, even long distance arrows, immediately gives away the position of the attacker? Would a subtle psychic scream (XGtE, p. 163) be traceable to the caster despite having no visible ties to the caster?

Xanathar's Guide to Everything has optional rules for this situation

Dungeon Master's Tools > Spellcasting > Perceiving a Caster at Work

If the need for a spell’s components has been removed by a special ability, such as the sorcerer’s Subtle Spell feature or the Innate Spellcasting trait possessed by many creatures, the casting of the spell is imperceptible. If an imperceptible casting produces a perceptible effect, it’s normally impossible to determine who cast the spell in the absence of other evidence.

The actual casting of the spell is hidden¹ by virtue of the subtle spell metamagic.

The effect of casting the fireball spell is to create

A bright streak flashes from your pointing finger to a point you choose within range and then blossoms with a low roar into an explosion of flame.

The streak of light coming from your finger is pretty clearly "other evidence", however it has some caveats as to how strong this evidence is. Your finger does not have to be pointing from an outstretched arm, not does it need to be obvious.

The bright streak however is an obvious and perceptible effect of the spell. If the Sorcerer was behind the cleric the cleric would absolutely notice a streak of light coming from behind them, but they may not notice who it came from. So would the majority of the rest of the party.

_{1. Note there is a potential argument to be made that the arcane focus is still perceptible under normal circumstances, however the special ring item you have created for the PC removes this from consideration from a detection perspective.}

Passive Perception does not give you 360 degree vision

PHB > Ability Checks:

Your Wisdom (Perception) check lets you spot, hear, or otherwise detect the presence of something. It measures your general awareness of your surroundings and the keenness of your senses. For example, you might try to hear a conversation through a closed door, eavesdrop under an open window, or hear monsters moving stealthily in the forest. Or you might try to spot things that are obscured or easy to miss, whether they are orcs lying in ambush on a road, thugs hiding in the shadows of an alley, or candlelight under a closed secret door.

If the Sorcerer were in any way intelligent they would know the streak of light is produced by the fireball and would wait to cast it until no one in the party was looking at them.

Thus the cleric would not "automatically" know where it is coming from due to their passive perception.

Casting fireball is not an attack

The rules you mention about unseen attackers state

If you are hidden — both unseen and unheard — when you make an attack, you give away your location when the attack hits or misses.

The game rules define attacks pretty clearly (in the same section of the PHB as the unseen attackers rules).

If there's ever any question whether something you're doing counts as an attack, the rule is simple: if you're making an attack roll, you're making an attack.

Fireball requires no attack roll, thus it is not an attack.

As a result the unseen attackers rule section does not apply.

Would a subtle psychic scream be imperceptible?

Yes it would. It requires only somatic components and produces no perceptible effect that would indicate it's point of origin.

Unclear, but it doesn't disguise fireball by default

It is worth noting that the spell description is in addition and, more importantly, separate from material, verbal, and somatic components.

For example, the 2019 Sage Advice Compendium states, in regards to suggestion,

Verbal components are mystic words (PH, 203), not normal speech. The spell’s suggestion is an intelligible utterance that is separate from the verbal component. The command spell is the simplest example of this principle. The utterance of the verbal component is separate from, and precedes, any verbal utterance that would bring about the spell’s effect.

I use this evidence to conclude that in the case of fireball, the bright streak from your pointed feature is a required portion of the spell's effect, outside of the verbal and somatic components that the metamagic removes.

Casting a spell attack, notably, does automatically reveal you

On PHB 195, under Unseen Attackers and Targets:

If you are hidden - both unseen and unheard - when you make an attack, you give away your location when the attack hits or misses.

The section does not specify for spells with a save DC (or other effects), so the default would be that it doesn't reveal you.

Subtle Spell only covers somatic components, not other spell effects

By the wording of Subtle Spell:

When you cast a spell, you can spend 1 sorcery point to cast it without any somatic or verbal components.

If we look at the basic rules on Somatic Spellcasting Components, we're given:

Spellcasting gestures might include a forceful gesticulation or an intricate set of gestures. If a spell requires a somatic component, the caster must have free use of at least one hand to perform these gestures.

Somatic components are specific hand gesture requirements necessarily for all somatic spells. This is separate from gestures specified within the spell itself, which are more akin to being part of the effect of the spell. Subtle Spell means you don't need to do the intricate hand signs or verbal parts to cast it, but once cast the effect takes place, which includes "A bright streak flashes from your pointing finger to a point you choose within range".

How perceivable/traceable this is, however, is up to the DM and the situation. I'd rule similar to how you did in the example you gave since even if you rule the hand does nothing, the bright flashing streak isn't a somatic component, and it has to come from somewhere.