The answer is simple: you hold their hand.

(Source: The experience of raising a toddler who is currently wielding a highlighter under my desk while I write this. I'm keeping her fed with sufficient paper; so far the walls and carpet are unadulterated)

One of the more terrifying things for a parent in today's society is cars. Cars move fast. You don't realize how fast they move until you are responsible for the life of a little booger monster of vile... angel. Your angel has no idea what 15mph will do to a body, much less 45mph. Your angel gets what is coming to them is completely innocent and needs to be protected in a remarkably deadly world.

Excuse me one moment. The sharpie has disappeared from my desk. I really need to move those to a higher shelf. I do this to myself, honestly.

The solution every parent learns is to hold the child's hand. Not only does this provide a critical physical safety preventing them from tearing out like a helion in front of a car, but it also provides a remarkable amount of feedback. The human body is geared for touch. You learn a remarkable amount about what the child is about to do just by feeling the way they respond to your hand. You can sense what they are feeling and they can sense what you are feeling. It gives you a way to guide them along in the right direction without yanking their arm from its socket constantly. (At least on good days)

So I would expect a child sorcerer would be subjected to the same technique. You'd hold their hand so that you can sense what they are doing. And, if they're about to cast a really big spell, it provides a lever to manhandle gently ensure the child's larger spells don't complete.

As for the tantrums, a child can already do a remarkable amount of damage. We keep a lot of our strength in reserve. It's how a mother can lift a car off of their child. Well, during a tantrum, the child has access to their full strength reserve and can do some rather remarkable things. As such, I would expect a similar approach as we have to tantrums today. You remove the child from the scene, and make them sit outside until they calm down. I'm assuming that, in these situations, the parents know enough to not be caught off guard. Any parent that lets their guard down during a child's tantrum is only human going to learn a lesson very quickly.

I would expect parents to learn how to damp the effects of their children's magic. This would get into the particulars of your spellcrafting system, but if the parents can exert a continuous effort to keep dangerous spells from materializing, that would be a major start, and fit in well with the dead tired fatigue joys of parenting.

Tantrums do cause disasters, on a semi-regular basis. However, once the parents have thus demonstrated their inability to control sufficiently powerful toddlers, the children get taken from them and placed under the control of the magically-powerful sorcerors who run the government, who adopt them into their families.

This both preserves the power structure of the current government by ensuring that no magically powerful individuals arise outside of it and ensures that all such sorcerors have a family tie to the current government, it also minimizes the damage that such children would cause. It's likely that these societies would treat such an adoption as an honor, since it would elevate the children from the common masses to the wizard nobility.

Children's "power" scales with age

Look how hard heavyweight boxers get punched. How could a child ever learn boxing if they would get punched that hard!

Well, 5 year olds learning the basics of boxing aren't in the ring with Wlad Klitschko, they are in the ring with other 5 year olds. Those five year olds can't hit hard enough to do any real lasting damage to another 5 year old.

Similarly, a 5 year old sorcerer-to-be simply doesn't have the mental or mystical development to cast a spell as powerful as a fireball. The child sorcerer is not threatened by the spellcasting ability of an adult sorcerer. A five year old sorcerer is actually not a danger to a five year old. Therefore, it is safe for the five year old to practice his or her spells. The most they can do is cause some mischief.

Has magical capability been a part of this race and society 'forever?'

If it is an evolved rather than recently god-granted(or such) capability of the species, it stands to reason that many previous systems have existed that, by the existence of your current one, have failed.

Your sorcerors and muggles live hand in hand today but did not and do not always, nonetheless your society has found an equilibrium.

Over thousands of years of selection, sorcerors who are prone to the most extreme of behavioral and emotional tendencies have purged themselves from the gene pool, being a clear and present danger to not only themselves but those around them.

The trials and tribulations societies would have undergone to reach an egalitarian point (a thoroughly unnatural state) would have necessitated not only the children to experiment with their magic, but society to experiment also on ways of controlling it.

Folklore and magical theory(heh) have many routes to impairing magical expression, Christopher Nuttall used 'Zeros' in general and particular spellforms woven into a person's magic, others use cold iron, null-magic zones, pacts, etc

But aside from those thematic forms, it is natural that a child of 2 or 3 years capable of the application of significant force is more likely to injure themselves than anyone else, lacking as they do the dexterity or experience to safely direct that force.

In the absence though of particular impairment methods or selection-by-death characteristics, the behavioral instruction of children likely to develop magical capability would begin at the very earliest opportunity 'Do not use magic to do evil' would not just be a catchphrase, but a necessary bedrock of instruction and indoctrination. Before guardians taught their charge not to touch hot things, they would inculcate restraint and self control, we allow children freedom because we can, a society with 2 yr old sorcerors does not have that luxury when any one of them can become a killer, or a dead child, at any time.

Cort Ammon's answer is nicer but the child is 'realistically' (imo)far more dangerous than the car, being as it is less predictable except insofar that a normal child will certainly try to deliberately cause harm to somebody at some point.

One way around this would be to massively restrict the impact of 'instinctual' magic, to require magical expression be orderly and precise in order to function effectively.

Good luck.

If magic-wielding toddlers were such a menace to the muggles population, I'm sure there would be entrepreneurial adult magicians who will sell magic amulets to the muggles to protect them from toddler magic. Due to the huge demand among muggles, and plethora of adult magicians, the amulets will be reasonably priced and even imported from third world countries. As toddlers grow into teens and adults they may learn more powerful magic to bypass those amulets, but that would be equivalent to teen thugs buying handguns to attack others, and it can be dealt with by a powerful guild of criminal justice sorcerers.

Anti-magic stuff

You can put anti-magic bracelets on your kids so that their magic is suppressed. They take it off in magic lessons and at home sometimes but keep them the rest of the time and are only authorized to remove them when adult.

The magic bracelets also have a magic detector so that when child has a fit so strong the bracelet can't handle it, magic police can get to the place to handle it.

Aside from the already marvelous answer about just good parenting @Cort Ammon.

For an analogy take a look at learning to walk... in an Adult. This happens after serious life events such as car crashes, or strokes (by no means the only ways). Adults have well developed muscles (quite) capable of causing harm. It still takes at least a few months to learn the co-ordination required to cause harm. Severely affected adults take years more. The muscles themselves are strong, but in an uncoordinated state they are effectively useless.

So treat magic the same way. It is a thousand muscles. Some people have many more muscles, some have far bigger muscles. Even if the child has a great many muscles of a great size, for the most part any venting is just going to cancel itself out. This is because the Child lacks the co-ordination to actual make the magic effective.

In this sense a really powerful toddler might unleash the flood gates of hell, only for hell to basically fight itself, and the only real effect is a few puffs of air moving in random directions. Certainly enough to truly frighten most animals, but nothing truly dangerous. Unless of course the adults sorely need an education in a child proof house/workplace...

A prodigy would be like many prodigious children, they would start to become apparent around 4ish. At this point the control over their magic would be 2-3 years in advance of their peers. Yes they could actually hurt at this point. The parents (even magically weak/nil parents) would by now know how to handle their kid. They weren't spending those years idle, they are older, cannier, and have been intelligently finding help. If the parent hasn't, then its parent training time, and society would have kicked in some social mechanisms for controlling the child (as cruel as that might sound, otherwise society would have perished).