Troop Transports blow by my ships guarding planet

I've never tried using guard or sentry and while in a perfect world I agree that it would be nice for these functions to work as expected I don't find this lack to be any great difficulty because leaving a single even unarmed ship in orbit will stop a troop transport from attacking a planet.

As to why this function may not appear to work perhaps there's a particular distance at which this function gets activated. If the opponents troop transport ends it's previous turn outside this distance but has enough speed to reach the planet in it's next turn then the behavior you've described is perfectly understandable. You could also simply scan for enemy troop transports yourself and attack those within range of your planets. That's what I do anyway.

A fast enough troop transport can just blast through an outlying picket before they can react, but like mumbles said, you should have kept a couple of ships in orbit ("on" your planet) just in case they break through. My suggestion: use packs of obsolete ships and don't build an orbital fleet manager, that way the enemy has to burn through a bunch of turns killing them one by one if they try to rush through, which gives your outlying pickets a chance to turn around and give them an azzfull of laz0r

You can also use Evil Stormbringer's patented method of merely putting a tiny hull in orbit around each planet. This ship has no engines and no weapons, it's merely a tiny hull with nothing else on it, but it stops an unarmed transport from attacking the planet. It has zero maintenance and can be built in one turn on any planet even with no industry other than the initial colony building. It's a great idea and a very useful tactic.

One caveat: these tactics do assume you've got picketing forces ready to swoop in if and when the enemy sends battlecruisers to wipe away that lone tiny hull like a hanging booger.