This concept was explored in Homeworld, and was a great aspect of the game. To adapt it to Sins however would take quite a bit of balancing. Here's what i think would make it work:
Troop ships would be constructed and purchased (based on resources, not AM) from carriers. That way they could be docked and launched mid battle so they aren't the first thing targeted by an enemy fleet.
They should be a little slower than bombers, but still faster than most ships. Their weakness should be flak frigates and fighters, but should have more armor and hp than bombers.
They should be able to latch on to cruisers and cap-ships only, smaller frigates are too agile and mobile. It should take a certain number of troopships (diff # for different ships) to begin a takeover process, the speed of which could be augmented by more troopships. Upon initiation of the take over process, the target vessel should act similar to being phased out, ie completely disabled but invulnerable (neither side should be able to fire on it and though many people may disagree with this, there are plenty of logical and balance reasons this should be in place). To stop the takeover process, a player must blast the troopships off the side of the targeted ship below the number it takes for take over. If the number of troopships falls below the amount needed for takeover, the process is reset and the ship is freed. Also each ship should have a maximum number of troop ships that can be attached at any given time. The level of a targeted cap-ship should also greatly affect the amount of time it takes to take over, making your level 10 cap ship quite difficult to lose in that fashion.
For each troopships attached to a target ship below the amount needed for take over, the targeted ship should suffer random system failures for brief periods of time, stuff like disabled sub-light engines for 15 seconds or disabled abilities for 30 seconds, etc.
Each race should have research available to boost the effects of troopships and reduce the effectiveness of troopships on their own vessels. (TEC might have something like "Armored Bulkheads"or vasari something like "molecular encryption, which lenthen the takeover time)
And to go with each race, only TEC should actually use manned vessels with marines on board. Advent should use some sort of remote psi nodes allowing psintegrates to slowly take over the minds of the crew, and vasari should use a ship that releases nano machines on a ship that target that ships systems and re-programs them to give you control.
This would make cap-ships with no supporting fleet much more vulnerable.
It would probably be a nightmare to code, but it would be way cool.
