I get the same problems with Zoomplayer and VLC. I found if I run VLC and pause a video, a second instance of Zoomplayer or VLC will play just fine. Kind of annoying but before that I was shutting down ObjectDock just to play videos.
Could this be because these media players are trying to use the same overlay that ObjectDock is using, not recognizing that it is in use? I think that for some reason when video players incorrectly assume that the first overlay is free and uses it, it actually gets marked as being busy/in use, causing 2nd and 3rd media player instances to move on to the next overlays that aren't being used by ObjectDock. This makes me think the problem could be worked around in the ObjectDock source - after all what are these media players doing to mark overlays as being used so that others don't try and use them at the same time? Couldn't ObjectDock do the same thing? Perhaps ObjectDock should register the overlay in the same manner that the video players do, if anything, to make sure that other video players see that it is in use.
I know the ATI x300 series and up support simultaneous overlays but my previous card, a GeForce2 GTS, didn't support more than one. Either way it makes sense to do some kind of workaround in ObjectDock, because at the very least upon seeing that the first video overlay is in use, a video player can fall back to software rendering if another hardware overlay is unavailable. Much better than everyone seeing black like it is now.