Yes, those are neutral defenders. Turning off pirates stops them from raiding. They will send out parties and attack player and AI owned planets.
There is one more way to fight neutrals as well, TEC have research for causing small neutral parties to spawn at random on enemy planets.