Tricky question... swapping brackish and freshwater tanks?

I have a 20 gallon established fresh, and a 10 gallon brackish. I want to put my puffer into the 20 fresh, and my platy and 3 neon tetras into the 10 gallon. Is there bacteria in each filter that needs to stay with each kind of fish? is this do-able? thanks!