While the rust was not helping, it shouldn't offer enough resistance to break a new screw. More likely the box threads either did not match the screw or the threads were damaged. I doubt you can get a screw extractor that small, so you'll need to drill out the broken screw. The proper drill size is ideally just a bit smaller than the root diameter (measured across the bottoms of the threads) of the screw. Try to drill accurately to not damage the box threads any more than they are.
It may take some fiddling with an awl to get all the screw bits out, or sometimes the cylinder comes out more or less intact. You may wish to treat the rust with rust converter just to reduce its spread, it will not inhibit the next step.
To install a new screw that will not bind, obtain a self-tapping screw to anchor the outlet. Ideally it should match the threads of the outlet screws, but honestly any similar sized screw will work since it will be the final anchoring screw and can be reused any number of times into the future for future replacements or other service work. Use generous amounts of penetrating oil when installing the screw.
(I'm pleased to see the 'gap' I saw in your other picture was not due to an odd sized box, but due to the edge of the material beyond the box. Just what material is the edge we are seeing? Just curious, doesn't really matter.)
