I've definitely had them do that when they are old.
Generally plumbers replace supply lines when doing any work. Given the price of the supplies lines, the potential cost of a leak and the general cost of a plumber's labor, it doesn't make sense to re-use them.
I typically don't worry about it (unless the supply is plastic) but if I have a spare I'll swap it otherwise just re-use. All the water supply pieces are rated as safe for potable water and the washers are generally there to stop leaks ( they don't have a constant stream of water passing over them ). If the water was constantly passing over the washers there would be nothing to rub off on your fingers.
As long as there is no leak I don't think it is worth putting too much mental anguish into the state of the washer. If you live in a brand new perfectly finished home then replace the supplies pipes / plan to replace the supply pipes when doing a project and you know they are old. The cost is negligible compared to repairing any water damage unless your bathroom / kitchen already has some amount of water damage.
I did have a plastic cut to size toilet supply crack and cause a flood so if I ever see those I replace them immediately. Not sure what the best practice would be around braided lines with washers.
