Investigating model mis-specification in simulation-based inference