The down side (if I wasn't in the TARDIS and the Doctor wasn't involved) would be a lack of electricity, lack of good bathrooms, heavy dresses, no computers or movies, and no air conditioning. Penny dreadfuls and The Strand would be good, at least.

Also stinky streets on account of all the horse-manure, and heavy air pollution from all the coal fires. On the other hand, it would be an amazing immersion in history, and you'd see the Victorian splendour as the Victorians saw it.