If you have not used 19th century census returns before please bear in mind that the households shown are often not in order within each street.  Even where house numbers are included these were different to current day numbers which were introduced the following century. For example census takers did not always walk up one side of a street and then down the other but may have criss-crossed to both sides and even then not in any particular order.

Nevertheless, some patterns of order are identifiable by using known or named buildings such as pubs and schools etc., It is possible for you to identify these patterns where they exist and so establish where a person lived.  To this end street maps of the period are included with some of the census years' transcripts (see transcripts for 1881).  

There may be some errors within these transcripts but they can always be checked by purchasing the original entries via websites that provide them although, of course, even the originals themselves were subject to human error.