It sounds like they were casting the result of a memcmp to a char. A char only has a range of -128 to 127. The resulting overflow means that an arbitrary password hash has a 1/255 chance of landing on 0, but you still have to try a bunch to hit one.
memcmp will probably return always the same value for the same parameters, which may not trigger the error. Trying different values may yield a value whose mod is 0.