That's very interesting. Have you collected statistical data on these bit errors? Is it always a single bit error?
I'm assuming you are using GeForce cards and not Tesla cards which have an ECC memory protection mode?
I've tried to collect some statistics on GPU memory errors rates but have found them to be normally extremely rare. The only time I've reproducibly seen them is due to faulty hardware, where the errors become highly reproducible and the GPU needs replacement. The other theoretical cause of bit flips is supposed to be random errors due to cosmic radiation but I've never been able to observe that using memory testing software (though I did only run the experiments in AWS).
Could it be that you have faulty or low grade GPUs? I assume these are all low-cost OEM parts, given your application? Or maybe there's something odd about your data center environment?
Regarding the GPU database application, I think the answer is to just use the Tesla grade GPU with ECC memory enabled.
Generally we prefer AMD cards as most (profitable) mining functions are memory bandwidth dominated. Usually it's a shader unit that gets unstable in the 70-80C range (note that most silicon is rated for higher ranges).
AMD's hardware specifications are more open too which lets one build your own shader compilers and get direct access to the iron.
I'm assuming you are using GeForce cards and not Tesla cards which have an ECC memory protection mode?
I've tried to collect some statistics on GPU memory errors rates but have found them to be normally extremely rare. The only time I've reproducibly seen them is due to faulty hardware, where the errors become highly reproducible and the GPU needs replacement. The other theoretical cause of bit flips is supposed to be random errors due to cosmic radiation but I've never been able to observe that using memory testing software (though I did only run the experiments in AWS).
Could it be that you have faulty or low grade GPUs? I assume these are all low-cost OEM parts, given your application? Or maybe there's something odd about your data center environment?
Regarding the GPU database application, I think the answer is to just use the Tesla grade GPU with ECC memory enabled.