Litter box issues

My ferret is litter box trained,he just uses the litter box 1 time and not again unless i have cleared all poo and pee,if there is a single turd when he needs to go in his litter box,hell use the corner in front of it,but then hell keep using that same very corner even if a turd is in that corner,the only reason at most 3 turds is in his cage is overnight,its bumped down to 1 turd in his box,that as soon as i wake up i have to remove before he wakes up and then the occasional turd in the corner in front of the litter box,i know the issue is the 1 turd in his box bc i clean it all day as soon as i see 1 i remove it and hell use box no problem....why is this will he be like this forever i have a ferret nation double unit coming in and another litter box, but i think its crazy 1 turd in his box is to much for him to use it again....luckily i stay at home otherwise there would be poo everywhere!