Let's see--what I'm interested in knowing about his schedule--
Very roughly, hour by hour--maybe do a particular day with a typical schedule so you are writing down what you are actually seeing happen instead of what you think you remember--record very very briefly, one or two words when you can.
When does he get fed; is his food always down or is he fed at certain times; when and where does he poop and pee and how often (fairly specific here--frustration can make for exaggeration
When and for how long are you out there; what the dog does in terms of pottying when you are there; what you do to guide him (leash, push, encourage, don't notice--sorta like that); how he responds to your guidance
Oh--you may have said this but I can't remember--are his poops are solid and formed or are they sometimes soft or soupy? Do they have an extremely offensive odor (or do they just smell like poop) What color are they? How big are they (I have one dog who poops more often than the other, but each one is smaller--I think the daily amount is actually about the same) Some of those symptoms could lead you to think that the dog has a physical illness which might need a vet check, some might point toward needing a different food, some may be this dog's idiosyncrasies)
As far as pee goes--do you ever see him dribble as he's walking, without seeming to know that he is going; does he wake up wet; does he try to pee without producing anything--those could be signs of a bladder infection. Not as common with male dogs, and unlikely if you are also having trouble with pooping too, but you never know.
One problem I'm seeing is that when you're there, you enforce him going to a certain spot, but you can't enforce that when you're not there (which probably ends up being the majority of the time) Every time he goes in the wrong place, he is basically learning that it is OK to go there.
It does seem that if you can get the poops down to a smaller frequency, a lot of this bother would solve itself--if you only have to pick up one or two a day, it's not nearly as annoying. Do you have a pooper scooper (a flat shovel-like scoop on a long handle with another long handled piece to push poop onto scoop)? It makes picking up a lot easier and therefore less annoying.