You may have several wires going to/from the + terminal of the battery. It helps to isolate which one(s) is sucking the 'trons.
ie: one from the alternator (yellow?), another to the fuse box (yellow), fat one to the starter and possibly another small one to the starter solenoid.
Undo the + terminal and take it off. Then separate all these individual wires.
Do you have a multimeter?....cheapo for $15. If so, set it to AMPS (current) and measure between each wire and the + terminal separately (series). See which one(s) register a reading.....and then you can narrow in on the problem.
If it's the yellow(?) one going to the fuse box, try taking one fuse out at a time and see if the current draw stops (check battery wire again). Then when you've found the problem circuit, either leave the fuse out, of it its an important one, search for the problem. If you have a problem in finding it, just undo the + terminal each time you leave the car
Most cars will have a small drain as mentioned (stereo/clock/alarm etc). The current draw from these things shouldn't be above 50 milliamps (0.050 Amps).
TIP 1: most multimeters have a 200 milliamp limit. As soon as you connect it to the wires, it will blow the fuse inside the meter (if the current draw is any higher, which is very probable in your case). I suggest you buy a multimeter with a 10 AMP option in addition to the standard 200mA setting. Measure the fault with this setting first. If it reads below 0.2 Amps (200mA), then you can use the smaller setting.
TIP 2: DON'T check the battery wires with any car doors open. The draw from interior lights will fry your multimeter!! Close the door after each time you check a fuse.
Good luck in chasing those gremlins!!
 if you don't have a multimeter, check for a small spark from each wire. ie: flick it onto the + terminal. Even a small 50mA current will cause a spark.