There are, in my experience, three major factors--make that four--that come into play with people we quite unkindly call 'hoovers' (because of their ability to suck their tanks down quickly).
1) As ikan noted, buoyancy and trim.
• If you are overweighted you will waste a lot of air pumping and dumping gas into and from your BCD in an effort to stay neutral.
• If you are underweighted you will have to breathe in fast and constantly exhale in order to stay down.
• If you have poor trim (you are not horizontal in the water) you expend a lot of energy pushing water ahead of you instead of slicing through it (think about how you spash somebody with your flat hand on the surface of the water), which means you are working harder and getting nowhere fast while you deplete your supply of air in the process.
2) Breathing technique. Shallow breathing leaves a lot of 'dead air' in your airways that is saturated with CO2. Excess CO2 in the respiratory system is what triggers the breathing impulse by the brain. Breathe all the air out of your lungs on every breath to get rid of all that excess CO2 and you will find that you can breathe more slowly and evenly, and therefore improve your air consumption.
3) Large lungs. Big men have big lungs, big muscles, etc. They simply consume more air! I had a dive instructor as a customer last week. This guy dives six days a week in Hawaii and has 7000 dives to his credit. He just couldn't stay down for a full one hour dive at our dive depths with a 12 l tank, and it was obvious this wasn't a newbie problem! Once I convinced him to dive with a 15 l tank, everybody was happy!
4) Fitness and health. Unfit people just breathe more heavily. Fitness is not a question of physical size--that is, slender people with lovely bodies can be horribly unfit, while people with a little lard on their frames can be very fit! If you get out of breath climbing stairs or walking up a slight hill, you should take measures to improve your cardiovascular fitness. Hand in hand with this is the temporary state of your health. When you're ill, or recovering from an illness, you will breathe down your tank faster. It's just a fact!