Virtually all the companies that I deal with have a "direct debit" feature where they will send the bills directly to your bank, eliminating envelopes, stamps, and hassle. Some even offer discounts for using this feature.
Even if some of the companies that you deal with don't have this feature, they will usually directly charge your credit cards, which then can be set up on the direct debit feature. To really save, pay the entire amount every month as mentioned earlier. No finance charges, no missed payments, no stamps, no wasted time, etc., etc.
As for using ATM's for money and paying their charges, why not use a card that costs you nothing, pays you to use it, is secure and convenient, and can't be lost or stolen like cash can? Paul mentioned that his credit card company gives discounts on specific types of purchases. There are some that will give you a straight cost savings and/or a rebate.
The secret is to get the system to work for you, not against you. Remember, if you visit a store that accepts credit cards, you are paying for that credit, even if you don't use it. It becomes a cost of business and we all pay the same price. This approach requires a great amount of discipline, but the results in both convenience and savings are worth it.