We chamfer holes for many reasons. To lead a part or another cutting tool into the hole, to remove burrs, to eliminate stress risers or improve appearance. Sometimes, however, chamfering tools can create problems of their own giving us poor results. Number one on the list is chatter. Chatter in this type of an operation is caused by some of the same issues in other machining processes.
1) Weak set-up.
2) Excessive tool pressure (dull tool or too many flutes)
3) Cutting speed too high.
4) Low feed rate.

Some possible solutions are, more rigid set-up, fluteless chamfer tool, slower cutting speed and increasing your feed rate. These are easy to say but are not always easy to accomplish and sometimes you are just stuck with what you have got. Some alternatives that you may want to think about are: 1) chamfering the hole with the spot drill before you drill the hole (just take the spot drill deep enough to create the desired chamfer diameter when you are spotting the hole). 2) Run the chamfering operation in another machine that has a slower spindle speed. 3) Chamfering by hand. 4) If all else fails and you are stuck with running a tool that you cannot eliminate chatter from, try placing a piece of paper under smaller chamfer tools or heavy paper (business card) under larger chamfer tools, between the tool and the work, then cut thru the paper and into the material. Once again, this will act to reduce the relief behind the cutting edge and help eliminate chatter.

Sometimes when we chamfer a hole we create a secondary burr that needs to be removed. I have a little hand held tool that has a handle on one end and 1" diameter 3 flute 120º counter sink on the other end. This is a great little home made tool that is easy to make and peels off those secondary burrs very nicely. If you are in a pinch, you can use a 118º drill bit held in a drill chuck.