If you want to do math, start right away - but take fewer classes than other majors would. Start with the honor series, if there is one at your school. If you go to an American school, you will probably have completed most of the major by end of second year. If you _really_ love math, think about taking all of the interesting classes in the department or related ones (philosophy). Or take a second major.
If you take my advice, remember the few classes part, please, or you'll be walking into a living hell. When you get the hang of it, you can take one or more extra if you want - you will be able to judge their difficulty by that point.
Math is probably the greatest field, but it can be ruined for you if you approach it in a bad way.