The risk of swine flu is worth going to your doctor over but not hyperventilating over. I'm in Melbourne, where we had the world's highest concentration of swine flu cases this winter (our winter - and hence our flu season - runs June 'til August). There was originally containment and quarantine, but after we hit about 2000 cases or so in our state they stopped concentrating on containment and focussed on supporting vulnerable demographics. In the end something like 60-70% of flu cases were swine flu.
So don't lose your head about it. I'm not saying there's no risk - I'm not a doctor, and my advice isn't definitive in any way - but unless you're in some sort of vulnerable demographic (pregnant, undergoing chemotherapy, etc.), it's probably not going to be any worse than a temperature for a day or two. Follow whatever advice they're proscribing in the States, but unless the PAX organisers say otherwise, I wouldn't be cancelling trips or anything.