Unfortunately, yes you do.
The destination fee is the cost it takes for a manufacturer to ship the car you just bought to your local dealer. It is also sometimes called the delivery fee. This fee is legit and is a non-negotiable sum passed on from the manufacturer to the dealer and then to you. The destination fee is generally included in the MSRP, but not always.
The destination fee isn't truly negotiable in the sense that it is a flat fee charged by the manufacturer. If you insist on haggling the destination fee, the dealer is probably just cutting the difference from your invoice.
Keep in mind that the transport costs of these cars have to account for hundreds of miles over land or from halfway across the Earth. Even if you happen to live close to the manufacturing plant, the manufacturers will never allow you to simply pick it up at the factory. The factory is not a place for service industry. They are not there to be salesmen and they do not have the staff to coddle visitors, nor the measures to ensure your safety. For the sake of fairness to their dealers, the manufacturers have to spread the delivery cost evenly across all cars, regardless of its distance from the originating factory.
There is no hidden profit built into a destination fee. It is a real cost to the dealer, and they're not trying to pick your pocket without you knowing (for once).