How hard is it to handle a double major in business and physics?

Whether physics is difficult or not depends on how comfortable you are with mathematics, manipulating/deriving equations (as opposed to memorizing a large number of facts), and problem-solving in general. If that's the sort of thing you enjoy (or think you might enjoy), then it might be a reasonable option. As Horatiu noted, you can take physics courses without majoring in it, so try a couple and see what you think. (Be aware that standard freshman physics, like calculus--which is probably either a prerequisite or a co-requirement--is frequently seen as a "weeder" class. If you have trouble with it, then physics might not be for you.)

Physics and business are not going to overlap a great deal, so if you do choose to double-major, be prepared for a lot of work and possibly a fairly restricted schedule for your remaining years in college--you may have to plan things out a couple of years in advance in order to get the scheduling to work out right. Both can be good foundations for multiple careers, but a bachelor's degree in either or both isn't necessarily going to open doors on its own. You'll want to do some internships at a minimum, and grad school (MBA, MS, or PhD) is necessary for a number of the more prestigious opportunities, though not all. But even for jobs in which it's optional, you'll be competing against at least a few candidates who do have advanced degrees, so experience via internships can be critical if you want to avoid grad school.

I happened to like both math and physics a lot, so neither was all that hard for me. But there are always exceptions--for example, group theory and thermodynamics did not come nearly as easily for me as everything else did. Some of that will depend on the professor and/or the textbook, too, so take more than one class before deciding, if possible.

The better question to ask is ... "Do I need a double major?"

I pursued a double major while in university at the expense of taking classes outside my comfort zone to expand my horizons ... an art history class, a dance class, an anthropology class, a physics class. Remind yourself that you don't need to major in a subject to take classes in that department. In my experience, having a double major didn't make me a stronger candidate in the eyes of recruiters, whether for employment or graduate studies. Major in the subject you are most passionate about and build the rest of your studies around that. While many of your peers may choose to double major because their friends are doing so, you can side-step that reflexive action to take control of your own studies and personal development.