The physics/science of it is essentially that by grinding it very coarse, you are creating a much larger pore space for water to infiltrate. More pore space means larger pores, which means more water in those pockets of coffee grinds to leach out the flavor (and caffeine). It also means less water gets trapped in the pore space when you drain it out.

The way cold brew works is that you are substituting heat for time - heat is essentially ripping the caffeine and flavor from the grounds, and a finer grind will result in a stronger flavor (hence, espresso grind is generally a stronger flavor than standard filter grind). More heat and pressure allows this to be done quicker, which is the principle behind espresso - very hot water forced through tightly packed very fine grounds - done in 10-20 seconds. Cold brew, by contrast, is a slow leaching process. The larger pores hold more water, and you slowly leach the flavor out of the grinds into the water in the pores. The longer you allow it to leach, the stronger the the coffee will be, but also the more bitter it will get. Similarly, the lower the ratio of water:coffee, the more concentrated the resulting brew will be.