Consider an infinite number of circles of diameter D. These circles can be laid on an infinite 2D plane. For optimum packing efficiency, the centre of each circle will coincide with the centre of a tessellating pattern of regular hexagons (think honeycomb). The hexagon's shortest distance across = D. (Or 2r).
The area of each hexagon is given by 6 x (r)^2 * tan(30deg) = (6/sqrt(3))*r^2 = 2*sqrt(3)*r^2
The area of the circle is then pi*r^2.
Area circle / Area hexagon = pi*r^2 / 2*sqrt(3)*r^2 = (pi/2*sqrt(3)) = 90.7%
The packing efficiency of your quarters into a jar won't be far off this. I suggest you measure the volume of a quarter, then do:
Volume of jar / Volume of quarter.
This gives you 100% packaging efficiency. You won't quite make the full 90.7% (because I've assumed infinite plane, etc...) but if your jar is reasonably wide you'll get close to it. It depends on how many coins intersect the edge of the jar.
At any rate - I have a value of 809 mm^3 as the volume of the quarter, and you give me 64 floz for the volume of the jar.
64floz = 1893000 mm^3.
So at 100% packaging efficiency: 2341 quarters, so around 2100 at 90% efficiency. (Which is $525).
The full maths is pretty complicated, but with a simplifying assumption you can actually make it rather trivial.