At the start of the episode, we see three statues of children on the fountain. Then, there's only two. The giggling noise when Rory came back with the coffee, was the confirmation that the child statue touched Rory. So that's how he got sent back. Rory himself never realised it.