It's a little hard to tell without more context, but I think what the speaker may have had in mind was the following. The Galilean group only includes transformations between frames in uniform rectilinear motion relative to one another. These frames are usually assumed to be inertial frames. To get the whole process started, you have to identify one frame that you consider to be inertial, and then the Galilean transformations provide a whole equivalence class of other inertial frames.
But you do have to get the process started, and in a Newtonian universe this is fundamentally difficult to do. For example, Newton didn't know that the solar system was accelerating toward the center of the galaxy. Although that particular acceleration is quite small, in principle these hidden accelerations can be arbitrarily large. For example, I could have a uniform wall of matter, somewhere in the universe, that doesn't emit light and has not yet been detected by telescopes. Such a wall creates a uniform gravitational field on both sides. If the wall is infinite in extent, then this uniform field extends to infinite distances from the wall. I can make the field as strong as I wish, simply by making the wall as massive as necessary.
These examples show that in order to make a Newtonian distinction between inertial and noninertial frames, you have to have an omniscient observer. Since omniscient observers don't really exist, it may be more reasonable to expand the symmetry group so that it connects frames that are in uniformly accelerated rectilinear motion relative to one another.
Another option would be to do what we do in general relativity, which is to redefine an inertial frame completely to be a free-falling frame. By this definition, the surface of the earth isn't even approximately inertial.
The reason that the gravitational field is not a physical observable is that in frames accelerating relative to one another, you detect a different gravitational field. For example, in a free-falling elevator, you experience apparent weightlessness, so you measure the gravitational field to be zero. Similarly, Newton had no hope of detecting the gravitational field of the galaxy through experiments confined to the earth.