We did that. Not letting them eat big chunks of dirt or mud, but not being too strict with clean hands, picking up food from floor etc. And yes few times for some reason they chugged in some small piece of outright dirt.

Its much easier when the amount of nasty diseases they can get from dirty hands is minimal on western world, compared to many parts of 3rd world countries with next to 0 sanitation and usually no sewage separation.

No allergies so far but its too early to tell.