It seems I was wrong and the phenomenon does really exist.

https://en.wikipedia.org/wiki/Prairie_madness

Some settlers specifically spoke of the wind that rushed through the prairie, which was loud, forceful, and alien compared to what settlers had experienced in their former lives.

https://core.tdar.org/document/441632/the-wind-cries-mary-th...

A number of conditions such as acute hyperacusis can cause increased sensitivity to environmental sounds. These conditions can result from high stress and have been known to cause behavior consistent with descriptions of prairie madness such as depression, insomnia, and violent behavior.

I also found this interesting case report:

https://casereports.bmj.com/content/2012/bcr-2012-006300.sho...

Wind direction appeared to be related to the patient's energy levels; these were significantly lower when the wind blew from the southeast. This effect could not be explained by other weather parameters. Decreases in energy in turn predicted increases in anxiety. The reverse effect was observed as well, with increases in anxiety predicting decreases in energy, indicating a positive feedback loop.

It seems plausible that wind from a specific direction through and across certain building structures could produce sound that is causing these variances.

Mythbusters assemble!

Hmm. Actually, ”le vent des fous” seems to be referring to a wind from the south, supposedly bringing in heat and humidity from the Mediterranean.

https://en.wikipedia.org/wiki/Valence,_Drôme#Climate

Valence enjoys a humid subtropical climate, whose main characteristic is an almost constant wind …. When it comes from the south, it usually announces the arrival of stormy disturbances. It is then called le vent du midi ou le vent des fous [the midi wind or the uncaring wind] because, for some people, it makes the atmosphere painful to bear, especially in the summer.