Skip to content

Comment on Megatron-Turing NLG 530B, the World’s Largest Generative Language Modelparent

Comments

oh finally something that I learned a lot about :)

Such research is the area of computational neuroscience - one thing that such people do is try to model parts of the brain (or just a single neuron) with computers.

A Neuron (=nerve cell in the brain) is a very complex beast. In rough terms they work like this: They collect signals (electrical impulses) via their small appendages called dendrites. when the sum of the signals reaches a certain threshold a large electrical impulse is generated at the cell body that will travel trough its "output" appendage (called axon) that connected to another neuron's cell body or to its dendrite.

Neurons display a dazzling variety in all these parameters:

- In morphology, e.g. they can look like a pine tree http://www.scholarpedia.org/article/Pyramidal_neuron (I really recommend scholarpedia, also this article has a nice animation on how electrical impulses propagate) or like a sea urchin.

- it really matters where the cell gets its impulse from: A neuron stimulated near its cell body will be much more sensitive to the input than being stimulated far away.

- Their response characteristics are wildly varied too. Some give off one large impulse, some a quick burst of impulses. Some are preventing others from giving out impulses from stimulation (inhibitor neurons)

- This whole mess can be modulated with chemical compounds that are released by the body -- some make some neurons more sensitive, some less.

- Also we still discover every year some new mechanism that modulates how they function.

The issue is that this results in such a complex system that a modern PC cant even simulate 1 detailed neuron model realtime (these tools are open source, try them out! for example https://neuron.yale.edu/ ). Now we know that we're simulating things that likely do not matter (e.g. we don't need a neuron model that consist of 10.000+ segments), but we do not know which parts we need to remove to have a faithful simulation. Also we might simply simulate some parts wrong because our knowledge of the subject is not enough.

But on the upside we've reached some great things already, for example we know how our brain calculates from our head and eye position the orientation of the things we're looking at

AboutSource Built by g1lg1l

Hackerly is an independent reader for Hacker News, built on the public HN API. Not affiliated with Y Combinator.