Showing posts with label instrumental (operant) learning. Show all posts
Showing posts with label instrumental (operant) learning. Show all posts

Tuesday, June 16, 2020

#66. Neuromodulators as Peril Specialists [Neuroscience]

NE

Red: theory; black, fact.



“Life is Difficulty”


The basic theory

I conjecture that the primordial function of any type of transmitter substance acting on the g-protein-coupled cell-surface receptors or nuclear receptors of neurons was to coordinate the whole-organism response to some class of perils.
 

Complications

Glutamate, GABA, and acetylcholine are usually considered neurotransmitters, not neuromodulators, but all three have G-protein-coupled receptors in addition to ionotropic receptors and are thus both.
In thermoregulation, hypothalamic glutamate and GABA act on the body via the serotonergic raphe pallidus nuclei. The implied connection with predation (See table) would be due to the fact that animals become torpid at extremes of temperature and thus easy prey. The larger predator would have a smaller surface to volume ratio and thus slower warming and cooling after leaving its refugium to hunt. The predator thermal advantage would have been the selection pressure for thermal sensitivity in the anti predation system, which eventually became upstream of temperature regulation effectors generally. 
The functional assignments suggested in Table 1 would mostly pertain to a very primordial brain. The implication is that any modern biological function of the neuromodulator substance other than organizing the response to a specific type of peril was elaborated out of the primordial function over long-term evolution, which can act opportunistically to confer new functions on preexisting adaptations.
An example of such elaboration is shown for dopamine in the inferred social role. A pre-adaptation for this role split may have been breast-feeding.

Table 1.

 Peril  Substance  Failure mode
Extremes of heat and cold glutamate and GABA  ?
Predator serotonin depression
Parasite histamine phobia
Rival conspecific noradrenaline paranoia
Social isolation

Wednesday, August 3, 2016

#11. Revised Motor Scheme [neuroscience]


How skilled behavior may be generated, on the assumption that it is acquired by an experimentation-like process.

Red, theory; black, fact.

Please find above a revised version of the motor control theory presented in the last blog. The revision was necessitated by the fact that there is no logical reason why a motor command cannot go to both sides of the body at once to produce a mid line-symmetrical movement. The prediction is that mid-line-symmetrical movements are acquired one side at a time whenever the controlling corticofugal pathway allows the two sides to move independently.

Saturday, July 30, 2016

#10. The Two–test-tube Experiment: Part II [neuroscience]

Red, theory; black, fact.

At this point we have a problem. The experimenting-brain theory predicts zero hard-wired asymmetries between the hemispheres. However, the accepted theory of hemispheric dominance postulates that this arrangement allows us to do two things at once, one task with the left hemisphere and the other task with the right. The accepted theory is basically a parsimony argument. However, this argument predicts huge differences between the hemispheres, not the subtle ones actually found.

My solution is that hard-wired hemispheric dominance must be seen as an imperfection of symmetry in the framework of the experimenting brain caused by the human brain being still in the process of evolving, combined with the hypothesis that brain-expanding mutations individually produce small and asymmetric expansions. (See Post 45.) Our left-hemispheric speech apparatus is the most asymmetric part of our brain and these ideas predict that we are due for another mutation that will expand the right side, thereby matching up the two sides, resulting in an improvement in the efficiency of operant conditioning of speech behavior.

These ideas also explain why speech defects such as lisping and stuttering are so common and slow to resolve, even in children, who are supposed to be geniuses at speech acquisition.
This is how the brain would have to work if fragments of skilled behaviors are randomly stored in memory on the left or right side, reflecting the possibility that the two hemispheres play experiment versus control, respectively, during learning.
The illustration shows the theory of motor control I was driven to by the implications of the theory of the dichotomously experimenting brain already outlined. It shows how hemispheric dominance can be reversed independently of the side of the body that should perform the movement specified by the applicable rule of conduct in the controlling hemisphere. The triangular device is a summer that converges the motor outputs of both hemispheres into a common output stream that is subsequently gated into the appropriate side of the body. This arrangement cannot create contention because at any given time, only one hemisphere is active. Anatomically, and from stroke studies, it certainly appears that the outputs of the hemispheres must be crossed, with the left hemisphere only controlling the right body and vice-versa.

However, my theory predicts that in healthy individuals, either hemisphere can control either side of the body, and the laterality of control can switch freely and rapidly during skilled performance so as to always use the best rule of conduct at any given time, regardless of the hemisphere in which it was originally created during REM sleep.

The first bit is calculated and stored in the basal ganglia. It would be output from the reticular substantia nigra (SNr) and gate sensory input to thalamus to favor one hemisphere or the other, by means of actions at the reticular thalamus and intermediate grey of the superior colliculus. The second bit would be stored in the cerebellar hemispheres and gate motor output to one side of the body or the other, at the red nucleus. Conceivably, the two parts of the red nucleus, the parvocellular and the magnocellular, correspond to the adder and switch, respectively, that are shown in the illustration.

Under these assumptions, the corpus callosum is needed only to distribute priming signals from the motor/premotor cortices to activate the rule that will be next to fire, without regard for which side that rule happens to be on. The callosum would never be required to carry signals forward from sensory to motor areas. I see that as the time-critical step, and it would never depend on getting signals through the corpus callosum, which is considered to be a signaling bottleneck.

How would the basal ganglia identify the "best" rule of conduct in a given context? I see the dopaminergic compact substantia nigra (SNc) as the most likely place for a hemisphere-specific "goodness" value to be calculated after each rule firing, using hypothalamic servo-error signals processed through the habenula as the main input for this. The half of the SNc located in the inactive hemisphere would be shut down by inhibitory GABAergic inputs from the adjacent SNr. The dopaminergic nigrostriatal projection would permanently potentiate simultaneously-active corticostriatal inputs (carrying context information) to medium spiny neurons (MSNs) of enkephalin type via a crossed projection, and to MSNs of substance-P type via uncrossed projections. The former MSN type innervates the external globus pallius (GPe), and the latter type innervates the SNr. These latter two nuclei are inhibitory and innervate each other. 

I conjecture that this arrangement sets up a winner-take-all kind of competition between GPe and SNr, with choice of the winner being exquisitely sensitive to small historical differences in dopaminergic tone between hemispheres. The "winner" is the side of the SNr that shuts down sensory input to the hemisphere on that side. The mutually inhibitory arrangement could also plausibly implement hysteresis, which means that once one hemisphere is shut down, it stays shut down without the need for an ongoing signal from the striatum to keep it shut down.

Each time the cerebral cortex outputs a motor command, a copy goes to the subthalamic nucleus (STN) and could plausibly serve as the timing signal for a "refresh" of the hemispheric dominance decision based on the latest context information from cortex. The STN signal presumably removes the hysteresis mentioned above, very temporarily, then lets the system settle down again into possibly a new state.

We now need a system that decides that something is wrong, and that the time to experiment has arrived. This could plausibly be the role of the large, cholinergic inter neurons of the striatum. They have a diverse array of inputs that could potentially signal trouble with the status quo, and could implement a decision to experiment simply by reversing the hemispheric dominance prevailing at the time. Presumably, they would do this by a cholinergic action on the surrounding MSNs of both types.

Finally, there is the second main output of the basal ganglia to consider, the inner pallidal segment (GPi). This structure is well developed in primates such as humans but is rudimentary in rodents and even in the cat, a carnivore. It sends its output forward, to motor thalamus. I conjecture that its role is to organize the brain's knowledge base to resemble block-structured programs. All the instructions in a block would be simultaneously primed by this projection. The block identifier may be some hash of the corticostriatal context information. A small group of cells just outside the striatum called the claustrum seems to have the connections necessary for preparing this hash. Jump rules, that is, rules of conduct for jumping between blocks, would not output motor commands, but block identifiers, which would be maintained online by hysteresis effects in the basal ganglia.

The cortical representation of jump rules would likely be located in medial areas, such as Brodmann 23, 24, 31, and 32. BA23-24 is classed as limbic system, and BA31-32 is situated between this and neocortex. This arrangement suggests that, seen as a computer, the brain is capable of executing programs with three levels of indentation, not counting whatever levels may be encoded as chromatin marks in the serotonergic neurons. Dynamic changes in hemispheric dominance might have to occur independently in neocortex, medial cortex, and limbic system.

Sunday, July 24, 2016

#9. The Two–test-tube Experiment: Part I [neuroscience]

Your Brain is Like This.

Red, theory; black, fact.

The the motivating challenge of this post is to explain the hemispheric organization of the human brain. That is, why we seem to have two very similar brains in our heads, the left side and the right side.

Systems that rely on the principle of trial-and-error must experiment. The genetic intelligence mentioned in previous posts would have to experiment by mutation/natural selection. The synaptic intelligence would have to experiment by operant conditioning. I propose that both these experimentation processes long ago evolved into something slick and simplified that can be compared to the two–test-tube experiment beloved of lab devils everywhere.

Remember that an experiment must have a control, because "everything is relative." Therefore, the simplest and fastest experiment in chemistry that has any generality is the two-test-tube experiment; one tube for the "intervention," and one tube for the control. Put mixture c in both tubes, and add chemical x to only the intervention tube. Run the reaction, then hold the two test tubes up to the light and compare the contents visually (Remember that ultimately, the visual system only detects contrasts.) Draw your conclusions.

My theory is basically this: the two hemispheres of the brain are like the two test tubes. Moreover, the two copies of a given chromosome in a diploid cell are also like the two test tubes. In both systems, which is which varies randomly from experiment to experiment, to prevent phenomena analogous to screen burn. The hemisphere that is dominant for a particular action is the last one that produced an improved result when control passed to it from the other. The allele that is dominant is the last one that produced an improved result when it got control from the other. Chromosomes and hemispheres will mutually inhibit to produce winner-take-all dynamics in which at any given time only one is exposed to selection, but it is fully exposed. 

These flip-flops do not necessarily involve the whole system, but may be happening independently in each sub-region of a hemisphere or chromosome (e.g., Brodmann areas, alleles). Some objective function, expressing the goodness of the organism's overall adaptation, must be recalculated after each flip-flop, and additional flip flopping suppressed if an improvement is found when the new value is compared to a copy of the old value held in memory. In case of a worsening of the objective function, you quickly flip back to the allele or hemisphere that formerly had control, then suppress further flip flopping for awhile, as before. 

The foregoing implies multiple sub-functions, and these ideas will not be compelling unless I specify a brain structure that could plausibly carry out each sub-function. For example, the process of comparing values of the objective function achieved by left and right hemispheres in the same context would be mediated by the nigrostriatal projection, which has a crossed component as well as an uncrossed component. More on this next post.

Saturday, June 18, 2016

#5. Why We Dream [neuroscience]

NE
Red, theory; black, fact.

The Melancholy Fields








Something I still remember from Psych 101 is the prof's statement that "operant conditioning" is the basis of all voluntary behavior. The process was discovered in lab animals such as pigeons by B.F. Skinner in the 1950s and can briefly be stated as "If the ends are achieved, the means will be repeated." (Gandhi said something similar about revolutionary governments.)

I Dream of the Gruffalo. Pareidolia as dream imagery.

Let's say The Organism is in a supermarket checkout line and can't get the opposite sides of a plastic grocery bag unstuck from each other no matter how it rubs, blows, stretches, picks at, or pinches the bag. At great length, a rubbing behavior by chance happens near the sweet spot next to the handle, and the bag opens at once. Thereafter, when in the same situation, The Organism goes straight to the sweet spot and rubs, for a great savings in time and aggravation. This is operant conditioning, which is just trial-and-error, like evolution itself, only faster. Notice how it must begin: with trying moves randomly--behavioral mutations. However, the process is not really random like a DNA mutation. The Organism never tries kicking out his foot, for example, when it is the hand that is holding the bag. Clearly, common sense plays a role in getting the bag open, but any STEM-educated person will want to know just what this "common sense" is and how you would program it. Ideally, you want the  creativity and genius of pure randomness, AND the assurance of not doing anything crazy or even lethal just because some random-move generator suggested it. You vet those suggestions.

That, in a nutshell, is dreaming: vetting random moves against our accumulated better judgment to see if they are safe--stocking the brain with pre-vetted random moves for use the next day when stuck. This is why the emotions associated with dreaming are more often unpleasant than pleasant: there are more ways to go wrong than to go right (This is why my illustrations for this post are melancholy and monster-haunted.) The vetting is best done in advance (e.g., while we sleep) because there's no time in the heat of the action the next day, and trial-and-error with certified-safe "random" moves is already time-consuming without having to do the vetting on the spot as well.

Dreams are loosely associated with brain electrical events called "PGO waves," which begin with a burst of action potentials ("nerve impulses") in a few small brainstem neuron clusters, then spread to the visual thalamus, then to the primary visual cortex. I theorize that each PGO wave creates a new random move that is installed by default in memory in cerebral cortex, and is then tested in the inner theater of dreaming to see what the consequences would be. In the event of a disaster foreseen, the move is scrubbed from memory, or better yet, added as a "don't do" to the store of accumulated wisdom. Repeat all night.

If memory is organized like an AI knowledge base, then each random move would actually be a connection from a randomly-selected but known stimulus to a randomly-selected but known response, amounting to adding a novel if-then rule to the knowledge base. Some of the responses in question could be strictly internal to the brain, raising or lowering the firing thresholds of still other rules.

In "Evolution in Four Dimensions" [1st ed.] Jablonka and Lamb make the point that epigenetic, cultural, and symbolic processes can come up with something much better than purely random mutations: variation that has been subjected to a variety of screening processes.

Nightmares involving feelings of dread superimposed on experiencing routine activities may serve to disrupt routine assumptions that are not serving you well (that is, you may be barking up the wrong tree).