- Imbalanced Decision Hierarchy in Addicts Emerging from Drug-Hijacked Dopamine Spiraling Circuit

Figure S1,A sample decision hierarchy with five levels of abstraction. Figure S2, The corresponding neural circuit for the three discussed value learning algorithms is a hierarchical decision structure. A, Using a simple TD-learning algorithm (equation S7), the prediction error signal in each level of abstraction is computed independently from other levels. B, In the model proposed by Haruno and Kawato (4) (equation S8), the value of the temporally-advanced state comes from one higher level of abstraction. C, in our model (equation S9) the value of the temporally-advanced state is substituted with a combination of the reward and Q-value of the performed action at a higher level of abstraction. Figure S3, Our model predicts different sites of action of drugs on the reward-learning circuit: sites 1 to 3. Drugs affecting sites 4 to 6, in contrast, will not result in the behavioral and neurobiological patterns produced by simulation of the model for drugs, but will produce results similar to the case of natural rewards. Figure S4, The task used for simulating the uncertainty-based competition mechanism among the levels of the hierarchy for taking control over behavior. Figure S5, Simulation result, showing gradual shift of control over behavior from higher to lower levels of the hierarchy. Q(s,a) and U(s,a) show the estimated value and uncertainty of the state-action pairs, respectively.

(PDF)