Neural Network Architectures Continued

Adaptivity and Learning

• What is the difference?
• Many different characterizations for adaptivity depending on time span, compare:
• "Have you adapted to the low temperature in this room yet?"
• "She was quick to adapt to the environment at her new school"
• "Eventually organisms adapted to the higher concentration of oxygen in the atmosphere"
• Remember: adaptivity is a mark of intelligence
• want adaptive systems for most tasks (why? - because we usually cannot program in all possible future scenarios and eventualities in which the system has to function)
• Rough characterization: "adaptivity" refers to the degree to which an agent is able to adjust its behavior according to environmental changes
• Time frame matters!
• One way to be adaptive: learning
• What are others?
• Distinguish:
• supervised
• feedback
• monitoring of learning progress
• correct answers provided
• example: error propagation (e.g., backpropagation in connectionist networks)
• unsupervised
• with or without feedback
• example with feedback (discrete or continuous): reinforcement learning (e.g., Q learning)
• example no feedback: associative learning (e.g., Hebbian learning)
• Which learning method is appropriate for what task?
• first, how much feedback to you have for the task at hand?
• what kind of feedback?
• the right answer (note that this could be very involved: e.g., providing detail information about trajectories of physical bodies)
• just some goodness measure (this is very common on in nature in higher developed animals)
• no useful feedback (form associations on your own; learn "causal links", temporal learning)
• is the feedback sufficient for the learning task (i.e., can you actually learn what you are supposed to given that kind of feedback)?
• how long should learning go on, when is the learning goal reached? (criteria--obviously this does not apply to all learning techniques)
• timing of the learning (fast, slow, what exactly is the time span)?
• problems of "overlearning" (e.g., overgeneralization) may arise or learning may be too slow
• how does the control system have to be adjusted to incorporate the learning method, i.e., what kinds of changes will can be made to the architecture
• parameter settings of components
• modification of links between or embeddings of components (=architectural change!)
• Side issue: what exactly is the difference between changing parameters of components and changing the architecture?

Learning Associations

• Hebbian learning is used to learn associations between active nodes (biological plausibility--synapse growth)
• The "Hebbian rule": Delta(t) = eta * o_j(t) * a_j(t)
• "eta" is called "learning rate"
• the product between one unit's output and the other unit's activation is a measure of the co-activation
• Update weights: w_ij(t+1) = Delta(t) -w_ij(t)
• Problem: weight explosion!
• Various ways to fix it:
• make eta a function
• e.g., a decreasing function of time: eta(t)
• make eta depending on the weight value (use a maximum weight values to constrain it):
• eta_ij(t) = (max_w_ij - w_ij(t)) for w_ij(t)>=0,
• eta_ij(t) = (w_ij(t) - min_w_ij) for w_ij(t)<0
• add a "forgetting" term to the Hebbian rule: - epsilon * Average(a_i) * w_ij(t)

Exercise:

• Design a connectionist network for an agent that makes the agent avoid bumping into walls (assume that the agent has two sonar sensors and two motors, in the style of Braitenberg vehicles, say)
• first, define the inputs and outputs you will have available on the robotic platform (i.e., the sensory and motor signals)
• then sketch the architecture and write down the components you are going to use, this includes
• deciding what kinds of components you will need and how many of each--remember: these components are the "behaviors" in the behavior-based approach (note that we deviate here from the Pfeiffer and Scheier book, where components are not restricted to be behaviors, but rather may implement behaviors together, the magic "emergence"!)
• drawing the components and the connections among them including sensor and motor connections
• third, specify any parameter settings that will provide enough detail for the architecture to be implementable (i.e., specify weight connections, activation and output functions, transducers, etc.)
• Add a Hebbian learning component to the architecture, which will allow it to learn the above behavior
• decide which connection can be altered during learning
• decide on the learning rule and fix various parameters (e.g., the learning rate, update frequency, etc.)