Patent Application Titled “Computational Method For Temporal Pooling And Correlation” Published Online (USPTO 20190095211)

Network Business Daily |

2019 APR 15 (NewsRx) -- By a News Reporter-Staff News Editor at Network Business Daily -- According to news reporting originating from Washington, D.C., by NewsRx journalists, a patent application by the inventors Imaino, Wayne I. (San Jose, Ca); Ozcan, Ahmet S. (San Jose, Ca); Scott, J. Campbell (Los Gatos, Ca), filed on , was made available online on .

The assignee for this patent application is International Business Machines Corporation (Armonk, New York, United States).

Reporters obtained the following quote from the background information supplied by the inventors: “Field of Invention

“The present invention relates generally to the field of artificial neural networks. More specifically, the present invention is related to computational methods for temporal pooling and correlation.

“Discussion of Related Art

“Artificial neural networks (ANNs) are increasingly successful in addressing problems in artificial intelligence, particularly those that require large amounts of data and long times to process. Such networks are typically organized in a hierarchical fashion, with data being fed forward from lower levels of the hierarchy to those above (see FIG. 1). FIG. 1 shows an example of a hierarchical network consisting of 7 regions on three levels, where the solid arrows show the feed-forward pathways of data from lower to higher levels, and where the receiving region pools and correlates the data coming from below. In the simplest case, the data represent static features, and the role of the network is to signal that a certain feature, or set of features, is present. Optionally, the signal may be fed back down the hierarchy (dotted arrows in FIG. 1) to provide additional context to the lower regions as they process their input.

“As a specific example, consider a static image consisting of several pixels. Data about each pixel, color and (relative) position are ‘pooled,’ i.e., combined into a composite dataset, and fed into the lowest level of the hierarchy. In the description of this invention, we use the terms pool, pooled, pooling, etc. exclusively to indicate the merging or combination of data from different sources.

“If a certain pixel is black, and the pixels above and below it are also black, but the pixels to the left and right of all three are white, then the three pixels together indicate (part of) a vertical line. All nine pixels (three black and six white) are correlated, in the sense that this is a frequently occurring pattern. The lowest regions in the hierarchy correlate their pooled data to form a (new) representation of each specific feature. The representation of the vertical line is passed upwards to the next level of the hierarchy where it may, for example, be combined with a horizontal line. Depending on their relative locations, these two features together may form part of a more complex feature, such as an L-shaped corner, a cross like a + sign, a T-shape or perhaps part of the letter E. By successive steps of such synthesis, increasingly complex features are represented at higher levels of the hierarchical network.

“When the data are serial in time, as is most often the case, the network should recognize not only the object, but also how it is moving and therefore where it is most likely to be next. Consider the case of a ball thrown towards the observer. The static representation is quite simple--it is merely a circle. However, as time goes by the circle grows larger, and its position in the field of view, or relative to the background, or more likely both, changes. The observer determines the motion of the ball from this succession of input data, and is able to estimate when the ball will reach it, and perhaps what must be done to avoid it, to catch it, to hit it, etc.

“The network which processes this time-dependent data must form a more complex representation. At the lowest level, the feature may still be represented only as a circle, but the attributes of that circle (its size and position) are changing--there is a temporal sequence of observations that, taken together indicate ‘ball approaching.’

“Briefly, ANNs comprise a number of ‘nodes’ which are programmed to act somewhat like the neurons of the brain. Each node is connected to many other nodes through connections that act as synapses. The activity of each node is passed via axons to each of its synapses. The synapses are assigned a certain weight, which indicates what fraction of the activity is passed into the dendrites of the target node. Mathematically this is expressed as a product, x.sub.i w.sub.ij, where x.sub.i is the activity of node i in level x and w.sub.ij is the weight of the synapse connecting nodes i and j, where j is the index for a different level of the hierarchy with nodes y. The basic operation of each node is to sum the weighted inputs from all the nodes to which it is connected, and then to apply a non-linear (‘squashing’ or threshold) function (.sigma.) to determine its own activity, i.e., y.sub.j=.sigma.(.SIGMA.sub.ix.sub.iw.sub.ij). Learning is accomplished by adjusting the weights (w) in order to minimize an error or cost function.

“Static ANNs simply feed data upwards through the hierarchy. In recurrent neural networks (RNNs), time dependent behavior is processed by taking the output activity at time t of each node in the network, and routing it to the inputs, at time t+1 of (potentially) any other node. There are many variants of RNN, depending on where the time-varying output is routed. RNNs have proved to be useful in the analysis of time-varying data, but they are not designed to form representations that have clear semantic meaning, such as ‘ball approaching.’

“A very different type of hierarchical network is described by Hawkins and his colleagues at Numenta. In essence, they describe a network of networks. The sub-networks, which we call ‘regions’, (see FIG. 1) contain many neurons which are arranged in structures that correspond to the layers and mini-columns of the mammalian cortex. Each layer of neurons is responsible for a specific set of cognitive functions. In particular, as with cortical layer-4 (L4) of the brain, there are neurons whose dendrites receive bottom-up data from levels lower in the hierarchy, as well as from core cells of the thalamus. The Hebbian learning rules which modify the weights of the synapses of the L4 dendrites lead to correlations among the inputs; i.e., axons that are active at the same time tend to connect to the same L4 neuron. Conversely, the firing of that neuron signals the correlation of activity among its connected axons. L4 is a correlator.

“When L4 neurons fire, they activate some of the neurons in the mini-columns of layers 2 and 3 above them. (The historical distinction between layers 2 and 3 is no longer made in modern neuro-science; this part of the cortex is simply identified as L2/3.) L2/3 functions as a sequence memory. The lateral connections between neurons in different mini-columns do not cause immediate firing, rather they are modulating in their behavior: the receiving neuron is put in a state that makes it more likely to fire when it receives input from the L4 neuron at the bottom of its mini-column. This may be termed ‘prediction’ and it is akin to biological depolarization. If the column is then indeed subsequently active, the learning rules reinforce the weights of the synapses that contributed to the correct prediction. Because there are multiple cells in each mini-column, only a small number of which fire in response to the L4 input, the specific set of correctly predicted cells encodes information about the previously active cells responsible for the prediction. In other words, the previously active cells provide the context for the cells that are now active, and so on back through time. L2/3 learns a sequence of firing patterns.

“However, there is little semantic meaning of any single firing pattern in the sequence--additional processing is required in order to form a new representation of the entire sequence. This is achieved by temporal correlation of the firing activity of the L2/3 neurons, and is the subject of the current invention.

“It is customary to call the firing pattern of a set of neurons a Sparse Distributed Representation (SDR). This is nothing more than a large binary array, most often a vector, of ones and zeroes. Each element in the array represents a single neuron. When the neuron is active, its bit is one (1) otherwise it is zero (0). The vector is ‘sparse’, because only a small fraction of the bits, less than 20%, and more typically less than 5%, are on at the same time. It is ‘distributed’ because the active bits are spread throughout the entire vector.

“On their website, Numenta describes a temporal pooler as it relates to their implementation of Hierarchical Temporal Memory (HTM). The Numenta pooler is considerably different from the current invention, and much more complicated. It requires the feed-forward (FF--up the hierarchy) of two distinct SDRs. The FF inputs are processed separately and combined as a weighted sum, then passed to a thresholding function. The result is given a ‘persistence’ which increases if the activity continues and decays otherwise. The most persistent results are fed into a second pooler. This algorithm not only has many steps, there are many adjustable parameters: the summing weight, the threshold, the decay rate of the persistence, the increase for continued activity and the parameters of the second pooler.

“Theyel et al. provide data from neuro-physiological studies showing that there are two pathways for feed-forward from one cortical region to another: either directly, or indirectly through the thalamus. Thus, the receiving cortical region receives (at least) two versions of the same data. The trans-thalamic pathway passes through more intermediate neurons and therefore (it is assumed) takes longer to reach its destination. This realization forms the inspiration for the current invention.

“Embodiments of the present invention are an improvement over prior art systems and methods.”

In addition to obtaining background information on this patent application, NewsRx editors also obtained the inventors’ summary information for this patent application: “In one embodiment, the present invention provides a computational method for the simulation of a hierarchical artificial neural network (ANN), wherein a single correlator pools, during a single time-step, two or more consecutive feed-forward inputs from previously predicted and now active neurons of one or more lower levels.

“In another embodiment, the present invention provides a computational method for the simulation of a hierarchical artificial neural network (ANN), the method comprising: (a) correlating two pooled feed-forward inputs, S(t), from time step, t, and S(t-1), from time-step, t-1 for all times t; (b) indirectly learning correlation between input S(t) and S(t-t’), where t’ is a positive integer that is .gtoreq.2; and © outputting correlations learned in (a) and (b).

“In yet another embodiment, the present invention provides a computational method for the simulation of a hierarchical artificial neural network (ANN), the method comprising: (a) correlating three pooled feed-forward inputs, S(t), from time step, t, S(t-1), from time-step, t-1, and S(t-2), from time step, t-2, for all times t; (b) indirectly learning correlation between input S(t) and S(t-t’), where t’ is a positive integer that is .gtoreq.3; and © outputting correlations learned in (a) and (b).

“In another embodiment, the present invention provides an article of manufacture storing computer readable program code implementing a computational method for the simulation of a hierarchical artificial neural network (ANN), the medium comprising: (a) computer readable program code correlating two pooled feed-forward inputs, S(t), from time step, t, and S(t-1), from time-step, t-1 for all times t; (b) computer readable program code indirectly learning correlation between input S(t) and S(t-t’), where t’ is a positive integer that is .gtoreq.2; and © computer readable program code outputting correlations learned in (a) and (b).

“In yet another embodiment, the present invention provides an article of manufacture storing computer readable program code implementing a computational method for the simulation of a hierarchical artificial neural network (ANN), the medium comprising: (a) computer readable program code correlating three pooled feed-forward inputs, S(t), from time step, t, S(t-1), from time-step, t-1, and S(t-2), from time step, t-2, for all times t; (b) computer readable program code indirectly learning correlation between input S(t) and S(t-t’), where t’ is a positive integer that is .gtoreq.3; and © computer readable program code outputting correlations learned in (a) and (b).”

The claims supplied by the inventors are:

“1. A computational method for the simulation of a hierarchical artificial neural network (ANN), wherein a single correlator pools, during a single time-step, two or more consecutive feed-forward inputs from previously predicted and now active neurons of one or more lower levels.

“2. The method of claim 1, wherein the single correlation is a static correlator.

“3. The method of claim 1, wherein pooling of feed-forward inputs is done by logical OR of consecutive feed-forward inputs.

“4. The method of claim 1, wherein pooling of feed-forward inputs is done by concatenating consecutive inputs.

“5. The method of claim 1, wherein a transformation operation is applied to each feed-forward input prior to pooling.

“6. The method of claim 5, wherein the transformation operation is any of the following: a permutation operation, a logical XOR operation, or a logical AND operation.

“7. A computational method for the simulation of a hierarchical artificial neural network (ANN), the method comprising: (a) correlating two pooled feed-forward inputs, S(t), from time step, t, and S(t-1), from time-step, t-1 for all times t; (b) indirectly learning correlation between input S(t) and S(t-t’), where t’ is a positive integer that is .gtoreq.2; and © outputting correlations learned in (a) and (b).

“8. The method of claim 7, wherein the output is a sparse distributed representation (SDR) matrix.

“9. The method of claim 7, wherein the correlating step in (a) is done by a static correlator.

“10. The method of claim 7, wherein pooling of feed-forward inputs is done by logical OR of consecutive feed-forward inputs.

“11. The method of claim 7, wherein pooling of feed-forward inputs is done by concatenating consecutive inputs.

“12. The method of claim 7, wherein a transformation operation is applied to each feed-forward input prior to pooling.

“13. The method of claim 12, wherein the transformation operation is any of the following: a permutation operation, a logical XOR operation, or a logical AND operation.

“14. A computational method for the simulation of a hierarchical artificial neural network (ANN), the method comprising: (a) correlating three pooled feed-forward inputs, S(t), from time step, t, S(t-1), from time-step, t-1, and S(t-2), from time step, t-2, for all times t; (b) indirectly learning correlation between input S(t) and S(t-t’), where t’ is a positive integer that is .gtoreq.3; and © outputting correlations learned in (a) and (b).

“15. The method of claim 14, wherein the output is a sparse distributed representation (SDR) matrix.

“16. The method of claim 14, wherein the correlating step in (a) is done by a static correlator.

“17. The method of claim 14, wherein pooling of feed-forward inputs is done by logical OR of consecutive feed-forward inputs.

“18. The method of claim 14, wherein pooling of feed-forward inputs is done by concatenating consecutive inputs.

“19. The method of claim 14, wherein a transformation operation is applied to each feed-forward input prior to pooling.

“20. The method of claim 19, wherein the transformation operation is any of the following: a permutation operation, a logical XOR operation, or a logical AND operation.

“21. An article of manufacture storing computer readable program code implementing a computational method for the simulation of a hierarchical artificial neural network (ANN), the medium comprising: (a) computer readable program code correlating two pooled feed-forward inputs, S(t), from time step, t, and S(t-1), from time-step, t-1 for all times t; (b) computer readable program code indirectly learning correlation between input S(t) and S(t-t’), where t’ is a positive integer that is .gtoreq.2; and © computer readable program code outputting correlations learned in (a) and (b).

“22. An article of manufacture storing computer readable program code implementing a computational method for the simulation of a hierarchical artificial neural network (ANN), the medium comprising: (a) computer readable program code correlating three pooled feed-forward inputs, S(t), from time step, t, S(t-1), from time-step, t-1, and S(t-2), from time step, t-2, for all times t; (b) computer readable program code indirectly learning correlation between input S(t) and S(t-t’), where t’ is a positive integer that is .gtoreq.3; and © computer readable program code outputting correlations learned in (a) and (b).”

For more information, see this patent application: Imaino, Wayne I.; Ozcan, Ahmet S.; Scott, J. Campbell. Computational Method For Temporal Pooling And Correlation. Filed and posted . Patent URL: http://appft.uspto.gov/netacgi/nph-Parser?Sect1=PTO1&Sect2=HITOFF&d=PG01&p=1&u=%2Fnetahtml%2FPTO%2Fsrchnum.html&r=1&f=G&l=50&s1=%2220190095211%22.PGNR.&OS=DN/20190095211&RS=DN/20190095211

(Our reports deliver fact-based news of research and discoveries from around the world.)

DISCLOSURE: The views and opinions expressed in this article are those of the authors, and do not represent the views of equities.com. Readers should not consider statements made by the author as formal recommendations and should consult their financial advisor before making any investment decisions. To read our full disclosure, please go to: http://www.equities.com/disclaimer

Comments

Watchlist

Symbol Last Price Change % Change
AAPL

     
AMZN

     
HD

     
JPM

     
IBM

     
BA

     
WMT

     
DIS

     
XOM

     

World Economic Forum at Davos 2019 - Dr Oliver Krause Founder Untitled Inc Part 1

Matt Bird sits down with Dr Oliver Krause, Founder Untitled Inc, in part 1 of this 2 part interview at the World Economic Forum at Davos 2019