Bernard Widrow

All Publications

• The Hebbian-LMS Learning Algorithm IEEE COMPUTATIONAL INTELLIGENCE MAGAZINE Widrow, B., Kim, Y., Park, D. 2015; 10 (4): 37-53

  View details for DOI 10.1109/MCI.2015.2471216

  View details for Web of Science ID 000363206100005

• The Back-Prop and No-Prop Training Algorithms COGNITIVE COMPUTATION Widrow, B. 2015
• Cognitive memory NEURAL NETWORKS Widrow, B., Aragon, J. C. 2013; 41: 3-14

  Abstract

  Regarding the workings of the human mind, memory and pattern recognition seem to be intertwined. You generally do not have one without the other. Taking inspiration from life experience, a new form of computer memory has been devised. Certain conjectures about human memory are keys to the central idea. The design of a practical and useful "cognitive" memory system is contemplated, a memory system that may also serve as a model for many aspects of human memory. The new memory does not function like a computer memory where specific data is stored in specific numbered registers and retrieval is done by reading the contents of the specified memory register, or done by matching key words as with a document search. Incoming sensory data would be stored at the next available empty memory location, and indeed could be stored redundantly at several empty locations. The stored sensory data would neither have key words nor would it be located in known or specified memory locations. Sensory inputs concerning a single object or subject are stored together as patterns in a single "file folder" or "memory folder". When the contents of the folder are retrieved, sights, sounds, tactile feel, smell, etc., are obtained all at the same time. Retrieval would be initiated by a query or a prompt signal from a current set of sensory inputs or patterns. A search through the memory would be made to locate stored data that correlates with or relates to the prompt input. The search would be done by a retrieval system whose first stage makes use of autoassociative artificial neural networks and whose second stage relies on exhaustive search. Applications of cognitive memory systems have been made to visual aircraft identification, aircraft navigation, and human facial recognition. Concerning human memory, reasons are given why it is unlikely that long-term memory is stored in the synapses of the brain's neural networks. Reasons are given suggesting that long-term memory is stored in DNA or RNA. Neural networks are an important component of the human memory system, and their purpose is for information retrieval, not for information storage. The brain's neural networks are analog devices, subject to drift and unplanned change. Only with constant training is reliable action possible. Good training time is during sleep and while awake and making use of one's memory. A cognitive memory is a learning system. Learning involves storage of patterns or data in a cognitive memory. The learning process for cognitive memory is unsupervised, i.e. autonomous.

  View details for DOI 10.1016/j.neunet.2013.01.016

  View details for Web of Science ID 000318209900002

  View details for PubMedID 23453302

• The No-Prop algorithm: a new learning algorithm for multilayer neural networks. Neural networks Widrow, B., Greenblatt, A., Kim, Y., Park, D. 2013; 37: 182-188

  Abstract

  A new learning algorithm for multilayer neural networks that we have named No-Propagation (No-Prop) is hereby introduced. With this algorithm, the weights of the hidden-layer neurons are set and fixed with random values. Only the weights of the output-layer neurons are trained, using steepest descent to minimize mean square error, with the LMS algorithm of Widrow and Hoff. The purpose of introducing nonlinearity with the hidden layers is examined from the point of view of Least Mean Square Error Capacity (LMS Capacity), which is defined as the maximum number of distinct patterns that can be trained into the network with zero error. This is shown to be equal to the number of weights of each of the output-layer neurons. The No-Prop algorithm and the Back-Prop algorithm are compared. Our experience with No-Prop is limited, but from the several examples presented here, it seems that the performance regarding training and generalization of both algorithms is essentially the same when the number of training patterns is less than or equal to LMS Capacity. When the number of training patterns exceeds Capacity, Back-Prop is generally the better performer. But equivalent performance can be obtained with No-Prop by increasing the network Capacity by increasing the number of neurons in the hidden layer that drives the output layer. The No-Prop algorithm is much simpler and easier to implement than Back-Prop. Also, it converges much faster. It is too early to definitively say where to use one or the other of these algorithms. This is still a work in progress.

  View details for DOI 10.1016/j.neunet.2012.09.020

  View details for PubMedID 23140797

• Quantization Noise: Round Off Error in Digital Computation, Signal Processing, Control and Communications Widrow, B., Kollar, I. Cambridge University Press. 2008
• Statistical efficiency of adaptive algorithms INNS/IEEE International Joint Conference on Neural Networks (IJCNN 03) Widrow, B., Kamenetsky, M. PERGAMON-ELSEVIER SCIENCE LTD. 2003: 735–44

  Abstract

  The statistical efficiency of a learning algorithm applied to the adaptation of a given set of variable weights is defined as the ratio of the quality of the converged solution to the amount of data used in training the weights. Statistical efficiency is computed by averaging over an ensemble of learning experiences. A high quality solution is very close to optimal, while a low quality solution corresponds to noisy weights and less than optimal performance. In this work, two gradient descent adaptive algorithms are compared, the LMS algorithm and the LMS/Newton algorithm. LMS is simple and practical, and is used in many applications worldwide. LMS/Newton is based on Newton's method and the LMS algorithm. LMS/Newton is optimal in the least squares sense. It maximizes the quality of its adaptive solution while minimizing the use of training data. Many least squares adaptive algorithms have been devised over the years, but no other least squares algorithm can give better performance, on average, than LMS/Newton. LMS is easily implemented, but LMS/Newton, although of great mathematical interest, cannot be implemented in most practical applications. Because of its optimality, LMS/Newton serves as a benchmark for all least squares adaptive algorithms. The performances of LMS and LMS/Newton are compared, and it is found that under many circumstances, both algorithms provide equal performance. For example, when both algorithms are tested with statistically nonstationary input signals, their average performances are equal. When adapting with stationary input signals and with random initial conditions, their respective learning times are on average equal. However, under worst-case initial conditions, the learning time of LMS can be much greater than that of LMS/Newton, and this is the principal disadvantage of the LMS algorithm. But the strong points of LMS are ease of implementation and optimal performance under important practical conditions. For these reasons, the LMS algorithm has enjoyed very widespread application. It is used in almost every modem for channel equalization and echo cancelling. Furthermore, it is related to the famous backpropagation algorithm used for training neural networks.

  View details for DOI 10.1016/S0893-6080(03)00126-6

  View details for Web of Science ID 000184011900028

  View details for PubMedID 12850029

• Least-Mean-Square Adaptive Filters Widrow, B., Haykin, S. Wiley-Interscience. 2003
• Statistical Efficiency of Adaptive Algorithms Neural Networks Widrow, B., Kamenetsky, M. 2003: 735-744
• Neurointerfaces IEEE Transactions on Control Systems Technology Widrow, B. 2002: 221-228
• A Microphone Array for Hearing Aids IEEE Circuits and Systems Magazine Widrow, B. 2001: 26-32
• Adaptive inverse control based on linear and nonlinear adaptive filtering International Workshop on Neural Networks for Identification, Control, Robotics, and Signal/Image Processing Widrow, B., Plett, G. L. IEEE COMPUTER SOC. 1996: 30–38

  View details for Web of Science ID A1996BG16U00004

• Nonlinear Control with Neural Networks Backpropagation: Theory, Architectures, and Applications Chauvin, Y., D. Rumelhart, D. Erlbaum Associates. 1995
• Noise Canceling and Channel Equalization Handbook of Brain Theory and Neural Networks Widrow, B., Lehr, M. MIT Press. 1995
• 30 Years of Adaptive Neural Networks: Perceptron, Madaline, and Backpropagation Neural Networks: Theoretical Foundations and Analysis Lau, C. IEEE Press. 1992
• 30 Years of Adaptive Neural Networks: Perceptron, Madaline, and Backpropagation Artificial Neural Networks: Paradigms, Applications, and Hardware Implemenation Sanchez-Sinencio, E., Lau, C. IEEE Press. 1992: 82–108
• 30 Years of Adaptive Neural Networks: Perceptron, Madaline, and Backpropagation Widrow, B., Lehr, M. 1990: 1415–42
• Fundamental Relations Between the LMS Algorithm and the DFT IEEE Transactions on Circuits and Systems Widrow, B., Baudrenghien, P., Vetterli, M., Titchener, P. 1987: 814-820
• Adaptive Signal Processing Widrow, B., Stearns, S. Prentice Hall. 1985
• On the Statistical Efficiency of the LMS Algorithm with Nonstationary Inputs IEEE Transactions on Information Theory Widrow, B., Walach E. 1984: 211-221
• Adaptive Filters Aspects of Network and System Theory Kalman, R., DeClaris, N. Holt, Rinehart and Winston. 1971
• Adaptive Antenna Systems [a citation classic] Widrow, B., Mantey, P. 1967: 2143–59
• Statistical Analysis of Amplitude-Quantized Sampled-Data Systems AIEE Transactions on Applications and Industry Widrow, B. 1961: 1-14
• Adaptive Switching Circuits RE WESCON Convention Record Widrow, B. 1960: 96–104
• A Study of Rough Amplitude Quantization by Means of Nyquist Sampling Theory IRE Transactions on Circuit Theory Widrow, B. 1956; CT-3(4): 266-276