Lee, Daniel D
Email Address
ORCID
Disciplines
Search Results
Now showing 1 - 10 of 21
Publication Multiplicative Updates for Nonnegative Quadratic Programming(2007-01-01) Lin, Yuanqing; Sha, Fei; Lee, Daniel D; Saul, Lawrence K.Many problems in neural computation and statistical learning involve optimizations with nonnegativity constraints. In this article, we study convex problems in quadratic programming where the optimization is confined to an axis-aligned region in the nonnegative orthant. For these problems, we derive multiplicative updates that improve the value of the objective function at each iteration and converge monotonically to the global minimum. The updates have a simple closed form and do not involve any heuristics or free parameters that must be tuned to ensure convergence. Despite their simplicity, they differ strikingly in form from other multiplicative updates used in machine learning.We provide complete proofs of convergence for these updates and describe their application to problems in signal processing and pattern recognition.Publication An Information Maximization Approach to Overcomplete and Recurrent Representations(2000-11-27) Shriki, Oren; Sompolinsky, Haim; Lee, Daniel DThe principle of maximizing mutual information is applied to learning overcomplete and recurrent representations. The underlying model consists of a network of input units driving a larger number of output units with recurrent interactions. In the limit of zero noise, the network is deterministic and the mutual information can be related to the entropy of the output units. Maximizing this entropy with respect to both the feedforward connections as well as the recurrent interactions results in simple learning rules for both sets of parameters. The conventional independent components (ICA) learning algorithm can be recovered as a special case where there is an equal number of output units and no recurrent connections. The application of these new learning rules is illustrated on a simple two-dimensional input example.Publication Multiplicative Updates for Large Margin Classifiers(2003-08-24) Saul, Lawrence K; Sha, Fei; Lee, Daniel DVarious problems in nonnegative quadratic programming arise in the training of large margin classifiers. We derive multiplicative updates for these problems that converge monotonically to the desired solutions for hard and soft margin classifiers. The updates differ strikingly in form from other multiplicative updates used in machine learning. In this paper, we provide complete proofs of convergence for these updates and extend previous work to incorporate sum and box constraints in addition to nonnegativity.Publication Room Impulse Response Estimation using Sparse Online Prediction and Absolute Loss(2006-05-24) Crammer, Koby; Lee, Daniel DThe need to accurately and efficiently estimate room impulse responses arises in many acoustic signal processing applications. In this work, we present a general family of algorithms which contain the conventional normalized least mean squares (NLMS) algorithm as a special case. Specific members of this family yield estimates which are robust both to different noise models and choice of parameters. We demonstrate the merits of our approach to accurately estimate sparse room impulse responses in simulations with speech signals.Publication Bayesian Regularization and Nonnegative Deconvolution for Time Delay Estimation(2004-12-13) Lin, Yuanqing; Lee, Daniel DBayesian Regularization and Nonnegative Deconvolution (BRAND) is proposed for estimating time delays of acoustic signals in reverberant environments. Sparsity of the nonnegative filter coefficients is enforced using an L1-norm regularization. A probabilistic generative model is used to simultaneously estimate the regularization parameters and filter coefficients from the signal data. Iterative update rules are derived under a Bayesian framework using the Expectation-Maximization procedure. The resulting time delay estimation algorithm is demonstrated on noisy acoustic data.Publication Real time voice processing with audiovisual feedback : toward autonomous agents with perfect pitch(2002-12-09) Saul, Lawrence K; Lee, Daniel D; Isbell, Charles L; LeCun, YannWe have implemented a real time front end for detecting voiced speech and estimating its fundamental frequency. The front end performs the signal processing for voice-driven agents that attend to the pitch contours of human speech and provide continuous audiovisual feedback. The algorithm we use for pitch tracking has several distinguishing features: it makes no use of FFTs or autocorrelation at the pitch period; it updates the pitch incrementally on a sample-by-sample basis; it avoids peak picking and does not require interpolation in time or frequency to obtain high resolution estimates; and it works reliably over a four octave range, in real time, without the need for postprocessing to produce smooth contours. The algorithm is based on two simple ideas in neural computation: the introduction of a purposeful nonlinearity, and the error signal of a least squares fit. The pitch tracker is used in two real time multimedia applications: a voice-to-MIDI player that synthesizes electronic music from vocalized melodies, and an audiovisual Karaoke machine with multimodal feedback. Both applications run on a laptop and display the user’s pitch scrolling across the screen as he or she sings into the computer.Publication Statistical signal processing with nonnegativity constraints(2003-09-01) Saul, Lawrence K; Sha, Fei; Lee, Daniel DNonnegativity constraints arise frequently in statistical learning and pattern recognition. Multiplicative updates provide natural solutions to optimizations involving these constraints. One well known set of multiplicative updates is given by the Expectation-Maximization algorithm for hidden Markov models, as used in automatic speech recognition. Recently, we have derived similar algorithms for nonnegative deconvolution and nonnegative quadratic programming. These algorithms have applications to low-level problems in voice processing, such as fundamental frequency estimation, as well as high-level problems, such as the training of large margin classifiers. In this paper, we describe these algorithms and the ideas that connect them.Publication Equilibrium Properties of Temporally Asymmetric Hebbian Plasticity(2001-01-08) Rubin, Jonathan; Lee, Daniel D; Sompolinsky, H.A theory of temporally asymmetric Hebb rules, which depress or potentiate synapses depending upon whether the postsynaptic cell fires before or after the presynaptic one, is presented. Using the Fokker-Planck formalism, we show that the equilibrium synaptic distribution induced by such rules is highly sensitive to the manner in which bounds on the allowed range of synaptic values are imposed. In a biologically plausible multiplicative model, the synapses in asynchronous networks reach a distribution that is invariant to the firing rates of either the presynaptic or postsynaptic cells. When these cells are temporally correlated, the synaptic strength varies smoothly with the degree and phase of their synchrony.Publication Blind Sparse-nonnegative (BSN) Channel Identification for Acousitic Time-Difference-of-Arrival Estimation(2007-10-01) Lin, Yuanqing; Chen, Jingdong; Lee, Daniel D; Kim, YoungmooEstimating time-difference-of-arrival (TDOA) remains a challenging task when acoustic environments are reverberant and noisy. Blind channel identification approaches for TDOA estimation explicitly model multipath reflections and have been demonstrated to be effective in dealing with reverberation. Unfortunately, existing blind channel identification algorithms are sensitive to ambient noise. This papers hows how to resolve the noise sensitivity issue by exploiting prior knowledge about an acoustic room impulse response (RIR), namely, an acoustic RIR can be modeled by a sparse-nonnegative FIR filter. This paper shows how to formulate a single-input two-output blind channel identification into a least square convex optimization, and how to incorporate the sparsity and nonnegativity priors so that the resulting optimization remains convex and can be solved efficiently. The proposed blind sparse-nonnegative (BSN) channel identification approach for TDOA estimation is not only robust to reverberation, but also robust to ambient noise, as demonstrated by simulations and experiments in real acoustic environments.Publication Multiplicative updates for nonnegative quadratic programming in support vector machines(2002-12-10) Sha, Fei; Saul, Lawrence K; Lee, Daniel DWe derive multiplicative updates for solving the nonnegative quadratic programming problem in support vector machines (SVMs). The updates have a simple closed form, and we prove that they converge monotonically to the solution of the maximum margin hyperplane. The updates optimize the traditionally proposed objective function for SVMs. They do not involve any heuristics such as choosing a learning rate or deciding which variables to update at each iteration. They can be used to adjust all the quadratic programming variables in parallel with a guarantee of improvement at each iteration. We analyze the asymptotic convergence of the updates and show that the coefficients of non-support vectors decay geometrically to zero at a rate that depends on their margins. In practice, the updates converge very rapidly to good classifiers.