Independent Researcher
AI Insights
- The authors also provide a new perspective on the role of noise in diffusion models, highlighting its importance in reducing overfitting and improving generalization. (ML: 0.96)
- The paper concludes by discussing the implications of their findings for the development of more efficient and effective diffusion models. (ML: 0.96)
- Dimensionality: A measure of the number of features or variables in a dataset. (ML: 0.95)
- Diffusion model: A type of generative model that uses a Markov chain to transform data from one distribution to another. (ML: 0.95)
- Learning dynamics: The process by which a neural network learns to minimize its loss function during training. (ML: 0.94)
- The authors use a combination of mathematical derivations and numerical experiments to demonstrate the importance of considering the dimensionality of the data when training diffusion models. (ML: 0.93)
- The paper provides a detailed analysis of the learning dynamics in diffusion models, including the derivation of the optimal loss and its relation to the dimensionality of the data. (ML: 0.92)
- Covariance matrix: A square matrix that describes the covariance between different variables in a dataset. (ML: 0.90)
- Optimal loss: The minimum value of the loss function achieved by a diffusion model under certain conditions. (ML: 0.87)
- The paper shows that the optimal loss is related to the eigenvalues of the covariance matrix of the data, which can be used to guide the choice of hyperparameters in diffusion models. (ML: 0.83)
Abstract
Recent advances in diffusion and flow matching models have highlighted a shift in the preferred prediction target -- moving from noise ($\varepsilon$) and velocity (v) to direct data (x) prediction -- particularly in high-dimensional settings. However, a formal explanation of why the optimal target depends on the specific properties of the data remains elusive. In this work, we provide a theoretical framework based on a generalized prediction formulation that accommodates arbitrary output targets, of which $\varepsilon$-, v-, and x-prediction are special cases. We derive the analytical relationship between data's geometry and the optimal prediction target, offering a rigorous justification for why x-prediction becomes superior when the ambient dimension significantly exceeds the data's intrinsic dimension. Furthermore, while our theory identifies dimensionality as the governing factor for the optimal prediction target, the intrinsic dimension of manifold-bound data is typically intractable to estimate in practice. To bridge this gap, we propose k-Diff, a framework that employs a data-driven approach to learn the optimal prediction parameter k directly from data, bypassing the need for explicit dimension estimation. Extensive experiments in both latent-space and pixel-space image generation demonstrate that k-Diff consistently outperforms fixed-target baselines across varying architectures and data scales, providing a principled and automated approach to enhancing generative performance.
Recent advances in diffusion and flow matching models have highlighted a shift in the preferred prediction target -- moving from noise ($\varepsilon$) and velocity (v) to direct data (x) prediction -- particularly in high-dimensional settings. However, a formal explanation of why the optimal target depends on the specific properties of the data remains elusive. In this work, we provide a theoretical framework based on a generalized prediction formulation that accommodates arbitrary output targets, of which $\varepsilon$-, v-, and x-prediction are special cases. We derive the analytical relationship between data's geometry and the optimal prediction target, offering a rigorous justification for why x-prediction becomes superior when the ambient dimension significantly exceeds the data's intrinsic dimension. Furthermore, while our theory identifies dimensionality as the governing factor for the optimal prediction target, the intrinsic dimension of manifold-bound data is typically intractable to estimate in practice. To bridge this gap, we propose k-Diff, a framework that employs a data-driven approach to learn the optimal prediction parameter k directly from data, bypassing the need for explicit dimension estimation. Extensive experiments in both latent-space and pixel-space image generation demonstrate that k-Diff consistently outperforms fixed-target baselines across varying architectures and data scales, providing a principled and automated approach to enhancing generative performance.
Why we are recommending this paper?
Due to your Interest in Diffusion Models
This paper directly addresses diffusion models, a core interest, and explores prediction targets, aligning with your focus on Deep Learning Models and Diffusion Models. Understanding how to optimize prediction targets within diffusion models is crucial for your research.
Due to your Interest in Diffusion Models
This paper directly addresses diffusion models, a core interest, and explores prediction targets, aligning with your focus on Deep Learning Models and Diffusion Models. Understanding how to optimize prediction targets within diffusion models is crucial for your research.
Vanderbilt University
AI Insights
- PHDME is a type of deep learning model that uses physics-informed neural networks to learn how complex systems behave over time. (ML: 0.71)
- It's trained on synthetic data generated from high-resolution PDE solvers, and it learns to predict the future behavior of the system by propagating the initial conditions through time using a Hamiltonian dynamics framework. (ML: 0.67)
- PHDME (Physics-Informed Hamiltonian Diffusion Models) is a deep learning model that uses physics-informed neural networks to learn the dynamics of complex systems. (ML: 0.67)
- The model is trained on synthetic data generated from high-resolution PDE solvers, and it learns to predict the future behavior of the system by propagating the initial conditions through time using a Hamiltonian dynamics framework. (ML: 0.65)
Abstract
Diffusion models provide expressive priors for forecasting trajectories of dynamical systems, but are typically unreliable in the sparse data regime. Physics-informed machine learning (PIML) improves reliability in such settings; however, most methods require \emph{explicit governing equations} during training, which are often only partially known due to complex and nonlinear dynamics. We introduce \textbf{PHDME}, a port-Hamiltonian diffusion framework designed for \emph{sparse observations} and \emph{incomplete physics}. PHDME leverages port-Hamiltonian structural prior but does not require full knowledge of the closed-form governing equations. Our approach first trains a Gaussian process distributed Port-Hamiltonian system (GP-dPHS) on limited observations to capture an energy-based representation of the dynamics. The GP-dPHS is then used to generate a physically consistent artificial dataset for diffusion training, and to inform the diffusion model with a structured physics residual loss. After training, the diffusion model acts as an amortized sampler and forecaster for fast trajectory generation. Finally, we apply split conformal calibration to provide uncertainty statements for the generated predictions. Experiments on PDE benchmarks and a real-world spring system show improved accuracy and physical consistency under data scarcity.
Diffusion models provide expressive priors for forecasting trajectories of dynamical systems, but are typically unreliable in the sparse data regime. Physics-informed machine learning (PIML) improves reliability in such settings; however, most methods require \emph{explicit governing equations} during training, which are often only partially known due to complex and nonlinear dynamics. We introduce \textbf{PHDME}, a port-Hamiltonian diffusion framework designed for \emph{sparse observations} and \emph{incomplete physics}. PHDME leverages port-Hamiltonian structural prior but does not require full knowledge of the closed-form governing equations. Our approach first trains a Gaussian process distributed Port-Hamiltonian system (GP-dPHS) on limited observations to capture an energy-based representation of the dynamics. The GP-dPHS is then used to generate a physically consistent artificial dataset for diffusion training, and to inform the diffusion model with a structured physics residual loss. After training, the diffusion model acts as an amortized sampler and forecaster for fast trajectory generation. Finally, we apply split conformal calibration to provide uncertainty statements for the generated predictions. Experiments on PDE benchmarks and a real-world spring system show improved accuracy and physical consistency under data scarcity.
Why we are recommending this paper?
Due to your Interest in Diffusion Models
This work investigates the use of physics-informed machine learning with diffusion models, a promising area for improving reliability in sparse data scenarios – a key area of interest for you. The focus on Mixture of Experts and their connection to diffusion models is highly relevant.
Due to your Interest in Diffusion Models
This work investigates the use of physics-informed machine learning with diffusion models, a promising area for improving reliability in sparse data scenarios – a key area of interest for you. The focus on Mixture of Experts and their connection to diffusion models is highly relevant.
Nanjing University of Aeronautics and Astronautics
AI Insights
- Imagine you're trying to write a story, but your language model keeps making mistakes. (ML: 0.99)
- The paper does not provide a comprehensive evaluation of the proposed methods, and more research is needed to fully understand their strengths and weaknesses. (ML: 0.98)
- The paper also discusses the importance of evaluating LLMs on tasks that require reasoning and factuality, rather than just fluency and coherence. (ML: 0.98)
- These methods help the model focus on the right information in the input sequence, resulting in higher-quality text generation. (ML: 0.97)
- Contrastive decoding: a method that uses contrastive learning to improve the quality of generated text by minimizing the difference between the generated text and the true text. (ML: 0.94)
- Classifier-free guidance: a method that uses a classifier-free approach to guide the model's attention mechanism, allowing it to focus on relevant information in the input sequence. (ML: 0.94)
- The paper concludes that contrastive decoding and classifier-free guidance are effective methods for improving the performance of LLMs in tasks such as text generation, reasoning, and factuality. (ML: 0.94)
- The paper discusses various methods for improving the performance of large language models (LLMs) in tasks such as text generation, reasoning, and factuality. (ML: 0.93)
- The paper discusses various methods for improving the performance of large language models (LLMs) in tasks such as text generation, reasoning, and factuality. (ML: 0.93)
- The paper explores new ways to make language models better at writing stories, by using techniques like contrastive learning and classifier-free guidance. (ML: 0.93)
- The paper cites several recent studies on LLMs, including work on contrastive learning, classifier-free guidance, and multi-token prediction. (ML: 0.92)
Abstract
Contrastive Decoding (CD) enhances the generation quality of large language models (LLMs) but incurs significant additional computational overhead due to the need for an auxiliary model. Existing internal self-contrastive decoding methods, such as Decoding by Contrasting Layers (DoLa), focus on discrepancies across different layers, which are notably unstable on small-scale models. In this work, based on the observation that LLMs exhibit local preferences, we propose a novel contrastive guidance strategy along the temporal dimension, namely Temporal Guidance (TeGu). Our method ingeniously leverages Multi-Token Prediction (MTP) to construct weaker amateur predictions for model self-contrast. To standardize the implementation of this mechanism, we further introduce a lightweight Conditional MTP Projector (cMTPP), which avoids maintaining multiple independent networks as required by other MTP modules. Across various model series and benchmarks, TeGu achieves significant performance improvements while maintaining low additional memory consumption and computational overhead.
Contrastive Decoding (CD) enhances the generation quality of large language models (LLMs) but incurs significant additional computational overhead due to the need for an auxiliary model. Existing internal self-contrastive decoding methods, such as Decoding by Contrasting Layers (DoLa), focus on discrepancies across different layers, which are notably unstable on small-scale models. In this work, based on the observation that LLMs exhibit local preferences, we propose a novel contrastive guidance strategy along the temporal dimension, namely Temporal Guidance (TeGu). Our method ingeniously leverages Multi-Token Prediction (MTP) to construct weaker amateur predictions for model self-contrast. To standardize the implementation of this mechanism, we further introduce a lightweight Conditional MTP Projector (cMTPP), which avoids maintaining multiple independent networks as required by other MTP modules. Across various model series and benchmarks, TeGu achieves significant performance improvements while maintaining low additional memory consumption and computational overhead.
Why we are recommending this paper?
Due to your Interest in Large Language Models
Given your interest in Large Language Models, this paper's exploration of Contrastive Decoding offers a valuable insight into enhancing LLM generation quality, directly addressing your interest in Deep Learning Models. The focus on computational efficiency is also a relevant consideration.
Due to your Interest in Large Language Models
Given your interest in Large Language Models, this paper's exploration of Contrastive Decoding offers a valuable insight into enhancing LLM generation quality, directly addressing your interest in Deep Learning Models. The focus on computational efficiency is also a relevant consideration.
University of Vienna
AI Insights
- The adaptive choice of recomputations may not always be optimal. (ML: 0.97)
- Transformer-based models have been widely used for natural language processing tasks, but their computational cost can be significant. (ML: 0.96)
- LAMP: Look-Ahead Mixed-Precision Transformer-based models: a type of neural network architecture used for natural language processing tasks The proposed method, LAMP, is shown to be effective in reducing the computational cost of transformer-based models while maintaining their accuracy. (ML: 0.94)
- The results indicate that the adaptive choice of recomputations in LAMP evaluation is critical for mixed-precision inference. (ML: 0.93)
- The method's performance is demonstrated on various datasets, including those with permuted sequences of tokens and random recomputations. (ML: 0.93)
- The method's performance may degrade if the threshold of LAMP is set too high or too low. (ML: 0.88)
- The results demonstrate that LAMP's performance is robust across various datasets and scenarios. (ML: 0.81)
- The proposed LAMP inference method is shown to be effective in reducing the computational cost of transformer-based models while maintaining their accuracy. (ML: 0.74)
Abstract
Mixed-precision computations are a hallmark of the current stage of AI, driving the progress in large language models towards efficient, locally deployable solutions. This article addresses the floating-point computation of compositionally-rich functions, concentrating on transformer inference. Based on the rounding error analysis of a composition $f(g(\mathrm{x}))$, we provide an adaptive strategy that selects a small subset of components of $g(\mathrm{x})$ to be computed more accurately while all other computations can be carried out with lower accuracy. We then explain how this strategy can be applied to different compositions within a transformer and illustrate its overall effect on transformer inference. We study the effectiveness of this algorithm numerically on GPT-2 models and demonstrate that already very low recomputation rates allow for improvements of up to two orders of magnitude in accuracy.
Mixed-precision computations are a hallmark of the current stage of AI, driving the progress in large language models towards efficient, locally deployable solutions. This article addresses the floating-point computation of compositionally-rich functions, concentrating on transformer inference. Based on the rounding error analysis of a composition $f(g(\mathrm{x}))$, we provide an adaptive strategy that selects a small subset of components of $g(\mathrm{x})$ to be computed more accurately while all other computations can be carried out with lower accuracy. We then explain how this strategy can be applied to different compositions within a transformer and illustrate its overall effect on transformer inference. We study the effectiveness of this algorithm numerically on GPT-2 models and demonstrate that already very low recomputation rates allow for improvements of up to two orders of magnitude in accuracy.
Why we are recommending this paper?
Due to your Interest in Large Language Models
This paper tackles the efficient deployment of Large Language Models using mixed-precision computation, a critical area for scaling these models – aligning with your interest in Deep Learning Optimization. The focus on transformer inference is particularly pertinent.
Due to your Interest in Large Language Models
This paper tackles the efficient deployment of Large Language Models using mixed-precision computation, a critical area for scaling these models – aligning with your interest in Deep Learning Optimization. The focus on transformer inference is particularly pertinent.
Princeton University
AI Insights
- The authors also discuss the importance of learning rate tuning for biases in MoE models. (ML: 0.95)
- Hyperparameter transfer: The process of transferring hyperparameters from one model to another, often in the context of scaling models. (ML: 0.93)
- The authors propose a method to scale MoE models while maintaining their performance and efficiency. (ML: 0.93)
- Learning rate tuning for biases is crucial for achieving optimal performance in MoE models. (ML: 0.92)
- Mixture-of-Experts (MoE) model: A type of neural network architecture that consists of multiple experts, each responsible for a specific task or function. (ML: 0.90)
- The paper explores the concept of hyperparameter transfer for Mixture-of-Experts (MoE) models. (ML: 0.90)
- They provide experimental results demonstrating the effectiveness of their proposed method. (ML: 0.85)
- The proposed method allows for efficient scaling of MoE models while maintaining their performance and efficiency. (ML: 0.85)
- The experimental results demonstrate the effectiveness of the proposed method in scaling MoE models. (ML: 0.84)
- They introduce a new parameterization scheme that allows for efficient scaling of MoE models. (ML: 0.70)
Abstract
Mixture-of-Experts (MoE) layers have emerged as an important tool in scaling up modern neural networks by decoupling total trainable parameters from activated parameters in the forward pass for each token. However, sparse MoEs add complexity to training due to (i) new trainable parameters (router weights) that, like all other parameter groups, require hyperparameter (HP) tuning; (ii) new architecture scale dimensions (number of and size of experts) that must be chosen and potentially taken large. To make HP selection cheap and reliable, we propose a new parameterization for transformer models with MoE layers when scaling model width, depth, number of experts, and expert (hidden) size. Our parameterization is justified by a novel dynamical mean-field theory (DMFT) analysis. When varying different model dimensions trained at a fixed token budget, we find empirically that our parameterization enables reliable HP transfer across models from 51M to over 2B total parameters. We further take HPs identified from sweeping small models on a short token horizon to train larger models on longer horizons and report performant model behaviors.
Mixture-of-Experts (MoE) layers have emerged as an important tool in scaling up modern neural networks by decoupling total trainable parameters from activated parameters in the forward pass for each token. However, sparse MoEs add complexity to training due to (i) new trainable parameters (router weights) that, like all other parameter groups, require hyperparameter (HP) tuning; (ii) new architecture scale dimensions (number of and size of experts) that must be chosen and potentially taken large. To make HP selection cheap and reliable, we propose a new parameterization for transformer models with MoE layers when scaling model width, depth, number of experts, and expert (hidden) size. Our parameterization is justified by a novel dynamical mean-field theory (DMFT) analysis. When varying different model dimensions trained at a fixed token budget, we find empirically that our parameterization enables reliable HP transfer across models from 51M to over 2B total parameters. We further take HPs identified from sweeping small models on a short token horizon to train larger models on longer horizons and report performant model behaviors.
Why we are recommending this paper?
Due to your Interest in Mixture of Experts
This paper investigates Mixture-of-Experts layers, a significant advancement in scaling neural networks, directly addressing your interest in Deep Learning Architectures and Deep Learning Models. The focus on training complexity is a key area for exploration.
Due to your Interest in Mixture of Experts
This paper investigates Mixture-of-Experts layers, a significant advancement in scaling neural networks, directly addressing your interest in Deep Learning Architectures and Deep Learning Models. The focus on training complexity is a key area for exploration.
Hokkaido University
AI Insights
- L2R addresses the issue of unstable expert selection by introducing a bounded query-magnitude transform. (ML: 0.97)
- Mixture-of-experts models: A type of neural network architecture where multiple experts are combined to produce the final output. (ML: 0.92)
- The paper presents a novel approach to addressing the issue of unstable expert selection in mixture-of-experts models. (ML: 0.91)
- Low-Rank and Lipschitz-Controlled Routing (L2R): A new routing method for mixture-of-experts models that addresses unstable expert selection. (ML: 0.89)
- SIPS (Saturation with Inhibition of Positive Saturation): A new scoring rule that balances between magnitude effects and angular contrast. (ML: 0.88)
- A key contribution of the paper is the introduction of SIPS (Saturation with Inhibition of Positive Saturation), a new scoring rule that balances between magnitude effects and angular contrast. (ML: 0.87)
- The paper provides a comprehensive analysis of the effect of different scoring rules on routing dynamics and expert selection. (ML: 0.85)
- The paper presents a new routing method for mixture-of-experts models called Low-Rank and Lipschitz-Controlled Routing (L2R). (ML: 0.85)
- L2R shows promising results in improving routing dynamics and expert selection. (ML: 0.81)
- The bounded magnitude term is controlled by a parameter β, which can be adjusted to balance between stability and selectivity. (ML: 0.76)
Abstract
Mixture-of-Experts (MoE) models scale neural networks by conditionally activating a small subset of experts, where the router plays a central role in determining expert specialization and overall model performance. However, many modern MoE systems still adopt linear routers in raw high-dimensional representation spaces, where representation mismatch, angular concentration, and scale-sensitive scoring can jointly undermine routing discriminability and stable expert specialization. In this work, we propose Low-rank \& Lipschitz-controlled Routing (L2R), a unified routing framework that reshapes both the routing space and scoring geometry. L2R performs expert assignment in a shared low-rank latent routing space and introduces Saturated Inner-Product Scoring (SIPS) to explicitly control the Lipschitz behavior of routing functions, yielding smoother and more stable routing geometry. In addition, L2R incorporates a parameter-efficient multi-anchor routing mechanism to enhance expert expressiveness. Extensive experiments on a large-scale language MoE model and a vision MoE setting on ImageNet demonstrate that L2R consistently improves routing stability, expert specialization, and overall model performance.
Mixture-of-Experts (MoE) models scale neural networks by conditionally activating a small subset of experts, where the router plays a central role in determining expert specialization and overall model performance. However, many modern MoE systems still adopt linear routers in raw high-dimensional representation spaces, where representation mismatch, angular concentration, and scale-sensitive scoring can jointly undermine routing discriminability and stable expert specialization. In this work, we propose Low-rank \& Lipschitz-controlled Routing (L2R), a unified routing framework that reshapes both the routing space and scoring geometry. L2R performs expert assignment in a shared low-rank latent routing space and introduces Saturated Inner-Product Scoring (SIPS) to explicitly control the Lipschitz behavior of routing functions, yielding smoother and more stable routing geometry. In addition, L2R incorporates a parameter-efficient multi-anchor routing mechanism to enhance expert expressiveness. Extensive experiments on a large-scale language MoE model and a vision MoE setting on ImageNet demonstrate that L2R consistently improves routing stability, expert specialization, and overall model performance.
Why we are recommending this paper?
Due to your Interest in Mixture of Experts
Due to your Interest in Mixture of Experts
Universit Paris Cit , CNRS
AI Insights
- The authors show that any feedforward network with ReLU activations can be viewed as a place-independent IFS, and they extend this result to other types of neural networks, including residual blocks and MoE models. (ML: 0.92)
- The paper discusses the interpretation of deep neural networks as iterated function systems (IFSs) and provides a general framework for analyzing their convergence properties. (ML: 0.89)
- The paper provides several examples of neural network architectures that can be interpreted as IFSs, including ResNet with Softplus activation, Transformer block, and MoE model. (ML: 0.88)
- The authors use the Hutchinson operator to analyze the convergence properties of IFSs and show that they can be used to bound the Wasserstein distance between the output of a neural network and its fixed point. (ML: 0.87)
- Definition 1: A Markov recursion is a sequence of random variables {Xn} defined by X0 = x and Xt+1 = w(Xt, Θ), where w is a function that depends on the current state Xt and the parameter Θ. (ML: 0.82)
- Definition 3: A place-dependent IFS (P-IFS) is an IFS {wξ} where each wξ depends on the current state x and the parameter Θ. (ML: 0.80)
- Definition 2: An iterated function system (IFS) is a collection of functions {wξ} indexed by ξ ∈ I, where each wξ is a Lipschitz map from X to itself. (ML: 0.80)
- They also introduce the concept of strong average Lipschitz contractivity for place-dependent IFSs and provide conditions under which it holds. (ML: 0.75)
- Definition 5: A P-IFS {wξ} is strongly average-contractive if sup_x∈X ∑_{ξ∈I} pξ(x)cξ ≤ c < 1. (ML: 0.67)
- Definition 4: The Hutchinson operator T is a contraction on the space of probability measures PP(X) with respect to the Wasserstein distance W2 if there exists a constant c < 1 such that W2(T(µ), T(ν)) ≤ cW2(µ, ν) for all µ, ν ∈ PP(X). (ML: 0.67)
- The Hutchinson operator T is defined as T(µ) = ∑_{ξ∈I} pwξ#µq. (ML: 0.49)
Abstract
Deep neural networks (DNNs) achieve remarkable performance on a wide range of tasks, yet their mathematical analysis remains fragmented: stability and generalization are typically studied in disparate frameworks and on a case-by-case basis. Architecturally, DNNs rely on the recursive application of parametrized functions, a mechanism that can be unstable and difficult to train, making stability a primary concern. Even when training succeeds, there are few rigorous results on how well such models generalize beyond the observed data, especially in the generative setting. In this work, we leverage the theory of stochastic Iterated Function Systems (IFS) and show that two important deep architectures can be viewed as, or canonically associated with, place-dependent IFS. This connection allows us to import results from random dynamical systems to (i) establish the existence and uniqueness of invariant measures under suitable contractivity assumptions, and (ii) derive a Wasserstein generalization bound for generative modeling. The bound naturally leads to a new training objective that directly controls the collage-type approximation error between the data distribution and its image under the learned transfer operator. We illustrate the theory on a controlled 2D example and empirically evaluate the proposed objective on standard image datasets (MNIST, CelebA, CIFAR-10).
Deep neural networks (DNNs) achieve remarkable performance on a wide range of tasks, yet their mathematical analysis remains fragmented: stability and generalization are typically studied in disparate frameworks and on a case-by-case basis. Architecturally, DNNs rely on the recursive application of parametrized functions, a mechanism that can be unstable and difficult to train, making stability a primary concern. Even when training succeeds, there are few rigorous results on how well such models generalize beyond the observed data, especially in the generative setting. In this work, we leverage the theory of stochastic Iterated Function Systems (IFS) and show that two important deep architectures can be viewed as, or canonically associated with, place-dependent IFS. This connection allows us to import results from random dynamical systems to (i) establish the existence and uniqueness of invariant measures under suitable contractivity assumptions, and (ii) derive a Wasserstein generalization bound for generative modeling. The bound naturally leads to a new training objective that directly controls the collage-type approximation error between the data distribution and its image under the learned transfer operator. We illustrate the theory on a controlled 2D example and empirically evaluate the proposed objective on standard image datasets (MNIST, CelebA, CIFAR-10).
Why we are recommending this paper?
Due to your Interest in Deep Learning Models
Due to your Interest in Deep Learning Models
BarIlan University
AI Insights
- FM-RBM: Fully-Multinomial Restricted Boltzmann Machine RBM: Restricted Boltzmann Machine dv, dh: visible and hidden sizes of the RBM ek: one-hot vector with 1 at its index k E(v, h): energy of the FM-FBM Xl, Xm: number of classes al, bm: bias terms for visible and hidden units wlm: weights between visible and hidden units Salakhutdinov et al., 2007: proposed the Multinomial-Binary RBM The Fully-Multinomial Restricted Boltzmann Machine (FM-RBM) is a type of RBM that can handle multinomial input and output. (ML: 0.76)
- The energy function of the FM-RBM is derived, and its conditional probability for visible units is calculated. (ML: 0.52)
Abstract
Unsupervised ensemble learning emerged to address the challenge of combining multiple learners' predictions without access to ground truth labels or additional data. This paradigm is crucial in scenarios where evaluating individual classifier performance or understanding their strengths is challenging due to limited information. We propose a novel deep energy-based method for constructing an accurate meta-learner using only the predictions of individual learners, potentially capable of capturing complex dependence structures between them. Our approach requires no labeled data, learner features, or problem-specific information, and has theoretical guarantees for when learners are conditionally independent. We demonstrate superior performance across diverse ensemble scenarios, including challenging mixture of experts settings. Our experiments span standard ensemble datasets and curated datasets designed to test how the model fuses expertise from multiple sources. These results highlight the potential of unsupervised ensemble learning to harness collective intelligence, especially in data-scarce or privacy-sensitive environments.
Unsupervised ensemble learning emerged to address the challenge of combining multiple learners' predictions without access to ground truth labels or additional data. This paradigm is crucial in scenarios where evaluating individual classifier performance or understanding their strengths is challenging due to limited information. We propose a novel deep energy-based method for constructing an accurate meta-learner using only the predictions of individual learners, potentially capable of capturing complex dependence structures between them. Our approach requires no labeled data, learner features, or problem-specific information, and has theoretical guarantees for when learners are conditionally independent. We demonstrate superior performance across diverse ensemble scenarios, including challenging mixture of experts settings. Our experiments span standard ensemble datasets and curated datasets designed to test how the model fuses expertise from multiple sources. These results highlight the potential of unsupervised ensemble learning to harness collective intelligence, especially in data-scarce or privacy-sensitive environments.
Why we are recommending this paper?
Due to your Interest in Deep Learning Models
Due to your Interest in Deep Learning Models
Queen Mary University of London
AI Insights
- The solution decomposes beamforming optimization into three sub-questions and solves them with three sub-modules: an AE for CSI feature extraction, an RL agent for finding the optimal beampattern, and a DNN network for reconstructing the desirable beamforming. (ML: 0.82)
- The proposed three-stage solution outperforms the baseline algorithm by optimizing beampattern and reconstructing beamforming. (ML: 0.69)
- AE: Autoencoder CSI: Channel State Information DNN: Deep Neural Network ISAC: Integrated Sensing and Communication MIMO: Multiple-Input Multiple-Output OFDM: Orthogonal Frequency Division Multiplexing The proposed three-stage solution is effective in optimizing beampattern and reconstructing beamforming, leading to improved performance compared to the baseline algorithm. (ML: 0.69)
- The solution's ability to decompose complex optimization problems into manageable sub-questions and solve them with specialized modules makes it a promising approach for ISAC systems. (ML: 0.62)
Abstract
In this paper, a general ISAC system where the base station (BS) communicates with multiple users and performs target detection is considered. Then, a sum communication rate maximization problem is formulated, subjected to the constraints of transmit power and the minimum sensing rates of users. To solve this problem, we develop a framework that leverages deep learning algorithms to provide a three-stage solution for ISAC beamforming. The three-stage beamforming optimization solution includes three modules: 1) an unsupervised learning based feature extraction algorithm is proposed to extract fixed-size latent features while keeping its essential information from the variable channel state information (CSI); 2) a reinforcement learning (RL) based beampattern optimization algorithm is proposed to search the desired beampattern according to the extracted features; 3) a supervised learning based beamforming reconstruction algorithm is proposed to reconstruct the beamforming vector from beampattern given by the RL agent. Simulation results demonstrate that the proposed three-stage solution outperforms the baseline RL algorithm by optimizing the intuitional beampattern rather than beamforming.
In this paper, a general ISAC system where the base station (BS) communicates with multiple users and performs target detection is considered. Then, a sum communication rate maximization problem is formulated, subjected to the constraints of transmit power and the minimum sensing rates of users. To solve this problem, we develop a framework that leverages deep learning algorithms to provide a three-stage solution for ISAC beamforming. The three-stage beamforming optimization solution includes three modules: 1) an unsupervised learning based feature extraction algorithm is proposed to extract fixed-size latent features while keeping its essential information from the variable channel state information (CSI); 2) a reinforcement learning (RL) based beampattern optimization algorithm is proposed to search the desired beampattern according to the extracted features; 3) a supervised learning based beamforming reconstruction algorithm is proposed to reconstruct the beamforming vector from beampattern given by the RL agent. Simulation results demonstrate that the proposed three-stage solution outperforms the baseline RL algorithm by optimizing the intuitional beampattern rather than beamforming.
Why we are recommending this paper?
Due to your Interest in Deep Learning Optimization
Due to your Interest in Deep Learning Optimization
University of California, Davis
AI Insights
- Addressing bias in medical AI is crucial for ensuring fairness and accuracy in decision-making processes. (ML: 0.99)
- Causal representations: A type of representation that captures the causal relationships between variables, enabling more robust reasoning in complex scenarios. (ML: 0.98)
- The paper highlights the significance of addressing bias in medical AI and provides a call for open science to mitigate these biases. (ML: 0.98)
- The paper discusses the importance of causal representations in medical multimodal large language models for robust reasoning in complex clinical scenarios. (ML: 0.97)
- The authors propose a novel approach to extend causal representations to medical multimodal large language models, enabling more robust reasoning in complex clinical scenarios. (ML: 0.97)
- The authors discuss various methods for handling missing data with graph representation learning, including mutual information neural estimation. (ML: 0.96)
- Medical multimodal large language models: Models that can process and reason about multiple types of medical data, such as text, images, and audio. (ML: 0.96)
- The proposed method involves learning disentangled causal substructures from graph neural networks, which can handle missing data and provide interpretable results. (ML: 0.92)
- Disentangled causal substructures: A method for learning disentangled causal substructures from graph neural networks, which can handle missing data and provide interpretable results. (ML: 0.90)
- The proposed approach has the potential to improve the robustness of medical multimodal large language models in complex clinical scenarios. (ML: 0.90)
Abstract
Medical multimodal representation learning aims to integrate heterogeneous data into unified patient representations to support clinical outcome prediction. However, real-world medical datasets commonly contain systematic biases from multiple sources, which poses significant challenges for medical multimodal representation learning. Existing approaches typically focus on effective multimodal fusion, neglecting inherent biased features that affect the generalization ability. To address these challenges, we propose a Dual-Stream Feature Decorrelation Framework that identifies and handles the biases through structural causal analysis introduced by latent confounders. Our method employs a causal-biased decorrelation framework with dual-stream neural networks to disentangle causal features from spurious correlations, utilizing generalized cross-entropy loss and mutual information minimization for effective decorrelation. The framework is model-agnostic and can be integrated into existing medical multimodal learning methods. Comprehensive experiments on MIMIC-IV, eICU, and ADNI datasets demonstrate consistent performance improvements.
Medical multimodal representation learning aims to integrate heterogeneous data into unified patient representations to support clinical outcome prediction. However, real-world medical datasets commonly contain systematic biases from multiple sources, which poses significant challenges for medical multimodal representation learning. Existing approaches typically focus on effective multimodal fusion, neglecting inherent biased features that affect the generalization ability. To address these challenges, we propose a Dual-Stream Feature Decorrelation Framework that identifies and handles the biases through structural causal analysis introduced by latent confounders. Our method employs a causal-biased decorrelation framework with dual-stream neural networks to disentangle causal features from spurious correlations, utilizing generalized cross-entropy loss and mutual information minimization for effective decorrelation. The framework is model-agnostic and can be integrated into existing medical multimodal learning methods. Comprehensive experiments on MIMIC-IV, eICU, and ADNI datasets demonstrate consistent performance improvements.
Why we are recommending this paper?
Due to your Interest in Multimodal Learning
Due to your Interest in Multimodal Learning
Tsinghua University
AI Insights
- Diffusion Modeling: A generative paradigm that generates data by reversing a continuous corruption process. (ML: 0.95)
- The training objective for standard AR models is to minimize the negative log-likelihood (NLL). (ML: 0.92)
- Autoregressive (AR) modeling: A generative paradigm that predicts tokens sequentially, one at a time. (ML: 0.91)
- Masked Autoregressive (MAR) modeling: A variant of AR modeling that uses a mask to predict missing tokens. (ML: 0.91)
- Diffusion Modeling generates data by reversing a continuous corruption process, with two frameworks: Denoising Diffusion Probabilistic Models (DDPM) and the deterministic ODE framework (Flow Matching). (ML: 0.90)
- Autoregressive (AR), Diffusion Modeling, and Masked Autoregressive (MAR) modeling are three primary generative paradigms utilized in UMMs. (ML: 0.90)
- Denoising Diffusion Probabilistic Models (DDPM): A framework for generating data by reversing a continuous corruption process, using the reparameterization trick. (ML: 0.87)
- Unified Multimodal Model (UMM): A model that can handle multiple input modalities and generate output in various formats. (ML: 0.86)
- Unified Multimodal Models (UMMs) have been a significant area of research in recent years. (ML: 0.85)
- MaskGit employs a masked prediction strategy to enable parallel token prediction and utilize bidirectional context. (ML: 0.80)
Abstract
Unified Multimodal Models (UMMs) integrate both visual understanding and generation within a single framework. Their ultimate aspiration is to create a cycle where understanding and generation mutually reinforce each other. While recent post-training methods have successfully leveraged understanding to enhance generation, the reverse direction of utilizing generation to improve understanding remains largely unexplored. In this work, we propose UniMRG (Unified Multi-Representation Generation), a simple yet effective architecture-agnostic post-training method. UniMRG enhances the understanding capabilities of UMMs by incorporating auxiliary generation tasks. Specifically, we train UMMs to generate multiple intrinsic representations of input images, namely pixel (reconstruction), depth (geometry), and segmentation (structure), alongside standard visual understanding objectives. By synthesizing these diverse representations, UMMs capture complementary information regarding appearance, spatial relations, and structural layout. Consequently, UMMs develop a deeper and more comprehensive understanding of visual inputs. Extensive experiments across diverse UMM architectures demonstrate that our method notably enhances fine-grained perception, reduces hallucinations, and improves spatial understanding, while simultaneously boosting generation capabilities.
Unified Multimodal Models (UMMs) integrate both visual understanding and generation within a single framework. Their ultimate aspiration is to create a cycle where understanding and generation mutually reinforce each other. While recent post-training methods have successfully leveraged understanding to enhance generation, the reverse direction of utilizing generation to improve understanding remains largely unexplored. In this work, we propose UniMRG (Unified Multi-Representation Generation), a simple yet effective architecture-agnostic post-training method. UniMRG enhances the understanding capabilities of UMMs by incorporating auxiliary generation tasks. Specifically, we train UMMs to generate multiple intrinsic representations of input images, namely pixel (reconstruction), depth (geometry), and segmentation (structure), alongside standard visual understanding objectives. By synthesizing these diverse representations, UMMs capture complementary information regarding appearance, spatial relations, and structural layout. Consequently, UMMs develop a deeper and more comprehensive understanding of visual inputs. Extensive experiments across diverse UMM architectures demonstrate that our method notably enhances fine-grained perception, reduces hallucinations, and improves spatial understanding, while simultaneously boosting generation capabilities.
Why we are recommending this paper?
Due to your Interest in Multimodal Learning
Due to your Interest in Multimodal Learning
Harvard University
AI Insights
- Milestones serve dual pedagogical and validation purposes, providing motivation through historical framing and demonstrating implementation correctness through real-world task performance. (ML: 0.98)
- Each module concludes with systems reasoning prompts measuring conceptual understanding beyond syntactic correctness. (ML: 0.97)
- Milestones are designed to be challenging but achievable, allowing students to demonstrate their understanding of complex concepts through real-world tasks. (ML: 0.96)
- Assessment validates both isolated correctness and cross-module integration. (ML: 0.96)
- The TinyTorch framework is designed for teaching machine learning concepts through hands-on implementation and analysis. (ML: 0.95)
- Reflect: Systems Analysis Questions. (ML: 0.94)
- TinyTorch follows a consistent Build-Use-Reflect cycle, integrating implementation, application, and systems reasoning to address multiple learning objectives. (ML: 0.94)
- It's a pedagogical tool aimed at bridging the gap between theoretical understanding and practical application. (ML: 0.94)
- Students implement components in Jupyter notebooks with scaffolded guidance. (ML: 0.91)
- TinyTorch's design emphasizes systems thinking, encouraging students to analyze and understand the relationships between components, rather than just focusing on individual functions. (ML: 0.87)
- The framework includes six historical milestones that recreate actual breakthroughs using exclusively student code, validating success through task-appropriate performance. (ML: 0.85)
- The framework is built with a focus on explicit dependencies, making it easier for students to understand where each module fits in the larger architecture. (ML: 0.83)
- Use: Integration Testing Beyond Unit Tests. (ML: 0.77)
- Build: Implementation with Explicit Dependencies. (ML: 0.66)
Abstract
Machine learning education faces a fundamental gap: students learn algorithms without understanding the systems that execute them. They study gradient descent without measuring memory, attention mechanisms without analyzing O(N^2) scaling, optimizer theory without knowing why Adam requires 3x the memory of SGD. This "algorithm-systems divide" produces practitioners who can train models but cannot debug memory failures, optimize inference latency, or reason about deployment trade-offs--the very skills industry demands as "ML systems engineering." We present TinyTorch, a 20-module curriculum that closes this gap through "implementation-based systems pedagogy": students construct PyTorch's core components (tensors, autograd, optimizers, CNNs, transformers) in pure Python, building a complete framework where every operation they invoke is code they wrote. The design employs three patterns: "progressive disclosure" of complexity, "systems-first integration" of profiling from the first module, and "build-to-validate milestones" recreating 67 years of ML breakthroughs--from Perceptron (1958) through Transformers (2017) to MLPerf-style benchmarking. Requiring only 4GB RAM and no GPU, TinyTorch demonstrates that deep ML systems understanding is achievable without specialized hardware. The curriculum is available open-source at mlsysbook.ai/tinytorch.
Machine learning education faces a fundamental gap: students learn algorithms without understanding the systems that execute them. They study gradient descent without measuring memory, attention mechanisms without analyzing O(N^2) scaling, optimizer theory without knowing why Adam requires 3x the memory of SGD. This "algorithm-systems divide" produces practitioners who can train models but cannot debug memory failures, optimize inference latency, or reason about deployment trade-offs--the very skills industry demands as "ML systems engineering." We present TinyTorch, a 20-module curriculum that closes this gap through "implementation-based systems pedagogy": students construct PyTorch's core components (tensors, autograd, optimizers, CNNs, transformers) in pure Python, building a complete framework where every operation they invoke is code they wrote. The design employs three patterns: "progressive disclosure" of complexity, "systems-first integration" of profiling from the first module, and "build-to-validate milestones" recreating 67 years of ML breakthroughs--from Perceptron (1958) through Transformers (2017) to MLPerf-style benchmarking. Requiring only 4GB RAM and no GPU, TinyTorch demonstrates that deep ML systems understanding is achievable without specialized hardware. The curriculum is available open-source at mlsysbook.ai/tinytorch.
Why we are recommending this paper?
Due to your Interest in Deep Learning
Due to your Interest in Deep Learning
Interests not found
We did not find any papers that match the below interests.
Try other terms also consider if the content exists in arxiv.org.
- Deep Learning Architectures
You can edit or add more interests any time.
💬 Help Shape Our Pricing
We're exploring pricing options to make this project sustainable. Take 3 minutes to share what you'd be willing to pay (if anything). Your input guides our future investment.
Help us improve your experience!
This project is on its early stages your feedback can be pivotal on the future of the project.
Let us know what you think about this week's papers and suggestions!
Give Feedback